hadoop mapreduce开发实践之HDFS文件分发by streaming-balich

nainai1

$ ./runstreaming_cachefile.sh　　18/01/26 15:38:27 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
　　18/01/26 15:38:28 WARN streaming.StreamJob: -cacheFile option is deprecated, please use -files instead.
　　18/01/26 15:38:28 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead.
　　18/01/26 15:38:28 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
　　packageJobJar: [./mapper.py, ./reducer.py, /tmp/hadoop-unjar1709565523181962236/] [] /tmp/streamjob6164905989972408041.jar tmpDir=null
　　18/01/26 15:38:29 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
　　18/01/26 15:38:29 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
　　18/01/26 15:38:31 INFO mapred.FileInputFormat: Total input paths to process : 1
　　18/01/26 15:38:31 INFO mapreduce.JobSubmitter: number of splits:2
　　18/01/26 15:38:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1516345010544_0012
　　18/01/26 15:38:32 INFO impl.YarnClientImpl: Submitted application application_1516345010544_0012
　　18/01/26 15:38:32 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1516345010544_0012/
　　18/01/26 15:38:32 INFO mapreduce.Job: Running job: job_1516345010544_0012
　　18/01/26 15:38:40 INFO mapreduce.Job: Job job_1516345010544_0012 running in uber mode : false
　　18/01/26 15:38:40 INFO mapreduce.Job:  map 0% reduce 0%
　　18/01/26 15:38:49 INFO mapreduce.Job:  map 50% reduce 0%
　　18/01/26 15:38:50 INFO mapreduce.Job:  map 100% reduce 0%
　　18/01/26 15:38:57 INFO mapreduce.Job:  map 100% reduce 100%
　　18/01/26 15:38:57 INFO mapreduce.Job: Job job_1516345010544_0012 completed successfully
　　18/01/26 15:38:57 INFO mapreduce.Job: Counters: 49
　　File System Counters
　　FILE: Number of bytes read=73950
　　FILE: Number of bytes written=582590
　　FILE: Number of read operations=0
　　FILE: Number of large read operations=0
　　FILE: Number of write operations=0
　　HDFS: Number of bytes read=636501
　　HDFS: Number of bytes written=27
　　HDFS: Number of read operations=9
　　HDFS: Number of large read operations=0
　　HDFS: Number of write operations=2
　　Job Counters
　　Launched map tasks=2
　　Launched reduce tasks=1
　　Data-local map tasks=2
　　Total time spent by all maps in occupied slots (ms)=12921
　　Total time spent by all reduces in occupied slots (ms)=5641
　　Total time spent by all map tasks (ms)=12921
　　Total time spent by all reduce tasks (ms)=5641
　　Total vcore-milliseconds taken by all map tasks=12921
　　Total vcore-milliseconds taken by all reduce tasks=5641
　　Total megabyte-milliseconds taken by all map tasks=13231104
　　Total megabyte-milliseconds taken by all reduce tasks=5776384
　　Map-Reduce Framework
　　Map input records=2866
　　Map output records=9243
　　Map output bytes=55458
　　Map output materialized bytes=73956
　　Input split bytes=198
　　Combine input records=0
　　Combine output records=0
　　Reduce input groups=3
　　Reduce shuffle bytes=73956
　　Reduce input records=9243
　　Reduce output records=3
　　Spilled Records=18486
　　Shuffled Maps =2
　　Failed Shuffles=0
　　Merged Map outputs=2
　　GC time elapsed (ms)=360
　　CPU time spent (ms)=3910
　　Physical memory (bytes) snapshot=719896576
　　Virtual memory (bytes) snapshot=8331550720
　　Total committed heap usage (bytes)=602931200
　　Shuffle Errors
　　BAD_ID=0
　　CONNECTION=0
　　IO_ERROR=0
　　WRONG_LENGTH=0
　　WRONG_MAP=0
　　WRONG_REDUCE=0
　　File Input Format Counters
　　Bytes Read=636303
　　File Output Format Counters
　　Bytes Written=27
　　18/01/26 15:38:57 INFO streaming.StreamJob: Output directory: /output/wordcount/wordwhitecachefiletest