hadoop-2.6.0基准测试

814247614

　　hadoop-2.6.0基准测试
　　1.测试程序的帮助信息
　　[hadoop@tong1 hadoop-2.6.0]$ hadoop  jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar
　　An example program must be given as the first argument.
　　Valid program names are:
　　DFSCIOTest: Distributed i/o benchmark of libhdfs.
　　DistributedFSCheck: Distributed checkup of the file system consistency.
　　JHLogAnalyzer: Job History Log analyzer.

　　MRReliabilityTest: A program that tests the>　　SliveTest: HDFS Stress Test and Live Data Verification.
　　TestDFSIO: Distributed i/o benchmark.
　　fail: a job that always fails
　　filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
　　largesorter: Large-Sort tester
　　loadgen: Generic map/reduce load generator
　　mapredtest: A map/reduce test check.
　　minicluster: Single process HDFS and MR cluster.
　　mrbench: A map/reduce benchmark that can create many small jobs
　　nnbench: A benchmark that stresses the namenode.
　　sleep: A job that sleeps at each map and reduce task.

　　testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does>　　testfilesystem: A test for FileSystem read/write.
　　testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
　　testsequencefile: A test for flat files of binary key value pairs.
　　testsequencefileinputformat: A test for sequence file input format.
　　testtextinputformat: A test for text input format.
　　threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill
　　[hadoop@tong1 hadoop-2.6.0]$ hadoop  jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar  TestDFSIO
　　15/02/03 15:28:32 INFO fs.TestDFSIO: TestDFSIO.1.7
　　Missing arguments.

　　Usage: TestDFSIO [genericOptions] -read [-random | -backward | -skip [-skipSize>　　[hadoop@tong1 hadoop-2.6.0]$
　　2.测试hadoop写的速度
　　向HDFS文件系统中写入数据,10个文件,每个文件10MB,文件存放到/benchmarks/TestDFSIO/io_data中
　　[hadoop@tong1 hadoop-2.6.0]$ hadoop  jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 10MB
　　15/02/03 15:38:21 INFO fs.TestDFSIO: TestDFSIO.1.7
　　15/02/03 15:38:21 INFO fs.TestDFSIO: nrFiles = 10
　　15/02/03 15:38:21 INFO fs.TestDFSIO: nrBytes (MB) = 10.0
　　15/02/03 15:38:21 INFO fs.TestDFSIO: bufferSize = 1000000
　　15/02/03 15:38:21 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO

　　SLF4J: Failed to load>　　SLF4J: Defaulting to no-operation (NOP) logger implementation
　　SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

　　15/02/03 15:38:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java>　　15/02/03 15:38:22 INFO fs.TestDFSIO: creating control file: 10485760 bytes, 10 files
　　15/02/03 15:38:30 INFO fs.TestDFSIO: created control files for: 10 files
　　15/02/03 15:38:30 INFO client.RMProxy: Connecting to ResourceManager at tong1/192.168.1.247:8032
　　15/02/03 15:38:30 INFO client.RMProxy: Connecting to ResourceManager at tong1/192.168.1.247:8032
　　15/02/03 15:38:31 INFO mapred.FileInputFormat: Total input paths to process : 10
　　15/02/03 15:38:31 INFO mapreduce.JobSubmitter: number of splits:10
　　15/02/03 15:38:31 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1422946799080_0004
　　15/02/03 15:38:32 INFO impl.YarnClientImpl: Submitted application application_1422946799080_0004
　　15/02/03 15:38:32 INFO mapreduce.Job: The url to track the job: http://tong1:8088/proxy/application_1422946799080_0004/
　　15/02/03 15:38:32 INFO mapreduce.Job: Running job: job_1422946799080_0004
　　15/02/03 15:38:39 INFO mapreduce.Job: Job job_1422946799080_0004 running in uber mode : false
　　15/02/03 15:38:39 INFO mapreduce.Job:  map 0% reduce 0%
　　15/02/03 15:38:50 INFO mapreduce.Job:  map 10% reduce 0%
　　15/02/03 15:38:52 INFO mapreduce.Job:  map 20% reduce 0%
　　15/02/03 15:39:02 INFO mapreduce.Job:  map 20% reduce 7%
　　15/02/03 15:39:08 INFO mapreduce.Job:  map 30% reduce 10%
　　15/02/03 15:39:55 INFO mapreduce.Job:  map 37% reduce 10%
　　15/02/03 15:39:59 INFO mapreduce.Job:  map 43% reduce 10%
　　15/02/03 15:40:03 INFO mapreduce.Job:  map 50% reduce 10%
　　15/02/03 15:40:13 INFO mapreduce.Job:  map 57% reduce 10%
　　15/02/03 15:40:33 INFO mapreduce.Job:  map 63% reduce 10%
　　15/02/03 15:40:43 INFO mapreduce.Job:  map 70% reduce 10%
　　15/02/03 15:40:50 INFO mapreduce.Job:  map 73% reduce 10%
　　15/02/03 15:40:51 INFO mapreduce.Job:  map 73% reduce 13%
　　15/02/03 15:40:52 INFO mapreduce.Job:  map 77% reduce 13%
　　15/02/03 15:40:54 INFO mapreduce.Job:  map 80% reduce 17%
　　15/02/03 15:40:57 INFO mapreduce.Job:  map 80% reduce 20%
　　15/02/03 15:41:00 INFO mapreduce.Job:  map 83% reduce 20%
　　15/02/03 15:41:01 INFO mapreduce.Job:  map 90% reduce 20%
　　15/02/03 15:41:03 INFO mapreduce.Job:  map 90% reduce 23%
　　15/02/03 15:41:05 INFO mapreduce.Job:  map 93% reduce 23%
　　15/02/03 15:41:09 INFO mapreduce.Job:  map 93% reduce 27%
　　15/02/03 15:41:15 INFO mapreduce.Job:  map 97% reduce 27%
　　15/02/03 15:41:18 INFO mapreduce.Job:  map 97% reduce 30%
　　15/02/03 15:41:34 INFO mapreduce.Job:  map 100% reduce 30%
　　15/02/03 15:41:45 INFO mapreduce.Job:  map 100% reduce 100%
　　15/02/03 15:41:47 INFO mapreduce.Job: Job job_1422946799080_0004 completed successfully
　　15/02/03 15:41:47 INFO mapreduce.Job: Counters: 53
　　File System Counters
　　FILE: Number of bytes read=842
　　FILE: Number of bytes written=1168035
　　FILE: Number of read operations=0
　　FILE: Number of large read operations=0
　　FILE: Number of write operations=0
　　HDFS: Number of bytes read=2310
　　HDFS: Number of bytes written=104857677
　　HDFS: Number of read operations=43
　　HDFS: Number of large read operations=0
　　HDFS: Number of write operations=12
　　Job Counters
　　Failed map tasks=1
　　Killed map tasks=8
　　Launched map tasks=19
　　Launched reduce tasks=1
　　Other local map tasks=1
　　Data-local map tasks=9
　　Rack-local map tasks=9
　　Total time spent by all maps in occupied slots (ms)=1501978
　　Total time spent by all reduces in occupied slots (ms)=172661
　　Total time spent by all map tasks (ms)=1501978
　　Total time spent by all reduce tasks (ms)=172661
　　Total vcore-seconds taken by all map tasks=1501978
　　Total vcore-seconds taken by all reduce tasks=172661
　　Total megabyte-seconds taken by all map tasks=1538025472
　　Total megabyte-seconds taken by all reduce tasks=176804864
　　Map-Reduce Framework
　　Map input records=10
　　Map output records=50
　　Map output bytes=736
　　Map output materialized bytes=896
　　Input split bytes=1190
　　Combine input records=0
　　Combine output records=0
　　Reduce input groups=5
　　Reduce shuffle bytes=896
　　Reduce input records=50
　　Reduce output records=5
　　Spilled Records=100
　　Shuffled Maps =10
　　Failed Shuffles=0
　　Merged Map outputs=10
　　GC time elapsed (ms)=8949
　　CPU time spent (ms)=24290
　　Physical memory (bytes) snapshot=2860949504
　　Virtual memory (bytes) snapshot=22915002368
　　Total committed heap usage (bytes)=2033188864
　　Shuffle Errors
　　BAD_ID=0
　　CONNECTION=0
　　IO_ERROR=0
　　WRONG_LENGTH=0
　　WRONG_MAP=0
　　WRONG_REDUCE=0
　　File Input Format Counters
　　Bytes Read=1120
　　File Output Format Counters
　　Bytes Written=77
　　15/02/03 15:41:47 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
　　15/02/03 15:41:47 INFO fs.TestDFSIO:            Date & time: Tue Feb 03 15:41:47 CST 2015
　　15/02/03 15:41:47 INFO fs.TestDFSIO:        Number of files: 10
　　15/02/03 15:41:47 INFO fs.TestDFSIO: Total MBytes processed: 100.0
　　15/02/03 15:41:47 INFO fs.TestDFSIO:      Throughput mb/sec: 0.4277928455924503
　　15/02/03 15:41:47 INFO fs.TestDFSIO: Average IO rate mb/sec: 2.8419787883758545
　　15/02/03 15:41:47 INFO fs.TestDFSIO:  IO rate std deviation: 3.5636629341100754
　　15/02/03 15:41:47 INFO fs.TestDFSIO:     Test exec time sec: 197.297
　　15/02/03 15:41:47 INFO fs.TestDFSIO:
　　[hadoop@tong1 hadoop-2.6.0]$ cat TestDFSIO_results.log        --查看写入的结果
　　----- TestDFSIO ----- : write
　　Date & time: Tue Feb 03 15:41:47 CST 2015
　　Number of files: 10
　　Total MBytes processed: 100.0
　　Throughput mb/sec: 0.4277928455924503
　　Average IO rate mb/sec: 2.8419787883758545
　　IO rate std deviation: 3.5636629341100754
　　Test exec time sec: 197.297
　　[hadoop@tong1 hadoop-2.6.0]$
　　3.测试hadoop读文件的速度
　　在HDFS文件系统中读入10个文件,每个文件10M
　　[hadoop@tong1 hadoop-2.6.0]$ hadoop  jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 10MB
　　15/02/03 15:51:53 INFO fs.TestDFSIO: TestDFSIO.1.7
　　15/02/03 15:51:53 INFO fs.TestDFSIO: nrFiles = 10
　　15/02/03 15:51:53 INFO fs.TestDFSIO: nrBytes (MB) = 10.0
　　15/02/03 15:51:53 INFO fs.TestDFSIO: bufferSize = 1000000
　　15/02/03 15:51:53 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO

　　SLF4J: Failed to load>　　SLF4J: Defaulting to no-operation (NOP) logger implementation
　　SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

　　15/02/03 15:51:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java>　　15/02/03 15:51:54 INFO fs.TestDFSIO: creating control file: 10485760 bytes, 10 files
　　15/02/03 15:51:56 INFO fs.TestDFSIO: created control files for: 10 files
　　15/02/03 15:51:56 INFO client.RMProxy: Connecting to ResourceManager at tong1/192.168.1.247:8032
　　15/02/03 15:51:56 INFO client.RMProxy: Connecting to ResourceManager at tong1/192.168.1.247:8032
　　15/02/03 15:51:57 INFO mapred.FileInputFormat: Total input paths to process : 10
　　15/02/03 15:51:57 INFO mapreduce.JobSubmitter: number of splits:10
　　15/02/03 15:51:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1422946799080_0005
　　15/02/03 15:51:57 INFO impl.YarnClientImpl: Submitted application application_1422946799080_0005
　　15/02/03 15:51:58 INFO mapreduce.Job: The url to track the job: http://tong1:8088/proxy/application_1422946799080_0005/
　　15/02/03 15:51:58 INFO mapreduce.Job: Running job: job_1422946799080_0005
　　15/02/03 15:52:10 INFO mapreduce.Job: Job job_1422946799080_0005 running in uber mode : false
　　15/02/03 15:52:10 INFO mapreduce.Job:  map 0% reduce 0%
　　15/02/03 15:52:21 INFO mapreduce.Job:  map 10% reduce 0%
　　15/02/03 15:52:27 INFO mapreduce.Job:  map 20% reduce 0%
　　15/02/03 15:52:29 INFO mapreduce.Job:  map 30% reduce 0%
　　15/02/03 15:52:30 INFO mapreduce.Job:  map 40% reduce 0%
　　15/02/03 15:52:32 INFO mapreduce.Job:  map 50% reduce 0%
　　15/02/03 15:52:38 INFO mapreduce.Job:  map 50% reduce 17%
　　15/02/03 15:52:39 INFO mapreduce.Job:  map 57% reduce 17%
　　15/02/03 15:52:40 INFO mapreduce.Job:  map 63% reduce 17%
　　15/02/03 15:52:41 INFO mapreduce.Job:  map 70% reduce 17%
　　15/02/03 15:52:42 INFO mapreduce.Job:  map 73% reduce 17%
　　15/02/03 15:52:43 INFO mapreduce.Job:  map 87% reduce 17%
　　15/02/03 15:52:44 INFO mapreduce.Job:  map 93% reduce 17%
　　15/02/03 15:52:45 INFO mapreduce.Job:  map 100% reduce 17%
　　15/02/03 15:52:51 INFO mapreduce.Job:  map 100% reduce 100%
　　15/02/03 15:52:57 INFO mapreduce.Job: Job job_1422946799080_0005 completed successfully
　　15/02/03 15:52:57 INFO mapreduce.Job: Counters: 50
　　File System Counters
　　FILE: Number of bytes read=816
　　FILE: Number of bytes written=1167961
　　FILE: Number of read operations=0
　　FILE: Number of large read operations=0
　　FILE: Number of write operations=0
　　HDFS: Number of bytes read=94374150
　　HDFS: Number of bytes written=64
　　HDFS: Number of read operations=53
　　HDFS: Number of large read operations=0
　　HDFS: Number of write operations=2
　　Job Counters
　　Launched map tasks=10
　　Launched reduce tasks=1
　　Data-local map tasks=7
　　Rack-local map tasks=4
　　Total time spent by all maps in occupied slots (ms)=236272
　　Total time spent by all reduces in occupied slots (ms)=27137
　　Total time spent by all map tasks (ms)=236272
　　Total time spent by all reduce tasks (ms)=27137
　　Total vcore-seconds taken by all map tasks=236272
　　Total vcore-seconds taken by all reduce tasks=27137
　　Total megabyte-seconds taken by all map tasks=241942528
　　Total megabyte-seconds taken by all reduce tasks=27788288
　　Map-Reduce Framework
　　Map input records=10
　　Map output records=50
　　Map output bytes=710
　　Map output materialized bytes=870
　　Input split bytes=1190
　　Combine input records=0
　　Combine output records=0
　　Reduce input groups=5
　　Reduce shuffle bytes=870
　　Reduce input records=50
　　Reduce output records=5
　　Spilled Records=100
　　Shuffled Maps =10
　　Failed Shuffles=0
　　Merged Map outputs=10
　　GC time elapsed (ms)=2853
　　CPU time spent (ms)=11890
　　Physical memory (bytes) snapshot=2568495104
　　Virtual memory (bytes) snapshot=22888431616
　　Total committed heap usage (bytes)=2021654528
　　Shuffle Errors
　　BAD_ID=0
　　CONNECTION=0
　　IO_ERROR=0
　　WRONG_LENGTH=0
　　WRONG_MAP=0
　　WRONG_REDUCE=0
　　File Input Format Counters
　　Bytes Read=1120
　　File Output Format Counters
　　Bytes Written=64
　　15/02/03 15:52:57 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
　　15/02/03 15:52:57 INFO fs.TestDFSIO:            Date & time: Tue Feb 03 15:52:57 CST 2015
　　15/02/03 15:52:57 INFO fs.TestDFSIO:        Number of files: 10
　　15/02/03 15:52:57 INFO fs.TestDFSIO: Total MBytes processed: 90.0
　　15/02/03 15:52:57 INFO fs.TestDFSIO:      Throughput mb/sec: 2.663509914175792
　　15/02/03 15:52:57 INFO fs.TestDFSIO: Average IO rate mb/sec: NaN
　　15/02/03 15:52:57 INFO fs.TestDFSIO:  IO rate std deviation: NaN
　　15/02/03 15:52:57 INFO fs.TestDFSIO:     Test exec time sec: 61.378
　　15/02/03 15:52:57 INFO fs.TestDFSIO:
　　[hadoop@tong1 hadoop-2.6.0]$ cat TestDFSIO_results.log        --查看结果
　　----- TestDFSIO ----- : write
　　Date & time: Tue Feb 03 15:41:47 CST 2015
　　Number of files: 10
　　Total MBytes processed: 100.0
　　Throughput mb/sec: 0.4277928455924503
　　Average IO rate mb/sec: 2.8419787883758545
　　IO rate std deviation: 3.5636629341100754
　　Test exec time sec: 197.297
　　----- TestDFSIO ----- : read
　　Date & time: Tue Feb 03 15:52:57 CST 2015
　　Number of files: 10
　　Total MBytes processed: 90.0
　　Throughput mb/sec: 2.663509914175792
　　Average IO rate mb/sec: NaN
　　IO rate std deviation: NaN
　　Test exec time sec: 61.378
　　[hadoop@tong1 hadoop-2.6.0]$
　　4.删除临时文件
　　[hadoop@tong1 hadoop-2.6.0]$ hadoop  jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar TestDFSIO -clean
　　15/02/03 15:55:19 INFO fs.TestDFSIO: TestDFSIO.1.7
　　15/02/03 15:55:19 INFO fs.TestDFSIO: nrFiles = 1
　　15/02/03 15:55:19 INFO fs.TestDFSIO: nrBytes (MB) = 1.0
　　15/02/03 15:55:19 INFO fs.TestDFSIO: bufferSize = 1000000
　　15/02/03 15:55:19 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO

　　SLF4J: Failed to load>　　SLF4J: Defaulting to no-operation (NOP) logger implementation
　　SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

　　15/02/03 15:55:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java>　　15/02/03 15:55:20 INFO fs.TestDFSIO: Cleaning up test files
　　[hadoop@tong1 hadoop-2.6.0]$