Hadoop集群基准测试

cmkzjw

　　hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar TestDFSIO -write -nrFiles 10 -size 100MB



14/07/21 21:29:42 INFO mapreduce.Job: Counters: 45
File System Counters
FILE: Number of bytes read=856
FILE: Number of bytes written=872121
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2320
HDFS: Number of bytes written=1048576079
HDFS: Number of read operations=43
HDFS: Number of large read operations=0
HDFS: Number of write operations=12
Job Counters
Killed map tasks=3
Launched map tasks=13
Launched reduce tasks=1
Data-local map tasks=6
Rack-local map tasks=7
Total time spent by all maps in occupied slots (ms)=1109317
Total time spent by all reduces in occupied slots (ms)=91582
Map-Reduce Framework
Map input records=10
Map output records=50
Map output bytes=750
Map output materialized bytes=910
Input split bytes=1200
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=910
Reduce input records=50
Reduce output records=5
Spilled Records=100
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=7691
CPU time spent (ms)=30730
Physical memory (bytes) snapshot=2555801600
Virtual memory (bytes) snapshot=11073003520
Total committed heap usage (bytes)=1874329600
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1120
File Output Format Counters
Bytes Written=79
14/07/21 21:29:42 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
14/07/21 21:29:42 INFO fs.TestDFSIO:            Date & time: Mon Jul 21 21:29:42 CST 2014
14/07/21 21:29:42 INFO fs.TestDFSIO:        Number of files: 10
14/07/21 21:29:42 INFO fs.TestDFSIO: Total MBytes processed: 1000.0
14/07/21 21:29:42 INFO fs.TestDFSIO:      Throughput mb/sec: 3.403479036270876
14/07/21 21:29:42 INFO fs.TestDFSIO: Average IO rate mb/sec: 4.484561443328857
14/07/21 21:29:42 INFO fs.TestDFSIO:  IO rate std deviation: 1.7674698292439088
14/07/21 21:29:42 INFO fs.TestDFSIO:     Test exec time sec: 190.944

　　
　　hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar TestDFSIO -write -nrFiles 10 -size 1GB



14/07/21 22:09:57 INFO mapreduce.Job: Counters: 47
File System Counters
FILE: Number of bytes read=875
FILE: Number of bytes written=872159
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2320
HDFS: Number of bytes written=10737418320
HDFS: Number of read operations=43
HDFS: Number of large read operations=0
HDFS: Number of write operations=12
Job Counters
Failed map tasks=7
Killed map tasks=1
Launched map tasks=18
Launched reduce tasks=1
Other local map tasks=8
Data-local map tasks=6
Rack-local map tasks=4
Total time spent by all maps in occupied slots (ms)=4322906
Total time spent by all reduces in occupied slots (ms)=155182
Map-Reduce Framework
Map input records=10
Map output records=50
Map output bytes=769
Map output materialized bytes=929
Input split bytes=1200
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=929
Reduce input records=50
Reduce output records=5
Spilled Records=100
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=26386
CPU time spent (ms)=119850
Physical memory (bytes) snapshot=2830766080
Virtual memory (bytes) snapshot=11052699648
Total committed heap usage (bytes)=2014642176
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1120
File Output Format Counters
Bytes Written=80
14/07/21 22:09:59 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
14/07/21 22:10:00 INFO fs.TestDFSIO:            Date & time: Mon Jul 21 22:09:59 CST 2014
14/07/21 22:10:00 INFO fs.TestDFSIO:        Number of files: 10
14/07/21 22:10:00 INFO fs.TestDFSIO: Total MBytes processed: 10240.0
14/07/21 22:10:00 INFO fs.TestDFSIO:      Throughput mb/sec: 6.388915232072354
14/07/21 22:10:00 INFO fs.TestDFSIO: Average IO rate mb/sec: 11.30741024017334
14/07/21 22:10:00 INFO fs.TestDFSIO:  IO rate std deviation: 7.81592713288423
14/07/21 22:10:00 INFO fs.TestDFSIO:     Test exec time sec: 556.575

　　
　　环境信息：
　　x220, i5-2430M, 4G*2, 320G
　　Windows 7 Ultimate, Vmware 8
　　Virtual Workstation: 2 cores, 2G, 40G
　　Hadoop 2.2.0