hadoop升级之fsck命令迎战miss block警告

lovegigi

继上次升级hadoop完毕后，集群启动正常，但是在访问Namenode的50070的界面上，发现了如下截图的警告信息：

如上异常，是什么意思呢？看了下官方的FAQ,大致意思就是，有几个块的数据，在现有的DataNode节点上，没有一个存储的，但是在NameNode的元数据里却存在。
怎么解决这个问题呢，下面介绍一个hadoop的健康监测命令fsck，
  fsck工具来检验HDFS中的文件是否正常可用。这个工具可以检测文件块是否在DataNode中丢失，是否低于或高于文件副本。 fack命令用法如下：
注：此处必须是启动hadoop hdfs的账号才有权查看
Usage: DFSck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]]
        <path>  检查的起始目录
        -move   将损坏的文件移到到/lost+found
        -delete 删除损坏的文件
        -files  打印出所有被检查的文件
        -openforwrite   打印出正在写的文件
        -list-corruptfileblocks print out list of missing blocks and files they belong to
        -blocks 打印出block报告
        -locations      打印出每个block的位置
        -racks  打印出data-node的网络拓扑结构
                默认情况下，fsck会忽略正在写的文件，使用-openforwrite可以汇报这种文件

下面是是一个运行的命令例子：
[search@fse01 hadoop]$ bin/hadoop fsck / -files
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Connecting to namenode via http://hadoop1:50070
FSCK started by search (auth:SIMPLE) from /127.0.0.1 for path / at Mon Jul 21 15:10:06 CST 2014
/ <dir>
/tmp <dir>
/tmp/hadoop-search <dir>
/tmp/hadoop-search/mapred <dir>
/tmp/hadoop-search/mapred/staging <dir>
/tmp/hadoop-search/mapred/staging/search <dir>
/tmp/hadoop-search/mapred/staging/search/.staging <dir>
/tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0001 <dir>
/tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006 <dir>
/tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar 13393150 bytes, 1 block(s):
/tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-2026656432152987415
MISSING 1 blocks of total size 13393150 B
/tmp/hadoop-search/mapred/staging/zhouming <dir>
/tmp/hadoop-search/mapred/staging/zhouming/.staging <dir>
/tmp/hadoop-search/mapred/staging/zhouming/.staging/job_201403111456_0028 <dir>
/tmp/hadoop-search/mapred/system <dir>
/tmp/hadoop-search/mapred/system/jobtracker.info 4 bytes, 1 block(s):
/tmp/hadoop-search/mapred/system/jobtracker.info: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-7234718182174847600
MISSING 1 blocks of total size 4 B
/tmp/hadoop-yarn <dir>
/tmp/hadoop-yarn/staging <dir>
/tmp/hadoop-yarn/staging/history <dir>
/tmp/hadoop-yarn/staging/history/done_intermediate <dir>
/tmp/hadoop-yarn/staging/history/done_intermediate/search <dir>
/tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0001-1405911635075-search-random%2Dwriter-1405912101075-40-0-SUCCEEDED-default.jhist 340102 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0001.summary 342 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0001_conf.xml 79756 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0003-1405921787298-search-com.dhgate.search.fse.proinfo.dataload.NewLoadData-1405921809468-1-1-SUCCEEDED-default.jhist 32989 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0003.summary 390 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0003_conf.xml 83089 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0004-1405922486108-search-Rebuild+Index%3Afsesearch%2D%3Ecollection1%2D%3Eshard1-1405922501203-1-0-SUCCEEDED-default.jhist 20173 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0004.summary 369 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0004_conf.xml 86592 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/search <dir>
/tmp/hadoop-yarn/staging/search/.staging <dir>
/tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002 <dir>
/tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.jar 30472539 bytes, 1 block(s):  Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742242_1099511720037. Target Replicas is 10 but found 4 replica(s).
/tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.split 116 bytes, 1 block(s):  Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742243_1099511720038. Target Replicas is 10 but found 4 replica(s).
/tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.splitmetainfo 31 bytes, 1 block(s):  OK
/tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.xml 72772 bytes, 1 block(s):  OK
/user <dir>
/user/search <dir>
/user/search/fse <dir>
/user/search/fse/kms <dir>
/user/search/fse/kms/in <dir>
/user/search/fse/kms/in/0 3 bytes, 1 block(s):  OK
/user/search/fse/kms/index <dir>
/user/search/fse/kms/index/index_shard_00_5e62b650 <dir>
/user/search/fse/kms/index/index_shard_00_5e62b650/_0.fdt 11069 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/_0.fdx 47 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/_0.fnm 992 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/_0.nvd 683 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/_0.nvm 68 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/_0.si 376 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.doc 50029 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.pos 101757 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.tim 62843 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.tip 2319 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/segments.gen 20 bytes, 1 block(s):  OK
/user/search/fse/kms/index/index_shard_00_5e62b650/segments_1 69 bytes, 1 block(s):  OK
/user/search/fse/kms/out <dir>
/user/search/fse/kms/out/0 <dir>
/user/search/fse/kms/out/0/00 425334 bytes, 1 block(s):  OK
/user/search/fse/proinfo <dir>
/user/search/fse/proinfo/schema.xml 32194 bytes, 1 block(s):  OK
/user/search/rand <dir>
/user/search/rand/_SUCCESS 0 bytes, 0 block(s):  OK
/user/search/rand/part-m-00000 1077280794 bytes, 9 block(s):  OK
/user/search/rand/part-m-00001 1077294346 bytes, 9 block(s):  OK
/user/search/rand/part-m-00002 1077276960 bytes, 9 block(s):  OK
/user/search/rand/part-m-00003 1077280905 bytes, 9 block(s):  OK
/user/search/rand/part-m-00004 1077288199 bytes, 9 block(s):  OK
/user/search/rand/part-m-00005 1077290933 bytes, 9 block(s):  OK
/user/search/rand/part-m-00006 1077270249 bytes, 9 block(s):  OK
/user/search/rand/part-m-00007 1077290630 bytes, 9 block(s):  OK
/user/search/rand/part-m-00008 1077279315 bytes, 9 block(s):  OK
/user/search/rand/part-m-00009 1077294441 bytes, 9 block(s):  OK
/user/search/rand/part-m-00010 1077288193 bytes, 9 block(s):  OK
/user/search/rand/part-m-00011 1077299978 bytes, 9 block(s):  OK
/user/search/rand/part-m-00012 1077274992 bytes, 9 block(s):  OK
/user/search/rand/part-m-00013 1077295278 bytes, 9 block(s):  OK
/user/search/rand/part-m-00014 1077280338 bytes, 9 block(s):  OK
/user/search/rand/part-m-00015 1077285350 bytes, 9 block(s):  OK
/user/search/rand/part-m-00016 1077276847 bytes, 9 block(s):  OK
/user/search/rand/part-m-00017 1077285962 bytes, 9 block(s):  OK
/user/search/rand/part-m-00018 1077280969 bytes, 9 block(s):  OK
/user/search/rand/part-m-00019 1077285691 bytes, 9 block(s):  OK
/user/search/rand/part-m-00020 1077292174 bytes, 9 block(s):  OK
/user/search/rand/part-m-00021 1077292141 bytes, 9 block(s):  OK
/user/search/rand/part-m-00022 1077285419 bytes, 9 block(s):  OK
/user/search/rand/part-m-00023 1077285751 bytes, 9 block(s):  OK
/user/search/rand/part-m-00024 1077282762 bytes, 9 block(s):  OK
/user/search/rand/part-m-00025 1077282468 bytes, 9 block(s):  OK
/user/search/rand/part-m-00026 1077283184 bytes, 9 block(s):  OK
/user/search/rand/part-m-00027 1077281056 bytes, 9 block(s):  OK
/user/search/rand/part-m-00028 1077292387 bytes, 9 block(s):  OK
/user/search/rand/part-m-00029 1077283341 bytes, 9 block(s):  OK
/user/search/rand/part-m-00030 1077288015 bytes, 9 block(s):  OK
/user/search/rand/part-m-00031 1077276923 bytes, 9 block(s):  OK
/user/search/rand/part-m-00032 1077296469 bytes, 9 block(s):  OK
/user/search/rand/part-m-00033 1077288224 bytes, 9 block(s):  OK
/user/search/rand/part-m-00034 1077278284 bytes, 9 block(s):  OK
/user/search/rand/part-m-00035 1077297217 bytes, 9 block(s):  OK
/user/search/rand/part-m-00036 1077289846 bytes, 9 block(s):  OK
/user/search/rand/part-m-00037 1077284545 bytes, 9 block(s):  OK
/user/search/rand/part-m-00038 1077290084 bytes, 9 block(s):  OK
/user/search/rand/part-m-00039 1077282121 bytes, 9 block(s):  OK
Status: CORRUPT
Total size:    43136702998 B
Total dirs:    32
Total files:   71
Total symlinks:                0
Total blocks (validated):      390 (avg. block size 110606930 B)
********************************
CORRUPT FILES:        2
MISSING BLOCKS:       2
MISSING SIZE:         13393154 B
CORRUPT BLOCKS:       2
********************************
Minimally replicated blocks:   388 (99.48718 %)
Over-replicated blocks:        0 (0.0 %)
Under-replicated blocks:       2 (0.51282054 %)
Mis-replicated blocks:         0 (0.0 %)
Default replication factor:    3
Average block replication:     2.9897435
Corrupt blocks:                2
Missing replicas:              12 (1.0075567 %)
Number of data-nodes:          4
Number of racks:               1
FSCK ended at Mon Jul 21 15:10:06 CST 2014 in 48 milliseconds

The filesystem under path '/' is CORRUPT
通过上面这个命令我们就可以监测出文件的存储状况，监测出有无效的或者缺失的block，下面我们就可以执行-delete命令，来清楚无效的信息块。


[search@fse01 hadoop]$ bin/hadoop  fsck /  -delete
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Connecting to namenode via http://hadoop1:50070
FSCK started by search (auth:SIMPLE) from /127.0.0.1 for path / at Mon Jul 21 15:11:39 CST 2014
.
/tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-2026656432152987415
/tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar: MISSING 1 blocks of total size 13393150 B..
/tmp/hadoop-search/mapred/system/jobtracker.info: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-7234718182174847600
/tmp/hadoop-search/mapred/system/jobtracker.info: MISSING 1 blocks of total size 4 B...........
/tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.jar:  Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742242_1099511720037. Target Replicas is 10 but found 4 replica(s).
.
/tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.split:  Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742243_1099511720038. Target Replicas is 10 but found 4 replica(s).
..........................................................Status: CORRUPT
Total size:    43136702998 B
Total dirs:    32
Total files:   71
Total symlinks:                0
Total blocks (validated):      390 (avg. block size 110606930 B)
********************************
CORRUPT FILES:        2
MISSING BLOCKS:       2
MISSING SIZE:         13393154 B
CORRUPT BLOCKS:       2
********************************
Minimally replicated blocks:   388 (99.48718 %)
Over-replicated blocks:        0 (0.0 %)
Under-replicated blocks:       2 (0.51282054 %)
Mis-replicated blocks:         0 (0.0 %)
Default replication factor:    3
Average block replication:     2.9897435
Corrupt blocks:                2
Missing replicas:              12 (1.0075567 %)
Number of data-nodes:          4
Number of racks:               1
FSCK ended at Mon Jul 21 15:11:39 CST 2014 in 45 milliseconds

The filesystem under path '/' is CORRUPT
最后一点，需要注意的是，这个命令在namenode文件信息较大的时候，会比较影响hadoop性能，所以应该慎用，通常可以在集群空闲的时间段，执行一次，查看整体的HDFS副本健康状况！