Hadoop study notes

yao000

Every datanode runs a block scanner, which periodically verifies all the blocks stored
on the datanode. This allows bad blocks to be detected and fixed before they are read
by clients. The DataBlockScanner maintains a list of blocks to verify and scans them one
by one for checksum errors. The scanner employs a throttling mechanism to preserve
disk bandwidth on the datanode.

Blocks are periodically verified every three weeks to guard against disk errors over time
(this is controlled by the dfs.datanode.scan.period.hours property, which defaults to
504 hours). Corrupt blocks are reported to the namenode to be fixed.
You can get a block verification report for a datanode by visiting the datanode’s web
interface at http://datanode:50075/blockScannerReport. Here’s an example of a report,
which should be self-explanatory:
Total Blocks                 :    194
Verified in last hour        :      0
Verified in last day         :     67
Verified in last week        :     94
Verified in last four weeks  :    187
Verified in SCAN_PERIOD      :    187
Not yet verified             :      7
Verified since restart       :     70
Scans since restart          :      1
Scan errors since restart    :      0
Transient scan errors        :      0
Current scan rate limit KBps :   1024
Progress this period         :      0%
Time left in cur period      :  99.47%

By specifying the listblocks parameter, http://datanode:50075/blockScannerReport?listblocks, the report is preceded by a list of all the blocks on the datanode along with
their latest verification status. Here is a snippet of the block list (lines are split to fit the
page):
blk_2880642477235589345_1712 : status : ok     type : local  scan time : 1328684200718   2012-02-08 01:56:40,718
blk_6862384560101574248_3203 : status : ok     type : none   scan time : 0               not yet verified
blk_-6204923618707049613_3146 : status : ok     type : none   scan time : 0               not yet verified
blk_8096385507793977436_1470 : status : ok     type : local  scan time : 1328726026095   2012-02-08 13:33:46,095
blk_-8383560827245098225_1470 : status : ok     type : local  scan time : 1328736508026   2012-02-08 16:28:28,026
blk_-1634356630613489001_3191 : status : ok     type : none   scan time : 0               not yet verified
blk_-6752468218406655007_3201 : status : ok     type : none   scan time : 0               not yet verified
blk_-1692843323764239407_1772 : status : ok     type : local  scan time : 1328742671906   2012-02-08 18:11:11,906
blk_3849616369028352463_3200 : status : ok     type : none   scan time : 0               not yet verified
blk_-34525423848470829_1226 : status : ok     type : local  scan time : 1328747018814   2012-02-08 19:23:38,814
blk_6305423182925037634_1226 : status : ok     type : local  scan time : 1328753126642   2012-02-08 21:05:26,642
blk_543251317099843969_3202 : status : ok     type : none   scan time : 0               not yet verified
blk_6417698981647840069_1833 : status : ok     type : local  scan time : 1328779874402   2012-02-09 04:31:14,402
blk_-7222269942471718886_3199 : status : ok     type : none   scan time : 0               not yet verified

check DataBlockScanner.java for detail