HADOOP报错Incompatible namespaceIDs

fdf

　　转自：blog.iyunv.com/wh62592855/article/details/5752199
　　今早一来，突然发现使用-put命令往HDFS里传数据传不上去了，抱一大堆错误，然后我使用bin/hadoop dfsadmin -report查看系统状态
　　admin@adw1:/home/admin/joe.wangh/hadoop-0.19.2>bin/hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: ?%

-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
　　
　　使用bin/stop-all.sh关闭HADOOP
　　
　　admin@adw1:/home/admin/joe.wangh/hadoop-0.19.2>bin/stop-all.sh
stopping jobtracker
172.16.197.192: stopping tasktracker
172.16.197.193: stopping tasktracker
stopping namenode
172.16.197.193: no datanode to stop
172.16.197.192: no datanode to stop
172.16.197.191: stopping secondarynamenode
　　
　　哦，看到了吧，发现datanode前面并没有启动起来。去DATANODE上查看一下日志
　　admin@adw2:/home/admin/joe.wangh/hadoop-0.19.2/logs>vi hadoop-admin-datanode-adw2.hst.ali.dw.alidc.net.log

************************************************************/
2010-07-21 10:12:11,987 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:Incompatible namespaceIDs in /home/admin/joe.wangh/hadoop/data/dfs.data.dir: namenode namespaceID = 898136669; datanode namespaceID = 2127444065
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:288)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:206)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1239)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1194)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1202)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1324)
......
　　
　　错误提示namespaceIDs不一致。
　　
　　下面给出两种解决办法，我使用的是第二种。
　　

Workaround 1: Start from scratch

I can testify that the following steps solve thiserror, but the side effects won't make you happy (me neither). The crudeworkaround I have found is to:

1.     stop the cluster

2.     delete the data directory on the problematic datanode:the directory is specified by dfs.data.dir in conf/hdfs-site.xml; if youfollowed this tutorial, the relevant directory is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data

3.     reformat the namenode (NOTE: all HDFS data is lostduring this process!)

4.     restart the cluster

When deleting all the HDFS data and starting fromscratch does not sound like a good idea (it might be ok during the initialsetup/testing), you might give the second approach a try.

Workaround 2: Updating namespaceID of problematicdatanodes

Big thanks to Jared Stehler for the followingsuggestion. I have not tested it myself yet, but feel free to try it out andsend me your feedback. This workaround is "minimally invasive" as youonly have to edit one file on the problematic datanodes:

1.     stop the datanode

2.     edit the value of namespaceID in /current/VERSIONto match the value of the current namenode

3.     restart the datanode

If you followed the instructions in my tutorials, thefull path of the relevant file is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data/current/VERSION(background: dfs.data.dir is by default set to ${hadoop.tmp.dir}/dfs/data, andwe set hadoop.tmp.dir to /usr/local/hadoop-datastore/hadoop-hadoop).

If you wonder how the contents of VERSION look like,here's one of mine:

#contents of /current/VERSION

namespaceID=393514426

storageID=DS-1706792599-10.10.10.1-50010-1204306713481

cTime=1215607609074

storageType=DATA_NODE

layoutVersion=-13

原因:每次namenode format会重新创建一个namenodeId,而tmp/dfs/data下包含了上次format下的id,namenode format清空了namenode下的数据,但是没有晴空datanode下的数据,导致启动时失败,所要做的就是每次fotmat前,清空tmp一下的所有目录.