hadoop错误- Too many open files

Matthewl

http://www.iyunv.com/wycg1984/archive/2010/04/27/1722431.html

下面是Hadoop使用过程中常见错误及解决方法的汇总 （英文部分会后续转为中文）

1. Too many open files错误
有時候 Map Reduce 的工作跑一跑，會發現 datanode 突然都陣亡，去看 log 會發現很多 Too many open files 的錯誤：   
2008-09-11 20:20:22,836 ERROR org.apache.hadoop.dfs.DataNode: 192.168.1.34:50010:DataXceiver: java.io.IOException: Too many open files   
at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)   
at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:68)   
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:52)   
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)   
at sun.nio.ch.Util.getTemporarySelector(Util.java:123)   
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92)   
at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1150)   
at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:994)   
at java.lang.Thread.run(Thread.java:619)   
這個發生的原因是同時間很多 client 要去跟 datanode 要東西，因此消耗太多的 file descriptor，那又因為我在用的 Linux 上面預設單一 process 能開的檔案只有 1,024 個，>   
於是就造成了這種結果。   
修正的方法是去 /etc/security/limits.conf 加上這行：

* - nofile 8192

　　讓單一 process 能同時開到 8,192 個檔案。改好後重開 datanode 就可以了。

2. 出现错误时采取的步骤:看Log，试试单机模式

• If you are having problems, check the logs in the logs directory to see if there are any Hadoop errors or Java Exceptions.
  
• Logs are named by machine and job they carry out in the cluster, and this can help you figure out which part of your configuration is giving you trouble.
  
• Even if you were very careful, the problem is probably with your configuration. Try running the grep example from the QuickStart. If it doesn't run then you need to check your configuration.
  
• If you can't get it to work on a real cluster, try it on a single-node.
  

3. 常见问题与解决
　　Symptom
Possible Problem
Possible Solution
　　You get an error that you cluster is in "safe mode"
Your cluster enters safe mode when it hasn't been able to verify that all the data nodes necessary to replicate your data are up and responding. Check the documentation to learn more about safe mode.

• First, wait a minute or two and then retry your command. If you just started your cluster, it's possible that it isn't fully initialized yet.
  
• If waiting a few minutes didn't help and you still get a "safe mode" error, check your logs to see if any of your data nodes didn't start correctly (either they have Java exceptions in their logs or they have messages stating that they are unable to contact some other node in your cluster). If this is the case you need to resolve the configuration issue (or possibly pick some new nodes) before you can continue.
  

　　You get a NoRouteToHostException in your logs or in stderr output from a command.
One of your nodes cannot be reached correctly. This may be a firewall issue, so you should report it to me.
The only workaround is to pick a new node to replace the unreachable one. Currently, I think that creusa is unreachable, but all other Linux boxes should be okay. None of the Macs will currently work in a cluster.
　　You get an error that "remote host identification has changed" when you try to ssh to localhost.
You have moved your single node cluster from one machine in the Berry Patch to another. The name localhost thus is pointing to a new machine, and your ssh client thinks that it might be a man-in-the-middle attack.
You can ask your login to skip checking the validity of localhost. You do this by setting NoHostAuthenticationForLocalhost to yes in ~/.ssh/config. You can accomplish this with the following command:

echo "NoHostAuthenticationForLocalhost yes" >>~/.ssh/config
　　Your DataNode is started and you can create directories with bin/hadoop dfs -mkdir, but you get an error message when you try to put files into the HDFS (e.g., when you run a command like bin/hadoop dfs -put).
Creating directories is only a function of the NameNode, so your DataNode is not exercised until you actually want to put some bytes into a file. If you are sure that the DataNode is started, then it could be that your DataNodes are out of disk space.

• Go to the HDFS info web page (open your web browser and go to http://namenode:dfs_info_port where namenode is the hostname of your NameNode and dfs_info_port is the port you chose dfs.info.port; if followed the QuickStart on your personal computer then this URL will be http://localhost:50070). Once at that page click on the number where it tells you how many DataNodes you have to look at a list of the DataNodes in your cluster.
  
• If it says you have used 100% of your space, then you need to free up room on local disk(s) of the DataNode(s).
  
• If you are on Windows then this number will not be accurate (there is some kind of bug either in Cygwin's df.exe or in Windows). Just free up some more space and you should be okay. On one Windows machine we tried the disk had 1GB free but Hadoop reported that it was 100% full. Then we freed up another 1GB and then it said that the disk was 99.15% full and started writing data into the HDFS again. We encountered this bug on Windows XP SP2.
  

　　You try to run the grep example from the QuickStart but you get an error message like this:

java.io.IOException: Not a file:
  hdfs://localhost:9000/user/ross/input/conf
　　
You may have created a directory inside the input directory in the HDFS. For example, this might happen if you run bin/hadoop dfs -put conf input twice in a row (this would create a subdirectory in input... why?).
The easiest way to get the example run is to just start over and make the input anew.

bin/hadoop dfs -rmr input
bin/hadoop dfs -put conf input
　　Your DataNodes won't start, and you see something like this in logs/*datanode*:

Incompatible namespaceIDs in /tmp/hadoop-ross/dfs/data
　　
Your Hadoop namespaceID became corrupted. Unfortunately the easiest thing to do reformat the HDFS.
You need to do something like this:

bin/stop-all.sh
rm -Rf /tmp/hadoop-your-username/*
bin/hadoop namenode -format
　　Be VERY careful with rm -Rf
　　When you try the grep example in the QuickStart, you get an error like the following:

org.apache.hadoop.mapred.InvalidInputException:
  Input path doesnt exist : /user/ross/input
　　
You haven't created an input directory containing one or more text files.

bin/hadoop dfs -put conf input
　　When you try the grep example in the QuickStart, you get an error like the following:

org.apache.hadoop.mapred.FileAlreadyExistsException:
  Output directory /user/ross/output already exists
　　
You might have already run the example once, creating an output directory. Hadoop doesn't like to overwrite files.
Remove the output directory before rerunning the example:

bin/hadoop dfs -rmr output
　　Alternatively you can change the output directory of the grep example, something like this:

bin/hadoop jar hadoop-*-examples.jar \
grep input output2 'dfs[a-z.]+'
　　You can run Hadoop jobs written in Java (like the grep example), but your HadoopStreaming jobs (such as the Python example that fetches web page titles) won't work.
You might have given only a relative path to the mapper and reducer programs. The tutorial originally just specified relative paths, but absolute paths are required if you are running in a real cluster.
Use absolute paths like this from the tutorial:

bin/hadoop jar contrib/hadoop-0.15.2-streaming.jar \
  -mapper  $HOME/proj/hadoop/multifetch.py         \
  -reducer $HOME/proj/hadoop/reducer.py            \
  -input   urls/*                                  \
  -output  titles