hadoop: Failed fetch notification Too many fetch-failures

Matthewl

　　本人是hadoop新手，在一次应用中配置出现了问题：就是hadoop 的map阶段正常，但是reduce却卡在18%哪里，一直要将近一个小时才能完成，查看日志如下：
2011-10-03 09:45:58,330 INFO org.apache.hadoop.mapred.JobInProgress: Choosing rack-local task task_201110022127_0003_m_000011
2011-10-0309:46:01,334 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000011_0' has completed task_201110022127_0003_m_000011 successfully.
2011-10-03 09:46:01,334INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:9  completedMapsInputSize:437327225  completedMapsOutputSize:193
2011-10-03 09:46:01,737 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000006_0' has completed task_201110022127_0003_m_000006 successfully.
2011-10-03 09:46:01,737INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:10  completedMapsInputSize:504436090  completedMapsOutputSize:215
2011-10-03 09:46:01,738 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000007_0' has completed task_201110022127_0003_m_000007 successfully.
2011-10-03 09:46:01,738INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:11  completedMapsInputSize:571544955  completedMapsOutputSize:237
2011-10-03 09:46:04,007 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000009_0' has completed task_201110022127_0003_m_000009 successfully.
2011-10-03 09:46:04,007INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:12  completedMapsInputSize:593329451  completedMapsOutputSize:258
2011-10-03 09:46:04,008 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000008_0' has completed task_201110022127_0003_m_000008 successfully.
2011-10-03 09:46:04,008INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:13  completedMapsInputSize:615113993  completedMapsOutputSize:279
2011-10-03 09:46:13,349 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task attempt_201110022127_0003_m_000000_0
2011-10-03 09:48:49,450 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task attempt_201110022127_0003_m_000000_0
2011-10-03 09:53:52,659 INFO org.apache.hadoop.mapred.JobInProgress:Failed fetch notification #3 for task attempt_201110022127_0003_m_000000_0
2011-10-03 09:53:52,659 INFO org.apache.hadoop.mapred.JobInProgress: Too many fetch-failuresfor output of task: attempt_201110022127_0003_m_000000_0 ... killing it
2011-10-0309:53:52,659 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201110022127_0003_m_000000_0: Too many fetch-failures
2011-10-0309:53:52,661 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201110022127_0003_m_000000_1' to tip task_201110022127_0003_m_000000, for tracker 'tracker_ubuntu-server:127.0.0.1/127.0.0.1:49740'
2011-10-03 09:53:52,661 INFO org.apache.hadoop.mapred.JobInProgress: Choosing rack-local task task_201110022127_0003_m_000000
2011-10-0309:53:53,107 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201110022127_0003_m_000000_0' from 'tracker_fangfei-desktop:127.0.0.1/127.0.0.1:54181'
2011-10-03 09:53:58,264 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201110022127_0003_m_000000_2' to tip task_201110022127_0003_m_000000, for tracker 'tracker_huangzhongyuan-desktop:127.0.0.1/127.0.0.1:48184'
2011-10-03 09:53:58,264 INFO org.apache.hadoop.mapred.JobInProgress: Choosing rack-local task task_201110022127_0003_m_000000
2011-10-0309:54:01,668 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000000_1' has completed task_201110022127_0003_m_000000 successfully.
2011-10-03 09:54:01,668INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:14  completedMapsInputSize:682222858  completedMapsOutputSize:301
2011-10-03 09:54:07,282 INFO org.apache.hadoop.mapred.JobTracker: Adding task (cleanup)'attempt_201110022127_0003_m_000000_2' to tip task_201110022127_0003_m_000000, for tracker 'tracker_huangzhongyuan-desktop:127.0.0.1/127.0.0.1:48184'
2011-10-0309:54:10,285 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201110022127_0003_m_000000_2' from 'tracker_huangzhongyuan
　　在网上google了很久没找到答案，后来有的说把/etc/hosts中的127.0.0.1的主机映射去掉，试了也不行。知道第二天上午google的时候才受到一点启发，突然想到，这个问题是发生在reduce阶段，而提示的消息应该是取不到map阶段的结果，既然在Failed fetch notification #1 for task attempt_201110022127_0003_m_000000_0中有取不到的任务分块的名字，说明namenode正常工作，namenode通知reduce节点进行reduce操作，而它却取不到，只能说明它没法和那些节点通信，又由于我在配置hadoop的时候用的是主机的名字，不是ip，所以想到解决办法应该是把各个datanode节点的映射互相加到/etc/hosts中。试了一下，果然正确。所以在此记录。

解决办法：把各个datanode节点的映射都加到每个datanode节点的/etc/hosts中