Steps to install Hadoop 2.x release (Yarn or Next-Gen) on multi-node cluster-Hei

kingbosster

　　In the previous post, we saw how to setup Hadoop 2.x on single-node. Here, we will see how to set up a multi-node cluster.

　　Hadoop 2.x>　　This post concentrates on installing Hadoop 2.x a.k.a. Yarn a.k.a. next-gen on a multi-node cluster.
　　Prerequisites:

• 　　Java 6 installed
  
• 　　Dedicated user for hadoop
  
• 　　SSH configured
  

　　Steps to install Hadoop 2.x:
　　1. Download tarball
　　You can download tarball for hadoop 2.x from here. Extract it to a folder say, /home/hduser/yarn on master and all the slaves. We assume dedicated user for Hadoop is “hduser”.
　　NOTE: Master and all the slaves must have the same user and hadoop directory on same path.
$ cd /home/hduser/yarn　　
$ sudo chown -R hduser:hadoop hadoop-2.0.1-alpha
　　2. Edit /etc/hosts
　　Add the association between the hostnames and the ip address for the master and the slaves on all the nodes in the /etc/hosts file. Make sure that the all the nodes in the cluster are able to ping to each other.
　　Important Change:
127.0.0.1 localhost localhost.localdomain my-laptop　　
127.0.1.1 my-laptop
　　If you have provided alias for localhost (as done in entries above), protocol buffers will try to connect to my-laptop from other hosts while making RPC calls which will fail.
　　Solution:
　　Assuming the machine (my-laptop) has ip address “10.3.3.43″, make an entry as follows in all the other machines:
10.3.3.43       my-laptop　　3. Password less SSH
　　Make sure that the master is able to do a password-less ssh to all the slaves.
　　4. Edit ~/.bashrc
export HADOOP_HOME=/home/hduser/yarn/hadoop-2.0.1-alpha　　
export HADOOP_MAPRED_HOME=$HADOOP_HOME
　　
export HADOOP_COMMON_HOME=$HADOOP_HOME
　　
export HADOOP_HDFS_HOME=$HADOOP_HOME
　　
export YARN_HOME=$HADOOP_HOME
　　
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
　　
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
　　5. Edit Hadoop environment files
　　Add JAVA_HOME to following files
　　Add following line at start of script in libexec/hadoop-config.sh :
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386/　　Add following lines at start of script in etc/hadoop/yarn-env.sh :
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386/　　
export HADOOP_HOME=/home/hduser/yarn/hadoop-2.0.1-alpha
　　
export HADOOP_MAPRED_HOME=$HADOOP_HOME
　　
export HADOOP_COMMON_HOME=$HADOOP_HOME
　　
export HADOOP_HDFS_HOME=$HADOOP_HOME
　　
export YARN_HOME=$HADOOP_HOME
　　
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
　　
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
　　Change the path as per your java installation.
　　6. Create Temp folder in HADOOP_HOME
$ mkdir -p $HADOOP_HOME/tmp　　7. Add properties in configuration files
　　Make changes as mentioned below in all the machines:
　　$HADOOP_CONF_DIR/core-site.xml
　　

　　

　　
　　fs.default.name
　　hdfs://master:9000
　　
　　
　　hadoop.tmp.dir
　　/home/hduser/yarn/hadoop-2.0.1-alpha/tmp
　　
　　

　　$HADOOP_CONF_DIR/hdfs-site.xml :
　　
　　
　　
　　dfs.replication
　　2
　　
　　
　　dfs.permissions
　　false
　　
　　
　　$HADOOP_CONF_DIR/mapred-site.xml :
　　

　　
　　mapreduce.framework.name
　　yarn
　　
　　

　　$HADOOP_CONF_DIR/yarn-site.xml :
　　
　　
　　yarn.nodemanager.aux-services
　　mapreduce.shuffle
　　
　　
　　yarn.nodemanager.aux-services.mapreduce.shuffle.class
　　org.apache.hadoop.mapred.ShuffleHandler
　　
　　
　　yarn.resourcemanager.resource-tracker.address
　　master:8025
　　
　　
　　yarn.resourcemanager.scheduler.address
　　master:8030
　　
　　
　　yarn.resourcemanager.address
　　master:8040
　　
　　
　　8. Add slaves
　　Add the slave entries in $HADOOP_CONF_DIR/slaves on master machine:
slave1　　
slave2
　　9. Format the namenode
$ bin/hadoop namenode -format　　10. Start Hadoop Daemons
$ sbin/hadoop-daemon.sh start namenode　　
$ sbin/hadoop-daemons.sh start datanode
　　
$ sbin/yarn-daemon.sh start resourcemanager
　　
$ sbin/yarn-daemons.sh start nodemanager
　　
$ sbin/mr-jobhistory-daemon.sh start historyserver
　　NOTE: For datanode and nodemanager, scripts are *-daemons.sh and not *-daemon.sh. daemon.sh does not lookup in slaves file and hence, will only start processes on master
　　11. Check installation
　　Check for jps output on slaves and master.
　　For master:
$ jps　　
6539 ResourceManager
　　
6451 DataNode
　　
8701 Jps
　　
6895 JobHistoryServer
　　
6234 NameNode
　　
6765 NodeManager
　　For slaves:
$ jps　　
8014 NodeManager
　　
7858 DataNode
　　
9868 Jps

　　If these services are not up, check the logs in $HADOOP_HOME/logs directory to>　　12. Run a demo application to verify installtion
$ mkdir in　　
$ cat > in/file
　　
This is one line
　　
This is another one
　　Add this directory to HDFS:
$ bin/hadoop dfs -copyFromLocal in /in　　Run wordcount example provided:
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.*-alpha.jar wordcount /in /out　　Check the output:
$ bin/hadoop dfs -cat /out/*　　
This 2
　　
another 1
　　
is 2
　　
line 1
　　
one 2
　　13. Web interface
　　1. http://master:50070/dfshealth.jsp
　　2. http://master:8088/cluster
　　3. http://master:19888/jobhistory (for Job History Server)
　　14. Stopping the daemons
$ sbin/mr-jobhistory-daemon.sh stop historyserver　　
$ sbin/yarn-daemons.sh stop nodemanager
　　
$ sbin/yarn-daemon.sh stop resourcemanager
　　
$ sbin/hadoop-daemons.sh stop datanode
　　
$ sbin/hadoop-daemon.sh stop namenode
　　15. Possible errors
　　If you get a exception stack trace similar to given below:
Container launch failed for container_1350204169962_0002_01_000004 : java.lang.reflect.UndeclaredThrowableException　　at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
　　at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:101)
　　at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:149)
　　at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
　　at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
　　at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
　　at java.lang.Thread.run(Thread.java:679)
　　
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "my-laptop":40365; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost
　　at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:187)
　　at $Proxy29.startContainer(Unknown Source)
　　at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:99)
　　... 5 more
　　
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "my-laptop":40365; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost
　　at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:740)
　　at org.apache.hadoop.ipc.Client$Connection.(Client.java:248)
　　at org.apache.hadoop.ipc.Client.getConnection(Client.java:1261)
　　at org.apache.hadoop.ipc.Client.call(Client.java:1141)
　　at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
　　... 7 more
　　
Caused by: java.net.UnknownHostException
　　... 11 more
　　Solution: Check the Important Change in Step 2 and apply the necessary changes.
　　Happy Coding!!!
　　- Rasesh Mori