Steps To Setup Hadoop 2.4.0 (Single Node Cluster) on CentOS/RHEL Written by Rahu

lixiaolong

　　Apache Hadoop 2.4.0 significant improvements over the previous stable>

Step 1. Install JAVA/JDK
　　Java is the primary requirement for running hadoop on any system, So make sure you have java installed on your system using following command.
#java -versionjava version "1.8.0_05"　　
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
　　
Java HotSpot(TM) Client VM (build 25.5-b02, mixed mode)
　　If you don’t have java installed on your system, use one of following link to install it first.
　　Install JAVA/JDK 8 on CentOS and RHEL 6/5
Step 2. Setup Hadoop User
　　We recommend to create a normal (nor root) account for hadoop working. So create a system account using following command.
# useradd hadoop　　
# passwd hadoop
　　After creating account, it also required to set up key based ssh to its own account. To do this use execute following commands.
# su - hadoop　　
$ ssh-keygen -t rsa
　　
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
　　
$ chmod 0600 ~/.ssh/authorized_keys
　　Lets verify key based login. Below command should not ask for password but first time it will prompt for adding RSA to the list of known hosts.
$ ssh localhost　　
$ exit
Step 3. Download Hadoop 2.4.0

　　Now download hadoop 2.4.0 source archive file using below command. You can also select>$ cd ~　　
$ wget http://apache.claz.org/hadoop/common/hadoop-2.4.0/hadoop-2.4.0.tar.gz
　　
$ tar xzf hadoop-2.4.0.tar.gz
Step 4. Configure Hadoop 2.4.0
4.1. Setup Environment Variables
　　First we need to set environment variable uses by hadoop. Edit ~/.bashrc file and append following values at end of file.
export HADOOP_HOME=/home/hadoop/hadoop　　
export HADOOP_INSTALL=$HADOOP_HOME
　　
export HADOOP_MAPRED_HOME=$HADOOP_HOME
　　
export HADOOP_COMMON_HOME=$HADOOP_HOME
　　
export HADOOP_HDFS_HOME=$HADOOP_HOME
　　
export YARN_HOME=$HADOOP_HOME
　　
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
　　
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
　　Now apply the changes in current running environment
$source ~/.bashrc　　Now edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh file and set JAVA_HOME environment variable
export JAVA_HOME=/opt/jdk1.8.0_05/4.2. Edit Configuration Files
　　Hadoop has many of configuration files, which need to configure as per requirements of your hadoop infrastructure. Lets start with the configuration with basic hadoop singlenode cluster setup. first navigate to below location
$ cd $HADOOP_HOME/etc/hadoopEdit core-site.xml
　　

　　
  fs.default.name
　　
    hdfs://localhost:9000
　　

　　

Edit hdfs-site.xml
　　

　　
dfs.replication
　　
1
　　

　　

　　

　　
  dfs.name.dir
　　
    file:///home/hadoop/hadoopdata/hdfs/namenode
　　

　　

　　

　　
  dfs.data.dir
　　
    file:///home/hadoop/hadoopdata/hdfs/datanode
　　

　　

Edit mapred-site.xml
　　

　　
  mapreduce.framework.name
　　
   yarn
　　

　　

Edit yarn-site.xml
　　

　　
  yarn.nodemanager.aux-services
　　
    mapreduce_shuffle
　　

　　

4.3. Format Namenode
　　Now format the namenode using following command, make sure that Storage directory is
$hdfs namenode -format　　[Sample output]
14/05/04 21:30:55 INFO namenode.NameNode: STARTUP_MSG:　　
/************************************************************
　　
STARTUP_MSG: Starting NameNode
　　
STARTUP_MSG:   host = svr1.tecadmin.net/192.168.1.11
　　
STARTUP_MSG:   args = [-format]
　　
STARTUP_MSG:   version = 2.4.0
　　
...
　　
...
　　
14/05/04 21:30:56 INFO common.Storage: Storage directory /home/hadoop/hadoopdata/hdfs/namenode has been successfully formatted.
　　
14/05/04 21:30:56 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
　　
14/05/04 21:30:56 INFO util.ExitUtil: Exiting with status 0
　　
14/05/04 21:30:56 INFO namenode.NameNode: SHUTDOWN_MSG:
　　
/************************************************************
　　
SHUTDOWN_MSG: Shutting down NameNode at svr1.tecadmin.net/192.168.1.11
　　
************************************************************/
Step 5. Start Hadoop Cluster
　　Lets start your hadoop cluster using the scripts provides by hadoop. Just navigate to your hadoop sbin directory and execute scripts one by one.
$ cd $HADOOP_HOME/sbin/　　Now run start-dfs.sh script.
$start-dfs.sh[Sample output]　　
14/05/04 21:37:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
　　
Starting namenodes on [localhost]
　　
localhost: starting namenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-namenode-svr1.tecadmin.net.out
　　
localhost: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-svr1.tecadmin.net.out
　　
Starting secondary namenodes [0.0.0.0]
　　
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-svr1.tecadmin.net.out
　　
14/05/04 21:38:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
　　Now run start-yarn.sh script.
$start-yarn.sh[Sample output]　　
starting yarn daemons
　　
starting resourcemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-svr1.tecadmin.net.out
　　
localhost: starting nodemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-svr1.tecadmin.net.out
Step 6. Access Hadoop Services in Browser
　　Hadoop NameNode started on port 50070 default. Access your server on port 50070 in your favorite web browser.
http://svr1.tecadmin.net:50070/

　　Now access port 8088 for getting the information about cluster and all applications
http://svr1.tecadmin.net:8088/

　　Access port 50090 for getting details about secondary namenode.
http://svr1.tecadmin.net:50090/

　　Access port 50075 to get details about DataNode
http://svr1.tecadmin.net:50075/

Step 7. Test Hadoop Setup
　　7.1 - Make the HDFS directories required using following commands.
$ bin/hdfs dfs -mkdir /user　　
$ bin/hdfs dfs -mkdir /user/hadoop
　　7.2 - Now copy all files from local file system /var/log/httpd to hadoop distributed file system using below command
$ bin/hdfs dfs -put /var/log/httpd logs　　7.3 - Now browse hadoop distributed file system by opening below url in browser.
http://svr1.tecadmin.net:50070/explorer.html#/user/hadoop/logs

　　7.4 - Now copy logs directory for hadoop distributed file system to local file system.
$ bin/hdfs dfs -get logs /tmp/logs　　
$ ls -l /tmp/logs/