centos6安装hadoop

087988

Hadoop是一个由Apache基金会所开发的分布式系统基础架构。
用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力进行高速运算和存储。
Hadoop实现了一个分布式文件系统（Hadoop Distributed File System），简称HDFS。HDFS有高容错性的特点，并且设计用来部署在低廉的（low-cost）硬件上；而且它提供高吞吐量（high throughput）来访问应用程序的数据，适合那些有着超大数据集（large data set）的应用程序。HDFS放宽了（relax）POSIX的要求，可以以流的形式访问（streaming access）文件系统中的数据。
Hadoop的框架最核心的设计就是：HDFS和MapReduce。HDFS为海量的数据提供了存储，则MapReduce为海量的数据提供了计算。

namenode  192.168.31.243

datenode  192.168.31.165

实验环境
centos6_x64

实验软件
jdk-6u31-linux-i586.bin
hadoop-1.0.0.tar.gz

软件安装
yum install -y rsync* openssh*
yum install -y ld-linux.so.2
groupadd hadoop
useradd hadoop -g hadoop
mkdir /usr/local/hadoop
mkdir -p /usr/local/java
service iptables stop

ssh-keygen -t rsa            192.168.31.243配置 （192.168.31.165配置相同）
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

scp -r /root/.ssh/id_rsa.pub 192.168.31.165:/root/.ssh/authorized_keys
scp -r /root/.ssh/id_rsa.pub 192.168.31.243:/root/.ssh/authorized_keys
scp -r jdk-6u31-linux-i586.bin hadoop-1.0.0.tar.gz 192.168.31.165:/root/

mv jdk-6u31-linux-i586.bin /usr/local/java/
cd /usr/local/java/
chmod +x jdk-6u31-linux-i586.bin
./jdk-6u31-linux-i586.bin

vim /etc/profile     最后一行追加配置
# set java environment
export JAVA_HOME=/usr/local/java/jdk1.6.0_31
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
# set hadoop path
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
source /etc/profile

java -version
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) Client VM (build 20.6-b01, mixed mode, sharing)

tar zxvf hadoop-1.0.0.tar.gz
mv hadoop-1.0.0 /usr/local/hadoop
chown -R hadoop:hadoop /usr/local/hadoop
ll /usr/local/hadoop/
drwxr-xr-x 14 hadoop hadoop 4096 Dec 16  2011 hadoop-1.0.0

cp /usr/local/hadoop/conf/hadoop-env.sh  /usr/local/hadoop/conf/hadoop-env.sh.bak
vim /usr/local/hadoop/conf/hadoop-env.sh
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME=/usr/local/java/jdk1.6.0_31  修改为

cd  /usr/local/hadoop/conf
cp core-site.xml hdfs-site.xml mapred-site.xml core-site.xml 这几个文件都备份一下

vim core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>                  红色为需要修改的地方

<property>

      <name>hadoop.tmp.dir</name>

      <value>/usr/local/hadoop/tmp</value>

      <description>A base for other temporary directories.</description>

    </property>

<!-- file system properties -->

    <property>

        <name>fs.default.name</name>

        <value>hdfs://192.168.31.243:9000</value>

    </property>

</configuration>

vim hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
   <name>dfs.replication</name>
   <value>3</value>
   </property>
</configuration>

vim mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
    <name> mapred.job.tracker</name>
    <value>http://192.168.21.243:9001</value>
  </property>
</configuration>

cp masters masters.bak

vim masters
localhost
192.168.31.243

cp slave slave.bak             192.168.31.165配置
vim /usr/local/hadoop/conf/slaves

localhost

192.168.31.165

scp -r  core-site.xml hdfs-site.xml mapred-site.xml 192.168.31.165:/usr/local/hadoop/conf/

/usr/local/hadoop/bin/hadoop namenode -format
Warning: $HADOOP_HOME is deprecated.
16/09/21 22:51:13 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = java.net.UnknownHostException: centos6: centos6
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.0.0
STARTUP_MSG:   build = https://svn.apache.org/repos/asf ... branches/branch-1.0 -r 1214675; compiled by 'hortonfo' on Thu Dec 15 16:36:35 UTC 2011
************************************************************/
16/09/21 22:51:14 INFO util.GSet: VM type       = 32-bit
16/09/21 22:51:14 INFO util.GSet: 2% max memory = 19.33375 MB
16/09/21 22:51:14 INFO util.GSet: capacity      = 2^22 = 4194304 entries
16/09/21 22:51:14 INFO util.GSet: recommended=4194304, actual=4194304
16/09/21 22:51:14 INFO namenode.FSNamesystem: fsOwner=root
16/09/21 22:51:14 INFO namenode.FSNamesystem: supergroup=supergroup
16/09/21 22:51:14 INFO namenode.FSNamesystem: isPermissionEnabled=true
16/09/21 22:51:14 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
16/09/21 22:51:14 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
16/09/21 22:51:14 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/09/21 22:51:14 INFO common.Storage: Image file of size 110 saved in 0 seconds.
16/09/21 22:51:14 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
16/09/21 22:51:14 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: centos6: centos6
************************************************************/

/usr/local/hadoop/bin/start-all.sh

Warning: $HADOOP_HOME is deprecated.
starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-namenode-centos6.out
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 81:d9:c6:54:a9:99:27:c0:f7:5f:c3:15:d5:84:a0:99.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
root@localhost's password:
localhost: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-datanode-centos6.out
The authenticity of host '192.168.31.243 (192.168.31.243)' can't be established.
RSA key fingerprint is 81:d9:c6:54:a9:99:27:c0:f7:5f:c3:15:d5:84:a0:99.
Are you sure you want to continue connecting (yes/no)? yes
192.168.31.243: Warning: Permanently added '192.168.31.243' (RSA) to the list of known hosts.
root@192.168.31.243's password:
192.168.31.243: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-secondarynamenode-centos6.out
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-jobtracker-centos6.out
root@localhost's password:
localhost: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-root-tasktracker-centos6.out

ll /usr/local/hadoop/tmp/
drwxr-xr-x 5 root root 4096 Sep 21 22:53 dfs
drwxr-xr-x 3 root root 4096 Sep 21 22:53 mapred  看到这两项证明没有错误

jps

3237 SecondaryNameNode
3011 NameNode
3467 Jps

netstat -tuplna | grep 500
tcp        0      0 :::50070                    :::*                        LISTEN      3011/java           
tcp        0      0 :::50090                    :::*                        LISTEN      3237/java           

http://192.168.31.243:50070/dfshealth.jsp