centos6.5 hadoop2.2.0

yunvn

　　规划:
cloud01、cloud02  Namenode HA
cloud03           YARN
cloud04\05\06     ZK节点、Datanode 




  机器名   IP地址                      进程
cloud01 192.168.1.201  jdk hadoop   NameNode、DFSZKFailoverController
cloud02 192.168.1.202  jdk hadoop   NameNode、DFSZKFailoverController
cloud03 192.168.1.203  jdk hadoop   ResourceManager  
cloud04 192.168.1.204  jdk hadoop   zookeeper   DataNode、NoderManager、JouranlNode、QuorumPeerMain    
cloud05 192.168.1.205  jdk hadoop   zookeeper   DataNode、NoderManager、JouranlNode、QuorumPeerMain
cloud06 192.168.1.206  jdk hadoop   zookeeper   DataNode、NoderManager、JouranlNode、QuorumPeerMain




说明:
---管理YARN yarn,Yet Another Resource Negotiator ，是 Hadoop 自 0.23.0 版本后新的 map-reduce 框架
---JouranlNode 数据实时同步  YARN 管理多个NodeManager  QuorumPeerMain是ZooKeeper的进程
hadoop2.0 namenode 有HA功能  需要安装zookeeper (Master选举)   ZK需要要奇数个节点
hadoop2.0 有两个NamdeNode组成，一个处于active，一个standby




安装步骤：
1、先在datanode 节点上安装jdk和zookeeper
1.1 下载jdk 解压到/usr/local目前下 /usr/local/jdk1.8.0_05
1.2 设置环境变量/etc/profile 
  export JAVA_HOME=/usr/local/jdk1.8.0_05
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin
export PATH
jps验证
1.3 创建mycloud目录，解压zookeeper-3.4.5.tar.gz 至这个文件目录下/mycloud/zookeeper-3.4.6   （之后hadoop也放到这个目录下）
 创建/mycloud/zookeeper-3.4.6/data 目录，在该目录下创建文件myid 填入相应的server.id号，一行一个id号 
    修改conf目录下的zoo.cfg     cp zoo_sample.cfg zoo.cfg
修改 zoo.cfg  dataDir=/mycloud/zookeeper-3.4.6/data
最后添加:
server.1=cloud04:2888:3888  通讯端口:选举端口
server.2=cloud05:2888:3888  通讯端口:选举端口
server.3=cloud06:2888:3888  通讯端口:选举端口
1.4 启动zk （ZK的三个节点都要启动）
    [iyunv@cloud04 bin]# ./zkServer.sh start
    jps验证:
   看到QuorumPeerMain进程，即启动成功。

查看ZK服务状态：
[iyunv@hnode03 bin]# ./zkServer.sh status
    JMX enabled by default
Using config: /mycloud/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: follower  / Mode: leader （两种模式）


  说明:leader是主节点，follower是从节约点
  
2、安装hadoop
下载 hadoop-2.2.0.tar.gz
    tar -xzvf hadoop-2.2.0.tar.gz -C /mycloud/


5个配置文件 hadoop-env.sh core-site.xml   hdfs-site.xml   mapred-site.xml yarn-site.xml
（生产中最好需要两台独立的机器来做NameNode的HA）


hadoop-env.sh   修改第27行   JAVA_HOME


core-site.xml 修改内容如下:
<configuration>
  <!--指定hdfs的nameservice为ns-->
  <property>  
  <name>fs.defaultFS</name>
  <value>hdfs://ns</value>
  </property>
  <!--指定hadoop的临时目录-->
    <property>  
  <name>hadoop.tmp.dir</name>
  <value>/mycloud/hadoop-2.2.0/temp</value>
  </property>
  <!--指定zookeeper地址-->
   <property>  
  <name>ha.zookeeper.quorum</name>
  <value>cloud04:2181,cloud05:2181,cloud06:2181</value>
  </property>
  </configuration>


hdfs-site.xml 修改内容如下:


<!-- Put site-specific property overrides in this file. -->
  <!--指定hdfs的nameservice为nsHA之后统一对外的名称-->
  <configuration>
  <property>  
  <name>dfs.nameservices</name>
  <value>ns</value>
  </property>
  <!--ns下面有两个NameNode，分别是cloud01和cloud01-->
    <property>  
  <name>dfs.ha.namenodes.ns</name>
  <value>cloud01,cloud02</value>
  </property>
  <!--cloud01的RPC通信地址-->
   <property>  
  <name>dfs.namenode.rpc-address.ns.cloud01</name>
  <value>cloud01:8020</value>
  </property>
    <!--cloud01的http通信地址-->
  <property>  
  <name>dfs.namenode.http-address.ns.cloud01</name>
  <value>cloud01:50070</value>
  </property>
    <!--cloud02的RPC通信地址-->
   <property>  
  <name>dfs.namenode.rpc-address.ns.cloud02</name>
  <value>cloud02:8020</value>
  </property>
    <!--cloud02的http通信地址-->
  <property>  
  <name>dfs.namenode.http-address.ns.cloud02</name>
  <value>cloud02:50070</value>
  </property>
  <!--指定NameNode的元数据在JournalNode上的存放位置-->
  <property>  
  <name>dfs.namenode.share.edits.dir</name>
  <value>qjournal://cloud04:8485;cloud05:8485;cloud06:8485/ns</value>
  </property>
   <!--指定JournalNode在本地磁盘存放数据的位置-->
  <property>  
  <name>dfs.journalnode.edits.dir</name>
  <value>/mycloud/hadoop-2.2.0/journal</value>
  </property>
  <!-- 开启NameNode失败自动切换-->
  <property>  
  <name>dfs.ha.automatic-failover.enabled</name>
  <value>true</value>
  </property>
  <!--配置失败自动切换实现方式-->
  <property>  
  <name>dfs.client.failover.proxy.provider.ns</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfigureFailoverProxyProvider</value>
  </property>
  <!--使用隔离机制-->
  <property>  
  <name>dfs.ha.fencing.methods</name>
  <value>sshfence</value>
  </property>
  <!-- 使用隔离机制时需要ssh免登录 -->
  <property>  
  <name>dfs.ha.fencing.ssh.private-key-files</name>
  <value>/root/.ssh/id_rsa</value>
  </property>
  </configuration>


vi mapred-site.xml
   <configuration>
    <-- 指定Mr框架为YARN方式 -->
     <property>  
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
     </property>
   </configuration>


vi yarn-site.xml
  <configuration>
    <-- 指定resourcemanager地址 -->
     <property>  
       <name>yarn.resourcemanager.hostname</name>
       <value>cloud03</value>
     </property>
    <-- 指定nodemanager启动时加载server的方式为mapreduce_shuffle -->
     <property>  
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
     </property>
</configuration>


3、ssh配置免密码登录
     #ssh-keygen -t rsa
     #ls 
     id_rsa id_rsa.pub
    #ssh-copy-id -i cloud02 （到目标机免登录，把公钥拷至目标机）操作时输入目标机的密码


cloud01-->cloud02 前配好的hadoop拷至02、03、04、05、06
     #scp -r /mycloud/ cloud02:/ 
     #scp -r /mycloud/ cloud03:/ 
     #scp -r /mycloud/ cloud04:/ 
     #scp -r /mycloud/ cloud05:/ 
     #scp -r /mycloud/ cloud06:/ 
     


 
4、配置环境变量
     在第一台机器上添加一个HADOOP_HOME环境变量
     /etc/profile


     export JAVA_HOME=/usr/local/jdk1.6.0_45
     export HADOOP_HOME=/project/hadoop-2.2.0
     export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin


source /etc/profile


将profile拷至其他节点，同步
scp /etc/profile cloud02:/etc
scp /etc/profile cloud03:/etc
scp /etc/profile cloud04:/etc
scp /etc/profile cloud05:/etc
scp /etc/profile cloud06:/etc


5、配置slaves节点（即运行JouralNode的节点，它的管理节点是:ResourceManager，也就是YARN）
vi /hadoop-2.2.0/etc/hadoop/slaves
cloud04
cloud05
cloud06


6、全部配置好之后，将hadoop-2.2.0拷贝到其他机器
scp -r /mycloud/hadoop-2.2.0 cloud02:/mycloud/
scp -r /mycloud/hadoop-2.2.0 cloud03:/mycloud/
scp -r /mycloud/hadoop-2.2.0 cloud04:/mycloud/
scp -r /mycloud/hadoop-2.2.0 cloud05:/mycloud/
scp -r /mycloud/hadoop-2.2.0 cloud06:/mycloud/


7、启动
7.1 先启动zk(每台机器)
cd /project/zookeeper-3.4.5/bin/
./zkServer.sh start
./zkServer.sh status
(一个leader，两个follower)


7.2 再启动journalnode（只需要在一台机器运行即可，其他机器将自动通过ssh运行。如果只运行自己的，是hadoop-deamon.sh，多台一起运行的是
                       hadoop-deamons.sh，运行的这些机器，也是安装了ZooKeeper的机器）
sbin/hadoop-deamons.sh start journalnode
验证:jps   查看Jouranlnode进程有没有运行


sbin/hadoop-deamons.sh start journalnode  (hadoop-deamon.sh运行本机journalnode   hadoop-deamons.sh 其他机器也一起运行)
[iyunv@cloud01 hadoop-2.2.0]# sbin/hadoop-daemons.sh start journalnode


8、格式化HDFS文件系统:
在cloud01（即NameNode的其中一个节点）上执行命令
#bin/hdfs haadmin DFSHAadmin –transitionToActive ns 验证HA条件


hadoop namenode -format
格式化后在根据core-site.xml中的hadoop
两个Namenode，格式化一个后，拷到另一个也作为格式化




9、将格式后之后，生成的HDFS的元数据拷贝至第2个NameNode节点
scp -r tmp/ cloud02:/project/hadoop-2.2.0/   （拷贝的目录tmp是执行格式化命令后生成的）




10、格式化zk（在cloud01上执行即可）
hdfs zkfc -formatZK
cd  /project/zookeeper-3.4.5/bin/
./zkCli.sh


ls /
[hadoop-ha,zookeeper]  看到多一个hadoop-ha目录，说明格式化成功


11、启动HDFS(在cloud01上执行即可)
sbin/start-dfs.sh


12、启动YARN(在cloud03上执行)
sbin/start-yarn.sh
jps 查看进程
可以查看到多一个ResourceManager进程，被管理的节点为NodeManager


测试环境与应用:


切换Namenode 的两个服务器需要相互免密码登录
强制kill  cloud01的NameNode进程后，cloud02自动转为Active  (cloud02到cloud01也必须是ssh免密码登录)


cloud02重新启动后，变为Standby
sbin/hadoop-daemon.sh start namenode


hadoop fs -put /root/install.log /  上传文件
hadoop fs -ls   查看文件
hadoop fs -get /install.log /home/111.log 下载文件


hadoop jar /project/hadoop-2.2.0/.../hadoop-mapreduce-examples-2.2.0.jar wordcount /file.txt  /resultout.txt
hadoop -fs -cat /resultout/part-r-0000 查看mapreduce结果文件








错误：


sbin/hadoop-daemon.sh start journalnode
starting journalnode, logging to /mycloud/hadoop-2.2.0/logs/hadoop-root-journalnode-cloud01.out
[Fatal Error] core-site.xml:20:4: 元素内容必须由格式正确的字符数据或标记组成。
Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/mycloud/hadoop-2.2.0/etc/hadoop/core-site.xml; lineNumber: 20; columnNumber: 4; 元素内容必须由格式正确的字符数据或标记组成。
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2152)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2001)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1918)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:817)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:792)
        at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1114)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:431)


原因: 将property 写成了porperty
  <property>  
  <name>fs.defaultFs</name>
  <value>hdfs://nnode</value>
  </property>
    S大写
    <name>fs.defaultFs</name>    <name>fs.defaultFS</name>
<--指定hdfs的nameservice为ns1--> 注释方法错误，少了叹号 <!--指定hdfs的nameservice为ns1-->



sbin/hadoop-daemon.sh start journalnode
starting journalnode, logging to /mycloud/hadoop-2.2.0/logs/hadoop-root-journalnode-cloud01.out
[Fatal Error] hdfs-site.xml:20:33: 注释中不允许出现字符串 "--"。
Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/mycloud/hadoop-2.2.0/etc/hadoop/hdfs-site.xml; lineNumber: 20; columnNumber: 33; 注释中不允许出现字符串 "--"。
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2152)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2001)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1918)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:817)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:792)
        at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1114)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:431)


    <!--指定hdfs的nameservice为nnode--HA之后统一对外的名称-->
原因：注释的内容中间还有 -- 




第一次格式化:HA没有true 见365，原因在hdfs-site.xml中
  <property>  
  <name>dfs.client.failover.proxy.provider.nnode</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfigureFailoverProxyProvider</value>
  </property> 这里配置错了，少了Provider


[iyunv@cloud01 hadoop-2.2.0]# hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.


14/06/15 22:47:28 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = cloud01/192.168.214.171
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.2.0
STARTUP_MSG:   classpath = /mycloud/hadoop-2.2.0/etc/hadoop:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jetty-6.1.26.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-math-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-logging-1.1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/paranamer-2.3.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/xz-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-io-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-lang-2.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/stax-api-1.0.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/guava-11.0.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-el-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jersey-json-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-digester-1.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-cli-1.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jersey-server-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jsch-0.1.42.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-codec-1.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jersey-core-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jsp-api-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/xmlenc-0.52.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jets3t-0.6.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/zookeeper-3.4.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/activation-1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/log4j-1.2.17.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/commons-net-3.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/junit-4.8.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/asm-3.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/servlet-api-2.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/jettison-1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/mycloud/hadoop-2.2.0/share/hadoop/common/hadoop-nfs-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-io-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/asm-3.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/xz-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/commons-io-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/junit-4.10.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/javax.inject-1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/guice-3.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/avro-1.7.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/asm-3.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/junit-4.10.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/mycloud/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/mycloud/hadoop-2.2.0/contrib/capacity-scheduler/*.jar:/mycloud/hadoop-2.2.0/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z
STARTUP_MSG:   java = 1.8.0_05
************************************************************/
14/06/15 22:47:28 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /mycloud/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/06/15 22:47:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-64625d97-d90d-490c-9d8d-f09a0735676d
14/06/15 22:47:29 INFO namenode.HostFileManager: read includes:
HostSet(
)
14/06/15 22:47:29 INFO namenode.HostFileManager: read excludes:
HostSet(
)
14/06/15 22:47:29 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
14/06/15 22:47:29 INFO util.GSet: Computing capacity for map BlocksMap
14/06/15 22:47:29 INFO util.GSet: VM type       = 64-bit
14/06/15 22:47:29 INFO util.GSet: 2.0% max memory = 966.7 MB
14/06/15 22:47:29 INFO util.GSet: capacity      = 2^21 = 2097152 entries
14/06/15 22:47:29 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
14/06/15 22:47:29 INFO blockmanagement.BlockManager: defaultReplication         = 3
14/06/15 22:47:29 INFO blockmanagement.BlockManager: maxReplication             = 512
14/06/15 22:47:29 INFO blockmanagement.BlockManager: minReplication             = 1
14/06/15 22:47:29 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
14/06/15 22:47:29 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
14/06/15 22:47:29 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
14/06/15 22:47:29 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
14/06/15 22:47:29 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
14/06/15 22:47:29 INFO namenode.FSNamesystem: supergroup          = supergroup
14/06/15 22:47:29 INFO namenode.FSNamesystem: isPermissionEnabled = true
14/06/15 22:47:29 INFO namenode.FSNamesystem: Determined nameservice ID: nnode
14/06/15 22:47:29 INFO namenode.FSNamesystem: HA Enabled: false
14/06/15 22:47:29 INFO namenode.FSNamesystem: Append Enabled: true
14/06/15 22:47:29 INFO util.GSet: Computing capacity for map INodeMap
14/06/15 22:47:29 INFO util.GSet: VM type       = 64-bit
14/06/15 22:47:29 INFO util.GSet: 1.0% max memory = 966.7 MB
14/06/15 22:47:29 INFO util.GSet: capacity      = 2^20 = 1048576 entries
14/06/15 22:47:29 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/06/15 22:47:29 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/06/15 22:47:29 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/06/15 22:47:29 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
14/06/15 22:47:29 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/06/15 22:47:29 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14/06/15 22:47:29 INFO util.GSet: Computing capacity for map Namenode Retry Cache
14/06/15 22:47:29 INFO util.GSet: VM type       = 64-bit
14/06/15 22:47:29 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB
14/06/15 22:47:29 INFO util.GSet: capacity      = 2^15 = 32768 entries
14/06/15 22:47:29 INFO common.Storage: Storage directory /mycloud/hadoop-2.2.0/temp/dfs/name has been successfully formatted.
14/06/15 22:47:29 INFO namenode.FSImage: Saving image file /mycloud/hadoop-2.2.0/temp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
14/06/15 22:47:29 INFO namenode.FSImage: Image file /mycloud/hadoop-2.2.0/temp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 196 bytes saved in 0 seconds.
14/06/15 22:47:29 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/06/15 22:47:29 INFO util.ExitUtil: Exiting with status 0
14/06/15 22:47:30 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at cloud01/192.168.214.171
************************************************************/
[iyunv@cloud01 hadoop-2.2.0]# 




终于成功了
************************************************************/
14/06/17 23:40:23 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /mycloud/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/06/17 23:40:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-47fe8dae-be4f-4f25-a542-1b43a720752b
14/06/17 23:40:24 INFO namenode.HostFileManager: read includes:
HostSet(
)
14/06/17 23:40:24 INFO namenode.HostFileManager: read excludes:
HostSet(
)
14/06/17 23:40:24 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
14/06/17 23:40:24 INFO util.GSet: Computing capacity for map BlocksMap
14/06/17 23:40:24 INFO util.GSet: VM type       = 64-bit
14/06/17 23:40:24 INFO util.GSet: 2.0% max memory = 966.7 MB
14/06/17 23:40:24 INFO util.GSet: capacity      = 2^21 = 2097152 entries
14/06/17 23:40:24 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
14/06/17 23:40:24 INFO blockmanagement.BlockManager: defaultReplication         = 3
14/06/17 23:40:24 INFO blockmanagement.BlockManager: maxReplication             = 512
14/06/17 23:40:24 INFO blockmanagement.BlockManager: minReplication             = 1
14/06/17 23:40:24 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
14/06/17 23:40:24 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
14/06/17 23:40:24 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
14/06/17 23:40:24 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
14/06/17 23:40:24 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
14/06/17 23:40:24 INFO namenode.FSNamesystem: supergroup          = supergroup
14/06/17 23:40:24 INFO namenode.FSNamesystem: isPermissionEnabled = true
14/06/17 23:40:24 INFO namenode.FSNamesystem: Determined nameservice ID: ns
14/06/17 23:40:24 INFO namenode.FSNamesystem: HA Enabled: true
14/06/17 23:40:24 INFO namenode.FSNamesystem: Append Enabled: true
14/06/17 23:40:24 INFO util.GSet: Computing capacity for map INodeMap
14/06/17 23:40:24 INFO util.GSet: VM type       = 64-bit
14/06/17 23:40:24 INFO util.GSet: 1.0% max memory = 966.7 MB
14/06/17 23:40:24 INFO util.GSet: capacity      = 2^20 = 1048576 entries
14/06/17 23:40:24 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/06/17 23:40:24 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/06/17 23:40:24 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/06/17 23:40:24 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
14/06/17 23:40:24 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/06/17 23:40:24 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14/06/17 23:40:24 INFO util.GSet: Computing capacity for map Namenode Retry Cache
14/06/17 23:40:24 INFO util.GSet: VM type       = 64-bit
14/06/17 23:40:24 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB
14/06/17 23:40:24 INFO util.GSet: capacity      = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /mycloud/hadoop-2.2.0/temp/dfs/name ? (Y or N) y
14/06/17 23:40:29 INFO common.Storage: Storage directory /mycloud/hadoop-2.2.0/temp/dfs/name has been successfully formatted.
14/06/17 23:40:29 INFO namenode.FSImage: Saving image file /mycloud/hadoop-2.2.0/temp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
14/06/17 23:40:29 INFO namenode.FSImage: Image file /mycloud/hadoop-2.2.0/temp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 196 bytes saved in 0 seconds.
14/06/17 23:40:29 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/06/17 23:40:29 INFO util.ExitUtil: Exiting with status 0
14/06/17 23:40:29 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at cloud01/192.168.214.171
************************************************************/