HADOOP docker(三):HDFS高可用实验

狐狸情

前言　　1.机器环境
　　2.配置HA
　　2.1 修改hdfs-site.xml
　　2.2 设置core-site.xml
　　3.配置手动HA
　　3.1 关闭YARN、HDFS
　　3.2 启动HDFS HA
　　4.配置自动HA
　　4.1 关闭集群
　　4.2 修改配置文件
　　4.3 启动HA
　　4.4 测试自动切换
前言
　　上一节学习了HDFS HA的原理，本节来做实验
1.机器环境
主机名IP角色hadoop1172.18.0.11NN1 ZK RMhadoop2172.18.0.12NN2 ZK RM JOBHISTORYhadoop3172.18.0.13DN ZK NDhadoop4172.18.0.14DN QJM1 NDhadoop5172.18.0.15DN QJM2 NDhadoop6172.18.0.16DN QJM3 ND　　目前已经安装了hdfs yarn zookeeper
2.配置HA
2.1 修改hdfs-site.xml

• <property>
  
• <name>dfs.nameservices</name>
  
• <value>dockercluster</value>
  
• </property>
  
• 
  
• <property>
  
• <name>dfs.ha.namenodes.dockercluster</name>
  
• <value>nn1,nn2</value>
  
• </property>
  
• 
  
• <property>
  
• <name>dfs.namenode.rpc-address.dockercluster.nn1</name>
  
• <value>hadoop1:8020</value>
  
• </property>
  
• <property>
  
• <name>dfs.namenode.rpc-address.dockercluster.nn2</name>
  
• <value>hadoop2:8020</value>
  
• </property>
  
• 
  
• <property>
  
• <name>dfs.namenode.http-address.dockercluster.nn1</name>
  
• <value>hadoop1:50070</value>
  
• </property>
  
• <property>
  
• <name>dfs.namenode.http-address.dockercluster.nn2</name>
  
• <value>hadoop2:50070</value>
  
• </property>
  
• 
  
• 
  
• <property>
  
• <name>dfs.namenode.shared.edits.dir</name>
  
• <value>qjournal://hadoop4:8485;hadoop5;hadoop6:8485/dockercluster</value>
  
• </property>
  
• 
  
• <property>
  
• <name>dfs.client.failover.proxy.provider.dockercluster</name>
  
• <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  
• </property>
  
• 
  
• <property>
  
• <name>dfs.ha.fencing.methods</name>
  
• <value>sshfence</value>
  
• </property>
  
• 
  
• <property>
  
• <name>dfs.ha.fencing.ssh.private-key-files</name>
  
• <value>/home/hdfs/.ssh/id_rsa</value>
  
• </property>
  
• 
  
• <property>
  
• <name>dfs.ha.fencing.ssh.connect-timeout</name>
  
• <value>30000</value>
  
• </property>
  
• 
  
• <property>
  
• <name>dfs.journalnode.edits.dir</name>
  
• <value>/opt/hadoop/hadoop-2.7.3/JNSdatadir</value>
  
• </property>
  

2.2 设置core-site.xml

• <property>
  
• <name>fs.defaultFS</name>
  
• <value>hdfs://dockercluster</value>
  
• </property>
  

　　将以上配置文件分发到各个节点上
3.配置手动HA
3.1 关闭YARN、HDFS
　　关闭yarn:

• resourcemanager上执行：$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager
  
• nodemanager上执行：$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager
  

　　关闭hdfs:

• namenode上执行：$HADOOP_HOME/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode
  
• datanode执行：$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode
  

　　如果配置了ssh免密码登录：

　　stop-yarn.sh
　　stop-dfs.sh

3.2 启动HDFS HA

• 　　启动journal节点
  　　在hadoop4 5 6上执行：
  

  　　hadoop-daemon.sh start journalnode
  

  
  
• 　　在standby namenode上执行初始化
  

• [hdfs@hadoop2 hadoop-2.7.3]$ hdfs namenode -bootstrapStandby
  
• .........
  
• 17/04/1918:24:17 INFO ipc.Client:Retrying connect to server: hadoop1/172.18.0.11:8020.Already tried 9 time(s);retry policy isRetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  
• 17/04/1918:24:17 FATAL ha.BootstrapStandby:Unable to fetch namespace information from active NN at hadoop1/172.18.0.11:8020:CallFrom hadoop2/172.18.0.12 to hadoop1:8020 failed on connection exception: java.net.ConnectException:Connection refused;For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
  
• 17/04/1918:24:17 INFO util.ExitUtil:Exitingwith status 2
  
• 17/04/1918:24:17 INFO namenode.NameNode: SHUTDOWN_MSG:
  
• /************************************************************
  
• SHUTDOWN_MSG: Shutting down NameNode at hadoop2/172.18.0.12
  

　　该语句把原namenode上的元数据目录复制到本节点上的相应目录中
　　执行这个语句报错了，连接不上hadoop1:8020.看来官网写的不对啊，操。在执行这一步之前要先启动原来的namenode.先向下继续。

• 初始化edit log　　在原来的namenode上执行：
  

• [hdfs@hadoop1 namenodedir]$ hdfs namenode -initializeSharedEdits
  
• .....
  
• 17/04/1918:28:24 INFO namenode.EditLogInputStream:Fast-forwarding stream '/opt/hadoop/hadoop-2.7.3/namenodedir/current/edits_0000000000000000001-0000000000000000256' to transaction>
  
• 17/04/1918:28:24 INFO namenode.FSEditLog:Starting log segment at 1
  
• 17/04/1918:28:24 INFO namenode.FSEditLog:Ending log segment 1
  
• 17/04/1918:28:24 INFO namenode.FSEditLog:Number of transactions:256Total time for transactions(ms):63Number of transactions batched inSyncs:0Number of syncs:1SyncTimes(ms):11
  
• 17/04/1918:28:24 INFO util.ExitUtil:Exitingwith status 0
  
• 17/04/1918:28:24 INFO namenode.NameNode: SHUTDOWN_MSG:
  
• /************************************************************
  
• SHUTDOWN_MSG: Shutting down NameNode at hadoop1/172.18.0.11
  
• ************************************************************
  

　　这个脚本会把本地的edit log复制到journal节点中去

• 启动原来的namenode
  

• $HADOOP_HOME/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
  

• 再次初始化standby namenode　　刚才初始化失败了，再来一次：
  

• =====================================================
  
• Re-format filesystem inStorageDirectory/opt/hadoop/hadoop-2.7.3/namenodedir ?(Y or N) y
  
• 17/04/1920:30:38 INFO common.Storage:Storage directory /opt/hadoop/hadoop-2.7.3/namenodedir has been successfully formatted.
  
• 17/04/1920:30:38 WARN common.Util:Path/opt/hadoop/hadoop-2.7.3/namenodedir should be specified as a URI in configuration files.Please update hdfs configuration.
  
• 17/04/1920:30:38 WARN common.Util:Path/opt/hadoop/hadoop-2.7.3/namenodedir should be specified as a URI in configuration files.Please update hdfs configuration.
  
• 17/04/1920:30:39 INFO namenode.TransferFsImage:Opening connection to http://hadoop1:50070/imagetransfer?getimage=1&txid=256&storageInfo=-63:51947955:0:CID-3adccc69-45b5-4b44-81b6-70ab593cc1ed
  
• 17/04/1920:30:39 INFO namenode.TransferFsImage:ImageTransfer timeout configured to 60000 milliseconds
  
• 17/04/1920:30:39 INFO namenode.TransferFsImage:Transfer took 0.00s at 500.00 KB/s
  
• 17/04/1920:30:39 INFO namenode.TransferFsImage:Downloaded file fsimage.ckpt_0000000000000000256>
  
• 17/04/1920:30:39 INFO util.ExitUtil:Exitingwith status 0
  
• 17/04/1920:30:39 INFO namenode.NameNode: SHUTDOWN_MSG:
  
• /************************************************************
  
• SHUTDOWN_MSG: Shutting down NameNode at hadoop2/172.18.0.12
  
• ************************************************************/
  

　　这次就行了！！
　　记住，选"Re-format filesystem in Storage Directory /opt/hadoop/hadoop-2.7.3/namenodedir ? (Y or N) Y"
　　7.启动standby namenode

• $HADOOP_HOME/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
  

　　这次启动成功了。
　　所以，正确的顺序是：启动journal->初始化edit log->启动nameode->初始化standby->启动standby
　　8.检查HA web页面：

　　看到两个namenode都是standby!
　　那手动切换一下，设置hadoop1上的namenode为active:

• [hdfs@hadoop2 hadoop]$ hdfs haadmin -failover --forceactive  nn1 nn2
  
• 17/04/1920:51:59 WARN ha.FailoverController:Serviceisnot ready to become active, but forcing:TheNameNodeisin safemode.The reported blocks 0 needs additional 16 blocks to reach the threshold 0.9990 of total blocks 16.
  
• The number of live datanodes 0 has reached the minimum number 0.Safe mode will be turned off automatically once the thresholds have been reached.
  
• Failoverfrom nn1 to nn2 successful
  

　　这样就把nn2切换为active的namenode了

　　为了验证sshfence的正确性，你可以反复的切几次~
4.配置自动HA
4.1 关闭集群
　　在两个namenode上：

• $HADOOP_HOME/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode
  

　　在所有datanode上：

• $HADOOP_HOME/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode
  

4.2 修改配置文件
　　1.在hdfs-site.xml中

• <property>
  
• <name>dfs.ha.automatic-failover.enabled</name>
  
• <value>true</value>
  
• </property>
  

　　2.在core-site.xml中：

• <property>
  
• <name>ha.zookeeper.quorum</name>
  
• <value>hadoop1:2181,hadoop2:2181,hadoop3:2181</value>
  
• </property>
  

　　将以上文件分发到各个节点。
4.3 启动HA
　　1.初始化ZKFC
　　在任意一个namenode上执行：

• [hdfs@hadoop1 hadoop]$ hdfs zkfc -formatZK
  
• ...........
  
• 17/04/1921:09:09 INFO zookeeper.ClientCnxn:Socket connection established to hadoop2/172.18.0.12:2181, initiating session
  
• 17/04/1921:09:09 INFO zookeeper.ClientCnxn:Session establishment complete on server hadoop2/172.18.0.12:2181, sessionid =0x25b6d24ae630000, negotiated timeout =5000
  
• 17/04/1921:09:09 INFO ha.ActiveStandbyElector:Session connected.
  
• 17/04/1921:09:10 INFO ha.ActiveStandbyElector:Successfully created /hadoop-ha/dockercluster in ZK.
  
• 17/04/1921:09:10 INFO zookeeper.ZooKeeper:Session:0x25b6d24ae630000 closed
  
• 17/04/1921:09:10 INFO zookeeper.ClientCnxn:EventThread shut down
  

　　2.启动ZKFC
　　在每个namenode上执行：

• $HADOOP_HOME/sbin/hadoop-daemon.sh --script $HADOOP_HOME/bin/hdfs start zkfc
  

　　注：如果配了SSH免密码登录，直接用start-dfs.sh启动集群即可
　　3.启动namenode

• $HADOOP_HOME/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
  

　　4.进web查看状态

4.4 测试自动切换
　　在hadoop1上直接kill n11:

• [hdfs@hadoop1 hadoop]$ jps
  
• 5588NameNode
  
• 5501DFSZKFailoverController
  
• 5695Jps
  
• [hdfs@hadoop1 hadoop]$ kill 5501
  

　　再看看：

　　说明自动切换正常！
null