|
集群的规划
Zookeeper集群:
192.168.142.12 (bigdata12)
192.168.142.13 (bigdata13)
192.168.142.14 (bigdata14)
Hadoop集群:
192.168.142.12 (bigdata12) NameNode1 ResourceManager1 Journalnode
192.168.142.13 (bigdata13) NameNode2 ResourceManager2 Journalnode
192.168.142.14 (bigdata14) DataNode1 NodeManager1
192.168.142.15 (bigdata15) DataNode2 NodeManager2
1、准备工作:
(1)、关闭防火墙:
查看防火墙的状态: systemctl status firewalld.service
关闭防火墙: systemctl stop firewalld.service
禁用防火墙(永久):systemctl disable firewalld.service
(2)、安装JDK及配置环境变量
tar -zxvf jdk-8u144-linux-x64.tar.gz -C ~/training/
设置环境变量:vi ~/.bash_profile
JAVA_HOME=/root/training/jdk1.8.0_144
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
export PATH
生效环境变量: source ~/.bash_profile
(3)、安装Hadoop及设置环境变量:
解压:tar -zxvf hadoop-2.7.3.tar.gz -C ~/training/
设置环境变量:
vi ~/.bash_profile
HADOOP_HOME=/root/training/hadoop-2.7.3
export HADOOP_HOME
PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export PATH
生效环境变量:
source ~/.bash_profile
(4)、设置免密码登录:
a、生成密钥:
ssh-keygen -t rsa
(存储在~/.ssh目录)
b、分发公钥:
ssh-copy-id -i /root/.ssh/id_rsa.pub root@bigdata12
c、验证:
ssh bigdata12
(5)、配置主机名:/etc/hosts文件
vi /etc/hosts
192.168.157.11 bigdata11
2、安装配置ZooKeeper集群:参考“ZooKeeper安装说明”
3、配置Hadoop集群:(在bigdata12上安装)
(1)修改hadoop-env.sh文件:(在bigdata12上执行)
export JAVA_HOME=/root/training/jdk1.8.0_144
(2)修改core-site.xml文件:(在bigdata12上执行)
fs.defaultFS
hdfs://ns1
hadoop.tmp.dir
/root/training/hadoop-2.7.3/tmp
ha.zookeeper.quorum
bigdata12:2181,bigdata13:2181,bigdata14:2181
(3)修改hdfs-site.xml(在bigdata12上执行)
dfs.nameservices
ns1
dfs.ha.namenodes.ns1
nn1,nn2
dfs.namenode.rpc-address.ns1.nn1
bigdata12:9000
dfs.namenode.http-address.ns1.nn1
bigdata12:50070
dfs.namenode.rpc-address.ns1.nn2
bigdata13:9000
dfs.namenode.http-address.ns1.nn2
bigdata13:50070
dfs.namenode.shared.edits.dir
qjournal://bigdata12:8485;bigdata13:8485;/ns1
dfs.journalnode.edits.dir
/root/training/hadoop-2.7.3/journal
dfs.ha.automatic-failover.enabled
true
dfs.client.failover.proxy.provider.ns1
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.ha.fencing.methods
sshfence
shell(/bin/true)
dfs.ha.fencing.ssh.private-key-files
/root/.ssh/id_rsa
dfs.ha.fencing.ssh.connect-timeout
30000
(4)、修改mapred-site.xml文件(在bigdata12上执行)
mapreduce.framework.name
yarn
(5)、修改yarn-site.xml文件(在bigdata12上执行)
yarn.resourcemanager.ha.enabled
true
yarn.resourcemanager.ha.rm-ids
rm1,rm2
yarn.resourcemanager.hostname.rm1
bigdata12
yarn.resourcemanager.hostname.rm2
bigdata13
yarn.resourcemanager.zk-address
bigdata12:2181,bigdata13:2181,bigdata14:2181
yarn.nodemanager.aux-services
mapreduce_shuffle
(6)、修改slaves文件(/root/training/hadoop-2.7.3/etc/hadoop),设置从节点
bigdata14
bigdata15
(7)、创建目录(在bigdata12上执行)
/root/training/hadoop-2.7.3/tmp
/root/training/hadoop-2.7.3/journal
(8)、将配置好的hadoop拷贝到其他节点
scp -r /root/training/hadoop-2.7.3/ root@bigdata13:/root/training/
scp -r /root/training/hadoop-2.7.3/ root@bigdata14:/root/training/
scp -r /root/training/hadoop-2.7.3/ root@bigdata15:/root/training/
(8)、启动ZooKeeper集群
zkServer.sh start
(9)、单独启动启动journalnode(bigdata12和bigdata13)
hadoop-daemon.sh start journalnode
(10)NameNode节点格式化HDFS(在bigdata12上执行)
hdfs namenode -format
(11)拷贝bigdata12的dfs目录到bigdata13
/root/training/hadoop-2.7.3/tmp/dfs拷贝到bigdata13的/root/training/hadoop-2.7.3/tmp
scp -r dfs/ root@bigdata13:/root/training/hadoop-2.7.3/tmp
(12)、格式化ZooKeeper(在bigdata12上执行)
hdfs zkfc -formatZK
日志:INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns1 in ZK.
(13)、启动Hadoop集群(在bigdata12或 bigdata13上执行)
start-all.sh
(14)、单独启动ResourceManager(bigdata12或 bigdata13上执行,取决于待启动)
yarn-daemon.sh start resourcemanager
|
|
|