Ansible实现自动decommission Hadoop node

ewe22

介绍如何用ansible的playbook实现自动decommission hadoop 的node ?

正文

• 对于如何部署安装ansible ，请看http://www.iyunv.com/thread-42283-1-1.html
  
  
• hadoop部署，请看http://www.iyunv.com/thread-42284-1-1.html
  
• 使用hadoop用户创建目录ansible_decom,里面的目录结构如下：
  
  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

ansible_decom/ ansible_decom/roles ansible_decom/roles/nn ansible_decom/roles/nn/tasks ansible_decom/roles/nn/tasks/main.yml ansible_decom/roles/decom ansible_decom/roles/decom/tasks ansible_decom/roles/decom/tasks/main.yml ansible_decom/roles/rm ansible_decom/roles/rm/tasks ansible_decom/roles/rm/tasks/main.yml ansible_decom/roles/get-status ansible_decom/roles/get-status/tasks ansible_decom/roles/get-status/tasks/main.yml ansible_decom/group_vars ansible_decom/group_vars/all ansible_decom/exclude-iplist ansible_decom/site.yml ansible_decom/hosts

以yml结尾的是playbook文件，其中site.yml是主文件，他会调用roles下面的子yml文件。
hosts文件定义的是主机列表。

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

[rmservers] 10.9.214.151 10.9.214.15 10.9.214.105 10.9.214.113 [nnservers] 10.9.214.151 10.9.214.15 10.9.214.105 10.9.214.113 [client] 10.9.214.149 [decomnodes] 10.9.214.149 10.9.214.160

exclude-iplist定义的是要decommission的主机列表。

1 2	10.9.214.160 10.9.214.149

site.yml

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

--- # this playbook displays decommission nodes - name: add node to mapred-exclude and refreshnode   hosts: rmservers   remote_user: hadoop   roles:     - rm - name: add node to hdfs-exclude and refreshnodes   hosts: nnservers   remote_user: hadoop   roles:     - nn - name: stop nodemanager and datanode service   hosts: decomnodes   remote_user: hadoop   roles:     - decom - name: get status for hosts   hosts: client   roles:     - get-status

roles/decom/tasks/main.yml

1 2 3 4 5 6

--- # stop services - name: stop nodemanager   command: /opt/hadoop-2.6.0/sbin/yarn-daemon.sh stop nodemanager - name: stop datanode   command: /opt/hadoop-2.6.0/sbin/hadoop-daemon.sh stop datanode

roles/nn/tasks/main.yml

1 2 3 4 5 6 7 8

--- # step1. add decom nodes to hdfs-excluse file # step2. refresh nodes - name: add decom nodes to hdfs-exclude file   copy: src=exclude-iplist dest=/opt/hadoop-2.6.0/etc/hadoop/ - shell: cd /opt/hadoop-2.6.0/etc/hadoop/ ; cat exclude-iplist >> hdfs-exclude ; rm exclude-iplist - name: refresh nodes   shell: /opt/hadoop-2.6.0/bin/hdfs dfsadmin -refreshNodes

roles/rm/tasks/main.yml

1 2 3 4 5 6 7 8 9

--- # step1. add decom nodes to mapred-excluse file # step2. refresh nodes - name: add decom nodes to mapred-exclude file   copy: src=exclude-iplist dest=/opt/hadoop-2.6.0/etc/hadoop/     - shell: cd /opt/hadoop-2.6.0/etc/hadoop/ ; cat exclude-iplist >> mapred-exclude ; rm exclude-iplist - name: refresh nodes   shell: /opt/hadoop-2.6.0/bin/yarn rmadmin -refreshNodes

4. 执行ansible的playbook
asible-playbook -i hosts site.yml

执行若有失败会报错的。

5. 完成之后查看http://10.9.214.151:50070看Live Node和Dead Node的数目和个数是不是变了。