自动实现Hadoop Decommission shell脚本版

fgswe

介绍
之前我有篇博文是介绍如何用ansible的playbook来自动化Hadoop Decommission的，本文介绍用shell脚本来实现。
脚本都放在跳板机上，远端服务器调用本地脚本完成所有操作，不需要拷贝文件到远端服务器。


正文
主脚本：decom.sh

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

#!/bin/bash iplist=/home/hadoop/shell_scripts/iplist #1. process iplist ,append them into exclude files # call append.sh for exclude_host in `cat $iplist` ; do     ssh hadoop@hadoopmaster "bash -s" < append.sh "$exclude_host" hdfs-exclude     ssh hadoop@hadoopmaster "bash -s" < append.sh "$exclude_host" mapred-exclude     ssh hadoop@hadoopmaster-standby "bash -s" < append.sh "$exclude_host" hdfs-exclude     ssh hadoop@hadoopmaster-standby "bash -s" < append.sh "$exclude_host" mapred-exclude done #2. ssh hadoop@hadoopmaster "bash -s" < refreshnodes.sh ssh hadoop@hadoopmaster-standby "bash -s" < refreshnodes.sh #3. stop nodemanager and datanode service ,maybe regionserver service too for client in `cat iplist`; do     ssh hadoop@"${client}" "bash -s"  < stopservice.sh done

分脚本：append.sh

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

#!/bin/bash conf_dir=/opt/hadoop-2.6.0/etc/hadoop/ backup_dir=/opt/hadoop-2.6.0/etc/hadoop/BAK/ exclude_host=$1 exclude_file=$2 function usage() {     echo -e "usage: $0 exclude file\nplease input the parameter ---- mapred-exclude or hdfs-exclude" } if [ $# -ne 2 ] ;then     usage     exit 1 elif [ "$exclude_file" != "mapred-exclude" -a "$exclude_file" != "hdfs-exclude" ];then     usage     exit 1 fi #if [ -d /apache/hadoop/conf ] ;then #    cd /apache/hadoop/conf #else #    echo "dir /apache/hadoop/conf doesnot exist , please check!" #    exit 3 #fi [ ! -d ${backup_dir} ] && mkdir ${backup_dir} # backup exclude file cp "${conf_dir}${exclude_file}" ${backup_dir}"${exclude_file}"-`date +%F.%H.%M.%S` # append hosts to exclude file     grep ${exclude_host} "${conf_dir}${exclude_file}" >/dev/null 2>&1     retval=$?     if [ $retval -ne 0 ];then         echo ${exclude_host} >> "${conf_dir}${exclude_file}"     else         echo "duplicated host: ${exclude_host}"     fi

分脚本：refreshnodes.sh

1 2 3 4 5 6 7 8 9 10 11 12 13 14

#!/bin/bash hadoop_bin_dir=/opt/hadoop-2.6.0/bin/ ${hadoop_bin_dir}yarn rmadmin -refreshNodes 2>/dev/null if [ $? -ne 0 ] ; then     echo "command yarn rmadmin -refreshNodes Failed on $(hostname)!!!"     exit 2 fi # wait a while to let mapreduce can switch jobs to other nodes sleep 2 ${hadoop_bin_dir}hadoop dfsadmin -refreshNodes 2>/dev/null if [ $? -ne 0 ] ; then         echo "command hadoop dfsadmin -refreshNodes Failed on $(hostname)!!!"         exit 3 fi

分脚本：stopservice.sh

1 2 3 4 5 6

#!/bin/bash hadoop_bin_dir=/opt/hadoop-2.6.0/sbin/ #svc -d /service/nodemanager #svc -d /service/datanode ${hadoop_bin_dir}yarn-daemon.sh stop nodemanager ${hadoop_bin_dir}hadoop-daemon.sh stop datanode

文件：iplist

1 2	10.9.214.160 10.9.214.149

操作：

1	bash decom.sh