Keepalived实现Redis Failover

李斯特

一、环境说明
操作系统版本：RHEL 5.4_64
redis版本：2.8.17
keepalived版本：1.1.15
master：10.142.130.81
slave：  10.142.130.82
Virtural IP Address (VIP) ：10.142.130.83（对外提供服务）

redis安装路径：/app/tomcat/redis
redis端口：6379
keepalived安装路径：/etc/keepalived

二、设计思路
1、当master与slave均运作正常时，VIP绑定在master上，master负责对外提供服务，slave进行replication，作为备用。
2、当master挂掉，slave正常时，VIP漂移至slave，由slave接管服务，同时关闭主从复制功能。
3、当master恢复正常后，从slave上同步数据，但不继续接管服务，此时master作为备用。
4、当slave挂掉，master正常，master接管服务，关闭主从复制功能。与此同时，位于slave恢复正常后，从master上同步数据，作为备用。
5、依次循环。
注意：master和slave上均开启rdb快照持久化。


三、具体配置步骤
1、在master和slave上分别安装redis
# su -
# tar zxf redis-2.8.17.tar.gz
# cd zxf redis-2.8.17
# make && make install
# cd ..
# mv redis-2.8.17 /app/tomcat/redis
# chown -R tomcat.app /app/tomcat/redis

2、配置redis
# su - tomcat
# cd /app/tomcat/redis
# mkdir {bin,rdb,conf,log}
# mv  redis.conf  sentinel.conf ./conf/
# find /app/tomcat/redis/ -maxdepth 1 -type f -delete
# cd src
# mv mkreleasehdr.sh  redis-benchmark  redis-check-aof  redis-check-dump  redis-cli  redis-sentinel  redis-server ../bin/

编辑/app/tomcat/redis/conf/redis.conf文件，修改如下内容：
daemonize yes   
pidfile /app/tomcat/redis/redis.pid
port 6376  
logfile  /app/tomcat/redis/log/redis.log
dir /app/tomcat/redis/rdbmaxmemory 16106127360   #给redis设定最大使用内存，这里是15G，根据实际情况而定。

3、在master和slave上分别安装keepalived


# su　　# tar zxf keepalived-1.1.15.tar.gz
　　# cd keepalived-1.1.15
　　# ./configure --sysconf=/etc --with-kernel-dir=/usr/src/kernels/2.6.18-164.el5-x86_64
　　# make
　　# make install
　　# cp /usr/local/sbin/keepalived /sbin/keepalived



4、master上的keepalived配置

# cd /etc/keepalived
# >keepalived.conf
# vi keepalived.conf  #添加如下内容
! Configuration File for keepalived

global_defs {
        router_id redis-master
}

vrrp_script Monitor_redis {
        script "/etc/keepalived/scripts/redis_monitor.sh"
        interval 2
        weight 2
}

vrrp_instance VI_1{
        state BACKUP
        interface bond0
        virtual_router_id 51
        mcast_src_ip 10.142.130.81
        priority  100
        nopreempt
        advert_int 1
        authentication {
        auth_type PASS
        auth_pass 1122
        }
        track_script {
                Monitor_redis
        }
        virtual_ipaddress {
                10.142.130.83
        }
        notify_master /etc/keepalived/scripts/redis_master.sh
        notify_backup /etc/keepalived/scripts/redis_backup.sh
        notify_fault  /etc/keepalived/scripts/redis_fault.sh
        notify_stop   /etc/keepalived/scripts/redis_stop.sh
}

# mkdir scripts
# cd scripts
# vi redis_monitor.sh  #redis监控脚本
#!/bin/bash   
value=$(/app/tomcat/redis/bin/redis-cli -h 10.142.130.81 -p 6379 get name)

if [ "${value}" == "test" ]; then
    exit 0
    else
    /etc/init.d/keepalived stop
    exit 1
fi            

# vi redis_master.sh #当状态为master时执行的脚本，用来关闭主从复制
#!/bin/bash   
REDISCLI="/app/tomcat/redis/bin/redis-cli"   
LOGFILE="/app/tomcat/redis/log/keepalived-redis-state.log"   
echo "[master]" >> ${LOGFILE}
date >> ${LOGFILE}   
echo "Being master...." >> ${LOGFILE} 2>&1   
echo "Run SLAVEOF NO ONE cmd ..." >> ${LOGFILE}
${REDISCLI} SLAVEOF NO ONE >> ${LOGFILE} 2>&1

# vi redis_backup.sh #当状态为backup时执行的脚本，用于开启主从复制
#!/bin/bash   
REDISCLI="/app/tomcat/redis/bin/redis-cli"   
LOGFILE="/app/tomcat/redis/log/keepalived-redis-state.log"   
echo "[backup]" >> ${LOGFILE}
date >> ${LOGFILE}
echo "Being slave...." >> ${LOGFILE} 2>&1   
sleep 10   
echo "Run SLAVEOF cmd ..." >> ${LOGFILE}
${REDISCLI} SLAVEOF 10.142.130.82 6379 >> ${LOGFILE}  2>&1

# vi redis_fault.sh
#!/bin/bash  
LOGFILE="/app/tomcat/redis/redis/log/keepalived-redis-state.log"
echo "[fault]" >> $LOGFILE  
date >> $LOGFILE

# vi redis_stop.sh
#!/bin/bash  
LOGFILE="/app/tomcat/redis/log/keepalived-redis-state.log"   
echo "[stop]" >> $LOGFILE  
date >> $LOGFILE

给脚本加上可执行权限chmod +x /etc/keepalived/scripts/*.sh


5、slave上的keepalived配置


# cd /etc/keepalived
# >keepalived.conf
# vi keepalived.conf  #添加如下内容! Configuration File for keepalived  

global_defs {  
  router_id redis_backup
}  

vrrp_script Monitor_redis {  
  script "/etc/keepalived/scripts/redis_monitor.sh"  
  interval 2   
  weight 2   
}  

vrrp_instance VI_1{   
  state BACKUP   
  interface bond0  
  virtual_router_id 51  
  mcast_src_ip 10.142.130.82
  priority  90  
  advert_int 1  
  authentication {  
  auth_type PASS  
  auth_pass 1122
}  
  track_script {  
   Monitor_redis  
}  
  virtual_ipaddress {  
   10.142.130.83
  }  
notify_master /etc/keepalived/scripts/redis_master.sh   
  notify_backup /etc/keepalived/scripts/redis_backup.sh   
  notify_fault  /etc/keepalived/scripts/redis_fault.sh   
  notify_stop   /etc/keepalived/scripts/redis_stop.sh   
}


# mkdir scripts
# cd scripts
# vi redis_monitor.sh  #redis监控脚本#!/bin/bash
value=$(/app/tomcat/redis/bin/redis-cli -h 10.142.130.82 -p 6379 get name)
if [ "${value}" == "test" ]; then
    exit 0
    else
    /etc/init.d/keepalived stop
    exit 1
fi

# vi redis_master.sh
#!/bin/bash   
REDISCLI="/app/tomcat/redis/bin/redis-cli"   
LOGFILE="/app/tomcat/redis/log/keepalived-redis-state.log"   

echo "[master]" >> ${LOGFILE}
date >> ${LOGFILE}   
echo "Being master...." >> ${LOGFILE} 2>&1   
echo "Run SLAVEOF NO ONE cmd ..." >> ${LOGFILE}
${REDISCLI} SLAVEOF NO ONE >> ${LOGFILE} 2>&1

# vi redis_backup.sh
#!/bin/bash   
REDISCLI="/app/tomcat/redis/bin/redis-cli"   
LOGFILE="/app/tomcat/redis/log/keepalived-redis-state.log"   
echo "[backup]" >> ${LOGFILE}
date >> ${LOGFILE}
echo "Being slave...." >> ${LOGFILE} 2>&1   
sleep 10   
echo "Run SLAVEOF cmd ..." >> ${LOGFILE}
${REDISCLI} SLAVEOF 10.142.130.81 6379 >> ${LOGFILE}  2>&1

# vi redis_fault.sh
#!/bin/bash  
LOGFILE="/app/tomcat/redis/redis/log/keepalived-redis-state.log"
echo "[fault]" >> ${LOGFILE}
date >> ${LOGFILE}

# vi redis_stop.sh
#!/bin/bash  
LOGFILE="/app/tomcat/redis/log/keepalived-redis-state.log"   
echo "[stop]" >> ${LOGFILE}
date >> ${LOGFILE}


给脚本加上可执行权限chmod +x /etc/keepalived/scripts/*.sh


四、启动
为方便后续管理，编写一个简单脚本重启redis
# vi restart-redis.sh
#!/bin/bash
ps -ef|awk '/app\/tomcat\/redis\/bin\/redis-server/{print $2}'|xargs kill -9
/app/tomcat/redis/bin/redis-server /app/tomcat/redis/conf/redis.conf

启动顺序
1、分别启动master和slave上redis
2、给master和slave上的redis设置检测的key和value
     # /app/tomcat/redis/bin/redis-cli set name test
3、先启动salve上的keepalived，启动方法，用root用户执行命令/etc/init.d/keepalived start
4、用同样的方法启动master上的keepalived

注意事项：
执行ip addr show bond0命令查看VIP绑定在哪个主机上，执行redis-cli info查看主从关系，确保VIP绑定的机器一定要是redis中的master。
如果不是，通过手动关闭keepalived进行调整，一定要保证VIP和主从复制关系正确。首次使用前调整好，后续自动切换基本不会有问题。


五、监控
master挂了，slave会接管服务，那如何去恢复master，让他恢复之后成为slave的角色呢。这就需要要每台主机上部署一个监控脚本，定时每分钟监测一次。脚本如下：
# cat monitor_redis.sh
#!/bin/bash
num_proc=$(/bin/ps -fe|grep [k]eepalived|wc -l)
active=$(/app/tomcat/redis/bin/redis-cli get name)
if [ "${num_proc}" != "3" ];then
        if [ "${active}" != "test" ];then
                /app/tomcat/bin/restart-redis.sh &>/dev/null
                if [ "$(/bin/ps -ef|grep [r]edis-server|wc -l)" == "1" -a "${active}" == "test" ];then
                        sleep 30
                        /usr/bin/sudo /etc/init.d/keepalived start &>/dev/null
                else
                        /app/tomcat/redis/bin/redis-cli set name test
                        [ "${active}" == "test" ] && /usr/bin/sudo /etc/init.d/keepalived start &>/dev/null
                fi
        else
                /usr/bin/sudo /etc/init.d/keepalived start &>/dev/null
        fi
fi

六、总结
1、keepalived的master与slave一般通过state和priority来指定，但是这两项设置满足不了我们上面的切换要求，这里是将state 都设置为BACKUP，然后以优先级riority的高低来决定     最初的master归属，另外优先级高的那台需要添加 nopreempt，这个参数的作用是keepalived恢复之后不主动抢占master的角色，等另一端挂掉之后会自动接管。
2、redis启动成功了之后并且能get设定的那个值后keepalived才能正常启动。
3、随着dump.rdb文件日益增大，redis重启后完全将数据加载进内存的时间会越来越长。monitor_redis.sh脚本中sleep 时间也需要作相应调整。