Redis主从自动failover-ylw6006

tedwhy

　　Redis主从架构持久化存在一个问题，即前次测试的结论，持久化需要配置在主实例上才能跨越实例保证数据不丢失，这样以来主实例在持久化数据到硬盘的过程中，势必会造成磁盘的I/O等待，经过实际测试，这个持久化写硬盘的过程给应用程序带来的影响无法忍受；因而在大多数场景下，会考虑把持久化配置在从实例上，当主实例宕机后，通过手动或者自动的方式将从实例提升为主实例，继续提供服务！当主实例恢复后，先从原从实例上同步数据，同步完成后再恢复到原始的主从状态！要实现这种的要求，需要有keepalive的配合，一方面keepalive提供了VIP，可以避免修改应用程序连接，同时redis实例的配置文件监听部分也需要修改为全网监听；另一方面keepalive定时调度脚本来监控主从实例的状态，根据具体情况进行切换！本文将重点介绍下使用keepalive实现redis主从自动failover！
　　环境介绍
　　
操作系统版本均为：rhel5.4 64bit
　　
redis版本：2.6.4
　　
redis实例端口均为：6379
　　
redis实例密码均为：123
　　
VIP：192.168.1.120
　　
主实例为server11(192.168.1.112)
　　
从实例为server12(192.168.1.113，开启快照持久化)
　　一：安装keepalive软件,server11安装完成后直接scp至server12上即可
　　

• [root@server11 ~]# wget http://keepalived.org/software/keepalived-1.1.19.tar.gz
  
• [root@server11 ~]# tar -zxvf ../tarbag/keepalived-1.1.19.tar.gz
  
• [root@server11 ~]# cd keepalived-1.1.19/
  
• [root@server11 ~]# ./configure --prefix=/usr/local/keepalived && make && make install
  

　　

　　二：配置主节点server11配置文件
　　

• [root@server11 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf
  
• ! Configuration File for keepalived
  
• 
  
• global_defs {
  
• router_id LVS_DEVEL
  
• }
  
• 
  
• vrrp_script Monitor_redis {
  
• script "/usr/local/scripts/redis_monitor.sh"
  
• interval 2
  
• weight 2
  
• }
  
• 
  
• vrrp_instance VI_1{
  
• state MASTER
  
• interface eth0
  
• virtual_router_id 51
  
• mcast_src_ip 192.168.1.112
  
• priority  100
  
• advert_int 1
  
• authentication {
  
• auth_type PASS
  
• auth_pass password_123
  
• }
  
• track_script {
  
• Monitor_redis
  
• }
  
• virtual_ipaddress {
  
• 192.168.1.120
  
• }
  
• notify_fault  /usr/local/scripts/redis_fault.sh
  
• notify_stop   /usr/local/scripts/redis_stop.sh
  
• 
  
• }
  

　　

　　三：配置从节点server12配置文件
　　

• [root@server12 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf
  
• ! Configuration File for keepalived
  
• 
  
• global_defs {
  
• router_id LVS_DEVEL
  
• }
  
• 
  
• vrrp_script Monitor_redis {
  
• script "/usr/local/scripts/redis_monitor.sh"
  
• interval 2
  
• weight 2
  
• }
  
• 
  
• vrrp_instance VI_1{
  
• state BACKUP
  
• interface eth0
  
• virtual_router_id 51
  
• mcast_src_ip 192.168.1.113
  
• priority  99
  
• advert_int 1
  
• authentication {
  
• auth_type PASS
  
• auth_pass password_123
  
• }
  
• track_script {
  
• Monitor_redis
  
• }
  
• virtual_ipaddress {
  
• 192.168.1.120
  
• }
  
• notify_master /usr/local/scripts/redis_master.sh
  
• notify_backup /usr/local/scripts/redis_backup.sh
  
• notify_fault  /usr/local/scripts/redis_fault.sh
  
• notify_stop   /usr/local/scripts/redis_stop.sh
  
• 
  
• }
  

　　

　　四：准备相关的脚本，主从实例上都需要存在这些脚本，同时注意脚本需要由可执行权限
　　

• [root@server11 ~]# cat /usr/local/scripts/redis_monitor.sh
  
• #!/bin/bash
  
• ALIVE=$(/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123 PING)
  
• 
  
• if [ "$ALIVE" == "PONG" ]; then
  
•     echo $ALIVE
  
•     exit 0
  
•     else
  
•     echo $ALIVE
  
•     killall -9 keepalived
  
•     service network restart
  
•     exit 1
  
• fi
  
• 
  
• [root@server11 ~]# sh /usr/local/scripts/redis_monitor.sh
  
• PONG
  
• 
  
• [root@server11 ~]# cat /usr/local/scripts/redis_master.sh
  
• #!/bin/bash
  
• REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123"
  
• LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"
  
• 
  
• echo "[master]" >> $LOGFILE
  
• date >> $LOGFILE
  
• echo "Being master...." >> $LOGFILE 2>&1
  
• echo "Run SLAVEOF cmd ..." >> $LOGFILE
  
• $REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE  2>&1
  
• sleep 10
  
• echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE
  
• $REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1
  
• 
  
• 
  
• [root@server11 ~]# cat /usr/local/scripts/redis_backup.sh
  
• #!/bin/bash
  
• REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123"
  
• LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"
  
• 
  
• echo "[backup]" >> $LOGFILE
  
• date >> $LOGFILE
  
• echo "Being slave...." >> $LOGFILE 2>&1
  
• sleep 15
  
• echo "Run SLAVEOF cmd ..." >> $LOGFILE
  
• $REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE  2>&1
  
• 
  
• [root@server11 ~]# cat /usr/local/scripts/redis_stop.sh
  
• #!/bin/bash
  
• LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"
  
• echo "[stop]" >> $LOGFILE
  
• date >> $LOGFILE
  
• [root@server11 ~]# cat /usr/local/scripts/redis_fault.sh　　#!/bin/bash
  　　LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"
  　　echo "[fault]" >> $LOGFILE
  　　date >> $LOGFILE
  

　　

　　五：主从实例分别启动keepalive进程，测试VIP是否正常(这里就要修改redis配置文件的监听地址为0.0.0.0)
　　

• [root@server11 ~]# /usr/local/keepalived/sbin/keepalived -D -f  /usr/local/keepalived/etc/keepalived/keepalived.conf
  
• [root@server11 ~]# tail -f /var/log/messages
  
• Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Configuration is using : 5499 Bytes
  
• Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Using LinkWatch kernel netlink reflector...
  
• Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]
  
• Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP_Script(Monitor_redis) succeeded
  
• Dec 12 09:25:50 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Transition to MASTER STATE
  
• Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Entering MASTER STATE
  
• Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) setting protocol VIPs.
  
• Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
  
• Dec 12 09:25:51 server11 avahi-daemon[4519]: Registering new address record for 192.168.1.120 on eth0.
  
• Dec 12 09:25:51 server11 Keepalived_healthcheckers[7710]: Netlink reflector reports IP 192.168.1.120 added
  
• Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: Netlink reflector reports IP 192.168.1.120 added
  
• Dec 12 09:25:56 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
  
• 
  
• [root@server11 ~]# ip a |grep 192
  
•     inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0
  
•     inet 192.168.1.120/32 scope global eth0
  
• 
  
• [root@server12 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf
  
• [root@server12 ~]# tail -f /var/log/messages
  
• Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Configuration is using : 5595 Bytes
  
• Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering BACKUP STATE
  
• Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Using LinkWatch kernel netlink reflector...
  
• Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]
  
• Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Script(Monitor_redis) succeeded
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'
  
• # Replication
  
• role:master
  
• connected_slaves:1
  
• slave0:192.168.1.113,6379,online
  

　　

　　六：主实例写入测试数据，该脚本原则上会写入25条测试数据，不过由于未优化redis默认并发数，会导致一些写入请求失败，最终功写入231839条测试数据，占内存总大小为25M左右，写入过程中可以观察主从实例的持久化文件变化情况，主实例的持久化文件维持在30k，从实例的则不断的扩展！
　　

• [root@server11 ~]# cat test.sh
  
• #!/bin/bash
  
• REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 -n 1 SET"
  
• ID=1
  
• while(($ID