Redis手动failover-ylw6006

cike0415

　　本文介绍redis主从环境下的手工failover操作及排错过程，实现主实例宕机的时候，将从实例提升为主实例，继续写入数据；等到原主实例恢复后，同步原从实例上的数据完成后，恢复初始的主从实例角色！
　　环境介绍
　　
操作系统版本均为：rhel5.4 64bit
　　
redis版本：2.6.4
　　
redis实例端口均为：6379
　　
redis实例密码均为：123
　　
主实例为server11(192.168.1.112)
　　
从实例为server12(192.168.1.113)
　　一：未配置持久化情况下的手工切换
　　
1：正常情况下，server11为主实例，server12为从实例，数据同步正常
　　

• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 info |grep -A 3 'Replication'
  
• # Replication
  
• role:master
  
• connected_slaves:1
  
• slave0:192.168.1.113,6379,online
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 config get save
  
• 1) "save"
  
• 2) ""
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 config get save
  
• 1) "save"
  
• 2) ""
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 set 5 e
  
• OK
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 get 5
  
• "e"
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 5
  
• "e"
  

　　

　　2：当主实例挂掉的时候，从实例可以正常查询，但无法写入数据
　　

• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 shutdown
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 get 5
  
• Could not connect to Redis at 192.168.1.112:6379: Connection refused
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 5
  
• "e"
  
• [root@server12 ~]#  /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 set 6 f
  
• (error) READONLY You can't write against a read only slave.
  

　　

　　3：将从实例提升为主实例，从而实现数据写入
　　

• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 SLAVEOF NO ONE
  
• OK
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 info |grep -A 3 'Replication'
  
• # Replication
  
• role:master
  
• connected_slaves:0
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 5
  
• "e"
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 set 6 f
  
• OK
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 6
  
• "f"
  

　　

　　4:主实例恢复后尝试从server12实例上获取最新的数据，实际测试表明这种方法不可行，最终导致server11和server12的数据不一致，如果强行恢复初始实例角色，则会导致数据丢失
　　

• [root@server11 ~]# /usr/local/redis2/bin/redis-server /usr/local/redis2/etc/redis.conf
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 info |grep -A 3 'Replication'
  
• # Replication
  
• role:master
  
• connected_slaves:0
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123  get 5
  
• (nil)
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123  get 6
  
• (nil)
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123  get 5
  
• "e"
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123  get 6
  
• "f"
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123 SLAVEOF 192.168.1.113 6379
  
• OK
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 info |grep -A 10 'Replication'
  
• # Replication
  
• role:slave
  
• master_host:192.168.1.113
  
• master_port:6379
  
• master_link_status:down
  
• master_last_io_seconds_ago:-1
  
• master_sync_in_progress:0
  
• master_link_down_since_seconds:517
  
• slave_priority:100
  
• slave_read_only:1
  
• connected_slaves:0
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 info |grep -A 3 'Replication'
  
• # Replication
  
• role:master
  
• connected_slaves:0
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 get 5
  
• (nil)
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 get 6
  
• (nil)
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 6
  
• "f"
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 5
  
• "e"
  

　　

　　二：开启从实例快照持久化下的测试
　　
1：恢复原测试环境后，开启从实例的快照持久化，因为是测试环境，所以设置60秒内如果有1条数据变更则保持一次快照
　　

• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 config get save
  
• 1) "save"
  
• 2) ""
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 config get save
  
• 1) "save"
  
• 2) "60 1"
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 info |grep -A 3 'Replication'
  
• # Replication
  
• role:master
  
• connected_slaves:1
  
• slave0:192.168.1.113,6379,online
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 info |grep -A 3 'Replication'
  
• # Replication
  
• role:slave
  
• master_host:192.168.1.112
  
• master_port:6379
  

　　

　　2：写入测试数据主从环境数据是否同步正常
　　

• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 set 5 e
  
• OK
  
• 
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 get 5
  
• "e"
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 5
  
• "e"
  

　　

　　3：模拟主实例宕机，手动将从实例提升为主实例，继续写入新数据
　　

• [root@server11 ~]# killall -9 redis-server
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 info |grep -A 3 'Replication'
  
• Could not connect to Redis at 192.168.1.112:6379: Connection refused
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 5
  
• "e"
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 set 6 f
  
• (error) READONLY You can't write against a read only slave
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 slaveof no one
  
• OK
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 info |grep -A 3 'Replication'
  
• # Replication
  
• role:master
  
• connected_slaves:0
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 5
  
• "e"
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 set 6 f
  
• OK
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 6
  
• "f"
  

　　

　　4:原主实例恢复后的数据同步及角色复原,这里同步数据采取将从实例的快照文件复制到主实例的方式实现
　　

• [root@server12 ~]# scp /usr/local/redis2/slave_dump.rdb  server11:/usr/local/redis2/master_dump.rdb
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-server /usr/local/redis2/etc/redis.conf
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 info |grep -A 2 'Replication'
  
• # Replication
  
• role:master
  
• connected_slaves:0
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 get 5
  
• "e"
  
• [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 get 6
  
• "f"
  
• 
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 slaveof 192.168.1.112 6379
  
• OK
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 info |grep -A 10 'Replication'
  
• # Replication
  
• role:slave
  
• master_host:192.168.1.112
  
• master_port:6379
  
• master_link_status:up
  
• master_last_io_seconds_ago:1
  
• master_sync_in_progress:0
  
• slave_priority:100
  
• slave_read_only:1
  
• connected_slaves:0
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 5
  
• "e"
  
• [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 get 6
  
• "f
  
• 
  
• [root@server11 ~]#  /usr/local/redis2/bin/redis-cli -h 192.168.1.112 -a 123 info |grep -A 3 'Replication'
  
• # Replication
  
• role:master
  
• connected_slaves:1
  
• slave0:192.168.1.113,6379,online
  

　　

　　后续扩展：本文实现的failover过程，到从实例提升到主实例阶段都是可以通过部署keepalive自动实现的，在最后原主实例数据同步和角色复原可以通过shell脚本来调度，下篇文章中将对此进行详细的介绍！