redis高可用之redis

半只蚂蚁

　　配置：
　　先搭建好redis主从环境：
　　主： 100.5.120.118  6379
　　从1  100.5.120.118  6380
　　从2  100.5.120.118  6381
　　从3  100.5.120.118  6394
　　sentinel机器：
　　100.5.120.119
　　1、获取安装包（2.8以后sentinel集成到了redis中，注意此处是版本2.8
　　之前测使用最新的3.0 切换存在问题，无法提升slave为主）
　　wget http://download.redis.io/releases/redis-2.8.11.tar.gz
　　2、安装：
　　tar -xvf  redis-2.8.11.tar.gz
　　cd redis-2.8.11
　　make
　　make test
　　make install
　　安装完成后，
　　进入到src目录
　　把 redis-sentinel命令拷贝到/usr/local/bin下
　　3、参数文件配置：
　　[iyunv@localhost ~]# cat /etc/sentinel.conf_new
　　port 26379
　　dir "/tmp"
　　sentinel monitor mymaster 100.5.120.118 6379 1
　　sentinel down-after-milliseconds mymaster 5000
　　#sentinel can-failover mymaster yes
　　sentinel failover-timeout mymaster 910000
　　sentinel parallel-syncs mymaster 2
　　
　　4、启动：
　　redis-sentinel  /etc/sentinel.conf_new
　　或者
　　redis-server    /etc/sentinel.conf_new --sentinel 
　　提示输出：
　　[iyunv@localhost ~]# redis-sentinel  /etc/sentinel.conf_new
　　_._                                                 
　　_.-``__ ''-._                                            
　　_.-``    `.  `_.  ''-._           Redis 2.8.11 (00000000/0) 64 bit
　　.-`` .-```.  ```\/    _.,_ ''-._                                  
　　(    '      ,       .-`  | `,    )     Running in sentinel mode
　　|`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
　　|    `-._   `._    /     _.-'    |     PID: 3311
　　`-._    `-._  `-./  _.-'    _.-'                                  
　　|`-._`-._    `-.__.-'    _.-'_.-'|                                 
　　|    `-._`-._        _.-'_.-'    |           http://redis.io       
　　`-._    `-._`-.__.-'_.-'    _.-'                                  
　　|`-._`-._    `-.__.-'    _.-'_.-'|                                 
　　|    `-._`-._        _.-'_.-'    |                                 
　　`-._    `-._`-.__.-'_.-'    _.-'                                  
　　`-._    `-.__.-'    _.-'                                      
　　`-._        _.-'                                          
　　`-.__.-'                                              
　　[3311] 27 May 15:50:13.340 # Sentinel runid is 00f8715dbff384753eb96ba969d963ed87de89bc
　　[3311] 27 May 15:50:13.340 # +monitor master mymaster 100.5.120.118 6379 quorum 1
　　[3311] 27 May 15:50:13.341 * +slave slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:13.341 * +slave slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:13.341 * +slave slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6379
　　
　　5、测试failover，关闭6379的redis：
　　[3311] 27 May 15:50:23.939 # +sdown master mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:23.939 # +odown master mymaster 100.5.120.118 6379 #quorum 1/1
　　[3311] 27 May 15:50:23.939 # +new-epoch 1
　　[3311] 27 May 15:50:23.939 # +try-failover master mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:23.948 # +vote-for-leader 00f8715dbff384753eb96ba969d963ed87de89bc 1
　　[3311] 27 May 15:50:23.948 # +elected-leader master mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:23.948 # +failover-state-select-slave master mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:24.048 # +selected-slave slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:24.048 * +failover-state-send-slaveof-noone slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:24.125 * +failover-state-wait-promotion slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:24.468 # +promoted-slave slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:24.468 # +failover-state-reconf-slaves master mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:24.526 * +slave-reconf-sent slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:24.526 * +slave-reconf-sent slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:25.489 * +slave-reconf-inprog slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:25.489 * +slave-reconf-done slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:25.489 * +slave-reconf-inprog slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:25.489 * +slave-reconf-done slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:25.588 # +failover-end master mymaster 100.5.120.118 6379
　　[3311] 27 May 15:50:25.588 # +switch-master mymaster 100.5.120.118 6379 100.5.120.118 6380
　　[3311] 27 May 15:50:25.588 * +slave slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 15:50:25.589 * +slave slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 15:50:25.590 * +slave slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 15:50:30.672 # +sdown slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 15:50:50.954 # -sdown slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　日志记录了完整的切换过程，6380升级为新主节点了。
　　登陆到6380上查看确认：
　　127.0.0.1:6380> info replication
　　# Replication
　　role:master
　　connected_slaves:3
　　slave0:ip=100.5.120.118,port=6381,state=online,offset=53624,lag=0
　　slave1:ip=100.5.120.118,port=6394,state=online,offset=53624,lag=1
　　master_repl_offset:53624
　　repl_backlog_active:1
　　repl_backlog_size:1048576
　　repl_backlog_first_byte_offset:2
　　repl_backlog_histlen:53623
　　6、启动6379端口的redis：
　　[3311] 27 May 15:51:00.912 * +convert-to-slave slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　可以看到6379自动转换成了6380的一个slave了。
　　
　　整个过程包括配置还是很简单的，遇到的问题就是版本问题：之前使用sentinel使用redis_version:3.0.1版本
　　发现无法完成正常切换，至于网上有人说必须使用aof关掉save才能完成切换，本人亲测跟是否开启aof无关，都能
　　正常切换
　　7、
　　sentinel可以通过命令配置：
　　添加集群：
　　sentinel monitor mymaster 100.5.120.118  6380 1
　　日志：
　　[3311] 27 May 16:52:01.853 # +monitor master mymaster 100.5.120.118 6380 quorum 1
　　[3311] 27 May 16:52:01.901 * +slave slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:52:01.901 * +slave slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:52:01.902 * +slave slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　删除集群：
　　sentinel remove mymaster
　　日志：[3311] 27 May 16:50:35.098 # -monitor master mymaster 100.5.120.118 6380
　　列出主节点信息：
　　sentinel masters
　　列出从节点信息：
　　sentinel slaves mymaster
　　强制集群failover：
　　sentinel failover mymaster
　　日志：
　　[3311] 27 May 16:56:21.927 # Executing user requested FAILOVER of 'mymaster'
　　[3311] 27 May 16:56:21.927 # +new-epoch 2
　　[3311] 27 May 16:56:21.927 # +try-failover master mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:21.934 # +vote-for-leader 00f8715dbff384753eb96ba969d963ed87de89bc 2
　　[3311] 27 May 16:56:21.934 # +elected-leader master mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:21.934 # +failover-state-select-slave master mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:21.993 # +selected-slave slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:21.993 * +failover-state-send-slaveof-noone slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:22.059 * +failover-state-wait-promotion slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:22.956 # +promoted-slave slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:22.956 # +failover-state-reconf-slaves master mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:23.009 * +slave-reconf-sent slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:23.995 * +slave-reconf-inprog slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:23.995 * +slave-reconf-done slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:24.060 * +slave-reconf-sent slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:25.026 * +slave-reconf-inprog slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:25.026 * +slave-reconf-done slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:25.126 # +failover-end master mymaster 100.5.120.118 6380
　　[3311] 27 May 16:56:25.126 # +switch-master mymaster 100.5.120.118 6380 100.5.120.118 6379
　　[3311] 27 May 16:56:25.126 * +slave slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 16:56:25.127 * +slave slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 16:56:25.127 * +slave slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[3311] 27 May 16:56:35.244 * +convert-to-slave slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　
　　8、sentinel可以做集群：
　　复制26379端口的配置文件修改成自己的端口：26380,26381启动就ok了：
　　26379:
　　Sentinel runid is 649f4731eb4eef48834b278712d6a79d08e19fe0
　　26380:
　　Sentinel runid is 9f57020ff3e36582b8d272efb09641a875534329
　　26381:
　　Sentinel runid is 308e61fca8a4bd4355b12a6f87ad9dd8788a0e25
　　启动日志：
　　[iyunv@localhost ~]# redis-sentinel /etc/sentinel.conf_new2
　　_._                                                 
　　_.-``__ ''-._                                            
　　_.-``    `.  `_.  ''-._           Redis 2.8.11 (00000000/0) 64 bit
　　.-`` .-```.  ```\/    _.,_ ''-._                                  
　　(    '      ,       .-`  | `,    )     Running in sentinel mode
　　|`-._`-...-` __...-.``-._|'` _.-'|     Port: 26380
　　|    `-._   `._    /     _.-'    |     PID: 28024
　　`-._    `-._  `-./  _.-'    _.-'                                  
　　|`-._`-._    `-.__.-'    _.-'_.-'|                                 
　　|    `-._`-._        _.-'_.-'    |           http://redis.io       
　　`-._    `-._`-.__.-'_.-'    _.-'                                  
　　|`-._`-._    `-.__.-'    _.-'_.-'|                                 
　　|    `-._`-._        _.-'_.-'    |                                 
　　`-._    `-._`-.__.-'_.-'    _.-'                                  
　　`-._    `-.__.-'    _.-'                                      
　　`-._        _.-'                                          
　　`-.__.-'                                              
　　[28024] 27 May 17:19:53.392 # Sentinel runid is 9f57020ff3e36582b8d272efb09641a875534329
　　[28024] 27 May 17:19:53.392 # +monitor master mymaster 100.5.120.118 6379 quorum 2
　　[28024] 27 May 17:19:53.393 * +slave slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:19:53.393 * +slave slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:19:53.393 * +slave slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:19:53.773 * +sentinel sentinel 100.5.120.119:26379 100.5.120.119 26379 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:19:59.971 * +sentinel sentinel 100.5.120.119:26381 100.5.120.119 26381 @ mymaster 100.5.120.118 6379
　　查看sentinel状态：
　　info sentinel
　　# Sentinel
　　sentinel_masters:1
　　sentinel_tilt:0
　　sentinel_running_scripts:0
　　sentinel_scripts_queue_length:0
　　master0:name=mymaster,status=ok,address=100.5.120.118:6380,slaves=3,sentinels=3
　　
　　关闭主节点的redis，模拟failover：
　　观察日志：
　　26379:
　　[28012] 27 May 17:21:46.058 # +sdown master mymaster 100.5.120.118 6379
　　[28012] 27 May 17:21:46.102 # +new-epoch 1
　　[28012] 27 May 17:21:46.102 # +vote-for-leader 9f57020ff3e36582b8d272efb09641a875534329 1
　　[28012] 27 May 17:21:46.159 # +odown master mymaster 100.5.120.118 6379 #quorum 3/2
　　[28012] 27 May 17:21:46.159 # Next failover delay: I will not start a failover before Wed May 27 17:52:06 2015
　　[28012] 27 May 17:21:48.119 # +config-update-from sentinel 100.5.120.119:26380 100.5.120.119 26380 @ mymaster 100.5.120.118 6379
　　[28012] 27 May 17:21:48.119 # +switch-master mymaster 100.5.120.118 6379 100.5.120.118 6380
　　[28012] 27 May 17:21:48.119 * +slave slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6380
　　[28012] 27 May 17:21:48.120 * +slave slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6380
　　[28012] 27 May 17:21:48.121 * +slave slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[28012] 27 May 17:21:53.145 # +sdown slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[28012] 27 May 17:24:40.046 # -sdown slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　26380:
　　[28024] 27 May 17:21:46.045 # +sdown master mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:46.100 # +odown master mymaster 100.5.120.118 6379 #quorum 2/2
　　[28024] 27 May 17:21:46.100 # +new-epoch 1
　　[28024] 27 May 17:21:46.100 # +try-failover master mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:46.101 # +vote-for-leader 9f57020ff3e36582b8d272efb09641a875534329 1
　　[28024] 27 May 17:21:46.102 # 100.5.120.119:26379 voted for 9f57020ff3e36582b8d272efb09641a875534329 1
　　[28024] 27 May 17:21:46.103 # 100.5.120.119:26381 voted for 9f57020ff3e36582b8d272efb09641a875534329 1
　　[28024] 27 May 17:21:46.163 # +elected-leader master mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:46.163 # +failover-state-select-slave master mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:46.264 # +selected-slave slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:46.264 * +failover-state-send-slaveof-noone slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:46.322 * +failover-state-wait-promotion slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:47.136 # +promoted-slave slave 100.5.120.118:6380 100.5.120.118 6380 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:47.136 # +failover-state-reconf-slaves master mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:47.217 * +slave-reconf-sent slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:47.217 * +slave-reconf-sent slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:48.186 * +slave-reconf-inprog slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:48.186 * +slave-reconf-done slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:48.186 * +slave-reconf-inprog slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:48.186 * +slave-reconf-done slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:48.251 # -odown master mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:48.251 # +failover-end master mymaster 100.5.120.118 6379
　　[28024] 27 May 17:21:48.251 # +switch-master mymaster 100.5.120.118 6379 100.5.120.118 6380
　　[28024] 27 May 17:21:48.252 * +slave slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6380
　　[28024] 27 May 17:21:48.252 * +slave slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6380
　　[28024] 27 May 17:21:48.253 * +slave slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[28024] 27 May 17:21:53.302 # +sdown slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[28024] 27 May 17:24:40.012 # -sdown slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　26381:
　　[28036] 27 May 17:21:45.978 # +sdown master mymaster 100.5.120.118 6379
　　[28036] 27 May 17:21:46.102 # +new-epoch 1
　　[28036] 27 May 17:21:46.103 # +vote-for-leader 9f57020ff3e36582b8d272efb09641a875534329 1
　　[28036] 27 May 17:21:47.092 # +odown master mymaster 100.5.120.118 6379 #quorum 3/2
　　[28036] 27 May 17:21:47.092 # Next failover delay: I will not start a failover before Wed May 27 17:52:06 2015
　　[28036] 27 May 17:21:48.119 # +config-update-from sentinel 100.5.120.119:26380 100.5.120.119 26380 @ mymaster 100.5.120.118 6379
　　[28036] 27 May 17:21:48.119 # +switch-master mymaster 100.5.120.118 6379 100.5.120.118 6380
　　[28036] 27 May 17:21:48.119 * +slave slave 100.5.120.118:6381 100.5.120.118 6381 @ mymaster 100.5.120.118 6380
　　[28036] 27 May 17:21:48.120 * +slave slave 100.5.120.118:6394 100.5.120.118 6394 @ mymaster 100.5.120.118 6380
　　[28036] 27 May 17:21:48.121 * +slave slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[28036] 27 May 17:21:53.153 # +sdown slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　[28036] 27 May 17:24:40.013 # -sdown slave 100.5.120.118:6379 100.5.120.118 6379 @ mymaster 100.5.120.118 6380
　　
　　可以看到三个节点的投票过程最后 26380节点的sentinel(获得3票)完成failover.
　　关闭26379的sentinel：
　　26380和26381输出：
　　[28024] 27 May 17:35:12.689 # +sdown sentinel 100.5.120.119:26379 100.5.120.119 26379 @ mymaster 100.5.120.118 6380
　　观察info输出：
　　# Sentinel
　　sentinel_masters:1
　　sentinel_tilt:0
　　sentinel_running_scripts:0
　　sentinel_scripts_queue_length:0
　　master0:name=mymaster,status=ok,address=100.5.120.118:6380,slaves=3,sentinels=3
　　没有自动剔除节点（bug还是功能缺失？）
　　测试2节点状态下的的failover，正常完成。
　　修改26379的端口为26382，然后启动sentinel：
　　正常加入集群，查看集群状态：
　　# Sentinel
　　sentinel_masters:1
　　sentinel_tilt:0
　　sentinel_running_scripts:0
　　sentinel_scripts_queue_length:0
　　master0:name=mymaster,status=ok,address=100.5.120.118:6379,slaves=3,sentinels=4
　　节点变成4个了..
　　改回到26379启动，节点还是4.
　　结论就是不同端口号启动的sentinel会累计到sentinels中。
　　
　　总结：
　　redis-sentinel提供了 redis主从的自动failover切换，节点的重启的自动加入，并且提供了自身的集群配置，
　　配置起来也很简单方便。结合vip的话，可以在简单的生产环境中推广使用.