SaltStack实例：Keepalived+Redis高可用架构

xiuxuan

　　    Keepalived+Redis是一种比较成熟的Redis高可用解决方案，通过keepalived实现虚拟ip（vip）的切换以及Reids进程监控，并控制Redis的主从同步状态。以下是架构图：
　　
　　环境介绍：
　　Saltstack-master: 192.168.39.185

　　Master：192.168.39.100
　　Slave：192.168.39.101
　　VIP：192.168.39.23
　　高可用状态情况：
　　1）当Master端与Slave端都运行正常时，Master负责服务，Slave作为备用。
　　2）当Master端运行不正常时，slave接管服务，同时关闭主从复制功能。
　　3）当Master端恢复正常时，则从slave同步数据，作为Slave的从，不切换成主。
　　4）当slave端挂掉，master端恢复成主。
　　以下通过saltstack完成keepalived+redis的高可用架构部署：
　　首先，修改master配置文件指定top file文件路径：

[root@centos7 ~]# vim /etc/salt/master
file_roots:
  base:
    - /srv/salt修改salt配置文件需要重启服务。
整个状态文件目录结构如下：

设置主从模式的Grains(Grains为模版文件设置变量，通常也可以用Pillar设置)：
设置master端和slave端的Grains，指定role、vip和redis的master_ip:

[root@centos7 ~]# salt "centos7-1" grains.setvals "{'role':'master','vip':'192.168.39.23','master_ip':'192.168.39.101'}"
[root@centos7 ~]# salt "centos7-2" grains.setvals "{'role':'slave','vip':'192.168.39.23','master_ip':'192.168.39.100'}"可以查看到Grains已经设置成功：

设置入口文件top.sls：
以list匹配方式对两台主机应用对应模块：
[root@centos7 salt]# cat top.sls
base:
  'centos7-1,centos7-2':
    - match: list
    - redis
    - keepalived一、部署Redis
安装redis并根据角色下发配置模版文件。
Redis安装入口文件：
[root@centos7 redis]# cat init.sls
include:
  - .install_redis
  - .redis_running
  - .redis_confRedis安装：
[root@centos7 redis]# cat install_redis.sls
{% for s in ["jemalloc-3.6.0-1.el7.x86_64.rpm","redis-3.2.12-1.el7.x86_64.rpm",] %}
copy_{{ s }}:
  file.managed:
    - name: /usr/local/src/{{ s }}
    - source: salt://redis/file/{{ s }}
    - user: root
    - group: root
    - template: jinja
    - mode: 755
{% endfor %}
install_redis:
  cmd.run:
    - name: cd /usr/local/src/ && yum install -y redis-3.2.12-1.el7.x86_64.rpm jemalloc-3.6.0-1.el7.x86_64.rpm
    - require:
      - file: copy_redis-3.2.12-1.el7.x86_64.rpm这里通过require指定依赖关系，将rpm都下发到节点下之后，进行安装redis。
Redis配置洗发以及创建目录：
[root@centos7 redis]# cat redis_conf.sls
redis_conf:
  file.managed:
    - name: /etc/redis.conf
    - source: salt://redis/templates/redis.j2
    - user: root
    - group: root
    - template: jinja
    - mode: 644
    - require:
      - cmd: install_redis
redis_dir:
  cmd.run:
    - name: mkdir -pv /data/redis && chown redis.redis /data/redis
    - unless: test -d /data/redis　　unless指定通过test命令判断目录是否存在，如果不存在则执行name。
　　Redis服务启动控制：
[root@centos7 redis]# cat redis_running.sls
redis_running:
  service.running:
    - name: redis
    - enable: True
    - require:
      - cmd: install_redis
    - watch:
      - file: redis_conf通过watch状态监控，如果就重启服务。
Redis配置文件模版：
[root@centos7 templates]# grep -v "#" redis.j2
bind 0.0.0.0
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
supervised no
pidfile /var/run/redis/redis.pid
loglevel notice
logfile /var/log/redis/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data/redis/
{% if grains['role'] == "slave" %}
slaveof {{grains['master_ip']}} 6379
{% endif %}
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes　　二、部署keepalived
　　keepalived入口文件：
[root@centos7 keepalived]# cat init.sls
include:
  - .install_keepalived
  - .keepalived_running
  - .keepalived_conf　　keepalived安装文件：
[root@centos7 keepalived]# cat install_keepalived.sls
install_keepalived:
  pkg.installed:
    - name: keepalivedpkg指定通过yum源进行安装。

　　keepalived配置下发以及创建目录：
[root@centos7 keepalived]# cat keepalived_conf.sls
keepalived_conf:
  file.managed:
    - name: /etc/keepalived/keepalived.conf
    - source: salt://keepalived/templates/keepalived.conf
    - user: root
    - group: root
    - template: jinja
    - mode: 644
    - require:
      - pkg: install_keepalived
      - cmd: scripts_dir
scripts_dir:
  cmd.run:
    - name: mkdir -pv /etc/keepalived/scripts/
    - unless: test -d /etc/keepalived/scripts/
{% for s in ["redis_backup.sh","redis_check.sh","redis_fault.sh","redis_master.sh","redis_stop.sh"] %}
keepalived_{{s}}:
  file.managed:
    - name: /etc/keepalived/scripts/{{s}}
    - source: salt://keepalived/templates/scripts/{{s}}
    - user: root
    - group: root
    - template: jinja
    - mode: 755
    - require:
      - pkg: install_keepalived
{% endfor %}keepalived服务启动文件：

[root@centos7 keepalived]# cat keepalived_running.sls
keepalived_running:
  service.running:
    - name: keepalived
    - enable: True
    - require:
      - pkg: install_keepalived
    - watch:
      - file: keepalived_confkeepalived模板文件和脚本文件：

keepalived.conf主配置文件根据role值动态下发配置文件：
[root@centos7 templates]# cat keepalived.conf
! Configuration File for keepalived
vrrp_script chk_redis {
    script "/etc/keepalived/scripts/redis_check.sh"
    interval 2
}
vrrp_instance VI_1 {
{% if grains['role'] == 'master' %}
    state MASTER
{% else %}
    state BACKUP
{% endif %}
    interface eth0
    virtual_router_id 51
{% if grains['role'] == "master" %}
    priority 101
{% else %}
    priority 100
{% endif %}
#    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass redis
    }
    track_script {
        chk_redis
    }
    virtual_ipaddress {
        {{ grains['vip'] }}
    }
    notify_master /etc/keepalived/scripts/redis_master.sh
    notify_backup /etc/keepalived/scripts/redis_backup.sh
    notify_fault /etc/keepalived/scripts/redis_fault.sh
    notify_stop /etc/keepalived/scripts/redis_stop.sh
}　　keepalived配置的几个脚本，当keeplaived在转换状态时会按照状态来呼叫：
　　当进入master状态时会呼叫notify_master；
　　当进入backup状态时会呼叫notify_backup；
　　当发现异常情况时进入fault状态呼叫notify_fault；
　　当keepalived程序终止时则呼叫notify_stop。
　　redis_backup.sh-Redis健康检测脚本：
[root@centos7 scripts]# cat redis_check.sh
#!/bin/bash
#
ALIVE=`/usr/bin/redis-cli PING`
if [ "$ALIVE" == "PONG" ];then
    echo $ALIVE
    exit 0
else
    echo $ALIVE
    exit 1
firedis_master.sh-切换master触发执行脚本：

[root@centos7 scripts]# cat redis_master.sh
#!/bin/bash
#
REDISCLI="/usr/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master..." >> $LOGFILE 2>&1
echo "run slaveof no one cmd..." >> $LOGFILE
$RESISCLI SLAVEOF {{ grains['master_ip']  }} 6479>> $LOGFILE 2>&1
$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1redis_backup.sh-切换成backup角色执行脚本：

[root@centos7 scripts]# cat redis_backup.sh
#!/bin/bash
#
REDISCLI="/usr/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[backup]" >> $LOGFILE
date >> $LOGFILE
echo "being salve..." >> $LOGFILE 2>&1
#sleep 25
echo "run slaveof cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF {{ grains['master_ip'] }} 6379 $LOGFILE 2>&1redis_fault.sh-进入fault状态执行脚本：
[root@centos7 scripts]# cat redis_fault.sh
#!/bin/bash
#
$LOGFILE=/var/log/keepalived-redis-state.log
echo "[fault]" >> $LOGFILE
date >> $LOGFILEredis_stop.sh-当keepalived服务终止时执行此脚本：

[root@centos7 scripts]# cat redis_stop.sh
#!/bin/bash
#
LOGFILE=/var/log/keepalived-redis-state.log
echo "[stop]" >> $LOGFILE
date >> $LOGFILE所有状态配置文件和脚本设置完成执行：
　　在执行时我们可以通过指定参数，模拟执行：
　　
[root@centos7 salt]# salt 'centos7-[1-2]' state.highstate test=True　　测试执行没有其他问题，执行安装：


[root@centos7 salt]# salt 'centos7-[1-2]' state.highstate执行结构如下：

会看到centos7-1节点，执行完成之后的状态：

　　测试keepalived+redis高可用：
　　登陆centos7-1查看vip情况:
　　在saltstack服务端通过连接vip测试登陆情况：
　　
　　关闭centos7-1下的redis服务：


[root@centos7-1:/usr/local/src]
# service redis stop
Redirecting to /bin/systemctl stop  redis.service
[root@centos7-1:/usr/local/src]
# service redis status
Redirecting to /bin/systemctl status  redis.service
● redis.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/redis.service.d
           └─limit.conf
   Active: inactive (dead) since Fri 2018-08-31 11:41:44 CST; 5s ago
  Process: 6414 ExecStop=/usr/libexec/redis-shutdown (code=exited, status=0/SUCCESS)
  Process: 6147 ExecStart=/usr/bin/redis-server /etc/redis.conf --supervised systemd (code=exited, status=0/SUCCESS)
Main PID: 6147 (code=exited, status=0/SUCCESS)
Aug 31 11:40:17 centos7-1 systemd[1]: Starting Redis persistent key-value database...
Aug 31 11:40:17 centos7-1 systemd[1]: Started Redis persistent key-value database.
Aug 31 11:41:44 centos7-1 systemd[1]: Stopping Redis persistent key-value database...
Aug 31 11:41:44 centos7-1 systemd[1]: Stopped Redis persistent key-value database.重新连接vip登陆redis查看设置name值：


　　登陆centos7-2下查看vip切换情况:
　　参考文献:《saltstack运维实战》