Tengine+keepalived 安装配置

ewe22

测试环境

系统：Centos-6.5-x86_64

主IP：192.168.1.220

备IP:192.168.219

VIP:192.168.1.226

一、安装Tengine

1.1创建目录并下载安装包

mkdir -p /data/ops/{app,packages,scripts}

cd /data/ops/packages/

wget http://tengine.taobao.org/download/tengine-2.1.0.tar.gz

1.2 安装tengine

tar zxf tengine-2.0.3.tar.gz && cd tengine-2.0.3/ && ./configure --prefix=/data/ops/app/tengine-2.0.3/ --with-http_stub_status_module --with-pcre --with-http_upstream_check_module && make && make install

/data/ops/app/tengine-2.0.3/sbin/nginx -t

部分模块说明：

--with-http_upstream_check_module

该模块可以为Tengine提供主动式后端服务器健康检查的功能。

该模块在Tengine-1.4.0版本以前没有默认开启，它可以在配置编译选项的时候开启：./configure --with-http_upstream_check_module

--with-pcre

设置PCRE库的源码路径

1.3 将tengine加入rc.local以便开机自启动

echo "/data/ops/app/tengine-2.0.3/sbin/nginx" >> /etc/rc.local

二、keepalived安装

2.1 创建目录并下载安装包

mkdir -p /data/ops/{packages,app} && cd /data/ops/packages

wget http://www.keepalived.org/software/keepalived-1.2.12.tar.gz

2.2 解压并安装keepalived

tar zxf keepalived-1.2.12.tar.gz && cd keepalived-1.2.12 && ./configure --prefix=/data/ops/app/keepalived && make && make install

2.3 配置keepalived配置文件路径，启动路径

拷贝启动文件到/etc/init.d目录下

cp -p /data/ops/app/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/

cp -p /data/ops/app/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

创建配置文件存放目录

mkdir /etc/keepalived

cp -p /data/ops/app/keepalived/sbin/keepalived /usr/sbin/

配置开启自启动级别

chkconfig --level 2345 keepalived on

三、keepalived配置

说明：备keepalived的配置文件只需去掉nopreempt，修改优先级（priority）的值改为：100(没有设置nopreempt的keepalived优先级必须低于设置nopreempt的keepalived，不然设置非抢占不生效)，修改router_id Nginx_Id_1为router_id Nginx_Id_2

3.1 添加主配置文件

cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

# written by :ops

global_defs {#全局配置

router_id Nginx_Id_1#运行 keepalive 机器的标示，注意每个机器需要唯一的标示

}

vrrp_script chk_nginx {

script "/data/ops/scripts/check_http/check_http.sh"#定义检测脚本

interval 5#检测间隔

weight 2

}

vrrp_instance Nginx1 {

state BACKUP#定义为backup

Nopreempt#启用非抢占模式

interface em1#实例绑定的网卡

virtual_router_id 51# 0-255 在同一个 instance 中一致 在整个 vrrp 中唯一

priority 150#优先级，优先级最大的会成为 master

authentication {#认证

auth_type PASS#认证的方式，支持PASS和AH

auth_pass hiveview#认证的密码

}

track_script {

chk_nginx

}

virtual_ipaddress {#指定漂移地址

192.168.0.226

}

}

注意：nopreempt就是在优先级高的主机回复后不抢占VIP，这样做为了减小keepalived主从切换带来的意外风险

3.2 Tengine健康检测脚本
cat /data/ops/scripts/check_http/check_http.sh
#!/bin/bash

DATE=`date +"%Y-%m-%d %H:%M:%S"`
NginxNum=`ss -lnt|grep -w :80|wc -l`
printf "+++$DATE\t`ss -lnt|grep -w :80`+++\n" >> /data/ops/scripts/check_http/check.log

if [ $NginxNum -eq 0 ]
then
        printf "+++THE START\t$DATE\t`ss -lnt|grep -w :80`+++\n" >> /data/ops/scripts/check_http/check.log
        /data/ops/app/tengine-2.1.0/sbin/nginx
        printf "+++THE END\t$DATE\t`ss -lnt|grep -w :80`+++\n" >> /data/ops/scripts/check_http/check.log
        sleep 2

        if [ `ss -lnt|grep -w :80|wc -l` -eq 0 ]
        then
                printf "+++WANT TO KILL KEEPALIVED\t$DATE\t`ss -lnt|grep -w :80`+++\n" >> /data/ops/scripts/check_http/check.log
                killall keepalived
        fi
Fi

四、故障切换与检测
4.1 故障转移
    从主（192.148.1.220）的角度来说，Keepalived会每5分钟运行一次check_http.sh脚本，当检测到nginx停掉后，尝试启动nginx，如果启动成功，VIP不发生变化，keepalived正常运行。如果启动失败，则停掉keepalived程序并将VIP切换到备的keealived服务器上。当主恢复正常后，因为配置了nopreempt非抢占模式，所以VIP还在备（192.148.1.219）上。

4.2 故障演示
主（192.148.1.220）上操作：
/etc/init.d/keepalived stop 停掉keepalived
4.1 在备上tailf /var/log/messages查看备机成了master，并获得了VIP（见红色字体部分）
May  3 03:17:41 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) Transition to MASTER STATE
May  3 03:17:42 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) Entering MASTER STATE
May  3 03:17:42 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) setting protocol VIPs.
May  3 03:17:42 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) Sending gratuitous ARPs on em1 for 192.148.1.226
May  3 03:17:42 OTT-AP-005 Keepalived_healthcheckers[17098]: Netlink reflector reports IP 192.148.1.226 added
May  3 03:17:47 OTT-AP-005 Keepalived_vrrp[17099]: VRRP_Instance(Nginx1) Sending gratuitous ARPs on em1 for 192.148.1.226

4.2 启动主的nginx在启动主的keepalived（见红色字体部分）
May  3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: Opening file '/etc/keepalived/keepalived.conf'.
May  3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: Configuration is using : 64313 Bytes
May  3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: Using LinkWatch kernel netlink reflector...
May  3 02:49:09 OTT-AP-004 Keepalived_healthcheckers[10199]: Opening file '/etc/keepalived/keepalived.conf'.
May  3 02:49:09 OTT-AP-004 Keepalived_healthcheckers[10199]: Configuration is using : 6313 Bytes
May  3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: VRRP_Instance(Nginx1) Entering BACKUP STATE
May  3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May  3 02:49:09 OTT-AP-004 Keepalived_healthcheckers[10199]: Using LinkWatch kernel netlink reflector...
May  3 02:49:09 OTT-AP-004 Keepalived_vrrp[10200]: VRRP_Script(chk_nginx) succeeded

通过上述可以看到，VIP成功漂移到备机，主机keepalived重新启动后，是备状态也没有抢占成主，说明非抢占也成功生效

4.3检测VIP状态变化
check_vip.sh脚本检测VIP是否在当前主机上记录当前VIP状态，并与上一次状态进行比较，如果两次状态不一样，则报警通知
#主上的脚本
#!/bin/bash
###check VIP

MasterServer="92.168.1.220"
SlaveServer="192.168.1.219"
smtpemailfrom="monitor@gongsiyouxiang.com"
emailto="monitor-roger@gongsiyouxiang.com"
subject="API VIP changed"
smtpserver=smtp.qiye.163.com
smtplogin=monitor@gongsiyouxiang.com
smtppass=jiankong123

if [ "`cat /data/ops/scripts/check_vip/status`" = "yes" ]
then
        Current_Date=`date +"%Y-%m-%d %H:%M:%S"`
        body="The API VIP drift from the MASTER SERVER:$MasterServer to SLAVE SERVER:$SlaveServer at $Current_Date"
        bodytwo="The API VIP drift from the SLAVE SERVER:$SlaveServer to MASTER SERVER:$MasterServer at $Current_Date"
        vip=`/sbin/ip addr|grep -o "124.192.140.226/32"`
        if [ ! -z "$vip" ]
        then
                /usr/bin/sendEmail -f $smtpemailfrom -t $emailto -u $subject -m $body -s $smtpserver:25 -xu $smtplogin -xp $smtppass >> /data/ops/scripts/check_vip/check_vip.log 2>&1

                echo "no" > /data/ops/scripts/check_vip/status

        fi
else

        Current_Date=`date +"%Y-%m-%d %H:%M:%S"`
        body="The API VIP drift from the MASTER SERVER:$MasterServer to SLAVE SERVER:$SlaveServer at $Current_Date"
        bodytwo="The API VIP drift from the SLAVE SERVER:$SlaveServer to MASTER SERVER:$MasterServer at $Current_Date"
        vip=`/sbin/ip addr|grep -o "124.192.140.226/32"`
        if [ -z "$vip" ]
        then
        /usr/bin/sendEmail -f $smtpemailfrom -t $emailto -u $subject -m $bodytwo -s $smtpserver:25 -xu $smtplogin -xp $smtppass >> /data/ops/scripts/check_vip/check_vip.log 2>&1
        echo "yes" > /data/ops/scripts/check_vip/status

        fi

fi

4.4 查看VIP 是否在当前主机上，使用ip add命令
#ip add
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether f0:1f:af:e5:ba:c5 brd ff:ff:ff:ff:ff:ff
    inet 124.192.140.220/26 brd 124.192.140.255 scope global em1
    inet 124.192.140.226/32 scope global em1
    inet6 fe80::f21f:afff:fee5:bac5/64 scope link
       valid_lft forever preferred_lft forever

五、无健康检测实例

说明：我们现在是一个VIP多种业务，既是nginx高可用也是redis高可用，但是我们不想因为一个业务有问题所有的都切换

5.1 主配置文件

# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

# written by :ops

global_defs {

router_id Redis_Id_3

}

vrrp_instance Tengine1 {

state BACKUP

nopreempt

interface em1

virtual_router_id 54

priority 150

authentication {

auth_type PASS

auth_pass hiveview

}

virtual_ipaddress {

192.168.1..252

}

}

狮子XL

五、无健康检测实例
...
请问第五部分是不是没写完呀. 怎么戛然而止了?