[转] keepalived+nginx 高可用性+负载均衡

wind-cold

　　nginx+keepalived高可用性负载均衡
　　一、前言
　　最近因业务扩展，需要将当前的apache 转为nginx(web), 再在web前端放置nginx(负载均衡)。同时结合keepalived 对前端nginx实现HA。
　　nginx 进程基于于Master+Slave(worker)多进程模型，自身具有非常稳定的子进程管理功能。在Master进程分配模式下，Master进程永 远不进行业务处理，只是进行任务分发，从而达到Master进程的存活高可靠性，Slave(worker)进程所有的业务信号都 由主进程发出，Slave(worker)进程所有的超时任务都会被Master中止，属于非阻塞式任务模型。
　　Keepalived是Linux下面实现VRRP 备份路由的高可靠性运行件。基于Keepalived设计的服务模式能够真正做到主服务器和备份服务器故障时IP瞬间无缝交接。二者结合，可以构架出比较稳定的软件lb方案。
　　
二、nginx+keepalived架构实战
　　采用两台服务器做nginx主备，后端采用两个realserver（可以随意扩展）。
　　1、 结构如下：
　　nginx master 172.31.1.100
nginx backup 172.31.1.135
vip 172.31.1.110
　　
　　


　　2、 安装配置
　　(1)、安装keepalived（在nginx的mater和backup都安装）
　　wgethttp://www.keepalived.org/software/keepalived-1.1.19.tar.gz
tar zxvf keepalived-1.1.19.tar.gz
cd keepalived-1.1.19
./configure --prefix=/usr/local/keepalived
make
make install
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
mkdir /etc/keepalived
cd /etc/keepalived/
　　
　　
配置nginx master的keepalived配置文件
　　[iyunv@zhang1 keepalived]# cat /etc/keepalived/keepalived.conf
############################################################
　　
vrrp_script      chk_http_port {
script "/etc/keepalived/nginx_pid.sh"         /* 监控脚本 */
interval 2                                                     /* 监控时间间隔 */
weight 2
}
global_defs {
         notification_email {
              
         }
         notification_email_from
         smtp_server 172.31.1.101
         smtp_connect_timeout 30
         router_id LVS_Twioo
}
　　vrrp_instance VI_1 {
         state MASTER                         /* 设置为主 */
         interface eth0                             /* 监控网卡 */
         virtual_router_id 51                    /* 这个两台服务器必须一样 */
         mcast_src_ip 172.31.1.100     /* 多播的源IP，设置为本机IP，与VIP同一网卡*/
         priority 100                                  /* 权重值 MASTRE 一定要高于 BAUCKUP */
         advert_int 1                                  /* VRRP Multicast广播周期秒数*/
         smtp_alert
　　authentication {
                 auth_type    PASS             /* 加密 */
                 auth_pass 123456           /* 加密的密码，两台服务器一定要一样，不然会出错 */
         }
　　track_script {
    chk_http_port                                 /* 执行监控的服务 */
}
　　virtual_ipaddress {
                 172.31.1.110
         }
}
############################################################
　　
　　
　　
　　
配置nginx backup的keepalived配置文件
　　[iyunv@zhang1 keepalived]# cat /etc/keepalived/keepalived.conf
############################################################
　　
vrrp_script        chk_http_port {
script "/etc/keepalived/nginx_pid.sh"      
interval 2                                    
weight 2
}
　　global_defs {
         notification_email {
               
         }
         notification_email_from
         smtp_server 172.31.1.101
         smtp_connect_timeout 30
         router_id LVS_Twioo
}
vrrp_instance VI_1 {
         state BACKUP
         interface eth0
         virtual_router_id 51
mcast_src_ip 172.31.1.100
         priority 80
         advert_int 1
         smtp_alert
　　authentication {
                 auth_type PASS
                 auth_pass 123456
         }
　　track_script {
    chk_http_port         
}
　　virtual_ipaddress {
                 172.31.1.110
         }
　　}
　　
　　
启动keepalived，查看虚拟IP是否绑定！
　　[iyunv@zhang1 ~]# /etc/rc.d/init.d/keepalived start
Starting keepalived:                                        [ OK ]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
     inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether 00:26:55:1a:e5:28 brd ff:ff:ff:ff:ff:ff
     inet 172.31.1.100/24 brd 172.31.1.255 scope global eth0
     inet 172.31.1.110/32 scope global eth0
     inet6 fe80::226:55ff:fe1a:e528/64 scope link
        valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether 00:26:55:1a:e5:2a brd ff:ff:ff:ff:ff:ff
4: sit0: <NOARP> mtu 1480 qdisc noop
     link/sit 0.0.0.0 brd 0.0.0.0
　　可以看到多了一个虚拟IP
　　
查看master keepalived日志 tailf /var/log/message
　　Jul 23 14:50:38 nginx Keepalived_vrrp: Registering Kernel netlink command channel
Jul 23 14:50:38 nginx Keepalived_vrrp: Registering gratutious ARP shared channel
Jul 23 14:50:38 nginx Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Jul 23 14:50:38 nginx Keepalived_vrrp: Configuration is using : 38512 Bytes
Jul 23 14:50:38 nginx Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Jul 23 14:50:38 nginx Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]
Jul 23 14:50:39 nginx Keepalived_vrrp: VRRP_Instance(VI_1)Transition to MASTER STATE
Jul 23 14:50:40 nginx Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Jul 23 14:50:40 nginx Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Jul 23 14:50:40 nginx Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.31.1.110
Jul 23 14:50:40 nginx Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:50:40 nginx Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:50:40 nginx Keepalived_vrrp: SMTP alert successfully sent.
　　可以看出状态为MASTER
　　
此时在BACKUP机上启动keepalived，再查看日志
　　Jul 23 14:58:30 SvrTest Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Jul 23 14:58:30 SvrTest Keepalived_vrrp: Configuration is using : 65052 Bytes
Jul 23 14:58:30 SvrTest Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Jul 23 14:58:30 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Jul 23 14:58:30 SvrTest Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]
Jul 23 14:58:30 SvrTest Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:58:30 SvrTest Keepalived_vrrp: SMTP alert successfully sent.
Jul 23 14:58:30 SvrTest Keepalived_vrrp: SMTP alert successfully sent.
　　
此时如果停止MASTER上的keepalived ,那么在BACKUP的日志会出现
　　Jul 23 14:59:45 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1)Transition to MASTER STATE
Jul 23 14:59:46 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1)Entering MASTER STATE
Jul 23 14:59:46 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Jul 23 14:59:46 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.31.1.110
Jul 23 14:59:46 SvrTest avahi-daemon[3160]: Registering new address record for 172.31.1.110 on eth0.
Jul 23 14:59:46 SvrTest Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:59:46 SvrTest Keepalived_vrrp: SMTP alert successfully sent.
　　此时BACKUP接管了VIP
　　
　　若再启动MASTER上的keepalived,BACKUP上的日志会出现：
　　Jul 23 14:56:37 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Jul 23 14:56:37 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Jul 23 14:56:37 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Jul 23 14:56:37 SvrTest Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:56:37 SvrTest avahi-daemon[3160]: Withdrawing address record for 172.31.1.110 on eth0.
Jul 23 14:56:37 SvrTest Keepalived_vrrp: SMTP alert successfully sent.
　　
则是说明有更高优先级的机器即MASTER启动，去除VIP，还给更优先级的机器
　　这样就实现了如果某一主机宕机，IP自动切换到备机上继续运行
　　
　　###################################    nginx_pid.sh    脚本
　　[iyunv@SvrTest keepalived]# cat /etc/keepalived/nginx_pid.sh
　　#!/bin/bash
　　NGINX_PROCESS=`ps -C nginx --no-header | wc -l`
　　if [ $NGINX_PROCESS -eq 0 ];then
　　/usr/local/nginx/sbin/nginx
　　sleep 3
         if [ `ps -C nginx --no-header | wc -l` -eq 0 ];then
　　/etc/init.d/keepalived stop
　　fi  
　　fi
　　
　　说明：
　　当主nginx 宕机后，会自动切换到 从nginx上，这个时候可以配合以个报警的脚不来通告运维人员
　　当主修复好后，需要手动启动keepalived 程序，以便将服务重新切换回主服务器。
　　报警脚步我抽个时间补上。