nginx+keepalievd，实现负载均衡和故障点切换。keepalived双机热备。

阿尔海南粉

　　接上文  学习日记4：nginx负载均衡（二）2012.01.08

　　......
　　到此，整体工作还差一个备用nginx proxy，其中涉及到keepalived，做心跳。

　　　　　
　　

　　黄线以上做完了。
　　
　　
　　本次主要做的是1加一台从nginx_proxy ，暂且命令为front2（为从）。之前的nginx_proxy命令为front（为主）
　　
　　1 在主nginx上配置keepalived。



cd /usr/local/src/
wget http://keepalived.org/software/keepalived-1.2.7.tar.gz
tar -xzvf keepalived-1.2.7.tar.gz
cd keepalived-1.2.7/
./configure --prefix=/usr/local/keepalived
　　这时候会报错的！



configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files.            !!!
　　缺少gcc，安装下面扩展



yum install gcc gcc-c++ autoconf automake
　　继续编译，下面还会报错，类似这样的错误。
　　



make: *** No targets specified and no makefile found.  Stop.


configure: error: ! OpenSSL is not properly installed on your system. configure: error: !
　　
　　对此，我们安装以下扩展即可解决。



yum -y install openssl-devel


yum -y install popt-devel
　　
　　在编译



./configure --prefix=/usr/local/keepalived
　　
　　这时候，会出现下面的结果，这个结果就表明是ok的。到此你的keepalived安装是成功的！





Keepalived configuration
------------------------
Keepalived version       : 1.2.7
Compiler                 : gcc
Compiler flags           : -g -O2
Extra Lib                : -lpopt -lssl -lcrypto
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
IPVS use libnl           : No
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
SNMP support             : No
Use Debug flags          : No

　　
　　继续下面的操作
　　





make
make install
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
mkdir /etc/keepalived
cd /etc/keepalived
vim keepalived.conf

　　
　　　
　　keepalived.conf配置文件比较重要
　　front1 ：主nginx的配置文件如下：





! Configuration File for keepalived
global_defs {
router_id LVS_01
}
vrrp_instance VI_1 {
state MASTER  #master 主nginx
interface eth1
virtual_router_id 51
mcast_src_ip 192.168.65.133 #本机ip地址也即主nginx地址
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.65.128 #vip 虚拟出来的ip地址 要求在内网中未被使用
}
}

　　
　　到此，主nginx配置完成。同样的操作，在从nginx上也配置一遍。
　　不过从nginx的配置文件如下：





[iyunv@localhost nginx]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_01
}
vrrp_instance VI_1 {
state BACKUP #slave
interface eth2
virtual_router_id 51
mcast_src_ip 192.168.65.137 #slave ip
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.65.128 #vip
}
}

　　
　　重启两边的keepalived服务：



[iyunv@localhost conf.d]# service keepalived restart
Stopping keepalived:                                       [FAILED]
Starting keepalived:                                       [  OK  ]
　　主nginx上，用





[iyunv@localhost conf.d]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:20:e9:20 brd ff:ff:ff:ff:ff:ff
inet 192.168.65.133/24 brd 192.168.65.255 scope global eth1
inet 192.168.65.128/32 scope global eth1
inet6 fe80::20c:29ff:fe20:e920/64 scope link
valid_lft forever preferred_lft forever

　　到此，我们的vip算是真正的绑定上了。
　　查看keepalived的日志：





root@localhost conf.d]# tail -f /var/log/messages
Jan  9 18:53:06 localhost Keepalived_healthcheckers[3355]: Using LinkWatch kernel netlink reflector...
Jan  9 18:53:06 localhost Keepalived_vrrp[3356]: Configuration is using : 34824 Bytes
Jan  9 18:53:06 localhost Keepalived_vrrp[3356]: Using LinkWatch kernel netlink reflector...
Jan  9 18:53:06 localhost Keepalived_vrrp[3356]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Jan  9 18:53:06 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan  9 18:53:07 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan  9 18:53:07 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan  9 18:53:07 localhost avahi-daemon[1765]: Registering new address record for 192.168.65.128 on eth1.IPv4.
Jan  9 18:53:07 localhost Keepalived_healthcheckers[3355]: Netlink reflector reports IP 192.168.65.128 added
Jan  9 18:53:07 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.65.128
Jan  9 18:53:12 localhost Keepalived_vrrp[3356]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.65.128

　　从nginx上，同样





[iyunv@localhost nginx]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:a7:81:c7 brd ff:ff:ff:ff:ff:ff
inet 192.168.65.137/24 brd 192.168.65.255 scope global eth2
inet6 fe80::20c:29ff:fea7:81c7/64 scope link
valid_lft forever preferred_lft forever

　　我们发现并没有绑定vip 128
　　同时；





[iyunv@localhost nginx]# tail -f /var/log/messages
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Registering Kernel netlink reflector
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Registering Kernel netlink command channel
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Configuration is using : 34822 Bytes
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Configuration is using : 4821 Bytes
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Using LinkWatch kernel netlink reflector...
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Using LinkWatch kernel netlink reflector...

　　
　　貌似从nginx似乎出了什么问题，先不着急。
　　这时候我们停止掉主nginx的服务，再看看效果怎么样！



killall nginx
　　再观察从nginx的状态变化





[iyunv@localhost nginx]# tail -f /var/log/messages
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Registering Kernel netlink reflector
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Registering Kernel netlink command channel
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Opening file '/etc/keepalived/keepalived.conf'.
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Configuration is using : 34822 Bytes
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Configuration is using : 4821 Bytes
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: Using LinkWatch kernel netlink reflector...
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan  9 19:16:25 localhost Keepalived_vrrp[22421]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Jan  9 19:16:25 localhost Keepalived_healthcheckers[22420]: Using LinkWatch kernel netlink reflector...
Jan  9 19:17:33 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan  9 19:17:34 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan  9 19:17:34 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan  9 19:17:34 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.65.128
Jan  9 19:17:34 localhost avahi-daemon[1810]: Registering new address record for 192.168.65.128 on eth2.IPv4.
Jan  9 19:17:34 localhost Keepalived_healthcheckers[22420]: Netlink reflector reports IP 192.168.65.128 added
Jan  9 19:17:39 localhost Keepalived_vrrp[22421]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.65.128

　　从第33秒后发现，备用nginx已经转为master状态。
　　再看





[iyunv@localhost nginx]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:a7:81:c7 brd ff:ff:ff:ff:ff:ff
inet 192.168.65.137/24 brd 192.168.65.255 scope global eth2
inet 192.168.65.128/32 scope global eth2
inet6 fe80::20c:29ff:fea7:81c7/64 scope link
valid_lft forever preferred_lft forever

　　发现没有？vip飘过来了！
　　

　　针对Keepalived的不足,用check_nginx.sh来监控nginx进程，实现真正意义上的负载均衡高可用.
此脚本思路其实也很简单，即放置在后台一直监控nginx进程；如进程消失，尝试重启nginx，
如是失败则立即停掉本机的keepalived服务，让另一台负载均衡器接手.

vi /shell/nginx_pid.sh
###################################################
#!/bin/bash
while :
do
nginxpid=`ps -C nginx --no-header | wc -l`
if [ $nginxpid -eq 0 ]
then
/usr/local/nginx-0.8.53/sbin/nginx
sleep 5
if [ $nginxpid -eq 0 ]
then
/etc/init.d/keepalived stop
fi
fi
sleep 5
done
###################################################

　　
　　
　　我们仍然可以继续访问我们的站点。至此。故障点切换宣告结束。