keepalived实现haproxy高可用详解

stbyruby

　　一，keepalived介绍
　　

　　keepalived是一个可以实现某些资源高可用的开源软件，其主要的组件包括core，check，vrrp，libipfwc，libipvs，这里说下各个组件的功能。

　　core：keepalived的核心组件，负责主进程的启动和维护以及加载解析配置文件等。

　　check：负责healthchecker,负责各种健康检查方式，和对应的配置解析以及LVS的配置解析。

　　vrrp：vrrpd的子进程。

　　libipfwc：结合iptables的ipchains库来使用。

　　libipvs：结合LVS使用。

　　keepalived启动后会生成3个进程，master主进程，VRRP子进程，healthchecker子进程。

　　VRRP协议是实现keepalived高可用的一个基础，下面说一下VRRP的实现原理：

　　VRRP虚拟路由(VRRP router)，VRRP是一个“选举”协议，它能够动态地将一个虚拟路由器的责任指定至同一个VRRP组中的其它路由器上，VRRP的优势：

　　冗余：可以使用多个路由器设备作为LAN客户端的默认网关，大大降低了默认网关成为单点故障的可能性；

　　
　　负载共享：允许来自LAN客户端的流量由多个路由器设备所共享；

　　
　　多VRRP组：在一个路由器物理接口上可配置多达255个VRRP组；

　　
　　多IP地址：基于接口别名在同一个物理接口上配置多个IP地址，从而支持在同一个物理接口上接入多个子网；

　　
　　抢占：在master故障时允许优先级更高的backup成为master；

　　
　　通告协议：使用IANA所指定的组播地址224.0.0.18进行VRRP通告；

　　
　　VRRP追踪：基于接口状态来改变其VRRP优先级来确定最佳的VRRP路由器成为master；

　　
　　

　　二，实验环境：

　　192.168.30.116 OS：Centos 6.4 x86_64   master.luojianlong.com
　　192.168.30.117 OS：Centos 6.4 x86_64   backup.luojianlong.com
　　keepalived版本：keepalived-1.2.7

　　

　　首先，分别在2台服务器上，安装keepalived,haproxy，由于系统版本是Centos 6.4，这2个软件已经被整合在内部了，所以使用yum来安装
[root@master ~]# yum -y install keepalived haproxy
[root@backup ~]# yum -y install keepalived haproxy　　

　　接下来，先配置master与backup服务器的优先级，virtual_router_id（同一个实例2台服务器必须相同），编辑配置文件:
　　
[root@master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
[root@backup ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
[root@master ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.30.230
}
}
[root@backup ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.30.230
}
}　　

　　global_defs：全局配置标识破；
　　notification_email：表示告警时发送的邮件地址；

　　notification_email_from：表示发送邮件的源地址；

　　smtp_server：发送邮件的smtp服务器地址；

　　router_id：机器标识；

　　vrrp_instance：定义一个vrrp实例；

　　state：state指定instance(Initial)的初始状态，就是说在配置好后，这台服务器的初始状态就是这里指定的，但这里指定的不算，还是得要通过竞选通过优先级来确定，里如果这里设置为master，但如若他的优先级不及另外一台，那么这台在发送通告时，会发送自己的优先级，另外一台发现优先级不如自己的高，那么他会就回抢占为master；

　　interface：实例绑定的网卡，因为在配置虚拟IP的时候必须是在已有的网卡上添加的；

　　
　　virtual router id：这里设置VRID，这里非常重要，相同的VRID为一个组，他将决定多播的MAC地址；

　　
　　priority 100：设置本节点的优先级，优先级高的为master；

　　
　　advert int：检查间隔，默认为1秒；

　　
　　virtual ipaddress：这里设置的就是VIP，也就是虚拟IP地址，他随着state的变化而增加删除，当state为master的时候就添加，当state为backup的时候删除，这里主要是有优先级来决定的，和state设置的值没有多大关系，这里可以设置多个IP地址；

　　
　　authentication：这里设置认证；

　　
　　auth type：认证方式，可以是PASS或AH两种认证方式；

　　
　　auth pass：认证密码；

　　
　　

　　启动俩台服务器的keepalived
[root@master ~]# service keepalived start
Starting keepalived:                                       [  OK  ]
[root@backup ~]# service keepalived start
Starting keepalived:                                       [  OK  ]
[root@master ~]# tail -f /var/log/messages
Jan 10 11:40:56 localhost Keepalived_healthcheckers[19368]: Using LinkWatch kernel netlink reflector...
Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Configuration is using : 63019 Bytes
Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Using LinkWatch kernel netlink reflector...
Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jan 10 11:40:57 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 10 11:40:58 localhost Keepalived_healthcheckers[19368]: Netlink reflector reports IP 192.168.30.230 added
Jan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 11:41:03 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 11:41:14 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
Jan 10 11:41:14 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
[root@master ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ff
inet 192.168.30.116/24 brd 192.168.30.255 scope global eth0
inet 192.168.30.230/32 scope global eth0
inet6 fe80::20c:29ff:fef3:fcba/64 scope link
valid_lft forever preferred_lft forever　　发现刚才定义的virtual ipaddress在master服务器上，因为优先级较高
　　

　　停止master服务器的keepalived服务器，看IP会不会转移到backup
[root@master ~]# service keepalived stop
Stopping keepalived:                                       [  OK  ]
[root@backup ~]# tail -f /var/log/messages
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Configuration is using : 63017 Bytes
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Using LinkWatch kernel netlink reflector...
Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Configuration is using : 7324 Bytes
Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Using LinkWatch kernel netlink reflector...
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advert
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 10 12:16:27 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 10 12:16:28 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 added
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:16:33 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
[root@backup ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:5b:50:f9 brd ff:ff:ff:ff:ff:ff
inet 192.168.30.117/24 brd 192.168.30.255 scope global eth0
inet 192.168.30.230/32 scope global eth0
inet6 fe80::20c:29ff:fe5b:50f9/64 scope link
valid_lft forever preferred_lft forever　　发现IP已经转移到backup服务器
　　下面重新启动master的keepalived
[root@master ~]# service keepalived start
Starting keepalived:                                       [  OK  ]
[root@backup ~]# tail -f /var/log/messages
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advert
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 10 12:16:27 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 10 12:16:28 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 added
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:16:33 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advert
Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) removing protocol VIPs.
Jan 10 12:18:20 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 removed
[root@master ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ff
inet 192.168.30.116/24 brd 192.168.30.255 scope global eth0
inet 192.168.30.230/32 scope global eth0
inet6 fe80::20c:29ff:fef3:fcba/64 scope link
valid_lft forever preferred_lft forever　　发现IP已经重新转移到master服务器
　　

　　现在编写haproxy状态检测脚本，来实现haproxy的健康检测：
[root@master ~]# cat haproxy_pid.sh
#!/bin/bash
while :
do
haproxypid=`ps -C haproxy --no-header | wc -l`
if [ $haproxypid -eq 0 ];then
service haproxy start
sleep 5
haproxypid=`ps -C haproxy --no-header | wc -l`
echo $haproxypid
if [ $haproxypid -eq 0 ];then
/etc/init.d/keepalived stop
fi
fi
sleep 5
done
# 启动backup的haproxy
[root@backup ~]# service haproxy start
Starting haproxy:                                          [  OK  ]　　模拟故障，先让httpd进程开启，修改haproxy监听端口为80，使得haproxy进程无法启动，看资源会不会转移到backup服务器
　　
　　
[root@master ~]# vi /etc/haproxy/haproxy.cfg
frontend  main *:5000 改为frontend  main *:80
# 启动httpd进程
[root@master ~]# scp -pr /etc/haproxy/haproxy.cfg root@192.168.30.117:/etc/haproxy/
[root@master ~]# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for master.luojianlong.com
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
[  OK  ]
[root@master ~]# netstat -antpl | grep :80
tcp        0      0 :::80                       :::*                        LISTEN      19965/httpd
#运行状态检测脚本
[root@master ~]# nohup /root/haproxy_pid.sh &
[root@master ~]# scp -pr haproxy_pid.sh root@192.168.30.117:/root/
[root@bakcup ~]# nohup /root/haproxy_pid.sh &
[root@master ~]# tail -f /var/log/messages
Jan 10 12:02:29 localhost Keepalived_vrrp[19849]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jan 10 12:02:29 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:02:30 localhost Keepalived_healthcheckers[19848]: Netlink reflector reports IP 192.168.30.230 added
Jan 10 12:02:35 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:14:49 localhost Keepalived[19847]: Stopping Keepalived v1.2.7 (02/21,2013)
Jan 10 12:14:49 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) sending 0 priority
Jan 10 12:14:49 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) removing protocol VIPs.
[root@master ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ff
inet 192.168.30.116/24 brd 192.168.30.255 scope global eth0
inet6 fe80::20c:29ff:fef3:fcba/64 scope link
valid_lft forever preferred_lft forever
[root@backup ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:5b:50:f9 brd ff:ff:ff:ff:ff:ff
inet 192.168.30.117/24 brd 192.168.30.255 scope global eth0
inet 192.168.30.230/32 scope global eth0
inet6 fe80::20c:29ff:fe5b:50f9/64 scope link
valid_lft forever preferred_lft forever
[root@backup ~]# ps aux | grep haproxy
haproxy  19054  0.0  0.0  18688  1280 ?        Ss   12:47   0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
root     19097  0.0  0.0 103248   828 pts/0    S+   12:58   0:00 grep haproxy　　发现IP已经转移到backup服务器，实现了keepalived对于haproxy故障的高可用。
　　

　　keepalived常见的启动报错：
　　
5913 May 16 15:26:04 localhost Keepalived_vrrp: ip address associated with VRID not present in received packet : 192.168.57.75
5914 May 16 15:26:04 localhost Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert
5915 May 16 15:26:04 localhost Keepalived_vrrp: bogus VRRP packet received on eth0 !!!
5916 May 16 15:26:04 localhost Keepalived_vrrp: VRRP_Instance(VI_1) ignoring received advertisment...
5917 May 16 15:26:05 localhost Keepalived_vrrp: ip address associated with VRID not present in received packet : 192.168.57.75
5918 May 16 15:26:05 localhost Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert
5919 May 16 15:26:05 localhost Keepalived_vrrp: bogus VRRP packet received on eth0 !!!
5920 May 16 15:26:05 localhost Keepalived_vrrp: VRRP_Instance(VI_1) ignoring received advertisment.　　解决方法：
　　在同一网段内virtual_router_id 值不能相同，如果相同会在messages中收到VRRP错误包 ，所以需要更改 virual_router_id。