LVS+Keepalived负载均衡及高可用WEB服务器集群
环境：操作系统Centos 6.5 X86_64(final)
LVS-Master:192.168.2.32
LVS-Backup:192.168.3.31
VIP:192.168.2.33
Web2:192.168.2.29
Web4:192.168.2.30
1、安装LVS軟件（分别在LVS-Master和LVS-Backup中）
1.1.查看是否有IPVS模块，LVS是Linux标准内核的一部分,直接被编译在内核中!
# modprobe -l | grep ipvs #使用modprobe命令查看
kernel/net/netfilter/ipvs/ip_vs.ko
kernel/net/netfilter/ipvs/ip_vs_rr.ko
kernel/net/netfilter/ipvs/ip_vs_wrr.ko
kernel/net/netfilter/ipvs/ip_vs_lc.ko
kernel/net/netfilter/ipvs/ip_vs_wlc.ko
kernel/net/netfilter/ipvs/ip_vs_lblc.ko
kernel/net/netfilter/ipvs/ip_vs_lblcr.ko
kernel/net/netfilter/ipvs/ip_vs_dh.ko
kernel/net/netfilter/ipvs/ip_vs_sh.ko
kernel/net/netfilter/ipvs/ip_vs_sed.ko
kernel/net/netfilter/ipvs/ip_vs_nq.ko
kernel/net/netfilter/ipvs/ip_vs_ftp.ko
kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko
1.2.安装IPVS管理工具
# yum install ipvsadm
2、安装keepalived（分别在LVS-Master和LVS-Backup中）
2.1.安装keepalived并配置。
# yum install keepalived
# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc #设置报警地址，可以设置多个，每行1个
sysadmin@firewall.loc #需安装sendmail，并开启邮件报警
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1 #设置SMTP Server地址
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER #指定Keepalived的角色，MASTER为主，BACKUP为备
interface eth0 #绑定真实的物理网卡
virtual_router_id 51 #虚拟路由编号，主备要一致
priority 100 #定义优先级，数字越大，优先级越高，主DR必须大于备用DR
advert_int 1 #检查间隔，默认为1s
authentication {
auth_type PASS #认证方式，PASS或AH两种
auth_pass 1111 #认证密码，主备服务器要一致
}
virtual_ipaddress {
192.168.2.33 #定义虚拟IP(VIP)为192.168.2.33，可多设，每行一个
}
}
virtual_server 192.168.2.33 80 { #定义对外提供服务的LVS的VIP以及port
delay_loop 6 #设置健康检查时间，单位是秒
lb_algo wlc #设置负载调度的算法为wlc
lb_kind DR #设置LVS实现负载的机制，有NAT、TUN、DR三个模式
persistence_timeout 50 #会话保持时间，单位为秒
protocol TCP #指定转发协议类型，有TCP和UDP两种
real_server 192.168.2.29 80 {#指定real server1的IP地址
weight 100 #配置节点权值，数字越大
TCP_CHECK {
connect_timeout 3 #表示3秒无响应，则超时
nb_get_retry 3 #表示重试次数
delay_before_retry 3 #表示重试间隔
}
}
real_server 192.168.2.30 80 {#指定real server2的IP地址
weight 100 #配置节点权值，数字越大
TCP_CHECK {
connect_timeout 3 #表示3秒无响应，则超时
nb_get_retry 3 #表示重试次数
delay_before_retry 3 #表示重试间隔
}
}
}
2.2.启动keepalived、设置keepalived服务自启动，检查是否生效。
# /etc/init.d/keepalived start
Starting keepalived:
# chkconfig keepalived on
# ipvsadm -L -n #查看ipvs规则是否生效
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.33:80 wlc persistent 50
-> 192.168.2.29:80 Route 100 0 0
-> 192.168.2.30:80 Route 100 0 0
3、在LVS-Backup上安装LVS、keepalived并配置
3.1.安装LVS、keepalived
# yum install ipvsadm
# yum install keepalived
3.2.配置LVS-Backup的keepalived
先从LVS－Master把修改过的keepalived配置文件复制过来。
# scp keepalived.conf root@192.168.2.31:/etc/keepalived/
# vi /etc/keepalived/keepalived.conf
只修改state为BACKUP和priority为90，MASTER为主机服务器，BACKUP为备用服务器。
state BACKUP
priority 90
3.3.启动LVS-Backup的keeplived服务，并检查是否生效。
# /etc/init.d/keepalived start
# chkconfig keepalived on
# ipvsadm -L -n #查看ipvs规则是否生效
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.2.33:80 wlc persistent 50
-> 192.168.2.29:80 Route 100 0 0
-> 192.168.2.30:80 Route 100 0 0
4、测试VIP，检查是否能主备切换。
4.1.启动keeplived后，先用ip addr显示主备服务器的VIP情况，当主备服务器同时启用keepalived时，只有主服务器拥有VIP地址，备服务器没有。
# ip addr
1: lo:mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:a6:00:13 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.32/24 brd 192.168.2.255 scope global eth0
inet 192.168.2.33/32 scope global eth0 #可以看到主服务器拥有192.168.2.33这个VIP地址
inet6 fe80::250:56ff:fea6:13/64 scope link
valid_lft forever preferred_lft forever
# ip addr
1: lo:mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 08:00:27:77:d3:82 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.31/24 brd 192.168.2.255 scope global eth0 #可以到看备服务器没有192.168.2.33这个VIP地址
inet6 fe80::a00:27ff:fe77:d382/64 scope link
valid_lft forever preferred_lft forever
4.2.当停止主服务器的keepalived服务，再查看下主备服务器VIP地址，发现VIP地址已从主服务器转移到了备服务器
# service keepalived stop
Stopping keepalived:
# ip addr
1: lo:mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:a6:00:13 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.32/24 brd 192.168.2.255 scope global eth0 #主服务器keepalived停止后，VIP地址将移除
inet6 fe80::250:56ff:fea6:13/64 scope link
valid_lft forever preferred_lft forever
# ip addr
1: lo:mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 08:00:27:77:d3:82 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.31/24 brd 192.168.2.255 scope global eth0
inet 192.168.2.33/32 scope global eth0 #可以看到VIP地址已转移到备服务器，继续提供服务
inet6 fe80::a00:27ff:fe77:d382/64 scope link
valid_lft forever preferred_lft forever
4.3.日志查看主备切换
当主服务器停止keepalived后，主备服务器相关日志
# service keepalived stop
Stopping keepalived:
主服务器日志
Sep2 17:39:13 LVS-Master kernel: IPVS: __ip_vs_del_service: enter
Sep2 17:39:13 LVS-Master Keepalived: Stopping Keepalived v1.2.7 (02/21,2013) #主服务器停掉后
Sep2 17:39:13 LVS-Master Keepalived_vrrp: VRRP_Instance(VI_1) sending 0 priority
Sep2 17:39:13 LVS-Master Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
备服务器日志
Sep2 17:39:14 LVS-Backup Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep2 17:39:15 LVS-Backup Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE #备转为MASTER，并设置VIP地址为192.168.2.33
Sep2 17:39:15 LVS-Backup Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep2 17:39:15 LVS-Backup Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.33
Sep2 17:39:15 LVS-Backup Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.33 added
Sep2 17:39:20 LVS-Backup Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.33
当主服务器重启keepalived后，主备服务器相关日志
# service keepalived start
Starting keepalived:
主服务器日志
Sep2 17:41:28 LVS-Master Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep2 17:41:29 LVS-Master Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE #主服务器重启后，从新接手MASTER角色，并拿回VIP地址192.168.2.33
Sep2 17:41:29 LVS-Master Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep2 17:41:29 LVS-Master Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.33 added
Sep2 17:41:29 LVS-Master Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.33
备服务器日志
Sep2 17:41:28 LVS-Backup Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Sep2 17:41:28 LVS-Backup Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE #备服务器从MASTER又变回了BACKUP，并移除VIP地址192.168.2.33
Sep2 17:41:28 LVS-Backup Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Sep2 17:41:28 LVS-Backup Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.33 removed
5、配置WEB服务器，即real server节点1和2
由于采用的是DR方式调度，Real_Server会以LVS的VIP来直接回复Client，所以需要在Real_Server的lo上开启LVS的VIP来与Client建立通信
配置web2
# vi /etc/init.d/lvsrs ＃编辑一个lvsrs脚本，并放到/etc/init.d/下
#!/bin/bash
#description : start Real Server
VIP=192.168.2.33
./etc/rc.d/init.d/functions
case "$1" in
start)
echo " Start LVS of Real Server "
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
;;
stop)
/sbin/ifconfig lo:0 down
echo "close LVS Director server"
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
# chmod 755 /etc/init.d/lvsrs #给新增lvsrs脚本一个755权限
# service lvsrs start #启动lvsrs服务，我环境启动时出错，给一个functions 755权限，按各自环境操作
/etc/init.d/lvsrs: line 7: ./etc/rc.d/init.d/functions: Permission denied
Start LVS of Real Server
# chmod 755 /etc/rc.d/init.d/functions#给functions 755权限
# service lvsrs start #启动lvsrs服务
Start LVS of Real Server
# ifconfig #查看已有一个VIP地址
lo:0 Link encap:Local Loopback
inet addr:192.168.2.33Mask:255.255.255.255
UP LOOPBACK RUNNINGMTU:16436Metric:1
# echo "service lvsrs start" >> /etc/rc.local #把lvsrs服务添加到开机启动，否则下次启动电脑后，还需手动启动。
# yum install httpd
# vi /var/www/html/index.html
Welcome to web2(192.168.2.29)
# service httpd start
配置web4
配置同上
# scp root@192.168.2.29:/etc/init.d/lvsrs /etc/init.d/
# chmod 755 /etc/rc.d/init.d/functions
# service lvsrs start
Start LVS of Real Server
# echo "service lvsrs start" >> /etc/rc.local
# ifconfig
lo:0 Link encap:Local Loopback
inet addr:192.168.2.33Mask:255.255.255.255
UP LOOPBACK RUNNINGMTU:16436Metric:1
# yum install httpd
# vi /var/www/html/index.html
Welcome to web4(192.168.2.30)
# service httpd start
6、负载均衡以及高可用测试
6.1.负载均衡测试
（1）、两台WEB服务器都工作正常，访问VIP地址时，可以被负载到两台WEB服务器上。
（2）、停掉一台WEB服务器的http服务后，访问VIP地址时，被负载到正常的WEB服务器上。
6.2.高可用测试
（1）、两台DR的keepalived服务器正常时，VIP地址在主服务器，提供正常服务。
（2）、当关闭主服务器的keepalived后，VIP地址自动转移到备服务器上，当主服务器重启keepalived服务后，VIP自动转移回主服务器，备服务器继续备用，实现了高可用负载均衡。
参考如下：
http://www.jizhuomi.com/software/351.html
http://nmshuishui.blog.运维网.com/1850554/1405745
http://czybl.blog.运维网.com/4283444/1536474
http://zhumeng8337797.blog.163.com/blog/static/1007689142011101392553489/
页:
[1]