使用keepalived打造轻量级的lvs高可用集群
使用keepalived打造轻量级的lvs高可用集群
一、VRRP协议
VRRP: Virtual Routing Redundent Rrotocol虚拟路由冗余协议
学过网络的朋友都知道，网络在设计的时候必须考虑到冗余容灾，包括线路冗余，设备冗余等，防止网络存在单点故障，那在路由器或三层交换机处实现冗余就显得尤为重要，在网络里面有个协议就是来做这事的，这个协议就是VRRP协议，Keepalived就是巧用VRRP协议来实现高可用性(HA)的。
下图为VRRP协议的工作模型图：
http://blog.运维网.com/attachment/201305/211426235.png
如上图所示，RouterA，B，C组成一个虚拟路由器。此虚拟路由器有自己的IP地址，局域网内的主机将虚拟路由器设置为缺省网关。RouterA、B、C中优先级高的路由器作为Master路由器，承担网关的功能。其余两台路由器作为Backup路由器。
1、备份组中路由器的优先级
VRRP根据优先级来确定备份组中每台路由器的角色(Master路由器或Backup路由器)。优先级越高则越有可能成为Master路由器
VRRP优先级的取值范围为0-255(数值越大优先级越高)，可配置的范围是1-254,0为系统保留，255是系统保留给IP地址拥有者。当路由器为IP地址拥有者时，其优先级始终为255，因此，当备份组内存在IP地址拥有者时，只要其工作正常，则为Master路由器。
2、备份组中的路由器的工作方式
备份组中的路由器具有以下两种工作方式：
非抢占模式：如果备份组中的路由器工作在非抢占模式下，则只要Master路由器没有出现故障，Backup路由器即使随后配置了更高的优先级也不会成为Master路由器。
抢占模式：如果备份组中的路由器工作在抢占模式下，它一旦发现自己的优先级比当前的Master路由器的优先级高，就会对外发送VRRP通告报文，导致备份组内路由器重新选举Master路由器，并最终取代原有的Master路由器。相应的，原来的Master路由器将会变成Backup路由器。
3、备份组中路由器的认证方式
simple：简单字符认证
md5: MD5认证
4、在一个物理设备上，可以配置多个组，靠组ID区别不同的组
具体介绍，参考下面的一篇博客：
http://liangbin332.blog.163.com/blog/static/119684536201051705315390/
二、搭建基于DR模型的lvs集群（web服务集群）
规划图：
http://blog.运维网.com/attachment/201305/213734609.png
1、配置realserver1
# vim /etc/sysconfig/network-scripts/ifcfg-eth0http://blog.运维网.com/attachment/201305/212319846.png
# service network restart
2、修改部分参数
# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
# echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce
# echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
# cat /proc/sys/net/ipv4/conf/all/arp_announce
2
# cat /proc/sys/net/ipv4/conf/all/arp_ignore
1 参数介绍：
kernel parameter：
arp_ignore: 接收到ARP请求时的响应级别
默认级别是0：只要本地配置的有相应地址，就予以响应
1：仅在请求的目标地址配置在请求到达的接口上的时候，才给予响应
arp_announce：定义将自己的地址及MAC地址向外通告时的通告级别
默认级别是0：将本机任何接口上的任何地址向外通告
1：试图仅向目标网络通告与其网络匹配的地址
2：仅向与本地接口上地址匹配的网络进行通告
3、在lo:0上配置vip,并配置路由条目
# ifconfig lo:0 172.16.25.1 broadcast 172.16.25.1 netmask 255.255.255.255 up
# route add -host 172.16.25.1 dev lo:0 4、安装web服务，并提供网页
# yum install httpd
# service httpd start
# echo "RS1.lsq.com" > /var/www/html/index.html
# curl http://172.16.25.7
RS1.lsq.com 5、ifconfig查看
http://blog.运维网.com/attachment/201305/213411888.png
http://blog.运维网.com/attachment/201305/213448896.png
同样的配置，配置realserver2
http://blog.运维网.com/attachment/201305/213616103.png
http://blog.运维网.com/attachment/201305/213633606.png
三、安装配置keepalived实现lvs的高可用
准备工作：
配置主、从director节点：
1、时间同步（主、从节点时间几个不得超过5秒）
2、主机名要和uname -n 一致，并通过/etc/hosts解析
3、SSH双机互信
配置主节点：
时间同步：
# ntpdate 172.16.0.1 #我们的ftp服务器
主机名要和uname -n 一致，并通过/etc/hosts解析：
# hostname node1.lsq.com
# vim /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1.lsq.com
GATEWAY=172.16.0.1
# vim /etc/hosts
添加两行：
172.16.25.10 node1.lsq.com node1
172.16.25.12 node2.lsq.com node2
SSH互信：
配置节点1(主节点)
# ssh-keygen -t rsa -f ~/.ssh/rsa_key -P ''
# ssh-copy-id -i .ssh/rsa_key.pub root@node2.lsq.com#把公钥发给从节点
1,2两点主从节点配置一致，不在演示
在从节点上建立SSH互信：
# ssh-keygen -t rsa -f ~/.ssh/rsa_key -P ''
# ssh-copy-id -i .ssh/rsa_key.pub root@node1.lsq.com#把公钥发给从节点 配置过程：
1、为director的主、备节点安装keepalived的rpm包，并提供配置文件
配置主节点
# yum -y --nogpgcheck localinstall keepalived-1.2.7-5.el5.i386.rpm
# scp keepalived-1.2.7-5.el5.i386.rpm node2:/root
# yum install ipvsadm #安装ipvs规则
# cd /etc/keepalived
# ls
keepalived.confkeepalived.conf.haproxy_examplenotify.sh
# cp keepalived.conf keepalived.conf.bak
# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email { #出现故障后向谁发送电子邮件
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 { #定义VRRP实例，即一个虚拟路由组
state MASTER #定义在初始状态下谁是主，谁是从
interface eth0 #在那个接口上配置地址和完成选举，使用ipaddr配置，不具备别名
virtual_router_id 86#组ID
priority 101 #优先级
advert_int 1 #通告时间间隔
authentication { #认证
auth_type PASS #PASS表示使用字符串认证
auth_pass keepalivedpass
}
virtual_ipaddress { #vip
172.16.25.1/16 dev eth0 label eth0:0
}
}
virtual_server 172.16.25.1 80 { #定义集群服务
delay_loop 6 #获取某个服务时的等待时长
lb_algo rr #负载均衡调度算法
lb_kind DR
nat_mask 255.255.0.0 #负载均衡集群类型
# persistence_timeout 50 #是不是支持持久链接，这里我们不使用
protocol TCP #tcp协议
real_server 172.16.25.7 80 { #realserver
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
connect_timeout 2 #3秒中探测一次健康状况
nb_get_retry 3 #探测三次都是不健康，就判定不健康
delay_before_retry 1 #探测一次后，再隔几秒钟后探测
}
}
real_server 172.16.25.8 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
}
# scp keepalived.conf node2:/etc/keepalived/
配置node2
# yum -y --nogpgcheck localinstall keepalived-1.2.7-5.el5.i386.rpm
# yum install ipvsadm #安装ipvs规则
# cd /etc/keepalived/
# ls
keepalived.confkeepalived.conf.haproxy_examplenotify.sh
# vim keepalived.conf 只需在主节点配置文件的基础上做如下修改即可：
将 state MASTER改为 state BACKUP
将 priority 101改为 priority 100
然后在主、备节点上启动keepalived服务
# service keepalived start
在director1上执行ifconfig命令
# ifconfig
http://blog.运维网.com/attachment/201305/220534651.png
查看ipvs规则
http://blog.运维网.com/attachment/201305/220657700.png
打开浏览器，访问以下vip
http://blog.运维网.com/attachment/201305/220731763.png
刷新页面
http://blog.运维网.com/attachment/201305/220817180.png
2、手动将RS1上的web服务关闭，查看是否能将服务移除
在RS1上关闭web服务
# service httpd stop
Stopping httpd:
在node1上查看规则
# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP172.16.25.1:80 rr
-> 172.16.25.8:80 Route 1 1 0
You have new mail in /var/spool/mail/root
查看邮件
Mail version 8.1 6/6/93.Type ? for help.
"/var/spool/mail/root": 6 messages 6 new
>N1 logwatch@localhost.lFri Mar 29 14:5345/1675"Logwatch for localhost.localdomain (Linux)"
N2 logwatch@localhost.lMon May6 21:4645/1672"Logwatch for localhost.localdomain (Linux)"
N3 logwatch@localhost.lTue May7 04:02 127/3683"Logwatch for localhost.localdomain (Linux)"
N4 logwatch@localhost.lThu May9 19:3569/2231"Logwatch for localhost.localdomain (Linux)"
N5 root@localhost.localThu May9 19:4119/837 "Anacron job for 'localhost.localdomain' cron.daily"
N6 keepalived@localhostFri May 17 14:3213/571 " Realserver :80 - DOWN"
& 6
Message 6:
From keepalived@localhost.localdomainFri May 17 14:32:40 2013
Date: Fri, 17 May 2013 06:32:40 +0000
From: keepalived@localhost.localdomain
Subject: Realserver :80 - DOWN
X-Mailer: Keepalived
=> CHECK failed on service : connection error/var/www/html/index.html
# scp /var/www/html/index.html node2:/var/www/html/
在director2上安装web服务，并启动
# yum install httpd
# service httpd start
在director的主、备节点上，编辑keepalived的配置文件
# pwd
/etc/keepalived
# vim keepalived.conf
定位至：virtual_server，在{}里面添加一行：
sorry_server 127.0.0.1 80
上述操作主、备节点一样，然后主、备节点都启动keepalived服务，使之生效
# service keepalived restart
Stopping keepalived:
Starting keepalived:
在realserver1和realserver2上都关闭web服务
# service httpd stop
Stopping httpd:
然后在director1(主)上查看ipvs规则：
# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP172.16.25.1:80 rr
-> 127.0.0.1:80 Local 1 0 0
You have new mail in /var/spool/mail/root
http://blog.运维网.com/attachment/201305/220012811.png
从新启动两个realserver上的web服务
# service httpd start
Starting httpd:
在director1上查看ipvs规则
# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP172.16.25.1:80 rr
-> 172.16.25.8:80 Route 1 0 0
-> 172.16.25.7:80 Route 1 0 0
You have new mail in /var/spool/mail/root
3、director主、备节点之间的高可用的实现
# vim keepalived.conf
在VRRP实例之外，添加如下内容：
vrrp_script chk_schedown {
script "[ -e /etc/keepalived/down ] && exit 1 || exit 0"#如果director主节点这个目录下存在down文件，就down掉主director，否则不down
interval 1 #检测一次的时间
weight -5 #如果存在down文件，主节点的优先级减5(减后要小于备节点的优先级才可)
fall 2 #如果失败了，需要检测2次
rise 1 #成功只需一次
}
然后还在配置文件中，添加如下内容：
定位至：vrrp_instance VI_1 { ，在vrrp实例中添加如下内容，：
track_script {
chk_schedown #定义上述脚本什么时候执行
}
以上操作director的主备节点的配置文件都要配置，内容一样
而后两个节点上都重启服务
# service keepalived restart
Stopping keepalived:
Starting keepalived:
在主director的/etc/keepalived目录下，touch一个down文件
# pwd
/etc/keepalived
# touch down
查看日志信息可以看出vip的漂移
# tail /var/log/messages
May 17 19:15:21 localhost Keepalived_healthcheckers: Netlink reflector reports IP 172.16.25.1 added
May 17 19:15:21 localhost avahi-daemon: Registering new address record for 172.16.25.1 on eth0.
May 17 19:15:26 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.16.25.1
May 17 19:17:11 localhost Keepalived_vrrp: VRRP_Script(chk_schedown) failed
May 17 19:17:13 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
May 17 19:17:13 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
May 17 19:17:13 localhost Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
May 17 19:17:13 localhost Keepalived_vrrp: Netlink reflector reports IP 172.16.25.1 removed
May 17 19:17:13 localhost Keepalived_healthcheckers: Netlink reflector reports IP 172.16.25.1 removed
May 17 19:17:13 localhost avahi-daemon: Withdrawing address record for 172.16.25.1 on eth0.
在director2上执行ifconfig http://blog.运维网.com/attachment/201305/221103800.png
4、写个脚本实现当vrrp事物发生时(主备节点切换)，发送警报邮件给指定的管理员
在/etc/keepalived下编写一个脚本
# pwd
/etc/keepalived
# vim new_notify.sh
#!/bin/bash
#
contact='root@localhost'
Usage() {
echo "Usage: `basename $0` {master|backup|fault} VIP"
}
Notify() {
subject="`hostname`'s state changed to $1"
mailbody="`date "+%F %T"`: `hostname`'s state change to $1, $VIP floating."
echo $mailbody | mail -s "$subject" $contact
}
[ $# -lt 2 ] && Usage && exit
VIP=$2
case $1 in
master)
Notify master
;;
backup)
Notify backup
;;
fault)
Notify fault
;;
*)
Usage
exit 1
;;
esac
# chmod +x new_notify.sh
# scp -p new_notify.sh node2:/etc/keepalived/ #复制到另外的节点上一份
配置director的两个节点上keepalived的配置文件：
# vim keepalived.conf
定位至：vrrp_instance VI_1 { ，在vrrp实例中添加如下内容，：
notify_master "/etc/keepalived/new_notify.sh master 172.16.25.1"
notify_backup "/etc/keepalived/new_notify.sh backup 172.16.25.1"
notify_fault "/etc/keepalived/new_notify.sh fault 172.16.25.1"
两个节点都启动keepalived服务
# service keepalived restart
Stopping keepalived:
Starting keepalived:
在director1上删除down文件
# ls
downkeepalived.confkeepalived.conf.bakkeepalived.conf.haproxy_examplenew_notify.shnotify.sh
# rm -rf down
# tail /var/log/maillog #可以根据邮件日志，查询邮件发送状态
执行mail，查看邮件
http://blog.运维网.com/attachment/201305/221705954.png
在director1上ifconfig
http://blog.运维网.com/attachment/201305/221719169.png
现在我们已经使用keepalived实现了lvs的高可用了，并且我们还自写脚本实现director主、备节点切换时可以发送警报邮件信息，是不是很easy，大家都来试试吧！
页:
[1]