lvs+keepalived搭建
环境：基于lvs/dr模式 通过wlc 加权轮叫 访问web服务器web1 192.168.4.7
web2 192.168.4.8
lvs-master 192.168.4.5
lvs-backup 192.168.4.6
VIP 192.168.4.10
配置：lvs-master
# cat keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth1
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.4.10
}
}
virtual_server 192.168.4.10 80 {
delay_loop 6
lb_algo wlc
lb_kind DR
persistence_timeout 60
protocol TCP
real_server 192.168.4.7 80 {
weight 3
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.4.8 80 {
weight 3
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
lvs-backup
! Configuration File for keepalived
global_defs {
notification_email {
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP
interface eth2
virtual_router_id 51
priority99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.4.10
}
}
virtual_server 192.168.4.10 80 {
delay_loop 6
lb_algo wlc
lb_kind DR
persistence_timeout 60
protocol TCP
real_server 192.168.4.7 80 {
weight 4
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.4.8 80 {
weight 6
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
web配置
# cat vip.sh
#!/bin/bash
# description: Config realserver lo and apply noarp
VIP=192.168.4.10
source/etc/rc.d/init.d/functions
case "$1" in
start)
/sbin/ifconfig lo:0 $VIPbroadcast $VIP netmask 255.255.255.255 up
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "VIPservice Start OK"
;;
stop)
/sbin/ifconfig lo:0 down
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "VIPservice Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
修改web的index.html
echo "web1"> /var/www/html/index.html
echo "web2"> /var/www/html/index.html
开始测试，启动web服务器的vip.sh
shvip.sh start 用ifconfig查看就出一个lo：0网卡信息
# ifconfig
eth2 Link encap:EthernetHWaddr 00:0C:29:04:2F:68
inet addr:192.168.4.7Bcast:192.168.4.255Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe04:2f68/64 Scope:Link
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
RX packets:3357 errors:0 dropped:0 overruns:0 frame:0
TX packets:538 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:217741 (212.6 KiB)TX bytes:45405 (44.3 KiB)
Interrupt:18 Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNINGMTU:16436Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b)TX bytes:0 (0.0 b)
lo:0 Link encap:Local Loopback
inet addr:192.168.4.10Mask:255.255.255.255
UP LOOPBACK RUNNINGMTU:16436Metric:1
启动master和backup的keepalived servicekeepalived start
执行 ipvsadm -lnc
通过访问192.168.4.10可以看到页面信息
web1
关掉master的服务，再去请求
master /var/log/messages日志
May 20 19:09:08 localhost Keepalived: Starting Keepalived v1.2.12 (04/30,2014)
May 20 19:09:08 localhost Keepalived: Starting Healthcheck child process, pid=19254
May 20 19:09:08 localhost Keepalived: Starting VRRP child process, pid=19255
May 20 19:09:08 localhost Keepalived_vrrp: Netlink reflector reports IP 192.168.4.6 added
May 20 19:09:08 localhost Keepalived_vrrp: Netlink reflector reports IP 192.168.218.5 added
May 20 19:09:08 localhost Keepalived_vrrp: Netlink reflector reports IP fe80::20c:29ff:fe25:11b2 added
May 20 19:09:08 localhost Keepalived_vrrp: Netlink reflector reports IP fe80::20c:29ff:fe25:11a8 added
May 20 19:09:08 localhost Keepalived_vrrp: Registering Kernel netlink reflector
May 20 19:09:08 localhost Keepalived_vrrp: Registering Kernel netlink command channel
May 20 19:09:08 localhost Keepalived_vrrp: Registering gratuitous ARP shared channel
May 20 19:09:08 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.4.6 added
May 20 19:09:08 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.218.5 added
May 20 19:09:08 localhost Keepalived_healthcheckers: Netlink reflector reports IP fe80::20c:29ff:fe25:11b2 added
May 20 19:09:08 localhost Keepalived_healthcheckers: Netlink reflector reports IP fe80::20c:29ff:fe25:11a8 added
May 20 19:09:08 localhost Keepalived_healthcheckers: Registering Kernel netlink reflector
May 20 19:09:08 localhost Keepalived_healthcheckers: Registering Kernel netlink command channel
May 20 19:09:08 localhost Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
May 20 19:09:08 localhost Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
May 20 19:09:08 localhost Keepalived_healthcheckers: Configuration is using : 14019 Bytes
May 20 19:09:08 localhost Keepalived_vrrp: Configuration is using : 37628 Bytes
May 20 19:09:08 localhost Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
May 20 19:09:08 localhost Keepalived_vrrp: VRRP sockpool:
May 20 19:09:08 localhost Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
May 20 19:09:08 localhost Keepalived_healthcheckers: Activating healthchecker for service :80
May 20 19:09:08 localhost Keepalived_healthcheckers: Activating healthchecker for service :80
May 20 19:09:09 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
May 20 19:09:10 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
May 20 19:09:10 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
May 20 19:09:10 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.4.10
May 20 19:09:10 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.4.10 added
May 20 19:09:15 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.4.10
May 20 19:18:08 localhost kernel: IPVS: __ip_vs_del_service: enter
May 20 19:18:08 localhost Keepalived: Stopping Keepalived v1.2.12 (04/30,2014)
May 20 19:18:08 localhost Keepalived_healthcheckers: Removing service :80 from VS :80
May 20 19:18:08 localhost Keepalived_healthcheckers: Removing service :80 from VS :80
May 20 19:18:08 localhost Keepalived_vrrp: VRRP_Instance(VI_1) sending 0 priority
May 20 19:18:08 localhost Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
backup的messages日志
May 20 21:26:43 example4 Keepalived: Starting Keepalived v1.1.15 (05/19,2014)
May 20 21:26:43 example4 Keepalived: Starting Healthcheck child process, pid=1948
May 20 21:26:43 example4 Keepalived: Starting VRRP child process, pid=1949
May 20 21:26:43 example4 Keepalived_healthcheckers: Using MII-BMSR NIC polling thread...
May 20 21:26:43 example4 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
May 20 21:26:43 example4 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.4.5 added
May 20 21:26:43 example4 Keepalived_healthcheckers: Registering Kernel netlink reflector
May 20 21:26:43 example4 Keepalived_healthcheckers: Registering Kernel netlink command channel
May 20 21:26:43 example4 Keepalived_vrrp: Netlink reflector reports IP 192.168.4.5 added
May 20 21:26:43 example4 Keepalived_vrrp: Registering Kernel netlink reflector
May 20 21:26:43 example4 Keepalived_vrrp: Registering Kernel netlink command channel
May 20 21:26:43 example4 Keepalived_vrrp: Registering gratutious ARP shared channel
May 20 21:27:03 example4 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
May 20 21:27:03 example4 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
May 20 21:27:03 example4 Keepalived_healthcheckers: Configuration is using : 11844 Bytes
May 20 21:27:03 example4 Keepalived_vrrp: Configuration is using : 36231 Bytes
May 20 21:27:03 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
May 20 21:27:03 example4 Keepalived_vrrp: VRRP sockpool:
May 20 21:27:03 example4 Keepalived_healthcheckers: Activating healtchecker for service
May 20 21:27:03 example4 Keepalived_healthcheckers: Activating healtchecker for service
May 20 21:27:03 example4 kernel: IPVS: scheduler registered.
May 20 21:27:49 example4 dhclient: DHCPREQUEST on eth2 to 192.168.4.254 port 67 (xid=0x76964cd)
May 20 21:27:49 example4 dhclient: DHCPACK from 192.168.4.254 (xid=0x76964cd)
May 20 21:27:49 example4 NetworkManager: <info> (eth2): DHCPv4 state changed renew -> renew
May 20 21:27:49 example4 NetworkManager: <info> address 192.168.4.5
May 20 21:27:49 example4 NetworkManager: <info> prefix 24 (255.255.255.0)
May 20 21:27:49 example4 NetworkManager: <info> gateway 192.168.4.2
May 20 21:27:49 example4 NetworkManager: <info> nameserver '192.168.4.2'
May 20 21:27:49 example4 NetworkManager: <info> domain name 'localdomain'
May 20 21:27:49 example4 dhclient: bound to 192.168.4.5 -- renewal in 856 seconds.
May 20 21:35:51 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
May 20 21:35:52 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
May 20 21:35:52 example4 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
May 20 21:35:52 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.4.10
May 20 21:35:52 example4 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.4.10 added
May 20 21:35:52 example4 Keepalived_vrrp: Netlink: skipping nl_cmd msg...
May 20 21:35:57 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.4.10
容易出错的地方，在配置keepalived.conf文件的时候 一定要看一下自己对外服务的到底是哪个网卡eth0还是eth1
再就是第一次配置nat模式的时候 ，原理理解不清楚，把web1和web2的网关指向了他们真实的网关，发现怎么也请求不到
# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP192.168.4.10:http wlc persistent 60
-> 192.168.4.8:http Route 6 0 0
-> 192.168.4.7:http Route 4 0 0
当时InActConn有数据，但是就是请求不到，后来才知道原来是web的网关许配置lvs的realip
lvs的activeconn是个一直让我很迷惑的东东.每次看到这个数巨大而真实机上的活动连接数并不是很高的时候,我都忍不住网上搜索一番,虽然大多时候总是无功而返,但是渐渐的总结出来了以下理论.
ActiveConn是活动连接数,也就是tcp连接状态的ESTABLISHED;InActConn是指除了ESTABLISHED以外的,所有的其它状态的tcp连接.那既然这样,为什么从lvs里看的ActiveConn会比在真实机上通过netstats看到的ESTABLISHED高很多呢?问得好!这也是笔者一直迷惑而渐渐清晰的一个问题.原来lvs自身也有一个默认超时时间.可以用ipvsadm -L --timeout查看,默认是900 120 300,分别是TCP TCPFIN UDP的时间.也就是说一条tcp的连接经过lvs后,lvs会把这台记录保存15分钟,而不管这条连接是不是已经失效!所以如果你的服务器在15分钟以内有大量的并发请求连进来的时候,你就会看到这个数值直线上升.
其实很多时候,我们看lvs的这个连接数是想知道现在的每台机器的真实连接数吧?怎么样做到这一点呢?其实知道现在的ActiveConn是怎样产生的,做到这一点就简单了.举个例子:比如你的lvs是用来负载网站,用的模式是dr,后台的web server用的nginx.这时候一条请求过来,在程序没有问题的情况下,一条连接最多也就五秒就断开了.这时候你可以这样设置:ipvsadm --set 5 10 300.设置tcp连接只保持5秒中.如果现在ActiveConn很高你会发现这个数值会很快降下来,直到降到和你用nginx的status看当前连接数的时候差不多.你可以继续增加或者减小5这个数值,直到真实机的status连接数和lvs里的ActiveConn一致.
页:
[1]