LVS+keepalived生产环境实现电信联通双VIP(DR模式)
server全部为centos6.81
2
# uname -a
Linux iiw_MASTER 2.6.32-642.4.2.el6.x86_64 #1 SMP Tue Aug 23 19:58:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
一:需求.由于前期经费有限,说白了就是没钱.
只能使用单主机做keepalived master,后端三台nginx提供web服务.开发80,443端口.
所需的ip为(安全原因ip都未虚构):
电信IP联通IP
VIP10.10.10.214 36.10.10.86
keepaliver master主机10.10.10.213 36.10.10.85
realserver1主机10.10.10.21536.10.10.87
realserver2主机10.10.10.21636.10.10.88
realserver3主机10.10.10.21736.10.10.89
二.流程
[*]masterserver主机(ip:10.10.10.213),安装lvs+keeplived.开启ip转发功能.
[*]
1
2
# cat /etc/sysctl.conf
net.ipv4.ip_forward = 1
[*]realserver安装nginx,并开启服务.略
[*]masterserver,realserver 防火墙开发80,443端口.开放虚拟路由广播:
1
2
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
4.后端realserver,VIP绑定脚本文件为:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ cat /etc/init.d/realserver.sh
#!/bin/bash
SNS_VIP=10.10.10.214#电信vip
SNS_VIP1=36.10.10.86#联通vip
. /etc/rc.d/init.d/functions
case "$1" in
start)
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP up
/sbin/route add -host $SNS_VIP dev lo:0
ifconfig lo:1 $SNS_VIP1 netmask 255.255.255.255 broadcast $SNS_VIP1 up
/sbin/route add -host $SNS_VIP1 dev lo:1
echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p > /dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
ifconfig lo:0 down
route del $SNS_VIP >/dev/null 2>&1
ifconfig lo:1 down
route del $SNS_VIP1 >/dev/null 2>&1
echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
开启脚本:
1
$ /etc/init.d/realserver.sh start
检测vip是否存在:
1
2
3
4
5
6
7
8
9
$ ifconfig
............................
lo:0 Link encap:Local Loopback
inet addr:10.10.10.214Mask:255.255.255.255
UP LOOPBACK RUNNINGMTU:65536Metric:1
lo:1 Link encap:Local Loopback
inet addr:36.10.10.86Mask:255.255.255.255
UP LOOPBACK RUNNINGMTU:65536Metric:1
可见我们所需的两个VIP已经绑定在网卡l0:0,l0:1上.
5.masterserver上keepalived的配置:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_MASTER
}
#电信vip设置.
vrrp_instance VI_1 {
state MASTER #注意此住为master状态
interface eth0 #检测eth0网卡
virtual_router_id 51 #注意id是51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.10.10.214
}
}
#联通VIP设置
vrrp_instance VI_2 {
state MASTER #注意此住为master状态
interface eth0#检测eth0网卡
virtual_router_id 52#复制的时候注意此处id是52,不要跟电信VIP相同
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
36.10.10.86
}
}
virtual_server 10.10.10.214 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mark 255.255.255.0
protocol TCP
real_server 10.10.10.215 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 10.10.10.216 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 10.10.10.217 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
virtual_server 10.10.10.214 443 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mark 255.255.255.0
protocol TCP
real_server 10.10.10.215 443 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}
real_server 10.10.10.216 443 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}
real_server 10.10.10.217 443 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}
}
virtual_server 36.10.10.86 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mark 255.255.255.0
protocol TCP
real_server 36.10.10.87 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 36.10.10.88 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 36.10.10.89 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
virtual_server 36.10.10.86 443 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mark 255.255.255.0
protocol TCP
real_server 36.10.10.87 443 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}
real_server 36.10.10.88 443 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}
real_server 36.10.10.89 443 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}
}
在此,我将两个VIP分别绑定在两个vrrp_instance上,并且每个vrrp_instance都设置为master.然后启动keepalived
1
# service keepalived start
检测vip:
1
2
3
4
5
6
7
8
9
10
11
12
13
# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 3c:4a:92:e0:d1:30 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.213/27 brd 10.10.10.223 scope global eth0
inet 36.10.10.85/27 brd 36.10.10.95 scope global eth0:0
inet 10.10.10.214/32 scope global eth0
inet 36.10.10.86/32 scope global eth0
inet6 fe80::3e4a:92ff:fee0:d130/64 scope li
可见我们所需的vip已经绑定在eth0上:
1
2
inet 10.10.10.214/32 scope global eth0
inet 36.10.10.86/32 scope global eth0
6.使用ipvsadm检测后端realserver是否ok轮询:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP36.10.10.86:80 wrr
-> 36.10.10.87:80 Route 1 0 18
-> 36.10.10.88:80 Route 1 0 17
-> 36.10.10.89:80 Route 1 0 17
TCP36.10.10.86:443 wrr
-> 36.10.10.87:443 Route 1 0 2
-> 36.10.10.88:443 Route 1 0 5
-> 36.10.10.89:443 Route 1 0 3
TCP10.10.10.214:80 wrr
-> 10.10.10.215:80 Route 1 7 25
-> 10.10.10.216:80 Route 1 11 25
-> 10.10.10.217:80 Route 1 7 25
TCP10.10.10.214:443 wrr
-> 10.10.10.215:443 Route 1 4 3
-> 10.10.10.216:443 Route 1 1 6
-> 10.10.10.217:443 Route 1 1 4
通过这个命令可见,后端三台realserver的nginx 80,443端口被完全识别.
7.通过hosts绑定网站为联通VIP36.10.10.86.通过AB软件测试:
1
2
root@huwei:/home/huwei# ulimit -SHn 51200
root@huwei:/home/huwei# ab -n 10000 -c 100 "http://www.iiii111.com"
测试,通过http访问网站1w次,使用watch "lpvsadm"查看会不会平均分布到后端realserver
1
# watch "ipvsadm"
1
2
3
4
5
6
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP36.10.10.86:http wrr
-> 36.10.10.87:http Route 10 3354
-> 36.10.10.88:http Route 10 3354
-> 36.10.10.89:http Route 10 3354
换https测试:
1
root@huwei:/home/huwei# ab -n 10000 -c 100 "https://www.iiii111.com"
结果:
1
2
3
4
TCP36.10.10.86:https wrr
-> 36.10.10.87:https Route 1 0 3366
-> 36.10.10.88:https Route 1 0 3364
-> 36.10.10.89:https Route 1 1 3366
可见,轮询成功.
换hosts绑定电信vip再测也正常.
到此结束.
总结:1.keepalived后端的nginx 80,443端口不要写错.
2.防火墙配置要正确.
3.后端服务器vip脚本先记得要启动,并检查vip ip有没有绑定好.
页:
[1]