LVS+keepalived生产环境实现电信联通双VIP(DR模式)

obeckham · 发表于 2018-12-30 13:16:07

　　参考:
　　http://duyunlong.blog.运维网.com/1054716/1118447
　　http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-DR.html
　　http://www.cnblogs.com/Eivll0m/p/4580826.html

　　防火墙配置参考:http://wgkgood.blog.运维网.com/1192594/1102870
　　server全部为centos6.8
[root@iiw_MASTER keepalived]# uname -a
Linux iiw_MASTER 2.6.32-642.4.2.el6.x86_64 #1 SMP Tue Aug 23 19:58:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux　　

　　一:需求.由于前期经费有限,说白了就是没钱.
　　只能使用单主机做keepalived master,后端三台nginx提供web服务.开发80,443端口.

　　所需的ip为(安全原因ip都未虚构):

	电信IP	联通IP
VIP	10.10.10.214	36.10.10.86
keepaliver master主机	10.10.10.213	36.10.10.85
realserver1主机	10.10.10.215	36.10.10.87
realserver2主机	10.10.10.216	36.10.10.88
realserver3主机	10.10.10.217	36.10.10.89

　　二.流程

　　masterserver主机(ip:10.10.10.213),安装lvs+keeplived.开启ip转发功能.
[root@iiw_MASTER keepalived]# cat /etc/sysctl.conf
net.ipv4.ip_forward = 1
　　realserver安装nginx,并开启服务.略
　　masterserver,realserver 防火墙开发80,443端口.开放虚拟路由广播:

　　

iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT　　4.后端realserver,VIP绑定脚本文件为:

[root@iiw_WEB1 ~]$ cat /etc/init.d/realserver.sh
#!/bin/bash
SNS_VIP=10.10.10.214  #电信vip
SNS_VIP1=36.10.10.86  #联通vip
. /etc/rc.d/init.d/functions
case "$1" in
start)
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP up
/sbin/route add -host $SNS_VIP dev lo:0
ifconfig lo:1 $SNS_VIP1 netmask 255.255.255.255 broadcast $SNS_VIP1 up
/sbin/route add -host $SNS_VIP1 dev lo:1
echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p > /dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
ifconfig lo:0 down
route del $SNS_VIP >/dev/null 2>&1
ifconfig lo:1 down
route del $SNS_VIP1 >/dev/null 2>&1
echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0　　开启脚本:
[root@iiw_WEB1 ~]$ /etc/init.d/realserver.sh start　　检测vip是否存在:
[root@iiw_WEB1 ~]$ ifconfig
............................
lo:0    Link encap:Local Loopback
      inet addr:10.10.10.214  Mask:255.255.255.255
      UP LOOPBACK RUNNING  MTU:65536  Metric:1
lo:1    Link encap:Local Loopback
      inet addr:36.10.10.86  Mask:255.255.255.255
      UP LOOPBACK RUNNING  MTU:65536  Metric:1　　可见我们所需的两个VIP已经绑定在网卡l0:0,l0:1上.
　　5.masterserver上keepalived的配置:
[root@iiw_MASTER keepalived]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
   acassen@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_MASTER
}
#电信vip设置.
vrrp_instance VI_1 {
state MASTER    #注意此住为master状态
interface eth0    #检测eth0网卡
virtual_router_id 51 #注意id是51
priority 100
advert_int 1
authentication {
      auth_type PASS
      auth_pass 1111
}
virtual_ipaddress {
      10.10.10.214
}
}
#联通VIP设置
vrrp_instance VI_2 {
state MASTER #注意此住为master状态
interface eth0  #检测eth0网卡
virtual_router_id 52  #复制的时候注意此处id是52,不要跟电信VIP相同
priority 100
advert_int 1
authentication {
      auth_type PASS
      auth_pass 1111
}
virtual_ipaddress {
      36.10.10.86
}
}
virtual_server 10.10.10.214 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mark 255.255.255.0
protocol TCP
real_server 10.10.10.215 80 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
      }
}
real_server 10.10.10.216 80 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
      }
}
real_server 10.10.10.217 80 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
      }
}
}
virtual_server 10.10.10.214 443 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mark 255.255.255.0
protocol TCP
real_server 10.10.10.215 443 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
      }
}
real_server 10.10.10.216 443 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
      }
}
real_server 10.10.10.217 443 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
      }
}
}
virtual_server 36.10.10.86 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mark 255.255.255.0
protocol TCP
real_server 36.10.10.87 80 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
      }
}
   real_server 36.10.10.88 80 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
      }
}
real_server 36.10.10.89 80 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
      }
}
}
virtual_server 36.10.10.86 443 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mark 255.255.255.0
protocol TCP
real_server 36.10.10.87 443 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
      }
}
real_server 36.10.10.88 443 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
      }
}
real_server 36.10.10.89 443 {
      weight 1
      TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
      }
}
}　　在此,我将两个VIP分别绑定在两个vrrp_instance上,并且每个vrrp_instance都设置为master.然后启动keepalived
[root@iiw_MASTER keepalived]# service keepalived start　　检测vip:
[root@iiw_MASTER keepalived]# ip add
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
   valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc mq state UP qlen 1000
link/ether 3c:4a:92:e0:d1:30 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.213/27 brd 10.10.10.223 scope global eth0
inet 36.10.10.85/27 brd 36.10.10.95 scope global eth0:0
inet 10.10.10.214/32 scope global eth0
inet 36.10.10.86/32 scope global eth0
inet6 fe80::3e4a:92ff:fee0:d130/64 scope li　　可见我们所需的vip已经绑定在eth0上:
inet 10.10.10.214/32 scope global eth0
inet 36.10.10.86/32 scope global eth0　　

　　6.使用ipvsadm检测后端realserver是否ok轮询:
[root@iiw_MASTER keepalived]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port          Forward Weight ActiveConn InActConn
TCP  36.10.10.86:80 wrr
  -> 36.10.10.87:80             Route 1    0       18
  -> 36.10.10.88:80             Route 1    0       17
  -> 36.10.10.89:80             Route 1    0       17
TCP  36.10.10.86:443 wrr
  -> 36.10.10.87:443          Route 1    0       2
  -> 36.10.10.88:443          Route 1    0       5
  -> 36.10.10.89:443          Route 1    0       3
TCP  10.10.10.214:80 wrr
  -> 10.10.10.215:80          Route 1    7       25
  -> 10.10.10.216:80          Route 1    11       25
  -> 10.10.10.217:80          Route 1    7       25
TCP  10.10.10.214:443 wrr
  -> 10.10.10.215:443          Route 1    4       3
  -> 10.10.10.216:443          Route 1    1       6
  -> 10.10.10.217:443          Route 1    1       4　　通过这个命令可见,后端三台realserver的nginx 80,443端口被完全识别.
　　

　　7.通过hosts绑定网站为联通VIP36.10.10.86.通过AB软件测试:
root@huwei:/home/huwei# ulimit -SHn 51200
root@huwei:/home/huwei# ab -n 10000 -c 100 "http://www.iiii111.com"　　测试,通过http访问网站1w次,使用watch "lpvsadm"查看会不会平均分布到后端realserver
[root@iiw_MASTER keepalived]# watch "ipvsadm"Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port          Forward Weight ActiveConn InActConn
TCP  36.10.10.86:http wrr
  -> 36.10.10.87:http          Route 1 0       3354
  -> 36.10.10.88:http          Route 1 0       3354
  -> 36.10.10.89:http          Route 1 0       3354　　换https测试:
root@huwei:/home/huwei# ab -n 10000 -c 100 "https://www.iiii111.com"　　结果:
TCP  36.10.10.86:https wrr
  -> 36.10.10.87:https          Route 1 0       3366
  -> 36.10.10.88:https          Route 1 0       3364
  -> 36.10.10.89:https          Route 1 1       3366　　可见,轮询成功.
　　换hosts绑定电信vip再测也正常.
　　到此结束.
　　

　　总结:1.keepalived后端的nginx 80,443端口不要写错.
　　2.防火墙配置要正确.
　　3.后端服务器vip脚本先记得要启动,并检查vip ip有没有绑定好.

账号		自动登录	找回密码
密码			立即注册

wirelessnetview好用的无线分析工具

Red Hat RHCE 8 (EX294) Cert Guide

Shell从入门到精通（阿良）

亿图图示专家(EDraw Max) V7.9 中文破解版

zabbix3.4.1安装部署+微信推送信息+大屏显

Red Hat OpenShift I: Containers & Kubern

2025 年，C++ 还能“硬核”多久？

[经验分享] LVS+keepalived生产环境实现电信联通双VIP(DR模式)

扫码加入运维网微信交流群