Keepalived+LVS-DR模式

23recfv

keepalived
- 通过在两台主、备调度器上实现Keepalived高可用解决调度器单点失败问题
- 主、备调度器上配置LVS，正常情况下主调度器工作
- 主调度器异常时，Keepalived启用备用调度器，保证了调度器的高可用
- 调度器保证了后台服务的高可用
一、keepalived+lvs-DR模式配置高可用负载均衡
拓扑如下：

二、服务器配置信息如下：
1、LVS_MASTER
[iyunv@master ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:5C:9B:35  
          inet addr:192.168.1.107  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe5c:9b35/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1167 errors:0 dropped:0 overruns:0 frame:0
          TX packets:417 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:99957 (97.6 KiB)  TX bytes:46517 (45.4 KiB)

2、LVS_BACKUP
[iyunv@backup ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:8B:95:97  
          inet addr:192.168.1.108  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe8b:9597/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:968 errors:0 dropped:0 overruns:0 frame:0
          TX packets:382 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:84935 (82.9 KiB)  TX bytes:50663 (49.4 KiB)

3、web1主机
root@realserver1 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:EB:5A:68  
          inet addr:192.168.1.109  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:feeb:5a68/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19546 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11171 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2870466 (2.7 MiB)  TX bytes:1742194 (1.6 MiB)
4、web2主机
[iyunv@realserver2 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:A9:D1:DC  
          inet addr:192.168.1.110  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fea9:d1dc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:20086 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12236 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3012984 (2.8 MiB)  TX bytes:1924957 (1.8 MiB)
三、在master和backup分别查看是否有ipvsadm软件

1、检查ipvsadm是否安装成功，可以直接输入
[iyunv@master ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[iyunv@backup ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
2、检查模板是否加入内核
[iyunv@master ~]# lsmod | grep ip_vs
ip_vs                 126534  0
libcrc32c               1246  1 ip_vs
ipv6                  335525  270 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6
[iyunv@backup ~]# lsmod | grep ip_vs
ip_vs                 126534  0
libcrc32c               1246  1 ip_vs
ipv6                  335525  270 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6
四、分别在web1和web2上面安装apache服务
开放端口80
[iyunv@realserver1 ~]# netstat -ntpl | grep :80
tcp        0      0 :::80                       :::*                        LISTEN      2841/httpd
[iyunv@realserver2 ~]# netstat -ntpl | grep :80
tcp        0      0 :::80                       :::*                        LISTEN      1602/httpd
五、在master主机（192.168.1.107）和backup主机（192.168.1.108）上安装keepalived
1、解压
[iyunv@master ~]# tar xf keepalived-1.2.7.tar.gz
2、配置
[iyunv@master ~]# cd keepalived-1.2.7
[iyunv@master keepalived-1.2.7]# ./configure --prefix=/home/keepalived/ --with-kernel-dir=/usr/src/kernels/2.6.32-573.7.1.el6.x86_64
报错：
  configure: error:
  !!! OpenSSL is not properly installed on your system. !!!
  !!! Can not include OpenSSL headers files.            !!!
[iyunv@master keepalived-1.2.7]# yum -y install openssl-devel
报错：
configure: error: Popt libraries is required
[iyunv@master keepalived-1.2.7]# yum -y install popt-devel
再次执行
[iyunv@master keepalived-1.2.7]# ./configure --prefix=/home/keepalived/ --with-kernel-dir=/usr/src/kernels/2.6.32-573.7.1.el6.x86_64

成功安装：

3、make
[iyunv@master keepalived-1.2.7]# make
4、make install
[iyunv@master keepalived-1.2.7]#make install
5、配置文件的路径
[iyunv@master keepalived-1.2.7]# cp /home/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[iyunv@master keepalived-1.2.7]# cp /home/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[iyunv@master keepalived-1.2.7]# mkdir /etc/keepalived
[iyunv@master keepalived-1.2.7]#cp /home/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[iyunv@master keepalived-1.2.7]# cp /home/keepalived/sbin/keepalived /sbin/
6、配置keepalived.conf文件（master）
[iyunv@master ~]# vim /etc/keepalived/keepalived.conf
----------------------------------------------------------------------------------------------------------------
#global define
global_defs {
router_id LVS     #全局配置，设置本机id
}
vrr_sync_group b1_group {
group {
  b1_one
}
}
vrrp_instance b1_one {
    state MASTER          #状态为master，从服务器为backup。必须大写
    interface eth0          #指定端口以提供虚拟ip的绑定
    virtual_router_id 38  #vrrp的组名，主从必须相同且不能与其他组相同
    priority 100               #指定优先级，值越大越优先
    advert_int 3              #组播信息发送间隔
    authentication {       #验证信息，两节点相同
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {         #执行虚拟ip地址
      192.168.1.113/24 dev eth0   
    }
}
virtual_server 192.168.1.113 80 {     #虚拟服务器的相应端口
    delay_loop 3    #延时时间
    lb_algo rr          #lvs的算法
    lb_kind DR        #lvs的模式
    persistence_timeout 1  #超时时间，在此时间内，同一用户发送多次请求，被转发到同一服务器上
    protocol TCP  #使用的协议
    real_server 192.168.1.109 80 { #真实主机的ip以及端口，另一个服务器上ip为110，这个为109
        weight 1      #权重，根据此进行请求的分发比例
        TCP_CHECK {
            connect_timeout 10
     nb_get_retry 3
     delay_before_retry 3
     connect_port 80
}
}
    real_server 192.168.1.110 80 {
weight 1
TCP_CHECK {
     connect_timeout 10
     nb_get_retry 3
     delay_before_retry 3
     connect_port 80
}
    }
}
----------------------------------------------------------------------------------------------------------------

7、配置keepalived.conf文件（BACKUP）
[iyunv@backup ~]# vim /etc/keepalived/keepalived.conf
----------------------------------------------------------------------------------------------------------------

#global define
global_defs {
router_id LVS
}
vrr_sync_group b1_group {
group {
  b1_one
}
}
vrrp_instance b1_one {
    state BACKUP
    interface eth0
    virtual_router_id 38
    priority 99
    advert_int 3
    authentication {
        auth_type PASS
        auth_pass 1111
    }  
    virtual_ipaddress {
      192.168.1.113/24 dev eth0
    }  
}
virtual_server 192.168.1.113 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP
    real_server 192.168.1.109 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }  
}
    real_server 192.168.1.110 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}
----------------------------------------------------------------------------------------------------------------

8、分别在主备上启动keepalived
[iyunv@master ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[iyunv@master ~]# chkconfig --add keepalived
[iyunv@master ~]# chkconfig keepalived on
[iyunv@master ~]# chkconfig keepalived --list
keepalived             0:off        1:off        2:on        3:on        4:on        5:on        6:off

[iyunv@master ~]# tail -f /var/log/messages
Oct 19 16:50:47 master Keepalived_vrrp[1601]: Netlink reflector reports IP 192.168.1.107 added
Oct 19 16:50:47 master Keepalived_vrrp[1601]: Netlink reflector reports IP fe80::20c:29ff:fe5c:9b35 added
Oct 19 16:50:47 master Keepalived_vrrp[1601]: Registering Kernel netlink reflector
Oct 19 16:50:47 master Keepalived_vrrp[1601]: Registering Kernel netlink command channel
Oct 19 16:50:47 master Keepalived_vrrp[1601]: Registering gratuitous ARP shared channel
Oct 19 16:50:47 master Keepalived_healthcheckers[1600]: Interface queue is empty
Oct 19 16:50:47 master Keepalived_healthcheckers[1600]: Netlink reflector reports IP 192.168.1.107 added
Oct 19 16:50:47 master Keepalived_healthcheckers[1600]: Netlink reflector reports IP fe80::20c:29ff:fe5c:9b35 added
Oct 19 16:50:47 master Keepalived_healthcheckers[1600]: Registering Kernel netlink reflector
Oct 19 16:50:47 master Keepalived_healthcheckers[1600]: Registering Kernel netlink command channel
Oct 19 16:51:08 master Keepalived_vrrp[1601]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 19 16:51:08 master Keepalived_vrrp[1601]: Configuration is using : 61515 Bytes
Oct 19 16:51:08 master Keepalived_vrrp[1601]: Using LinkWatch kernel netlink reflector...
Oct 19 16:51:08 master Keepalived_healthcheckers[1600]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 19 16:51:08 master Keepalived_healthcheckers[1600]: Configuration is using : 12492 Bytes
Oct 19 16:51:08 master Keepalived_healthcheckers[1600]: Using LinkWatch kernel netlink reflector...
Oct 19 16:51:08 master Keepalived_healthcheckers[1600]: Activating healthchecker for service [192.168.1.109]:80 监控两台主机
Oct 19 16:51:08 master Keepalived_healthcheckers[1600]: Activating healthchecker for service [192.168.1.110]:80
Oct 19 16:51:08 master Keepalived_vrrp[1601]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Oct 19 16:51:11 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) Transition to MASTER STATE
Oct 19 16:51:14 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) Entering MASTER STATE
Oct 19 16:51:14 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) setting protocol VIPs.
Oct 19 16:51:14 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) Sending gratuitous ARPs on eth0 for 192.168.1.113
Oct 19 16:51:14 master Keepalived_healthcheckers[1600]: Netlink reflector reports IP 192.168.1.113 added
Oct 19 16:51:19 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) Sending gratuitous ARPs on eth0 for 192.168.1.113
[iyunv@backup ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[iyunv@backup ~]# tail -f /var/log/messages
Oct 19 16:54:08 backup Keepalived_healthcheckers[1617]: Netlink reflector reports IP 192.168.1.108 added
Oct 19 16:54:08 backup Keepalived_vrrp[1618]: Interface queue is empty
Oct 19 16:54:08 backup Keepalived_healthcheckers[1617]: Netlink reflector reports IP fe80::20c:29ff:fe8b:9597 added
Oct 19 16:54:08 backup Keepalived_healthcheckers[1617]: Registering Kernel netlink reflector
Oct 19 16:54:08 backup Keepalived_healthcheckers[1617]: Registering Kernel netlink command channel
Oct 19 16:54:08 backup Keepalived_vrrp[1618]: Netlink reflector reports IP 192.168.1.108 added
Oct 19 16:54:08 backup Keepalived_vrrp[1618]: Netlink reflector reports IP fe80::20c:29ff:fe8b:9597 added
Oct 19 16:54:08 backup Keepalived_vrrp[1618]: Registering Kernel netlink reflector
Oct 19 16:54:08 backup Keepalived_vrrp[1618]: Registering Kernel netlink command channel
Oct 19 16:54:08 backup Keepalived_vrrp[1618]: Registering gratuitous ARP shared channel
Oct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 19 16:54:28 backup Keepalived_vrrp[1618]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 19 16:54:28 backup Keepalived_vrrp[1618]: Configuration is using : 61515 Bytes
Oct 19 16:54:28 backup Keepalived_vrrp[1618]: Using LinkWatch kernel netlink reflector...
Oct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Configuration is using : 12492 Bytes
Oct 19 16:54:28 backup Keepalived_vrrp[1618]: VRRP_Instance(b1_one) Entering BACKUP STATE
Oct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Using LinkWatch kernel netlink reflector...
Oct 19 16:54:28 backup Keepalived_vrrp[1618]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Oct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Activating healthchecker for service [192.168.1.109]:80
Oct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Activating healthchecker for service [192.168.1.110]:80
# PS:这里说的LVS.并不是用ipvsadm软件配置.而是直接用keepalived的virtual_server配置项控制的.
#  安装ipvsadm只是可以看到负载状况.其实只需要keepalived也可以实现负载均衡集群.
分别在主备服务器上查看VIP：
[iyunv@master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:5c:9b:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.107/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.113/24 scope global secondary eth0
    inet6 fe80::20c:29ff:fe5c:9b35/64 scope link
       valid_lft forever preferred_lft forever
[iyunv@backup ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:8b:95:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.108/24 brd 192.168.1.255 scope global eth0
    inet6 fe80::20c:29ff:fe8b:9597/64 scope link
       valid_lft forever preferred_lft forever
VIP目前在master上
六、在web1和web2主机上配置LVS-DR的相关配置

1、[iyunv@realserver1 ~]# vim /etc/init.d/lvsDR_node
---------------------------------------------------------------------------------------------------
#! /bin/sh  
# descript : start real server DR  
VIP=192.168.1.113
. /etc/rc.d/init.d/functions
case "$1" in
       start)
             echo "start LVS of RealServer DR"  
             /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
             echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore
             echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce
               ;;
       stop)
             /sbin/ifconfig lo:0 down
             echo "close LVS of RealServer DR"  
             echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
             echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
              ;;
         *)
              echo "Usage : $0 {start|stop}"  
              exit 1
esac
---------------------------------------------------------------------------------------------------

[iyunv@realserver1 ~]# chmod +x /etc/init.d/lvsDR_node
将启动脚本加入到 /etc/rc.local  开机启动
/etc/init.d/lvsDR_node start

七、web1和web2主机上开启httpd服务

[iyunv@realserver1 ~]# netstat -ntpl | grep httpd
tcp        0      0 :::80                       :::*                        LISTEN      1228/httpd
[iyunv@realserver2 ~]# netstat -ntpl | grep httpd
tcp        0      0 :::80                       :::*                        LISTEN      1238/httpd
八、测试是否可以负载均衡

编写测试脚本
#! /bin/sh  
for((i=1;i<=100;i++));do
curl http://192.168.1.113:80 >> /tmp/q;
done
结果：
[iyunv@master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.113:80 rr
  -> 192.168.1.109:80             Route   1      0          100      
  -> 192.168.1.110:80             Route   1      0          100
九、测试断开一个web服务或加入一个web服务

现在关闭web2主机（192.168.1.110）的web服务
查看日志可以看到：
Oct 19 20:39:59 master Keepalived_healthcheckers[1796]: TCP connection to [192.168.1.110]:80 failed !!!
Oct 19 20:39:59 master Keepalived_healthcheckers[1796]: Removing service [192.168.1.110]:80 from VS [192.168.1.113]:80
[iyunv@master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.113:80 rr
  -> 192.168.1.109:80             Route   1      0          0        ---只有1.109
现在再把web2主机（192.168.1.110）的web服务加进来：
查看日志可以看到：
Oct 19 20:42:51 master Keepalived_healthcheckers[1796]: TCP connection to [192.168.1.110]:80 success.
Oct 19 20:42:51 master Keepalived_healthcheckers[1796]: Adding service [192.168.1.110]:80 to VS [192.168.1.113]:80
[iyunv@master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.113:80 rr
  -> 192.168.1.109:80             Route   1      0          0         
  -> 192.168.1.110:80             Route   1      0          0

十、测试HA

现在模拟主（master）的keepalived主机死掉
备机的日志文件
Oct 19 20:45:33 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) Transition to MASTER STATE
Oct 19 20:45:36 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) Entering MASTER STATE
Oct 19 20:45:36 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) setting protocol VIPs.
Oct 19 20:45:36 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) Sending gratuitous ARPs on eth0 for 192.168.1.113
Oct 19 20:45:36 backup Keepalived_healthcheckers[1795]: Netlink reflector reports IP 192.168.1.113 added
Oct 19 20:45:41 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) Sending gratuitous ARPs on eth0 for 192.168.1.113
可以看到备机的keepalived已经开始工作.......
整个部署就完成了