论坛 › lvs › 负载均衡集群LVS实战篇

zzgzyyz 发表于 2019-1-4 11:45:58

负载均衡集群LVS实战篇

Keepalived+LVS的高可用集群系统
　　前面两章已经详细介绍了负载均衡LVS原理和安装，这篇文章实际操作一遍，让大家能更深刻理解，闲话不多说，直接进入正题。

一.拓扑图结构：
http://s4.运维网.com/wyfs02/M02/82/CA/wKiom1dhBbKQEqIOAAHZOQTsykA798.png-wh_500x0-wm_3-wmp_4-s_246440786.png
二.Keepalived的安装
1.主服务器的安装
# wget http://www.keepalived.org/software/keepalived-1.2.20.tar.gz



# tar -zxvf keepalived-1.2.20.tar.gz



# cd keepalived-1.2.20/



# ./configure --prefix=/usr/local/keepalived



configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files.            !!!
# yum -y install openssl-devel libnl-devel libnfnetlink-devel ipvsadm




# ./configure --prefix=/usr/local/keepalived



Keepalived configuration
------------------------
Keepalived version       : 1.2.20
Compiler               : gcc
Compiler flags         : -g -O2 -DFALLBACK_LIBNL1
Extra Lib                : -lssl -lcrypto -lcrypt-lnl
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
IPVS use libnl         : Yes
fwmark socket support    : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
Use VRRP authentication: Yes
SNMP keepalived support: No
SNMP checker support   : No
SNMP RFCv2 support       : No
SNMP RFCv3 support       : No
SHA1 support             : No
Use Debug flags          : No
libnl version            : 1
Use IPv4 devconf         : No
Use libiptc            : No
Use libipset             : No

# make && make install



# ln -s /usr/local/keepalived/sbin/keepalived /usr/bin/keepalived



# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived



# chmod 755 /etc/init.d/keepalived



# chkconfig keepalived on



# vim /etc/init.d/keepalived



# Source configuration file (we set KEEPALIVED_OPTIONS there)
. /etc/sysconfig/keepalived
改为：
# Source configuration file (we set KEEPALIVED_OPTIONS there)
. /usr/local/keepalived/etc/sysconfig/keepalived

# mkdir /etc/keepalived



# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf



# service keepalived restart
2.主服务器配置文件
# vim /etc/keepalived/keepalived.conf



! Configuration File for keepalived

global_defs {
   notification_email {
   65556505@qq.com#收件人邮箱
   }
   notification_email_from keepalived@localhost#发件人邮箱
   smtp_server 127.0.0.1#邮箱服务器地址
   smtp_connect_timeout 30#连接时间
   router_id LVS_DEVEL#标识，唯一性
}

vrrp_instance VI_1 {#实例
    state MASTER#主MASTER备BACKUP
    interface eno16777736#网卡
    virtual_router_id 51#路由标识
    priority 100#优先级
    advert_int 1#检查间隔
    authentication {
      auth_type PASS#验证类型
      auth_pass 1111#验证密码
    }
    virtual_ipaddress {
      192.168.1.100#vip
    }
}

virtual_server 192.168.1.100 80 {
    delay_loop 6#轮询间隔时间
    lb_algo rr#调度算法，rr|wrr|lc|wlc|lblc|sh|dh
    lb_kind DR#集群模式，NAT|DR|TUN
    persistence_timeout 50#会话保持时间
    protocol TCP#使用协议

    real_server 192.168.1.30 80 {
      weight 1#权重
      TCP_CHECK {#检查方式，有HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK
            connect_port    80#连接端口
            connect_timeout 3#连接超时时间
            nb_get_retry 3#重试次数
            delay_before_retry 3#重连时间间隔
      }
    }

    real_server 192.168.1.40 80 {
      weight 1
      TCP_CHECK {
            connect_port    80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
      }
    }
}
3.配置 IPVS
3.1.开启IP转发功能
# echo 1 >/proc/sys/net/ipv4/ip_forward   默认是0，关闭ip转发；这里需要开启，所以设置值为1。

3.2.配置重定向
# echo "0" >/proc/sys/net/ipv4/ip_forward



# echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects



# echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects



# echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects

3.3.清除ipvsadm表
# ipvsadm –C

3.4.使用ipvsadm安装web服务
# ipvsadm -A -t 192.168.1.100:80-s rr

3.5.增加二台web服务器
# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.30:80 -m -w 1



# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.40:80 -m -w 1

3.6查看结果
#ipvsadm -L-n


IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port         Forward Weight ActiveConn InActConn
TCP192.168.1.100:80 rr
-> 192.168.1.30:80            Route   1      0          1      
-> 192.168.1.40:80            Route   1      0          0

# service keepalived restart

4.主服务器IP查看
# ip add



1: lo:mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno16777736:mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:74:1f:4e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.10/24 brd 192.168.1.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 192.168.1.100/32 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe74:1f4e/64 scope link
       valid_lft forever preferred_lft forever

5.备服务器的安装
# wget http://www.keepalived.org/software/keepalived-1.2.20.tar.gz



# tar -zxvf keepalived-1.2.20.tar.gz



# cd keepalived-1.2.20/



# ./configure --prefix=/usr/local/keepalived



configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files.            !!!

# yum -y install openssl-devel libnl-devel libnfnetlink-devel ipvsadm



# ./configure --prefix=/usr/local/keepalived



Keepalived configuration
------------------------
Keepalived version       : 1.2.20
Compiler               : gcc
Compiler flags         : -g -O2 -DFALLBACK_LIBNL1
Extra Lib                : -lssl -lcrypto -lcrypt-lnl
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
IPVS use libnl         : Yes
fwmark socket support    : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
Use VRRP authentication: Yes
SNMP keepalived support: No
SNMP checker support   : No
SNMP RFCv2 support       : No
SNMP RFCv3 support       : No
SHA1 support             : No
Use Debug flags          : No
libnl version            : 1
Use IPv4 devconf         : No
Use libiptc            : No
Use libipset             : No

# make && make install



# ln -s /usr/local/keepalived/sbin/keepalived /usr/bin/keepalived



# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived



# chmod 755 /etc/init.d/keepalived



# chkconfig keepalived on



# vim /etc/init.d/keepalived



# mkdir /etc/keepalived



# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf



# service keepalived restart

6.备服务器配置文件
# vim /etc/keepalived/keepalived.conf



! Configuration File for keepalived

global_defs {
   notification_email {
   65556505@qq.com
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP
    interface eno16777736
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
      auth_type PASS
      auth_pass 1111
    }
    virtual_ipaddress {
      192.168.1.100
    }
}

virtual_server 192.168.1.100 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP

    real_server 192.168.1.30 80 {
      weight 1
      TCP_CHECK {
            connect_port    80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
      }
    }

    real_server 192.168.1.40 80 {
      weight 1
      TCP_CHECK {
            connect_port    80
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
      }
    }
}

7.配置 IPVS
7.1.开启IP转发功能
# echo 1 >/proc/sys/net/ipv4/ip_forward   

7.2.配置重定向
# echo "0" >/proc/sys/net/ipv4/ip_forward



# echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects



# echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects



# echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects

7.3.清除ipvsadm表
# ipvsadm –C

7.4.使用ipvsadm安装web服务
# ipvsadm -A -t 192.168.1.100:80-s rr

7.5.增加二台web服务器
# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.30:80 -m -w 1



# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.40:80 -m -w 1

7.6查看结果
#ipvsadm -L-n



IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port         Forward Weight ActiveConn InActConn
TCP192.168.1.100:80 rr
-> 192.168.1.30:80            Route   1      0          1      
-> 192.168.1.40:80            Route   1      0          0

# service keepalived restart

8.备服务器IP查看
# ip add



1: lo:mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno16777736:mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:85:b8:f8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.20/24 brd 192.168.1.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe85:b8f8/64 scope link
       valid_lft forever preferred_lft forever

备注：如果主备服务器都有VIP存在，请关闭selinux和防火墙
# setenforce 0
# systemctl stop firewalld.service

三.web服务器的配置
1.此处以默认安装好的web服务，如Nginx和Apache等等。只进行节点的配置步骤。简单安装apache服务，方便后面的测试。
# yum -y install httpd



# vim /var/www/html/index.html

web1 server



# systemctl start httpd.service

http://s2.运维网.com/wyfs02/M01/82/CB/wKiom1dhDMOB8fi8AABrn6NR5sE431.png-wh_500x0-wm_3-wmp_4-s_1235626271.png



# yum -y install httpd



# vim /var/www/html/index.html



web2 server



# systemctl start httpd.service

http://s5.运维网.com/wyfs02/M01/82/CA/wKioL1dhDeXz-RhtAABtNZipBx4615.png-wh_500x0-wm_3-wmp_4-s_1385612872.png

2.web1服务器LVS配置：
# vim /etc/init.d/lvsrs



#!/bin/bash
#description:start realserver
vip=192.168.1.100
source /etc/rc.d/init.d/functions
case $1 in
start)
      echo "Start Realserver"
      /sbin/ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
         /sbin/route add -host $VIP dev lo:0
      echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore
      echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
      echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore
      echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce
          sysctl -p >/dev/null 2>&1
;;
stop)
      echo "Stop Realserver"
      /sbin/ifconfig lo:0 down
         /sbin/route del $VIP >/dev/null 2>&1
      echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
      echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce
      echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
      echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
      echo "Usage: $0 (start | stop)"
exit 1
esac

# chmod 755 /etc/init.d/lvsrs



# service lvsrs start

3.web2服务器LVS配置：
# vim /etc/init.d/lvsrs



#!/bin/bash
#description:start realserver
vip=192.168.1.100
source /etc/rc.d/init.d/functions
case $1 in
start)
      echo "Start Realserver"
      /sbin/ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
         /sbin/route add -host $VIP dev lo:0
      echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore
      echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
      echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore
      echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce
          sysctl -p >/dev/null 2>&1
;;
stop)
      echo "Stop Realserver"
      /sbin/ifconfig lo:0 down
         /sbin/route del $VIP >/dev/null 2>&1
      echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore
      echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce
      echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
      echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
      echo "Usage: $0 (start | stop)"
exit 1
esac

# chmod 755 /etc/init.d/lvsrs



# service lvsrs start



到此为止，Keepalived+LVS的高可用集群系统搭建完成，下面进行测试。

四.高可用keepalived功能测试
1.先停掉主服务器上的keepalived服务，看看备服务器是否接管vip
# service keepalived stop



# ip add



1: lo:mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno16777736:mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:74:1f:4e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.10/24 brd 192.168.1.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe74:1f4e/64 scope link
       valid_lft forever preferred_lft forever

2.查看备服务器的ip
# ip add



1: lo:mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno16777736:mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:85:b8:f8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.20/24 brd 192.168.1.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 192.168.1.100/32 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe85:b8f8/64 scope link
       valid_lft forever preferred_lft forever

上面信息可以看到备服务器已经接管vip地址。

3.恢复主服务器keepalived服务。
# service keepalived start



# ip add



1: lo:mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno16777736:mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:74:1f:4e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.10/24 brd 192.168.1.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 192.168.1.100/32 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe74:1f4e/64 scope link
       valid_lft forever preferred_lft forever

上面信息可以看到主服务器已经重新接管vip地址。

4.再查看备服务器ip地址
# ip add



1: lo:mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno16777736:mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:85:b8:f8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.20/24 brd 192.168.1.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe85:b8f8/64 scope link
       valid_lft forever preferred_lft forever

上面信息可以看到主服务器已经释放了vip地址。

五.负载均衡LVS功能测试
打开浏览器输入http://192.168.1.100.不断刷新页面分别看到web1 server和web2 server表示LVS已经负载均衡了。

http://s5.运维网.com/wyfs02/M02/82/CB/wKiom1dhDODyOKwsAABrL2_Td9E939.png-wh_500x0-wm_3-wmp_4-s_154355495.png


http://s2.运维网.com/wyfs02/M02/82/CA/wKioL1dhDf3S1n7oAABqhzCzh8E458.png-wh_500x0-wm_3-wmp_4-s_2325343582.png

六.故障切换测试
1.停掉web1服务器的httpd服务
# systemctl stop httpd.service

2. 查看主服务器状态
# ipvsadm



IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port         Forward Weight ActiveConn InActConn
TCPwjb10000-master:http rr
-> 192.168.1.40:http            Route   1      0          0

3.查看备服务器状态
# ipvsadm



IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port         Forward Weight ActiveConn InActConn
TCP192.168.1.100:http rr
-> 192.168.1.40:http            Route   1      0          0

以上内容可以看出Keepalived检测web1服务器出现故障，将此服务器从集群中移除。

4.恢复web1服务器
# systemctl start httpd.service

5.再查看主服务器状态
# ipvsadm



IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port         Forward Weight ActiveConn InActConn
TCPwjb10000-master:http rr
-> 192.168.1.30:http            Route   1      0          0      
-> 192.168.1.40:http            Route   1      0          0

6.再查看备服务器状态
# ipvsadm



IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port         Forward Weight ActiveConn InActConn
TCP192.168.1.100:http rr
-> 192.168.1.30:http            Route   1      0          0         
-> 192.168.1.40:http            Route   1      0          0

节点恢复重新加入集群当中，浏览器刷新页面，可以重新看到web1 server和web2 server内容交替显示。
结束语：到此Keepalived+LVS的高可用集群系统实验完全结束。大家哪里不明白的欢迎留言。









　　

查看完整版本: 负载均衡集群LVS实战篇