负载均衡集群LVS实战篇

zzgzyyz

Keepalived+LVS的高可用集群系统
　　前面两章已经详细介绍了负载均衡LVS原理和安装，这篇文章实际操作一遍，让大家能更深刻理解，闲话不多说，直接进入正题。

一.拓扑图结构：

二.Keepalived的安装
1.主服务器的安装

[root@wjb10000-master ~]# wget http://www.keepalived.org/software/keepalived-1.2.20.tar.gz

[root@wjb10000-master ~]# tar -zxvf keepalived-1.2.20.tar.gz

[root@wjb10000-master ~]# cd keepalived-1.2.20/

[root@wjb10000-master keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived

configure: error:   !!! OpenSSL is not properly installed on your system. !!!   !!! Can not include OpenSSL headers files.            !!!

[root@wjb10000-master keepalived-1.2.20]# yum -y install openssl-devel libnl-devel libnfnetlink-devel ipvsadm

[root@wjb10000-master keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived

Keepalived configuration ------------------------ Keepalived version       : 1.2.20 Compiler                 : gcc Compiler flags           : -g -O2 -DFALLBACK_LIBNL1 Extra Lib                : -lssl -lcrypto -lcrypt  -lnl   Use IPVS Framework       : Yes IPVS sync daemon support : Yes IPVS use libnl           : Yes fwmark socket support    : Yes Use VRRP Framework       : Yes Use VRRP VMAC            : Yes Use VRRP authentication  : Yes SNMP keepalived support  : No SNMP checker support     : No SNMP RFCv2 support       : No SNMP RFCv3 support       : No SHA1 support             : No Use Debug flags          : No libnl version            : 1 Use IPv4 devconf         : No Use libiptc              : No Use libipset             : No

[root@wjb10000-master keepalived-1.2.20]# make && make install

[root@wjb10000-master keepalived-1.2.20]# ln -s /usr/local/keepalived/sbin/keepalived /usr/bin/keepalived

[root@wjb10000-master keepalived-1.2.20]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived

[root@wjb10000-master keepalived-1.2.20]# chmod 755 /etc/init.d/keepalived

[root@wjb10000-master keepalived-1.2.20]# chkconfig keepalived on

[root@wjb10000-master keepalived-1.2.20]# vim /etc/init.d/keepalived

# Source configuration file (we set KEEPALIVED_OPTIONS there) . /etc/sysconfig/keepalived 改为： # Source configuration file (we set KEEPALIVED_OPTIONS there) . /usr/local/keepalived/etc/sysconfig/keepalived

[root@wjb10000-master keepalived-1.2.20]# mkdir /etc/keepalived

[root@wjb10000-master keepalived-1.2.20]# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

[root@wjb10000-master keepalived-1.2.20]# service keepalived restart

2.主服务器配置文件

[root@wjb10000-master /]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived global_defs {    notification_email {      65556505@qq.com#收件人邮箱    }    notification_email_from keepalived@localhost#发件人邮箱    smtp_server 127.0.0.1#邮箱服务器地址    smtp_connect_timeout 30#连接时间    router_id LVS_DEVEL#标识，唯一性 } vrrp_instance VI_1 {#实例     state MASTER#主MASTER备BACKUP     interface eno16777736#网卡     virtual_router_id 51#路由标识     priority 100#优先级     advert_int 1#检查间隔     authentication {         auth_type PASS#验证类型         auth_pass 1111#验证密码     }     virtual_ipaddress {         192.168.1.100#vip     } } virtual_server 192.168.1.100 80 {     delay_loop 6#轮询间隔时间     lb_algo rr#调度算法，rr|wrr|lc|wlc|lblc|sh|dh     lb_kind DR#集群模式，NAT|DR|TUN     persistence_timeout 50#会话保持时间     protocol TCP#使用协议     real_server 192.168.1.30 80 {         weight 1#权重         TCP_CHECK {#检查方式，有HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK             connect_port    80#连接端口             connect_timeout 3#连接超时时间             nb_get_retry 3#重试次数             delay_before_retry 3#重连时间间隔         }       }       real_server 192.168.1.40 80 {         weight 1         TCP_CHECK {             connect_port    80             connect_timeout 3             nb_get_retry 3             delay_before_retry 3         }       }   }

3.配置 IPVS
3.1.开启IP转发功能

[root@wjb10000-master /]# echo 1 >/proc/sys/net/ipv4/ip_forward   默认是0，关闭ip转发；这里需要开启，所以设置值为1。

3.2.配置重定向

[root@wjb10000-master /]# echo "0" >/proc/sys/net/ipv4/ip_forward

[root@wjb10000-master /]# echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects

[root@wjb10000-master /]# echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects

[root@wjb10000-master /]# echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects

3.3.清除ipvsadm表

[root@wjb10000-master /]# ipvsadm –C

3.4.使用ipvsadm安装web服务

[root@wjb10000-master /]# ipvsadm -A -t 192.168.1.100:80-s rr

3.5.增加二台web服务器

[root@wjb10000-master /]# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.30:80 -m -w 1

[root@wjb10000-master /]# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.40:80 -m -w 1

3.6查看结果

[root@wjb10000-master ~]#  ipvsadm -L  -n

IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn TCP  192.168.1.100:80 rr   -> 192.168.1.30:80              Route   1      0          1           -> 192.168.1.40:80              Route   1      0          0

[root@wjb10000-master keepalived-1.2.20]# service keepalived restart

4.主服务器IP查看

[root@wjb10000-master /]# ip add

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000     link/ether 00:0c:29:74:1f:4e brd ff:ff:ff:ff:ff:ff     inet 192.168.1.10/24 brd 192.168.1.255 scope global eno16777736        valid_lft forever preferred_lft forever     inet 192.168.1.100/32 scope global eno16777736        valid_lft forever preferred_lft forever     inet6 fe80::20c:29ff:fe74:1f4e/64 scope link        valid_lft forever preferred_lft forever

5.备服务器的安装

[root@wjb10000-backup ~]# wget http://www.keepalived.org/software/keepalived-1.2.20.tar.gz

[root@wjb10000-backup ~]# tar -zxvf keepalived-1.2.20.tar.gz

[root@wjb10000-backup ~]# cd keepalived-1.2.20/

[root@wjb10000-backup keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived

configure: error:   !!! OpenSSL is not properly installed on your system. !!!   !!! Can not include OpenSSL headers files.            !!!

[root@wjb10000-backup keepalived-1.2.20]# yum -y install openssl-devel libnl-devel libnfnetlink-devel ipvsadm

[root@wjb10000-backup keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived

Keepalived configuration ------------------------ Keepalived version       : 1.2.20 Compiler                 : gcc Compiler flags           : -g -O2 -DFALLBACK_LIBNL1 Extra Lib                : -lssl -lcrypto -lcrypt  -lnl   Use IPVS Framework       : Yes IPVS sync daemon support : Yes IPVS use libnl           : Yes fwmark socket support    : Yes Use VRRP Framework       : Yes Use VRRP VMAC            : Yes Use VRRP authentication  : Yes SNMP keepalived support  : No SNMP checker support     : No SNMP RFCv2 support       : No SNMP RFCv3 support       : No SHA1 support             : No Use Debug flags          : No libnl version            : 1 Use IPv4 devconf         : No Use libiptc              : No Use libipset             : No

[root@wjb10000-backup keepalived-1.2.20]# make && make install

[root@wjb10000-backup keepalived-1.2.20]# ln -s /usr/local/keepalived/sbin/keepalived /usr/bin/keepalived

[root@wjb10000-backup keepalived-1.2.20]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived

[root@wjb10000-backup keepalived-1.2.20]# chmod 755 /etc/init.d/keepalived

[root@wjb10000-backup keepalived-1.2.20]# chkconfig keepalived on

[root@wjb10000-backup keepalived-1.2.20]# vim /etc/init.d/keepalived

[root@wjb10000-backup keepalived-1.2.20]# mkdir /etc/keepalived

[root@wjb10000-backup keepalived-1.2.20]# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

[root@wjb10000-backup keepalived-1.2.20]# service keepalived restart

6.备服务器配置文件

[root@wjb10000-backup /]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived global_defs {    notification_email {      65556505@qq.com    }    notification_email_from keepalived@localhost    smtp_server 127.0.0.1    smtp_connect_timeout 30    router_id LVS_DEVEL } vrrp_instance VI_1 {     state BACKUP     interface eno16777736     virtual_router_id 51     priority 99     advert_int 1     authentication {         auth_type PASS         auth_pass 1111     }     virtual_ipaddress {         192.168.1.100     } } virtual_server 192.168.1.100 80 {     delay_loop 6     lb_algo rr     lb_kind DR     persistence_timeout 50     protocol TCP     real_server 192.168.1.30 80 {         weight 1         TCP_CHECK {             connect_port    80             connect_timeout 3             nb_get_retry 3             delay_before_retry 3         }       }       real_server 192.168.1.40 80 {         weight 1         TCP_CHECK {             connect_port    80             connect_timeout 3             nb_get_retry 3             delay_before_retry 3         }       }   }

7.配置 IPVS
7.1.开启IP转发功能

[root@wjb10000-backup /]# echo 1 >/proc/sys/net/ipv4/ip_forward   

7.2.配置重定向

[root@wjb10000-backup /]# echo "0" >/proc/sys/net/ipv4/ip_forward

[root@wjb10000-backup /]# echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects

[root@wjb10000-backup /]# echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects

[root@wjb10000-backup /]# echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects

7.3.清除ipvsadm表

[root@wjb10000-backup /]# ipvsadm –C

7.4.使用ipvsadm安装web服务

[root@wjb10000-backup /]# ipvsadm -A -t 192.168.1.100:80-s rr

7.5.增加二台web服务器

[root@wjb10000-backup /]# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.30:80 -m -w 1

[root@wjb10000-backup /]# ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.40:80 -m -w 1

7.6查看结果

[root@wjb10000-backup /]#  ipvsadm -L  -n

IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn TCP  192.168.1.100:80 rr   -> 192.168.1.30:80              Route   1      0          1           -> 192.168.1.40:80              Route   1      0          0

[root@wjb10000-backup /]# service keepalived restart

8.备服务器IP查看

[root@wjb10000-backup /]# ip add

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000     link/ether 00:0c:29:85:b8:f8 brd ff:ff:ff:ff:ff:ff     inet 192.168.1.20/24 brd 192.168.1.255 scope global eno16777736        valid_lft forever preferred_lft forever     inet6 fe80::20c:29ff:fe85:b8f8/64 scope link        valid_lft forever preferred_lft forever

备注：如果主备服务器都有VIP存在，请关闭selinux和防火墙

# setenforce 0

# systemctl stop firewalld.service

三.web服务器的配置
1.此处以默认安装好的web服务，如Nginx和Apache等等。只进行节点的配置步骤。简单安装apache服务，方便后面的测试。

[root@wjb10000-web1 /]# yum -y install httpd

[root@wjb10000-web1 /]# vim /var/www/html/index.html

web1 server

[root@wjb10000-web1 /]# systemctl start httpd.service

[root@wjb10000-web2 /]# yum -y install httpd

[root@wjb10000-web2 /]# vim /var/www/html/index.html

web2 server

[root@wjb10000-web2 /]# systemctl start httpd.service

2.web1服务器LVS配置：

[root@wjb10000-web1 /]# vim /etc/init.d/lvsrs

#!/bin/bash #description:start realserver vip=192.168.1.100 source /etc/rc.d/init.d/functions case $1 in start)         echo "Start Realserver"         /sbin/ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up          /sbin/route add -host $VIP dev lo:0         echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore         echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce         echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore         echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce           sysctl -p >/dev/null 2>&1 ;; stop)         echo "Stop Realserver"         /sbin/ifconfig lo:0 down          /sbin/route del $VIP >/dev/null 2>&1         echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore         echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce         echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore         echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce ;; *)         echo "Usage: $0 (start | stop)" exit 1 esac

[root@wjb10000-web1 /]# chmod 755 /etc/init.d/lvsrs

[root@wjb10000-web1 /]# service lvsrs start

3.web2服务器LVS配置：

[root@wjb10000-web2 ~]# vim /etc/init.d/lvsrs

#!/bin/bash #description:start realserver vip=192.168.1.100 source /etc/rc.d/init.d/functions case $1 in start)         echo "Start Realserver"         /sbin/ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up          /sbin/route add -host $VIP dev lo:0         echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore         echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce         echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore         echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce           sysctl -p >/dev/null 2>&1 ;; stop)         echo "Stop Realserver"         /sbin/ifconfig lo:0 down          /sbin/route del $VIP >/dev/null 2>&1         echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore         echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce         echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore         echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce ;; *)         echo "Usage: $0 (start | stop)" exit 1 esac

[root@wjb10000-web2 ~]# chmod 755 /etc/init.d/lvsrs

[root@wjb10000-web2 ~]# service lvsrs start

到此为止，Keepalived+LVS的高可用集群系统搭建完成，下面进行测试。

四.高可用keepalived功能测试
1.先停掉主服务器上的keepalived服务，看看备服务器是否接管vip

[root@wjb10000-master ~]# service keepalived stop

[root@wjb10000-master ~]# ip add

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000     link/ether 00:0c:29:74:1f:4e brd ff:ff:ff:ff:ff:ff     inet 192.168.1.10/24 brd 192.168.1.255 scope global eno16777736        valid_lft forever preferred_lft forever     inet6 fe80::20c:29ff:fe74:1f4e/64 scope link        valid_lft forever preferred_lft forever

2.查看备服务器的ip

[root@wjb10000-backup ~]# ip add

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000     link/ether 00:0c:29:85:b8:f8 brd ff:ff:ff:ff:ff:ff     inet 192.168.1.20/24 brd 192.168.1.255 scope global eno16777736        valid_lft forever preferred_lft forever     inet 192.168.1.100/32 scope global eno16777736        valid_lft forever preferred_lft forever     inet6 fe80::20c:29ff:fe85:b8f8/64 scope link        valid_lft forever preferred_lft forever

上面信息可以看到备服务器已经接管vip地址。

3.恢复主服务器keepalived服务。

[root@wjb10000-master ~]# service keepalived start

[root@wjb10000-master ~]# ip add

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000     link/ether 00:0c:29:74:1f:4e brd ff:ff:ff:ff:ff:ff     inet 192.168.1.10/24 brd 192.168.1.255 scope global eno16777736        valid_lft forever preferred_lft forever     inet 192.168.1.100/32 scope global eno16777736        valid_lft forever preferred_lft forever     inet6 fe80::20c:29ff:fe74:1f4e/64 scope link        valid_lft forever preferred_lft forever

上面信息可以看到主服务器已经重新接管vip地址。

4.再查看备服务器ip地址

[root@wjb10000-backup ~]# ip add

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000     link/ether 00:0c:29:85:b8:f8 brd ff:ff:ff:ff:ff:ff     inet 192.168.1.20/24 brd 192.168.1.255 scope global eno16777736        valid_lft forever preferred_lft forever     inet6 fe80::20c:29ff:fe85:b8f8/64 scope link        valid_lft forever preferred_lft forever

上面信息可以看到主服务器已经释放了vip地址。

五.负载均衡LVS功能测试

打开浏览器输入http://192.168.1.100.不断刷新页面分别看到web1 server和web2 server表示LVS已经负载均衡了。

六.故障切换测试
1.停掉web1服务器的httpd服务

[root@wjb10000-web1 ~]# systemctl stop httpd.service

2. 查看主服务器状态

[root@wjb10000-master ~]# ipvsadm

IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn TCP  wjb10000-master:http rr   -> 192.168.1.40:http            Route   1      0          0

3.查看备服务器状态

[root@wjb10000-backup ~]# ipvsadm

IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn TCP  192.168.1.100:http rr   -> 192.168.1.40:http            Route   1      0          0

以上内容可以看出Keepalived检测web1服务器出现故障，将此服务器从集群中移除。

4.恢复web1服务器

[root@wjb10000-web1 ~]# systemctl start httpd.service

5.再查看主服务器状态

[root@wjb10000-master ~]# ipvsadm

IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn TCP  wjb10000-master:http rr   -> 192.168.1.30:http            Route   1      0          0           -> 192.168.1.40:http            Route   1      0          0

6.再查看备服务器状态

[root@wjb10000-backup ~]# ipvsadm

IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn TCP  192.168.1.100:http rr   -> 192.168.1.30:http            Route   1      0          0            -> 192.168.1.40:http            Route   1      0          0

节点恢复重新加入集群当中，浏览器刷新页面，可以重新看到web1 server和web2 server内容交替显示。

结束语：到此Keepalived+LVS的高可用集群系统实验完全结束。大家哪里不明白的欢迎留言。