设为首页 收藏本站

运维网

 找回密码
 立即注册
查看: 928|回复: 0

[经验分享] LVS/NAT + keepalived负载均衡实现

[复制链接]
hao0089

尚未签到
发表于 2018-12-30 11:11:11 | 显示全部楼层 |阅读模式
　　一、keepalived简介
　　keepalived是分布式部署解决系统高可用的软件，结合lvs(LinuxVirtual Server)使用，解决单机宕机的问题。
keepalived是一个基于VRRP协议来实现IPVS的高可用的解决方案。对于LVS负载均衡来说，如果前端的调度器direct发生故障，则后端的realserver是无法接受请求并响应的。因此，保证前端direct的高可用性是非常关键的，否则后端的服务器是无法进行服务的。而我们的keepalived就可以用来解决单点故障（如LVS的前端direct故障）问题。keepalived的主要工作原理是：运行keepalived的两台服务器，其中一台为MASTER，另一台为BACKUP，正常情况下，所有的数据转换功能和ARP请求响应都是由MASTER完成的，一旦MASTER发生故障，则BACKUP会马上接管MASTER的工作，这种切换时非常迅速的。
　　二、测试环境
　　下面拿4台虚拟机进行环境测试,实验环境为centos6.6 x86_64,在lvs NAT模式环境下只有前端两台keepalived调度机有公网ip，后端真实机只有内网ip，具体用途和ip如下
服务器类型

公网ip

内网ip

LVS VIP

192.168.214.70

192.168.211.254

Keepalived Master

192.168.214.76

192.168.211.76

Keepalived Backup

192.168.214.77

192.168.211.77

Realserver A


192.168.211.79

　　Realserver B

192.168.211.83

　　三、软件安装
　　1、安装lvs所需包ipvsadm
　　yum install -y ipvsadm
　　ln -s /usr/src/kernels/`uname -r`  /usr/src/linux
　　lsmod |grep ip_vs
　　#注意Centos 6.X安装lvs，使用1.26版本。并且需要先安装yuminstall libnl* popt* -y
　　执行ipvsadm(modprobe ip_vs)把ip_vs模块加载到内核
　　[root@test85 ~]# ipvsadm -L -n
　　IP Virtual Server version 1.2.1 (size=4096)
　　Prot LocalAddress:Port Scheduler Flags
　　-> RemoteAddress:Port          Forward Weight ActiveConn InActConn
　　#IP Virtual Server version 1.2.1 ---- ip_vs内核模块版本
　　2、安装keepalived
　　yum install -y keepalived
　　chkconfig keepalived on
　　注：在centos7系列系统中开机自动启动使用systemctl enable keepalived
　　四、开启路由转发
　　在lvs/NAT模式下前端两台调度机192.168.211.76和192.168.211.77上需要开启路由转发功能
　　vi /etc/sysctl.conf
　　修改net.ipv4.ip_forward = 0为net.ipv4.ip_forward= 1
　　保存退出后，使用systcl –p命令让其生效
　　五、keepalived配置
　　先看下211.76 keepalivedmaster主机上的配置
　　具体详细的参数说明见上篇LVS/DR + keepalived负载均衡实现
　　
[root@localhost ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     charles@test.com
   }
   notification_email_from ReportLog@test.com
   #smtp_server 127.0.0.1
   #smtp_connect_timeout 30
    router_id LVS_1
   #lvs_id LVS_EXAMPLE_01
}
vrrp_sync_group VG1 {
   group {
      VI_1
      VI_GATEWAY
   }
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0    #公网ip网口
    virtual_router_id 51
    priority 150
    advert_int 1
    lvs_sync_daemon_inteface eth0
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.214.70    #公网vip地址
    }
}
vrrp_instance VI_GATEWAY {
    state MASTER
    interface eth1         #内网ip网口
    lvs_sync_daemon_inteface eth1
    virtual_router_id 52
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.211.254  #内网vip地址
    }
}
virtual_server 192.168.214.70 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT         #使用lvs的NAT工作模式
    #nat_mask 255.255.255.0
    persistence_timeout 10
    protocol TCP
    real_server 192.168.211.79 80 {
        TCP_CHECK {
            connect_timeout 3
            connect_port 80
            nb_get_retry 3
            delay_before_retry 3
        }
    }
   real_server 192.168.211.83 80 {
        TCP_CHECK {
            connect_timeout 3
            connect_port 80
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}　　再看下211.77 keepalived BACKUP主机上的配置
　　
[root@localhost ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     charles@test.com
   }
   notification_email_from ReportLog@test.com
   #smtp_server 127.0.0.1
   #smtp_connect_timeout 30
   router_id LVS_1
   #lvs_id LVS_EXAMPLE_02
}
vrrp_sync_group VG1 {
   group {
      VI_1
      VI_GATEWAY
   }
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    lvs_sync_daemon_inteface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.214.70
    }
}
vrrp_instance VI_GATEWAY {
    state BACKUP
    interface eth1
    lvs_sync_daemon_inteface eth1
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.211.254
    }
}
virtual_server 192.168.214.70 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    #nat_mask 255.255.255.0
    persistence_timeout 10
    protocol TCP
real_server 192.168.211.79 80 {
        TCP_CHECK {
            connect_timeout 3
            connect_port 80
            nb_get_retry 3
            delay_before_retry 3
        }
    }
real_server 192.168.211.83 80 {
        TCP_CHECK {
            connect_timeout 3
            connect_port 80
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}　　六、后端真实机配置
　　后端两台web服务真实机192.168.211.79和192.168.211.83需要配置默认网关为vip内网ip地址192.168.211.254。
　　注意，线上环境如果更改默认网关为vip内网地址后，可能会造成连接不上服务器，需要提前添加相关静态路由。
　　七、启动keepalived服务及查看相关信息
　　在211.76和211.77上分别启动keepalived服务
　　在211.76keepalived Master主机上查看信息
　　通过ip addr可以看到vip 地址已经绑定在eth0和eth1网口
　　
[root@localhost ~]# ip addr
1: lo:  mtu65536 qdisc noqueue state UNKNOWN
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
   inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen1000
   link/ether 52:54:00:55:b2:d4 brd ff:ff:ff:ff:ff:ff
   inet 192.168.214.76/24 brd 192.168.214.255 scope global eth0
   inet 192.168.214.70/32 scope global eth0
   inet6 fe80::5054:ff:fe55:b2d4/64 scope link
      valid_lft forever preferred_lft forever
3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen1000
   link/ether 52:54:00:85:11:95 brd ff:ff:ff:ff:ff:ff
   inet 192.168.211.76/24 brd 192.168.211.255 scope global eth1
   inet 192.168.211.254/32 scope global eth1
   inet6 fe80::5054:ff:fe85:1195/64 scope link
      valid_lft forever preferred_lft forever
在211.76上查看日志信息，看到已成功进入keepalived主机模式
[root@localhost ~]# tail -f/var/log/messages
May 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_GATEWAY) setting protocol VIPs.
May 10 10:25:47 localhost Keepalived_healthcheckers[6209]:Netlink reflector reports IP 192.168.211.254 added
May 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_GATEWAY) Sending gratuitous ARPs oneth1 for 192.168.211.254
May 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Group(VG1) Syncing instances to MASTER state
May 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_1) EnteringMASTER STATE
May 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_1) setting protocol VIPs.
May 10 10:25:47 localhost Keepalived_healthcheckers[6209]:Netlink reflector reports IP 192.168.214.70 added
May 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.70
May 10 10:25:52 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_GATEWAY) Sendinggratuitous ARPs on eth1 for 192.168.211.254
May 10 10:25:52 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for192.168.214.70　　在211.77上查看日志信息，看到已成功进入keepalived备机模式
　　
[root@localhost ~]# tail -f/var/log/messages
May 10 10:27:54 localhostKeepalived_vrrp[6249]: Opening file '/etc/keepalived/keepalived.conf'.
May 10 10:27:54 localhostKeepalived_vrrp[6249]: Configuration is using : 69554 Bytes
May 10 10:27:54 localhostKeepalived_vrrp[6249]: Using LinkWatch kernel netlink reflector...
May 10 10:27:54 localhostKeepalived_vrrp[6249]: VRRP_Instance(VI_1) Entering BACKUP STATE
May 10 10:27:54 localhostKeepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Entering BACKUP STATE
May 10 10:27:54 localhost Keepalived_vrrp[6249]:VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 10 10:27:54 localhostKeepalived_vrrp[6249]: VRRP sockpool: [ifindex(3), proto(112), unicast(0),fd(12,13)]
May 10 10:27:54 localhostKeepalived_healthcheckers[6248]: Using LinkWatch kernel netlink reflector...
May 10 10:27:54 localhostKeepalived_healthcheckers[6248]: Activating healthchecker for service[192.168.211.79]:80
May 10 10:27:54 localhostKeepalived_healthcheckers[6248]: Activating healthchecker for service[192.168.211.83]:80　　通过ipvsadm -L –n 查看相应lvs连接信息
[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.214.70:80 rr persistent 10
  -> 192.168.211.79:80            Masq    1      0          0         
  -> 192.168.211.83:80            Masq    1      3          0

　　八、keepalived测试
　　使用vip地址192.168.214.70访问后端192.168.211.79和192.168.211.83的页面



　　正常访问没问题后，我们来模拟lvs集群故障，让主备进行自动切换下
　　首先把keepalived master主机211.76宕机，看备机能否接管过来，vip地址是否会漂移过来
　　在211.77上查看日志，发现备机成功切换成了主机状态，vip地址成功漂移了过来
　　[root@localhost ~]# ip addr
　　1: lo:  mtu65536 qdisc noqueue state UNKNOWN
　　link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
　　inet 127.0.0.1/8 scope host lo
　　inet6 ::1/128 scope host
　　valid_lft forever preferred_lft forever
　　2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen1000
　　link/ether 52:54:00:1b:a2:11 brd ff:ff:ff:ff:ff:ff
　　inet 192.168.214.77/24 brd 192.168.214.255 scope global eth0
　　inet 192.168.214.70/32 scope global eth0
　　inet6 fe80::5054:ff:fe1b:a211/64 scope link
　　valid_lft forever preferred_lft forever
　　3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
　　link/ether 52:54:00:64:47:7d brd ff:ff:ff:ff:ff:ff
　　inet 192.168.211.77/24 brd 192.168.211.255 scope global eth1
　　inet 192.168.211.254/32 scope global eth1
　　inet6 fe80::5054:ff:fe64:477d/64 scope link
　　valid_lft forever preferred_lft forever
[root@localhost ~]# tail -f /var/log/messages
May 10 10:27:54 localhost Keepalived_vrrp[6249]: Opening file '/etc/keepalived/keepalived.conf'.
May 10 10:27:54 localhost Keepalived_vrrp[6249]: Configuration is using : 69554 Bytes
May 10 10:27:54 localhost Keepalived_vrrp[6249]: Using LinkWatch kernel netlink reflector...
May 10 10:27:54 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Entering BACKUP STATE
May 10 10:27:54 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Entering BACKUP STATE
May 10 10:27:54 localhost Keepalived_vrrp[6249]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 10 10:27:54 localhost Keepalived_vrrp[6249]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(12,13)]
May 10 10:27:54 localhost Keepalived_healthcheckers[6248]: Using LinkWatch kernel netlink reflector...
May 10 10:27:54 localhost Keepalived_healthcheckers[6248]: Activating healthchecker for service [192.168.211.79]:80
May 10 10:27:54 localhost Keepalived_healthcheckers[6248]: Activating healthchecker for service [192.168.211.83]:80
May 10 11:09:26 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 10 11:09:26 localhost Keepalived_vrrp[6249]: VRRP_Group(VG1) Syncing instances to MASTER state
May 10 11:09:26 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Transition to MASTER STATE
May 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Entering MASTER STATE
May 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) setting protocol VIPs.
May 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Sending gratuitous ARPs on eth1 for 192.168.211.254
May 10 11:09:27 localhost Keepalived_healthcheckers[6248]: Netlink reflector reports IP 192.168.211.254 added
May 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Entering MASTER STATE
May 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) setting protocol VIPs.
May 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.70
May 10 11:09:27 localhost Keepalived_healthcheckers[6248]: Netlink reflector reports IP 192.168.214.70 added
May 10 11:09:32 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Sending gratuitous ARPs on eth1 for 192.168.211.254
May 10 11:09:32 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Sending　　如果想了解更多，请关注我们的公众号
公众号ID:opdevos

　　扫码关注





运维网声明 1、欢迎大家加入本站运维交流群:群②:261659950 群⑤:202807635 群⑦870801961 群⑧679858003
2、本站所有主题由该帖子作者发表，该帖子作者与运维网享有帖子相关版权
3、所有作品的著作权均归原作者享有，请您和我们一样尊重他人的著作权等合法权益。如果您对作品感到满意，请购买正版
4、禁止制作、复制、发布和传播具有反动、淫秽、色情、暴力、凶杀等内容的信息，一经发现立即删除。若您因此触犯法律，一切后果自负，我们对此不承担任何责任
5、所有资源均系网友上传或者通过网络收集，我们仅提供一个展示、介绍、观摩学习的平台，我们不对其内容的准确性、可靠性、正当性、安全性、合法性等负责，亦不承担任何法律责任
6、所有作品仅供您个人学习、研究或欣赏，不得用于商业或者其他用途，否则，一切后果均由您自己承担，我们对此不承担任何法律责任
7、如涉及侵犯版权等问题，请您及时通知我们，我们将立即采取措施予以解决
8、联系人Email：admin@iyunv.com 网址：www.yunweiku.com

所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其承担任何法律责任,如涉及侵犯版权等问题,请您及时通知我们,我们将立即处理,联系人Email:kefu@iyunv.com,QQ:1061981298 本贴地址：https://www.yunweiku.com/thread-657532-1-1.html 上篇帖子： ipvs 下篇帖子： keepalived高可用集群学习以及实验总结
回复

使用道具 举报

返回列表
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

扫码加入运维网微信交流群X

扫码加入运维网微信交流群

扫描二维码加入运维网微信交流群，最新一手资源尽在官方微信交流群！快快加入我们吧...

扫描微信二维码查看详情

客服 E-mail:iyunvcom@gmail.com

本站由青云提供云计算服务

运维网--中国最专业的运维工程师交流社区

豫ICP备20007574号-1 Copyright © 2012-2025

关于我们

发展历程

联系我们

关注我们

广告服务

广告案例

隐私条款

免责声明

用户登录

用户注册

版主守则

申请版主

手机版

金币获取

帮助中心

每日签到

Good Good

Study

Day Day

UP


客服E-mail:kefu@iyunv.com 客服QQ：1061981298


QQ群⑦：运维网交流群⑦ QQ群⑧：运维网交流群⑧ k8s群：运维网kubernetes交流群


提醒：禁止发布任何违反国家法律、法规的言论与图片等内容；本站内容均来自个人观点与网络等信息，非本站认同之观点.


本站大部分资源是网友从网上搜集分享而来，其版权均归原作者及其网站所有，我们尊重他人的合法权益，如有内容侵犯您的合法权益，请及时与我们联系进行核实删除！



合作伙伴: 青云cloud

快速回复 返回顶部 返回列表