haproxy+keepalived应用实战

lanying56123

　　1、四层与七层负载均衡的区别

　　所谓的四层就是 ISO 参考模型中的第四层。四层负载均衡也称为四层交换机，它主要是通过分析 IP 层及 TCP/UDP 层的流量实现的基于 IP 加端口的负载均衡。常见的基于四层的负
　　载均衡器有 LVS、 F5 等。以常见的 TCP 应用为例，负载均衡器在接收到第一个来自客户端的 SYN请求时，会通过设定的负载均衡算法选择一个最佳的后端服务器，同时将报文中目标 IP地址修改为后端服务器IP，然后直接转发给该后端服务器，这样一个负载均衡请求就完成了。从这个过程来看，一个TCP连接是客户端和服务器直接建立的，而负载均衡器只不过完成了一个类似路由器的转发动作。在某些负载均衡策略中，为保证后端服务器返回的报文可以正确传递给负载
　　均衡器，在转发报文的同时可能还会对报文原来的源地址进行修改
　　

　　同理，七层负载均衡器也称为七层交换机，位于 OSI 的最高层，即应用层，此时负载均衡器支持多种应用协议，常见的有 HTTP、SMTP等。七层负载均衡器可以根据报文内容，再配合负载均衡算法来选择后端服务器，因此也称为“内容交换器”。比如，对于 Web服务器的负载均衡，七层负载均衡器不但可以根据“ IP+端口”的方式进行负载分流，还可以根据网站的 URL、访问域名、浏览器类别、语言等决定负载均衡的策略。例如，有两台
　　Web 服务器分别对应中英文两个网站，两个域名分别是 A、 B，要实现访问 A 域名时进入中文网站，访问 B 域名时进入英文网站，这在四层负载均衡器中几乎是无法实现的，而七层负
　　载均衡可以根据客户端访问域名的不同选择对应的网页进行负载均衡处理。常见的七层负载
　　均衡器有 HAproxy、 Nginx 等。这里仍以常应用为例，由于负载均衡器要获取到报文的内容，因此只能先代替后端服务器和客户端建立连接，接着，才能收到客户端发送过来的报文内容，然后再根据该
　　报文中特定字段加上负载均衡器中设置的负载均衡算法来决定最终选择的内部服务器。纵观整个过程，七层负载均衡器在这种情况下类似于一个代理服务器。
　　

　　

　　七层负载均衡代理实现原理
　　对比四层负载均衡和七层负载均衡运行的整个过程，可以看出，在七层负载均衡模式下，负载均衡器与客户端及后端的服务器会分别建立一次 TCP 连接，而在四层负载均衡模式下，
　　仅建立一次 TCP 连接。由此可知，七层负载均衡对负载均衡设备的要求更高，而七层负载均衡的处理能力也必然低于四层模式的负载均衡。
　　

　　环境规划 centos6.7
　　172.16.80.132   haproxy+keepalived  master   vip  172.16.80.199

　　172.16.80.126   haproxy+keepalived  slave
　　172.16.80.125   提供web测试（apache  8080      nginx 80）

　　172.16.80.128   作为客户端访问测试
　　

　　2、haproxy  keepalived 安装
[root@Master-drbd ansible]# tree
.
├── hosts
├── roles
│ ├── haproxy
│ │ ├── defaults
│ │ ├── files
│ │ │ ├── haproxy-1.5.4-3.el6.x86_64.rpm
│ │ │ └── install_haproxy.sh
│ │ ├── handles
│ │ │ └── main.yml
│ │ ├── meta
│ │ ├── tasks
│ │ │ └── main.yml
│ │ ├── templates
│ │ │ └── haproxy.cfg.j2
│ │ └── vars
│ ├── keepalived-master
│ │ ├── defaults
│ │ ├── files
│ │ │ ├── install_keepalived_master.sh
│ │ │ ├── keepalived-1.2.13-5.el6_6.x86_64.rpm
│ │ │ ├── lm_sensors-libs-3.1.1-17.el6.x86_64.rpm
│ │ │ └── net-snmp-libs-5.5-57.el6_8.1.x86_64.rpm
│ │ ├── handles
│ │ │ └── main.yml
│ │ ├── meta
│ │ ├── tasks
│ │ │ └── main.yml
│ │ ├── templates
│ │ │ └── keepalived.conf.j2
│ │ └── vars
│ ├── keepalived-slave
│ │ ├── defaults
│ │ ├── files
│ │ │ ├── install_keepalived_slave.sh
│ │ │ ├── keepalived-1.2.13-5.el6_6.x86_64.rpm
│ │ │ ├── lm_sensors-libs-3.1.1-17.el6.x86_64.rpm
│ │ │ └── net-snmp-libs-5.5-57.el6_8.1.x86_64.rpm
│ │ ├── handles
│ │ │ └── main.yml
│ │ ├── meta
│ │ ├── tasks
│ │ │ └── main.yml
│ │ ├── templates
│ │ │ └── keepalived.conf.j2
│ │ └── vars
│ └── zabbix-agent
│     ├── defaults
│     ├── files
│     │ ├── install_zabbix_agent.sh
│     │ ├── zabbix-agent-3.0.4-1.el6.x86_64.rpm
│     │ ├── zabbix-release-3.0-1.el7.noarch.rpm
│     │ └── zabbix.repo
│     ├── handlers
│     │ └── main.yml
│     ├── meta
│     ├── tasks
│     │ └── main.yml
│     ├── templates
│     │ └── zabbix_agentd.conf.j2
│     └── vars
│         └── all
├── site.yml
└── zabbix_host.yml
33 directories, 30 files

[root@Master-drbd ansible]# ls
hosts  roles  site.yml  zabbix_host.yml
[root@Master-drbd ansible]# cat hosts
[zabbix-agent]
172.16.80.132
172.16.80.126
[haproxy]
172.16.80.132
172.16.80.126
[keepalived-master]
172.16.80.132
[keepalived-slave]
172.16.80.126
[root@Master-drbd ansible]# cat site.yml
- name: install zabbix agent
  hosts: zabbix-agent
  remote_user: root
  roles:
    - zabbix-agent
- name: install haproxy
  hosts: haproxy
  remote_user: root
  roles:
    - haproxy
- name: install keepalived
  hosts: keepalived-master,keepalived-slave
  remote_user: root
  roles:
    - keepalived-master
    - keepalived-slave
[root@Master-drbd ansible]# ansible-playbook -i hosts site.yml
PLAY [install zabbix agent] ****************************************************
TASK [setup] *******************************************************************
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [zabbix-agent : cp zabbix-agent-3.0.4-1.el6.x86_64.rpm to all client] *****
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [zabbix-agent : cp install_zabbix_agent.sh to all client] *****************
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [zabbix-agent : execute scripts to install zabbix agent] ******************
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [zabbix-agent : configure zabbix agent conf] ******************************
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [zabbix-agent : Start zabbix agent] ***************************************
ok: [172.16.80.132]
ok: [172.16.80.126]
PLAY [install haproxy] *********************************************************
TASK [setup] *******************************************************************
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [haproxy : cp haproxy-1.5.4-3.el6.x86_64.rpm to all client] ***************
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [haproxy : cp install_haproxy.sh to all client] ***************************
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [haproxy : execute scripts to install zabbix agent] ***********************
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [haproxy : configure haproxy conf] ****************************************
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [haproxy : Start haproxy] *************************************************
ok: [172.16.80.132]
ok: [172.16.80.126]
PLAY [install keepalived] ******************************************************
TASK [setup] *******************************************************************
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [keepalived-master : cp keepalived-1.2.13-5.el6_6.x86_64.rpm to all client] ***
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-master : cp lm_sensors-libs-3.1.1-17.el6.x86_64.rpmm to all client] ***
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-master : cp net-snmp-libs-5.5-57.el6_8.1.x86_64.rpm to all client] ***
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-master : cp install_keepalived_master.sh to all client] *******
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-master : execute scripts to install zabbix agent] *************
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-master : configure keepalived conf] ***************************
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-master : Start keepalived] ************************************
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-slave : cp keepalived-1.2.13-5.el6_6.x86_64.rpm to all client]
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [keepalived-slave : cp lm_sensors-libs-3.1.1-17.el6.x86_64.rpmm to all client] ***
ok: [172.16.80.126]
ok: [172.16.80.132]
TASK [keepalived-slave : cp net-snmp-libs-5.5-57.el6_8.1.x86_64.rpm to all client] ***
ok: [172.16.80.132]
ok: [172.16.80.126]
TASK [keepalived-slave : cp install_keepalived_slave.sh to all client] *********
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-slave : execute scripts to install zabbix agent] **************
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-slave : configure keepalived conf] ****************************
changed: [172.16.80.132]
changed: [172.16.80.126]
TASK [keepalived-slave : Start keepalived] *************************************
ok: [172.16.80.132]
ok: [172.16.80.126]
PLAY RECAP *********************************************************************
172.16.80.126              : ok=27   changed=12   unreachable=0    failed=0   
172.16.80.132              : ok=27   changed=12   unreachable=0    failed=0　　

　　

　　3、实际配置
主节点
[root@haproxy_master keepalived]# cat /etc/haproxy/haproxy.cfg
# Global settings
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    retries                 3
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout check           10s
    maxconn                 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  www
    bind *:80
    mode http
    option httplog
    option forwardfor
    option httpclose
    log global
    default_backend             app1
backend app1
    mode http
#    option redispatch
    option abortonclose
    balance     roundrobin
    option httpchk GET /
    server api-gw-1 172.16.80.125:80 check inter 20 rise 2 fall 5
    server api-gw-2 172.16.80.125:8080 check inter 2000 rise 2 fall 5
listen stats
  mode http
  bind 0.0.0.0:8888
  stats enable
  stats refresh 10s
  stats auth martin:123456
  stats uri /admin
  stats realm Global\ statistics
  
  [root@haproxy_master keepalived]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id haproxy_master
}
vrrp_script check_haproxy {
#   script "killall -0 haproxy"
script "/etc/keepalived/check_haproxy.sh"
   interval 3
}
vrrp_instance haproxy_ha {
    state BACKUP
    interface eth0
    virtual_router_id 80
    priority 100
    advert_int 2
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
    check_haproxy
}
    virtual_ipaddress {
       172.16.80.199/24
    }
}
[root@haproxy_slave keepalived]# cat check_haproxy.sh
#!/bin/bash
A=`ps -C haproxy --no-header |wc -l`
if [ $A -eq 0 ];then
  /etc/init.d/haproxy start
  sleep 3
  if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then
    /etc/init.d/keepalived stop
  fi
fi　　

　　备用节点
[root@haproxy_slave keepalived]# cat /etc/haproxy/haproxy.cfg
# Global settings
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    retries                 3
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout check           10s
    maxconn                 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  www
    bind *:80
    mode http
    option httplog
    option forwardfor
    option httpclose
    log global
    default_backend             app1
backend app1
    mode http
#    option redispatch
    option abortonclose
    balance     roundrobin
    option httpchk GET /
    server api-gw-1 172.16.80.125:80 check inter 20 rise 2 fall 5
    server api-gw-2 172.16.80.125:8080 check inter 2000 rise 2 fall 5
listen stats
  mode http
  bind 0.0.0.0:8888
  stats enable
  stats refresh 10s
  stats auth martin:123456
  stats uri /admin
  stats realm Global\ statistics
  
  [root@haproxy_slave keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id haproxy_slave
}
vrrp_script check_haproxy {
#########   script "killall -0 haproxy"
script "/etc/keepalived/check_haproxy.sh"
   interval 3
}
vrrp_instance haproxy_ha {
    state BACKUP
    interface eth0
    virtual_router_id 80
    priority 80
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
    check_haproxy
}
    virtual_ipaddress {
       172.16.80.199/24
    }
}　　

　　浏览器访问测试

　　

在客户端访问vip地址测试