Keepalived+Haproxy双主高可用负载均衡web和mysql综合案例

4tret

日期及版本：2014.5.4v1.0

架构图

实验目的：

1.Haproxy+Keepalived双主双机高可用模型，keepalived为Haproxy主从提供高可用保证haproxy-master若挂掉haproxy-backup能无缝接管，haproxy为后端Web提供负载均衡，缓解并发压力，实现WEB站点负载均衡+高可用性；

2. Haproxy反代web做动静分离；

3. Haproxy反代mysql 算法leastconn和roundrobin的不同效果；

系统环境：

OS：centos6.4 X64

apache: httpd-2.2.15-26.el6.centos.x86_64

haproxy: haproxy-1.4.22-3.el6.x86_64

keepalived：keepalived-1.2.7

基础配置：

Role Out-IP In-IP Host

haproxy VIP: 192.168.1.10

haproxy-master(haproxy1): 192.168.1.11 172.16.100.254 ha1.example.com

haproxy-backup(haproxy2): 192.168.1.12 172.16.100.253 ha2.example.com

apache1: 172.16.100.13 web1.example.com

apache2: 172.16.100.14 web2.example.com

实验步骤：因为都是双机配置，重复内容合并

1、配置后端web1，web2按IP修改

[iyunv@web1 ~]# echo '<h1>web1.example.com</h1>' > /var/www/html/index.html

[iyunv@ web1 html]# cp index.html index.php

[iyunv@web1 html]# cat >> index.php <<EOF

?

[iyunv@web1 html]# service httpd restart

2、配置Haproxy反向代理

[iyunv@ha1 ~]# sed -i '/net.ipv4.ip_forward/ s/\(.*= \).*/\11/' /etc/sysctl.conf

net.ipv4.ip_forward = 1

[iyunv@ha1 ~]# cd /etc/haproxy/

[iyunv@ha1 haproxy]# cp haproxy.cfg haproxy.cfg.bak

用rsyslog的理由：

1.防止系统崩溃无法获取系统日志分享崩溃原因，用rsyslog可以把日志传输到远程的日志服务器上

2.使用rsyslog日志可以减轻系统压力，因为使用rsyslog可以有效减轻系统的磁盘IO

3.rsyslog使用tcp传输非常可靠，可以对日志进行过滤，提取出有效的日志，rsyslog是轻量级的日志软件，在大量日志写的情况下，系统负载基本上在0.1以下

[iyunv@ ha1 haproxy ~]# vim /etc/rsyslog.conf

?

[iyunv@ha1 haproxy]# service rsyslog restart

3、双机公共设置ha1与ha2相同设置

[iyunv@ha1 haproxy]# vim haproxy.cfg

?

在web1，web2服务器配置中日志格式添加"X-Forwarded-For"特殊首部，以便后端服务器记录真实发起请求的客户端IP地址

[iyunv@web1 ~]# vim /etc/httpd/conf/httpd.conf

搜LogFormat 第一个字节段插入%{X-Forwarded-For}i

LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

[iyunv@web1 ~]# sed -i '/^LogFormat.*combined/ s#"#"\%{X-Forwarded-For}\i#' /etc/httpd/conf/httpd.conf

[iyunv@web1 html]# service httpd reload

[iyunv@web2 html]# service httpd restart

验证静态页面的负载均衡，动态页面长连接保持于同一后端服务器，动静分离。

Haproxy反代日志会详细显示前端客户端IP,Haproxy反代IP,根据动静分离ACL选择不同后端服务器，GET到不同页面。后端upstream服务器别名

[iyunv@ha1 haproxy]# tail /var/log/haproxy.log

May 4 01:15:25 localhost haproxy[1819]: 192.168.1.108:50854 [04/May/2014:01:15:25.571] webservers dynamic_servs/dynamic2 24/0/1/2/+27 200 +177 - - ---- 2/2/2/2/0 0/0 {192.168.1.11||http://192.168.1.11/} "GET /index.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1"

May 4 01:15:25 localhost haproxy[1819]: 192.168.1.108:50855 [04/May/2014:01:15:25.572] webservers dynamic_servs/dynamic2 25/0/0/2/+27 200 +177 - - ---- 1/1/1/1/0 0/0 {192.168.1.11||http://192.168.1.11/} "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1"

May 4 01:15:28 localhost haproxy[1819]: 192.168.1.108:50856 [04/May/2014:01:15:28.552] webservers static_servs/static1 5/0/1/1/+7 304 +147 - - ---- 3/3/1/1/0 0/0 {192.168.1.11||} "GET /index.html HTTP/1.1"

May 4 01:15:29 localhost haproxy[1819]: 192.168.1.108:50857 [04/May/2014:01:15:28.556] webservers static_servs/static2 799/0/1/1/+801 200 +265 - - ---- 2/2/1/1/0 0/0 {192.168.1.11||} "GET /index.html HTTP/1.1"

[iyunv@ha2 haproxy]# tail /var/log/haproxy.log

May 4 01:38:30 localhost haproxy[1733]: 192.168.1.108:51208 [04/May/2014:01:38:30.436] webservers dynamic_servs/dynamic2 24/0/0/4/+28 200 +177 - - ---- 2/2/2/2/0 0/0 {192.168.1.12||http://192.168.1.12/} "GET /index.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1"

May 4 01:38:30 localhost haproxy[1733]: 192.168.1.108:51209 [04/May/2014:01:38:30.436] webservers dynamic_servs/dynamic2 24/0/0/4/+28 200 +177 - - ---- 2/2/2/1/0 0/0 {192.168.1.12||http://192.168.1.12/} "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1"

May 4 01:39:04 localhost haproxy[1733]: 192.168.1.108:51219 [04/May/2014:01:39:04.178] webservers static_servs/static1 88/0/1/1/+90 200 +265 - - ---- 3/3/1/1/0 0/0 {192.168.1.12||} "GET /index.html HTTP/1.1"

May 4 01:39:38 localhost haproxy[1733]: 192.168.1.108:51229 [04/May/2014:01:39:38.161] webservers static_servs/static2 8/0/2/0/+10 200 +265 - - ---- 3/3/1/1/0 0/0 {192.168.1.12||} "GET /index.html HTTP/1.1"

各个后端服务器日志记录前端客户端IP,Haproxy反代内网卡(内网关)IP,Get到不同页面，动态页面还有haproxy反代外网卡IP

[iyunv@web1 ~]# tail /var/log/httpd/access_log

192.168.1.108172.16.100.254 - - [04/May/2014:01:14:01 +0800] "GET /index.html HTTP/1.1" 304 - "-"

[iyunv@web2 ~]# tail /var/log/httpd/access_log

192.168.1.108172.16.100.253 - - [04/May/2014:01:38:29 +0800] "GET /index.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2524 "http://192.168.1.12/"

192.168.1.108172.16.100.253 - - [04/May/2014:01:38:29 +0800] "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2146 "http://192.168.1.12/"

不同http://192.168.1.1x:8080/haproxyadmin?stats访问同一侦听stats主机的指定url,端口经过身份验证，显示管理页面。

配置keepalived主从服务器，主从相同实例高低相反

[iyunv@ha1 ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

192.168.1.11 keepalived主节点配置

[iyunv@ha1 ~]# vim /etc/keepalived/keepalived.conf

?

[iyunv@ha1 keepalived]# service keepalived start

[iyunv@ha2 keepalived]# vim keepalived.conf

?

[iyunv@ha2 keepalived]# service keepalived start

[iyunv@ha1 ~]# ip addr | grep eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

inet 192.168.1.11/24 brd 192.168.1.255 scope global eth0

inet 192.168.1.88/24 scope global secondary eth0

[iyunv@ha2 ~]# ip addr | grep eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

inet 192.168.1.12/24 brd 192.168.1.255 scope global eth0

inet 192.168.1.188/24 scope global secondary eth0

[iyunv@ha1 ~]# tail /var/log/messages

May 4 04:49:26 station11 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) Transition to MASTER STATE

May 4 04:49:27 station11 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) Entering MASTER STATE

May 4 04:49:35 station11 Keepalived_vrrp[3973]: VRRP_Instance(VI_2) Entering BACKUP STATE

[iyunv@ha2 ~]# tail /var/log/messages

May 4 04:49:35 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) Entering BACKUP STATE

May 4 04:49:35 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_2) Transition to MASTER STATE

May 4 04:49:36 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_2) Entering MASTER STATE

停止ha1服务器haproxy服务，keepalived进行重新选举，ha2提升VI_1从备到主，VIP切换到备用服务器

[iyunv@ha1 ~]# service haproxy stop

[iyunv@ha2 ~]# tail /var/log/messages

May 4 04:56:00 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) forcing a new MASTER election

May 4 04:56:01 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) Transition to MASTER STATE

May 4 04:56:02 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) Entering MASTER STATE

[iyunv@ha1 ~]# ip addr | grep eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

inet 192.168.1.11/24 brd 192.168.1.255 scope global eth0

[iyunv@ha2 ~]# ip addr | grep eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

inet 192.168.1.12/24 brd 192.168.1.255 scope global eth0

inet 192.168.1.188/24 scope global secondary eth0

inet 192.168.1.88/24 scope global secondary eth0

启动ha1服务器haproxy服务，VIP重新切换回主服务器

[iyunv@ha1 ~]# service haproxy start

[iyunv@ha1 ~]# ip addr | grep eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

inet 192.168.1.11/24 brd 192.168.1.255 scope global eth0

inet 192.168.1.88/24 scope global secondary eth0

[iyunv@ha2 ~]# ip addr | grep eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

inet 192.168.1.12/24 brd 192.168.1.255 scope global eth0

inet 192.168.1.188/24 scope global secondary eth0

[iyunv@ha1 ~]# tail /var/log/messages

May 4 05:07:41 ha1 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) forcing a new MASTER election

May 4 05:07:42 ha1 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) Transition to MASTER STATE

May 4 05:07:43 ha1 Keepalived_vrrp[3973]: VRRP_Instance(VI_1) Entering MASTER STATE

[iyunv@ha2 ~]# tail /var/log/messages

May 4 05:07:41 station12 Keepalived_vrrp[3594]: VRRP_Instance(VI_1) Entering BACKUP STATE

3、反代mysql 算法leastconn和roundrobin的不同效果？

[iyunv@web1 ~]# yum -y install mysql mysql-server mysql-devel

[iyunv@web1 ~]# service mysqld start

[iyunv@web1 ~]# mysqladmin -u root password redhat

[iyunv@web1 ~]# mysql -uroot -predhat

?

station14相同

[iyunv@ha1 haproxy]# vim haproxy.cfg

?

[iyunv@ha1 haproxy]# service haproxy reload

[iyunv@ha2 ~]# mysql -umysqlchecker -h192.168.1.11

mysql> select @@hostname;

| station14.example.com |

[iyunv@ha2 ~]# mysql -umysqlchecker -h192.168.1.11

mysql> select @@hostname;

| station13.example.com |

[iyunv@ha1 haproxy]# vim haproxy.cfg

balance leastconn 修改算法为最少连接数

[iyunv@ha1 haproxy]# service haproxy reload

当前最大连接数

[iyunv@web1 ~]# mysql -uroot -predhat -e "show status like '%connections%';"

| Connections | 6607 |

| Max_used_connections | 3 |

没有大量客户端进行测试，它只计算活动连接数，哪个后端服务器的连接最少就调度到哪台。