用NginX+keepalived实现高可用的负载均衡

efn阿克说

一、 规划和准备：

用途	IP
MASTER	10.1.1.100
BACKUP	10.1.1.150

两台接入服务器公用一个虚拟IP（VIP）：10.1.1.200
100\150两个主机配置虚拟IP:
# vi /etc/sysconfig/network-scripts/ifcfg-eth2:0      DEVICE=eth2:0      TYPE=Ethernet      ONBOOT=yes      BOOTPROTO=static      IPADDR=10.1.1.200      NETMASK=255.255.255.0# service network restart二、 安装：
两台接入服务器分别安装NginX和keepalived:
· 准备依赖包：

# yum -y install gcc pcre-devel zlib-devel openssl-devel# tar zxvf nginx-1.6.1.tar.gz
# cd nginx-1.6.1/
# ./configure --user=nginx --group=nginx --prefix=/usr/local/nginx --with-http_stub_status_module--with-http_ssl_module --with-http_gzip_static_module
# make && make install      
· 安装keepalived：
# tar zxvf keepalived-1.2.13.tar.gz
# ./configure --prefix=/usr/local/keepalived
# make && make install
# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
# chmod 755 /etc/init.d/keepalived
# chkconfig --add keepalived
# chkconfig keepalived on
# mkdir /etc/keepalived
# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/
# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
在configure正确的执行后，可以得到如下的输出：
Keepalived configuration
------------------------
Keepalived version       : 1.2.13
Compiler                 : gcc
Compiler flags           : -g -O2
Extra Lib                : -lpopt -lssl -lcrypto
Use IPVS Framework       : No
IPVS sync daemon support : No
Use VRRP Framework       : Yes
Use Debug flags          : No
Use VRRP Framwork VRRP框架，这基本上是必须的，Keepalived的核心进程vrrpd。· 加入启动服务
echo "/usr/local/nginx/sbin/nginx" >> /etc/rc.localecho "/etc/init.d/keepalived start" >> /etc/rc.local三、 配置：
　　

3.1 配置NginX
两台接入服务器的NginX的配置完全一样,主要是配置/usr/local/nginx/conf/nginx.conf的http。其中多域名指向是通过虚拟主机（配置http下面的server）实现；同一域名的不同虚拟目录通过每个server下面的不同location实现；到后端的服务器在http下面配置upstream,然后在server或location中通过proxypass引用。要实现前面规划的接入方式，http的配置如下：
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    upstream www.123.com {
      ip_hash;
      server 10.1.1.100:80;
      server 10.1.1.150:80;
    }
server {
      listen      80;
     server_name www.123.com;
      root  /data/www/html;
      index index.php index.html index.htm;

    location / {
      proxy_pass http://www.123.com;# 反向代理
      proxy_set_header        X-Real-IP       $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                       }      
    location  /nginx_status {
      stub_status on; #Nginx 状态监控配置
      access_log off;
                            }
       }
    }
验证方法：

·        首先用IP访问前表中各个应用服务器的url
·        再用域名和路径访问前表中各个应用系统的域名/虚拟路径


3.2 配置keepalived
按照上面的安装方法，keepalived的配置文件在/etc/keepalived/keepalived.conf。主、从服务器的配置相关联但有所不同。如下：


修改MASTER的keepalived.conf:
#vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs { #指定keepalived在发生切换时需要发送email到的对象，一行一个
   notification_email {
     root@localhost
   }
   notification_email_from root@localhost  #指定发件人
   smtp_server 127.0.0.1   #指定smtp服务器地址
   smtp_connect_timeout 30   #指定smtp连接超时时间
   router_id MY_KEEPALIVED   #运行keepalived机器的一个标识
}
vrrp_script chk_http_port {         
        script "/opt/nginx_pid.sh" ###监控脚本  
        interval 2 ###监控时间  
        weight 2  
        }  
vrrp_instance VI_1 {  
        state MASTER ### 设置为主  
        interface eth0 ### 监控网卡  
        virtual_router_id 51 ### 这个两台服务器必须一样  
        priority 101 ### 权重值MASTRE一定要高于BAUCKUP  
        authentication {  
        auth_type PASS ### 加密  
        auth_pass test ### 加密的密码，两台服务器一定要一样，不然会出错  
        }  
        track_script {  
        chk_http_port ### 执行监控的服务  
        }  
        virtual_ipaddress {                  
        10.1.1.200 ### VIP 地址         
        }  
}
修改BACKUPkeepalived.conf:
#vim /etc/keepalived/keepalived.conf  
! Configuration File for keepalived
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from root@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id MY_KEEPALIVED
}
vrrp_script chk_http_port {  
        script "/opt/nginx_pid.sh"  
        interval 2  
        weight 2  
}  
vrrp_instance VI_1 {  
        state BACKUP ### 设置为备份机  
        interface eth0  
        virtual_router_id 51 ### 与 MASTRE 设置值一样  
        priority 80 ### 比 MASTRE权重值低  
        authentication {  
        auth_type PASS  
        auth_pass test ### 密码 与 MASTRE 一样  
        }  
        track_script {  
        chk_http_port  
        }  
        virtual_ipaddress {  
        10.1.1.200
        }  
}
3.3 编写监控nginx监控脚本
vim /opt/nginx_pid.sh
#!/bin/bash
# varsion 0.0.2
# 根据一网友说这样做不科学，如果nginx服务起来了，但是我把keepalived 杀掉了，我的理由是，如果nginx死掉了，我觉得就很难在起来，再有就是nagios 当然要给你报警了啊。不过这位同学说的有道理，所以就稍加改了一下脚本
A=`ps -C nginx --no-header |wc -l`                ## 查看是否有 nginx进程 把值赋给变量A
if [ $A -eq 0 ];then                                         ## 如果没有进程值得为 零
                /usr/local/nginx/sbin/nginx
                sleep 3
                if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then
                       killall keepalived                        ## 则结束 keepalived 进程
                fi
fi
3.4 测试:
分别在两个服务器启动和日志信息可通过查看监控的日志
# cat /var/log/messages
监控 Nginx Mastaer 的日志
[plain] view plaincopy
May 12 17:33:44 localhost Keepalived_vrrp: Configuration is using : 35676 Bytes  
May 12 17:33:44 localhost Keepalived: Starting VRRP child process, pid=1245  
May 12 17:33:44 localhost Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(8,9)]  
May 12 17:33:45 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE  
May 12 17:33:46 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE  
May 12 17:33:46 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.  
May 12 17:33:46 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.1.200  
May 12 17:33:46 localhost avahi-daemon[2344]: Registering new address record for 10.1.1.200 on eth0.  
May 12 17:33:46 localhost Keepalived_vrrp: Netlink reflector reports IP 10.1.1.200 added  
监控 Nginx Backup 的日志
[plain] view plaincopy
May 11 22:28:21 localhost Keepalived: Starting Keepalived v1.1.15 (05/11,2010)  
May 11 22:28:21 localhost Keepalived_vrrp: Using MII-BMSR NIC polling thread...  
May 11 22:28:21 localhost Keepalived_vrrp: Registering Kernel netlink reflector  
May 11 22:28:21 localhost Keepalived_vrrp: Registering Kernel netlink command channel  
May 11 22:28:21 localhost Keepalived_vrrp: Registering gratutious ARP shared channel  
May 11 22:28:21 localhost Keepalived: Starting VRRP child process, pid=27040  
May 11 22:28:21 localhost Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.  
May 11 22:28:21 localhost Keepalived_vrrp: Configuration is using : 35538 Bytes  
May 11 22:28:21 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE  
May 11 22:28:21 localhost Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(7,8)]  
May 11 22:28:23 localhost Keepalived_vrrp: VRRP_Script(chk_http_port) succeeded  
看日志可以看出，两台服务器的 MASTRE 和 BACUKUP 已经都正常了
现在我们把Master的Nginx停掉.查看Backup的日志
[plain] view plaincopy
May 11 22:28:21 localhost Keepalived: Starting VRRP child process, pid=27040  
May 11 22:28:21 localhost Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.  
May 11 22:28:21 localhost Keepalived_vrrp: Configuration is using : 35538 Bytes  
May 11 22:28:21 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE  
May 11 22:28:21 localhost Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(7,8)]  
May 11 22:28:23 localhost Keepalived_vrrp: VRRP_Script(chk_http_port) succeeded  
May 11 22:29:25 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE  
May 11 22:29:26 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE  
May 11 22:29:26 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.  
May 11 22:29:26 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.1.200
可以看出backup已经变成master
现在再启动原来master的nginx,再查看backup的日志
[plain] view plaincopy
May 11 22:30:32 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert  
May 11 22:30:32 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE  
May 11 22:30:32 localhost Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.  
May 11 22:30:32 localhost avahi-daemon[2409]: Withdrawing address record for 10.1.1.200 on eth0.  
可以看出又变回了backup.
四、还可以做什么
对于简单重复性劳动，人总是容易犯错，这种事情最好交给机器去做。比如，在这个案例中，作为统一接入服务器，可能经常要修改nginx的配置、nginx下面的html文件等。而且，一定要保证集群中的每台服务器的配置相同。最好的做法是由配置管理服务器来管理，如果没有，也可以使用简单的linux文件同步来解决。
五、SSL配置
在nginx/conf下生成秘钥：

-rand -genkey -out myRSA.key -des3 --new -x509 -days  -key cert.key -  cert.*
#生成免密码文件openssl rsa -in cert.key -out cert.key.unsecure
如果要启用SSL，在nginx中进行如下配置：

这里是的相关配置