LVS+Keepalived实现四层负载及高可用

ms133

　　

　　最近要上线一个新项目，需要软负载，本来决定用nginx做反向代理，但装完nginx以后，发现nginx无法显示css、js，目前还在研究中，但由于项目较急，所以依然采用了经典架构LVS+Keepalived来实现。
　　环境：
　　LVS-01 192.168.199.117 MASTER
　　LVS-02 192.168.199.118 BACKUP
　　WEB-01 192.168.199.113
　　WEB-02 192.168.199.114
　　VIP 192.168.199.119
　　编译安装LVS
ln -s /usr/src/kernels/2.6.18-308.el5-x86_64/ /usr/src/linux
wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.26.tar.gz
tar zxf ipvsadm-1.26.tar.gz
cd ipvsadm-1.26
make
　　发现报错，报错结果如下：
make[1]: *** [libipvs.o] Error1
make[1]: Leaving directory`/root/tools/ipvsadm-1.26/libipvs'
make: *** [libs] Error 2

经查询，有好多种说法，均没有解决我的问题：
1、  有的说是需要做内核软链接 ln-s /usr/src/kernels/2.6.18-308.el5-x86_64/ /usr/src/linux
2、  有的说是要yuminstall gcc-*
3、还有的说是yum installlibnl* popt*
以上方法可能适用于其实环境，但是在我的机器上面均无效。折腾了好久，请教高手，又是查看http://www.linuxvirtualserver.org/Documents.html官方说明文档，最终认为版本与内核不匹配，总结为以下2点：
1、  ipvsadm-1.26版本适合kernel 2.6.28-rc3及以后
2、  选择好相应的ipvsadm后，还要选择相对应的keepalived，否则会出现启动keepalived后，没有realserver列表。大致显示就是这样：
IP Virtual Server version 1.2.1(size=4096)
Prot LocalAddress:PortScheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

　　好了，问题找到了，重新操作，下载低版本的ipvsadm，我的内核版本为2.6.18-308.el5-x86_64，用ipvsadm-1.24测试正常，keepalived的版本为keepalived-1.1.15，具体操作如下：
安装LVS
ln -s /usr/src/kernels/2.6.18-308.el5-x86_64/ /usr/src/linux
wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz
tar zxvf ipvsadm-1.24.tar.gz
cd ipvsadm-1.24
make
make install
cd ..
ipvsadm –version

安装keepalived
wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz
tar zxfkeepalived-1.1.15.tar.gz
cd keepalived-1.1.15
./configure
make
make install
cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/
cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived/
cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/sbin/keepalived /usr/sbin/
/etc/init.d/keepalived start
至此，安装完毕，下面配置文件。
　　LVS-01上面keepalived的配置文件，主与备基本一致，不同的为红色部分
vim /etc/keepalived/keepalived.conf
! Configuration File forkeepalived
global_defs {
   router_id LVS-01        #routeid
}
vrrp_instance LVS {
    state MASTER          #备上为BACKUP
    interface eth0
    virtual_router_id 251
    priority 100                #备节点上比主的数值要小
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.199.119
    }
}
virtual_server 192.168.199.11980 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    persistence_timeout 60           #会话保持
    protocol TCP

    real_server 192.168.199.113 80 {
        weight 1
        TCP_CHECK {
        connect_timeout 10
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
}
    }
    real_server 192.168.168.199.114 80 {
        weight 1
        TCP_CHECK {
        connect_timeout 10
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
        }
     }
}
　　启动keealived
/etc/init.d/keepalived restart
　　检查是否有VIP
[root@localhost ipvsadm-1.26]#ip add
1: lo: mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:70:47:f4 brdff:ff:ff:ff:ff:ff
    inet 192.168.199.117/24 brd 192.168.199.255scope global eth0
    inet 192.168.199.119/32 scope global eth0
    inet6 fe80::20c:29ff:fe70:47f4/64 scopelink
       valid_lft forever preferred_lft forever
3: sit0:  mtu 1480qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
　　检查是否有realserver列表
[root@localhost ipvsadm-1.26]#ipvsadm -Ln
IP Virtual Server version 1.2.1(size=4096)
Prot LocalAddress:PortScheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.199.119:80 wrr
  -> 192.168.199.113:80           Route   1     0          0
  -> 192.168.199.114:80           Route   1     0          0
　　最后别忘了，有realserver上面配置VIP和ARP抑制，这个用脚本就好啦，具体如下：
　　[root@localhost tools]# cat realserver.sh
#!/bin/sh
[ -f"/etc/init.d/functions" ] && . /etc/init.d/functions
VIP=(
192.168.199.119
)
start(){
#bind VIP
for ((i=0;i/dev/null2>&1
done
#ARP restrain
echo "1">/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1">/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/all/arp_announce
}

stop(){
for ((i=0;i/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0">/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0">/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0">/proc/sys/net/ipv4/conf/all/arp_announce
}

case "$1" in
start)
start
if [ $? -eq 0 ];then
   action "ipvs start sucess"  /bin/true
else
   action "ipvs start failed "/bin/false
fi
;;
stop)
stop
if [ $? -eq 0 ];then
   action "ipvs stop sucess"  /bin/true
else
   action "ipvs stop failed "/bin/false
fi
;;
*)
echo "USAGE:$0{start|stop}"
esac

执行完成后，查看VIP
[root@localhost tools]# ip add
1: lo: mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 192.168.199.119/32 brd 192.168.199.119scope global lo:0
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:41:7c:7e brdff:ff:ff:ff:ff:ff
    inet 192.168.199.114/24 brd 192.168.199.255scope global eth0
    inet6 fe80::20c:29ff:fe41:7c7e/64 scopelink
       valid_lft forever preferred_lft forever
3: sit0:  mtu 1480qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0
　　剩下的就是故障模拟与测试了，在此就不多写了。