用lvs+keepalive构建高可用的后端nginx+tomcat

765435

nginx和tomcat实现动静分离，加快访问网站的速度。
工作流程就是
    lvs---->keepalive---->nginx负载均衡---->tomcat
准备四台服务器：
1.   web1 :192.168.4.10
2.   web2 :192.168.4.11
3.   keep_master :192.168.4.20
4.   kepp_slave :192.168.4.21

web1和web2上： (web2一样)
nginx安装过程省略...
1.  配置ngnix
[iyunv@web1 opt]# vim /usr/local/nginx/conf/nginx.conf
user  nginx nginx;
worker_processes  1;
error_log  logs/error.log  info;
pid        logs/nginx.pid;
worker_rlimit_nofile 65535;
events {
    worker_connections  1024;
    use epoll;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    #tcp_nopush     on;
    server_names_hash_bucket_size 128;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 32k;
    client_max_body_size 300m;
    sendfile        on;
    tcp_nopush     on;
    keepalive_timeout  65;
    tcp_nodelay   on;
    server_tokens off;
    client_body_buffer_size 512k;
    proxy_connect_timeout 5;
    proxy_send_timeout 60;
    proxy_read_timeout 5;
    proxy_buffer_size  16k;
    proxy_buffers      4 64k;
    proxy_busy_buffers_size 128k;
    proxy_temp_file_write_size 128k;

    #keepalive_timeout  0;
    gzip  on;
    gzip_min_length 1k;
    gzip_buffers 4 16k;
    gzip_http_version 1.1;
    gzip_comp_level 2;
    gzip_types text/plain application/x -javascript text/css application/xml;
    gzip_vary on;

    server {
        listen       80;
        server_name  www.test123.com;
        index  index.html index.htm index.jsp index.do;
         root  /var/www/html;
        error_page  404              /404.html;
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
            }   
    location ~ \.(jsp|jspx|do)?$ {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP   $remote_addr;
        proxy_pass http://127.0.0.1:8080;
        }
        location ~ .*\.(htm|html|gif|jpg|jpeg|png|bmp|swf|zip|rar|doc|mp3|pdf)
        {  expires   30d;
              }
        location ~ .*\.(js|css)?$
        {  expires    1h;
               }   
}

开启nginx
[iyunv@web1 opt]# /usr/local/nginx/sbin/nginx
[iyunv@web2 opt]# /usr/local/nginx/sbin/nginx
[iyunv@web1 opt]#echo "192.168.4.10" >/var/www/html/index.html
[iyunv@web2 opt]#echo "192.168.4.12" >/var/www/html/index.htm
安装tomcat
安装过程省略...
2 . 配置tomcat（web2一样）
[iyunv@web1 opt]# vim /usr/local/tomcat/conf/server.xml

<Host name="www.test123.com"  appBase="a"
            unpackWARs="true" autoDeploy="true">
         


3 .创建jsp文件，测试
[iyunv@web1 opt]#vim /var/www/html/111.jsp
    Now time is: <%=new>

[iyunv@web2 opt]#vim /var/www/html/111.jsp
    Now time is: <%=new>

4.[iyunv@web1 opt]#/usr/local/tomcat/bin/startup.sh

  [iyunv@web2 opt]#/usr/local/tomcat/bin/startup.sh

5.编写realserver.sh脚本，在lo网卡上绑定VIP，设定ARP抵制功能
[iyunv@web1 opt]# cat realserver.sh
#!/bin/bash
snf_vip=192.168.4.100
case $1 in
start)
    ifconfig lo:0 $snf_vip netmask 255.255.255.255 broadcast $snf_vip
    /sbin/route add -host $snf_vip dev lo:0
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce        
    sysctl -p &>/dev/null
    echo -e "\033[32m start OK\033[0m"
    ;;
stop)
    ifconfig lo:0 down
    /sbin/route del $snf_vip &>/dev/null
    echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce        
    sysctl -p &>/dev/null
    echo -e "\033[32m stopd OK\033[0m"
    ;;
*)
    echo "Usage: $0 {start|stop}"
    exit 1
    ;;
esac
exit 0

6 . 运行脚本。
[iyunv@web1 opt]#./realserver.sh start
[iyunv@web2 opt]#./realserver.sh start

7 . keep_master和keep_slave上安装ipvsadm和keepalive(keep_slave一样)
[iyunv@ keep_master LoadBalancer]# rpm -ivh ipvsadm-1.26-4.el6.x86_64.rpm
[iyunv@lkeep_master ~]# yum install -y gcc kernel-devel openssl-devel popt-devel
[iyunv@lkeep_master ~]# tar -xzf keepalived-1.2.7.tar.gz
[iyunv@lkeep_master ~]# cd keepalived-1.2.7
[iyunv@lkeep_master keepalived-1.2.7]# ./configure --sysconf=/etc
[iyunv@lkeep_master keepalived-1.2.7]# make && make install
[iyunv@lkeep_master keepalived-1.2.7]# ln -s /usr/local/sbin/keepalived /sbin/
[iyunv@lkeep_master keepalived-1.2.7]# chkconfig keepalived on
[iyunv@keep_master opt]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     zte_lwz1@126.com
   }
   notification_email_from keep_lvs@mail.com
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.4.100
    }
}

virtual_server 192.168.4.100 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    persistence_timeout 60
    protocol TCP

    real_server 192.168.4.10 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
        connect_port 80
        }
    }
    real_server 192.168.4.11 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
        connect_port 80
        }
    }
}

在keep_slave上只需修改
state MASTER     ------>state SLAVE
router_id  lvs1  ------> router_id  lvs2
priority 100     ------> priority 80


8. [iyunv@web1 opt]#  vim /etc/rc.local  (web2一样）
ulimit -SHn 65535 65535
/usr/local/nginx/sbin/nginx
/usr/local/tomcat/bin/startup.sh
/usr/local/bin/realserver.sh start

9.
[iyunv@keep_master opt]#/etc/init.d/keepalived start
[iyunv@keep_slave opt]#/etc/init.d/keepalived start

10. 测试,在客户端测试
[iyunv@localhost  ~]#vim /etc/hosts
192.168.4.100   www.test123.com
[iyunv@localhost  ~]# curl www.test123.com
192.168.4.10

关闭[iyunv@web1 opt]# /usr/local/nginx/sbin/nginx -s stop
[iyunv@localhost  ~]# curl www.test123.com
192.168.4.11

11.访问jsp
[iyunv@localhost  ~]# curl www.test123.com/111.jsp
......

12.keepalive的主备测试就不测试了。
13. [iyunv@keep_master opt]#ipvsadm -Ln