论坛 › keepalived › nginx+keepalived 高可用兼负载均衡集群

hitl 发表于 2018-12-31 06:54:01

nginx+keepalived 高可用兼负载均衡集群

nginx+keepalived 高可用兼负载均衡集群


    　　版本
　　作者
　　联系方式
　　最后更新
　　v0.3
　　purplegrape
　　purplegrape4@gmail.com
　　2013-04-10

Nginx是一个高性能的web服务器，同时也是一个优秀的反向代理服务器，本文利用两台Dell R720 构建一个高可用兼负载均衡的Linux web集群。

原理

通过nginx分别搭建两个web服务器，监听在本地非80端口；
然后利用nginx构建一个包含两个节点的负载均衡池；
最后通过keepalived实现负载均衡池的高可用。两个节点同时运行时，备用节点可以承载一半的前端流量，一个节点宕机后，负载均衡器通过健康检查，将失效节点踢出集群。

环境：CentOS 6.4 x86_64

软件
版本
nginx
1.2.7
keepalived
1.2.7
php-fpm
5.4.13

主机名
IP地址
web01
192.168.122.10
web02
192.168.122.20
VIP
192.168.122.30


安装nginx、keepalived和php-fpm等（编译过程略）

yum install nginx keepalived php-fpm php-gd php-xml php-mysql php-pecl-memcached php-pecl-sphinx watchodg -y
chkconfig nginx on
chkconfig keepalived on
chkconfig php-fpm on
chkconfig watchdog on


配置nginx

nginx主配置文件 /etc/nginx/nginx.conf
usernginx;
worker_processes2;
worker_rlimit_nofile 65535;


error_log/var/log/nginx/error.log warn;
pid      /var/run/nginx.pid;


events {
    use epoll;
    worker_connections2048;
}


http {
    include       /etc/nginx/mime.types;
    default_typeapplication/octet-stream;

    log_formatmain'$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log/var/log/nginx/access.logmain;

    sendfile      on;
    server_tokens   off;

    #tcp_nopush   on;

    keepalive_timeout65;

    gzipon;
    gzip_static on;
    gzip_disable "msie6";
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 6;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x$
    gzip_buffers 16 8k;

client_max_body_size 20m;
client_body_buffer_size 128k;

      proxy_hide_header Vary;
      proxy_connect_timeout 600;
      proxy_read_timeout 600;
      proxy_send_timeout 600;
      proxy_buffer_size 16k;
      proxy_buffers 4 64k;
      proxy_busy_buffers_size 128k;
      #proxy_temp_file_write_size 128k

    proxy_temp_path /dev/shm/tmp;
    proxy_cache_path /dev/shm/proxycache/ levels=1:2 keys_zone=shmcache:10m inactive=2h max_size=500m;

    include /etc/nginx/conf.d/*.conf;
}



网站主配置文件/etc/nginx/conf.d/default.conf
upstream webservers {
ip_hash ;
server 192.168.122.10:88 max_fails=3 fail_timeout=3s weight=2 ;
server 192.168.122.20:88 max_fails=3 fail_timeout=3s weight=2 ;
}

server {
listen80 ;
server_name localhost ;
       access_log/var/log/nginx/web01.access.logmain;

location / {
proxy_redirect off;
proxy_pass http://webservers ;

             proxy_cache shmcache;
      proxy_cache_valid 200 302 1d;
      proxy_cache_valid 404 1h;
      proxy_cache_valid any 10m;
      proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_504;
}

location /status {
stub_status on;
access_log off;
allow 192.168.122.0/24;
}
}

server {
    listen       192.168.122.10:88;
    server_namelocalhost;
root   /usr/share/nginx/html;
indexindex.html index.htm index.php;

    #charset koi8-r;

    location / {
if (!-e $request_filename) {
rewrite ^/(.*)$ /index.php?q=$1 last;
}
    }

    #error_page404            /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504/50x.html;

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
location ~ \.php$ {
#try_files $uri = 404;
fastcgi_pass   127.0.0.1:9000;
#fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_indexindex.php;
fastcgi_paramSCRIPT_FILENAME$document_root$fastcgi_script_name;
fastcgi_buffer_size 128k;
fastcgi_buffers 256 16k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_read_timeout 240;
include      fastcgi_params;
}

if ($fastcgi_script_name ~ \..*\/.*php) {
return 403;
}

# deny access to hiden file . (filename begin with ".")
location ~ /\. {
access_log off;
log_not_found off;
deny all;
}

# deny access to bakup file .(any filename end with "~" )
location ~ ~$ {
access_log off;
log_not_found off;
deny all;
}

# cache image file
location ~* \.(jpg|jpeg|gif|png|css|js|ico|xml|swf)$ {
    expires         1d;
}

# don't log robotsand favion
location = /robots.txt { access_log off; log_not_found off; }
location = /favicon.ico { access_log off; log_not_found off; }

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    location ~ /\.ht {
      denyall;
    }
}

配置php-fpm
php-fpm全称php fastcgi process manager，用于管理php的fastcgi进程，自从php5.3.3之后集成进了php源码中。默认的php-fpm用户是apache，我们需要修改成nginx，代码：

sed -i 's/apache/nginx/g' /etc/php-fpm.d/www.conf


配置keepalived
keepalived是一个高可用软件，通过vrrp心跳来检测对方是否存活。

keepalived主配置文件
global_defs {
   notification_email {
   root@localhost
   }
   notification_email_from keepalived01@web01.test.org
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
      auth_type PASS
      auth_pass 1111
    }
    virtual_ipaddress {
      192.168.122.30 label eth0:1
    }
}

配置watchdog

watchdog是个健壮可靠的内核模块，即使在高负载的系统中仍然可以存活。它能够执行检测脚本，定时检查nginx和keepalived进程是否存在，如果不存在可以重新开启进程。

watchdog主配置文件/etc/watchdog.conf
min-memory= 1
repair-binary= /etc/watchdog.d/repair.sh
test-binary       = /etc/watchdog.d/test.sh
test-timeout= 5
watchdog-device= /dev/watchdog
admin= root
interval= 10
logtick         = 1
realtime= yes
priority= 1
pidfile= /var/run/syslogd.pid   
pidfile         = /var/run/nginx.pid
pidfile         = /var/run/keepalived.pid
pidfile         = /var/run/php-fpm/php-fpm.pid



测试脚本，权限751
#!/usr/bin/env bash
#author:purplegrape
#desc: shell script to check if keepalived/nginx/php-fpm down.

keepalived_proc=`pgrep keepalived |wc -l`
nginx_proc=`pgrep nginx|wc -l`
php_fpm_porc=`pgrep php-fpm|wc -l`

if [$keepalived_proc == 0 ];then
      exit 1
fi

if [$nginx_proc == 0 ];then
      exit 1
fi

if [$php_fpm_proc == 0 ];then
   exit 1
fi

修复脚本,权限751
#!/usr/bin/env bash
#author:purplegrape
#desc: shell script to repair keepalived/nginx/php-fpm.

keepalived_proc=`pgrep keepalived |wc -l`
nginx_proc=`pgrep nginx|wc -l`
php_fpm_proc=`pgrep php-fpm|wc -l`

if [$keepalived_proc == 0 ];then
      wall “keepalived is dead”
      /etc/init.d/keepalived restart
      sleep 30
   /etc/init.d/nginx restart
fi

if [$nginx_proc == 0 ];then
      if [$keepalived_proc != 0 ];then
         wall “nginx is dead”
         /etc/init.d/nginx restart
      fi
fi

if [ $php_fpm_proc == 0 ]; then
      wall “php-fpm is dead”
      /etc/init.d/php-fpm restart
fi

测试
打开浏览器，访问地址http://192.168.122.30

查看完整版本: nginx+keepalived 高可用兼负载均衡集群