nginx+keepalived 高可用兼负载均衡集群

hitl

nginx+keepalived 高可用兼负载均衡集群

版本	作者	联系方式	最后更新
v0.3	purplegrape	purplegrape4@gmail.com	2013-04-10

Nginx是一个高性能的web服务器，同时也是一个优秀的反向代理服务器，本文利用两台Dell R720 构建一个高可用兼负载均衡的Linux web集群。

原理

通过nginx分别搭建两个web服务器，监听在本地非80端口；
然后利用nginx构建一个包含两个节点的负载均衡池；
最后通过keepalived实现负载均衡池的高可用。两个节点同时运行时，备用节点可以承载一半的前端流量，一个节点宕机后，负载均衡器通过健康检查，将失效节点踢出集群。

环境：CentOS 6.4 x86_64

软件	版本
nginx	1.2.7
keepalived	1.2.7
php-fpm	5.4.13

主机名	IP地址
web01	192.168.122.10
web02	192.168.122.20
VIP	192.168.122.30

安装nginx、keepalived和php-fpm等（编译过程略）

yum install nginx keepalived php-fpm php-gd php-xml php-mysql php-pecl-memcached php-pecl-sphinx watchodg -y chkconfig nginx on chkconfig keepalived on chkconfig php-fpm on chkconfig watchdog on

配置nginx

nginx主配置文件 /etc/nginx/nginx.conf

user  nginx; worker_processes  2; worker_rlimit_nofile 65535; error_log  /var/log/nginx/error.log warn; pid        /var/run/nginx.pid; events {     use epoll;     worker_connections  2048; } http {     include       /etc/nginx/mime.types;     default_type  application/octet-stream;     log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '                       '$status $body_bytes_sent "$http_referer" '                       '"$http_user_agent" "$http_x_forwarded_for"';     #access_log  /var/log/nginx/access.log  main;     sendfile        on;     server_tokens   off;     #tcp_nopush     on;     keepalive_timeout  65;     gzip  on;     gzip_static on;     gzip_disable "msie6";     gzip_http_version 1.1;     gzip_vary on;     gzip_comp_level 6;     gzip_proxied any;     gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x$     gzip_buffers 16 8k; client_max_body_size 20m; client_body_buffer_size 128k;       proxy_hide_header Vary;       proxy_connect_timeout 600;       proxy_read_timeout 600;       proxy_send_timeout 600;       proxy_buffer_size 16k;       proxy_buffers 4 64k;       proxy_busy_buffers_size 128k;       #proxy_temp_file_write_size 128k     proxy_temp_path /dev/shm/tmp;     proxy_cache_path /dev/shm/proxycache/ levels=1:2 keys_zone=shmcache:10m inactive=2h max_size=500m;     include /etc/nginx/conf.d/*.conf; }

网站主配置文件/etc/nginx/conf.d/default.conf

upstream webservers { ip_hash ; server 192.168.122.10:88 max_fails=3 fail_timeout=3s weight=2 ; server 192.168.122.20:88 max_fails=3 fail_timeout=3s weight=2 ; } server { listen80 ; server_name localhost ;        access_log  /var/log/nginx/web01.access.log  main; location / { proxy_redirect off; proxy_pass http://webservers ;              proxy_cache shmcache;       proxy_cache_valid 200 302 1d;       proxy_cache_valid 404 1h;       proxy_cache_valid any 10m;       proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; proxy_set_header Host $host;   proxy_set_header X-Real-IP $remote_addr;   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;   proxy_next_upstream error timeout invalid_header http_500 http_502 http_504;   } location /status { stub_status on;   access_log off;   allow 192.168.122.0/24;   } } server {     listen       192.168.122.10:88;     server_name  localhost; root   /usr/share/nginx/html; indexindex.html index.htm index.php;     #charset koi8-r;     location / { if (!-e $request_filename) { rewrite ^/(.*)$ /index.php?q=$1 last; }     }     #error_page  404              /404.html;     # redirect server error pages to the static page /50x.html     #     error_page   500 502 503 504  /50x.html;     # proxy the PHP scripts to Apache listening on 127.0.0.1:80     #     #location ~ \.php$ {     #    proxy_pass   http://127.0.0.1;     #}     # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000     # location ~ \.php$ { #try_files $uri = 404; fastcgi_pass   127.0.0.1:9000; #fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock; fastcgi_index  index.php; fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name; fastcgi_buffer_size 128k; fastcgi_buffers 256 16k; fastcgi_busy_buffers_size 256k; fastcgi_temp_file_write_size 256k; fastcgi_read_timeout 240; include        fastcgi_params; } if ($fastcgi_script_name ~ \..*\/.*php) { return 403; } # deny access to hiden file . (filename begin with ".") location ~ /\. { access_log off; log_not_found off;   deny all; } # deny access to bakup file .(any filename end with "~" ) location ~ ~$ {   access_log off;   log_not_found off;   deny all;   } # cache image file location ~* \.(jpg|jpeg|gif|png|css|js|ico|xml|swf)$ {     expires           1d; } # don't log robots  and favion location = /robots.txt { access_log off; log_not_found off; }   location = /favicon.ico { access_log off; log_not_found off; }       # deny access to .htaccess files, if Apache's document root     # concurs with nginx's one     #     location ~ /\.ht {         deny  all;     } }

配置php-fpm
php-fpm全称php fastcgi process manager，用于管理php的fastcgi进程，自从php5.3.3之后集成进了php源码中。默认的php-fpm用户是apache，我们需要修改成nginx，代码：

sed -i 's/apache/nginx/g' /etc/php-fpm.d/www.conf

配置keepalived
keepalived是一个高可用软件，通过vrrp心跳来检测对方是否存活。

keepalived主配置文件

global_defs {    notification_email {      root@localhost    }    notification_email_from keepalived01@web01.test.org    smtp_server 127.0.0.1    smtp_connect_timeout 30 } vrrp_instance VI_1 {     state MASTER     interface eth0     virtual_router_id 51     priority 100     advert_int 1     authentication {         auth_type PASS         auth_pass 1111     }     virtual_ipaddress {         192.168.122.30 label eth0:1     } }

配置watchdog

watchdog是个健壮可靠的内核模块，即使在高负载的系统中仍然可以存活。它能够执行检测脚本，定时检查nginx和keepalived进程是否存在，如果不存在可以重新开启进程。

watchdog主配置文件/etc/watchdog.conf

min-memory= 1 repair-binary= /etc/watchdog.d/repair.sh test-binary       = /etc/watchdog.d/test.sh test-timeout= 5 watchdog-device= /dev/watchdog admin= root interval= 10 logtick         = 1 realtime= yes priority= 1 pidfile= /var/run/syslogd.pid    pidfile         = /var/run/nginx.pid pidfile         = /var/run/keepalived.pid pidfile         = /var/run/php-fpm/php-fpm.pid

测试脚本，权限751

#!/usr/bin/env bash #author:purplegrape #desc: shell script to check if keepalived/nginx/php-fpm down. keepalived_proc=`pgrep keepalived |wc -l` nginx_proc=`pgrep nginx|wc -l` php_fpm_porc=`pgrep php-fpm|wc -l` if [$keepalived_proc == 0 ];then       exit 1 fi if [$nginx_proc == 0 ];then       exit 1 fi if [$php_fpm_proc == 0 ];then      exit 1 fi

修复脚本,权限751

#!/usr/bin/env bash #author:purplegrape #desc: shell script to repair keepalived/nginx/php-fpm. keepalived_proc=`pgrep keepalived |wc -l` nginx_proc=`pgrep nginx|wc -l` php_fpm_proc=`pgrep php-fpm|wc -l` if [$keepalived_proc == 0 ];then       wall “keepalived is dead”       /etc/init.d/keepalived restart       sleep 30      /etc/init.d/nginx restart fi if [$nginx_proc == 0 ];then       if [$keepalived_proc != 0 ];then            wall “nginx is dead”            /etc/init.d/nginx restart       fi fi if [ $php_fpm_proc == 0 ]; then       wall “php-fpm is dead”       /etc/init.d/php-fpm restart fi

测试
打开浏览器，访问地址http://192.168.122.30