memcached演练(3) 使用apache+搭建双节点tomcat集群
上篇文章，已经明确了tomcat集群的必要性，主要从高可用和高并发量考虑。一般使用tomcat集群搭建，都要使用前置设一个nginx或apache作个反向代理，将请求转发给后端tomcat。使用tomcat集群，必然会引来session数据共享问题。如何解决这个问题？可借助memcached实现。本章主要搭建一个tomcat双节点集群，算“准备”篇吧。主要内容
[*] 测试单台tomcat的吞吐率
[*] 搭建apache+mod_jk+2个tomcat集群环境
环境准备
确认apache服务位置及版本号
# httpd -v
Server version: Apache/2.2.15 (Unix)
Server built: Feb 22 2013 11:17:44
# which httpd
/usr/sbin/httpd
#apache主目录:/etc/httpd 确认JDK 及tomcat服务
$ ls /u01/tomcat-sets
apache-tomcat-node1
$ cat /proc/meminfo
MemTotal: 1937148 kB
...
$ /u01/tomcat-sets/apache-tomcat-node1/bin/version.sh
Using CATALINA_BASE: /u01/tomcat-sets/apache-tomcat-node1
Using CATALINA_HOME: /u01/tomcat-sets/apache-tomcat-node1
Using CATALINA_TMPDIR: /u01/tomcat-sets/apache-tomcat-node1/temp
Using JRE_HOME: /usr/local/java/jdk1.6.0_45
...
Server version: Apache Tomcat/7.0.33
Server built: Nov 18 2012 04:15:21
Server number:7.0.33.0
OS Name: Linux
OS Version: 2.6.32-358.el6.i686
Architecture: i386
JVM Version: 1.6.0_45-b06
JVM Vendor: Sun Microsystems Inc. 1.测试单台tomcat的吞吐率
1.1准备一个静态测试资源（staticProtocal.html，大小50K）
资源可以通过http://192.168.163.146:8080/examples/staticProtocal.html访问
$ ll /u01/tomcat-sets/apache-tomcat-node1/webapps/examples
total 72
-rw-r--r--.1 hadoop hadoop1179 Nov 192012 index.html
drwxr-xr-x. 22 hadoop hadoop4096 Aug 10 18:48 jsp
drwxr-xr-x.3 hadoop hadoop4096 Aug 10 18:48 servlets
-rw-rw-r--.1 hadoop hadoop 50663 Aug 12 21:09 staticProtocal.html
drwxr-xr-x.7 hadoop hadoop4096 Aug 10 18:48 WEB-INF
drwxr-xr-x.2 hadoop hadoop4096 Aug 10 18:48 websocket 1.2启动apache-tomcat-node1，端口为默认的8080
1.3 在另外一台主机使用apache ab命令测试
C:\Apache24\bin>ab-n 10000 -c 4http://192.168.163.146:8080/examples/staticProtocal.html
This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.163.146 (be patient)
Completed 1000 requests
...
Completed 10000 requests
Finished 10000 requests
Server Software: Apache-Coyote/1.1
Server Hostname: 192.168.163.146
Server Port: 8080
Document Path: /examples/staticProtocal.html
Document Length: 50663 bytes
Concurrency Level: 4
Time taken for tests: 12.065 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 509120000 bytes
HTML transferred: 506630000 bytes
Requests per second: 828.81 [#/sec] (mean)
Time per request: 4.826 (mean)
Time per request: 1.207 (mean, across all concurrent requests)
Transfer rate: 41207.63 received
... 经过测试,单台tomcat的结果是：QPS:828.81
都有这样一个共识，apache处理静态文件的能力，比tomcat强大。是骡子是马，比比看。
1.4 简单编辑apache配置文件httpd.conf。(最好修改之前备份下)
$ cat /etc/httpd/conf/httpd.conf
...
ServerName localhost:80
DocumentRoot "/var/www/html"
... 1.5 将刚才的staticProtocal.html文件，复制到DocumentRoot“/var/www/html”目录下。
$ ls /var/www/html/
staticProtocal.html 1.6 启动httpd
$service httpd start
Starting httpd: 这时候可以通过链接访问资源：http://192.168.163.146/staticProtocal.html
1.7 测试apache吞吐量
C:\Apache24\bin>ab-n 10000 -c 4http://192.168.163.146/staticProtocal.html
This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.163.146 (be patient)
Completed 1000 requests
...
Finished 10000 requests
Server Software: Apache/2.2.15
Server Hostname: 192.168.163.146
Server Port: 80
Document Path: /staticProtocal.html
Document Length: 50663 bytes
Concurrency Level: 4
Time taken for tests: 11.902 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 509350000 bytes
HTML transferred: 506630000 bytes
Requests per second: 840.17 [#/sec] (mean)
Time per request: 4.761 (mean)
Time per request: 1.190 (mean, across all concurrent requests)
Transfer rate: 41790.94 received 经过测试,单台apache的结果是：QPS:840.17
显而易见，apache处理静态文件能力略高于tomcat.
接下来，那就利用apache搭建一个双节点tomcat集群，顺带验证下QPS能达到多少。这是下一小节内容。
2.搭建apache+mod_jk+2个tomcat集群环境
2.1下载mod_jk并安装
要安装httpd-devel,否则会configure: error: Cannot find the WebServer
#yum install httpd-devel
## 安装完httpd-devel之后，新添加目录/usr/sbin/apxs
##下载
# wget
##解压
# tar zxvf tomcat-connectors-1.2.41-src.tar.gz
# cd tomcat-connectors-1.2.41-src/native
# ./buildconf.sh --with-apxs=/usr/sbin/apxs
# make 到这里mod_jk.so已经生成了，最后将mod_jk.so放到apache的modules 目录下
cp /u01/software/tomcat-connectors-1.2.41-src/native/apache-2.0/mod_jk.so/etc/httpd/modules 2.2配置mod_jk
请参考：http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html
mod_jk：支持平台Linux, FreeBSD, AIX, HP-UX, MacOS X, Solaris，Apache 1.3 and/or 2.x
mod_jk: 支持Tomcat 3.2 to Tomcat 8.
redirector 使用协议：ajp12 and ajp13
ajp12 协议针对Tomcat 3.2.x and 3.3.x.
ajp13协议支持tomcat 4+
mod_jk:2个组成部分。
The Apache HTTP Server module。由平台决定，一般是mod_jk.so, mod_jk.nlm or MOD_JK.SRVPGM
workers.properties ：描述Tomcat 相关的host(s) and port(s)文件。
2.3启用jk_module模块
将下面内容片段，加入/etc/httpd/httpd.conf
# Load mod_jk module
LoadModule jk_modulemodules/mod_jk.so
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevelinfo
include "conf/mod_jk.conf" 2.4 配置后端tomcat 主机及端口映射,编辑或新建文件，/etc/httpd/workers.properties
配置说明参照：https://tomcat.apache.org/connectors-doc/reference/workers.html
worker.list=tomcat1
worker.tomcat1.port=8009
worker.tomcat1.host=127.0.0.1
worker.tomcat1.type=ajp13
#worker.tomcat1.lbfactor=1 现在达到的效果是apache反代1个tomcat。所以不需要配置额外的负载均衡等信息。
2.5 编辑conf/mod_jk.conf，增加资源映射
#LoadModule jk_module modules/mod_jk.so
JkMount/*.jsp tomcat1
JkMount/examples/* tomcat1
#JkWorkersFile conf/workers.properties 注释的两行，由于已经在httpd.conf中已经配置，所以不需要重复配置。
2.6 启动tomcat,及apache
笔者，在这一步经过测试，不能正常达到代理效果。
通过查看日志信息(/var/log/httpd/mod_jk.log)
init_jk::mod_jk.c (3574): Initializing shm:/etc/httpd/logs/jk-runtime-status.9739 errno=13. Load balancing workers will not function properly.
通过参照网上解决方案：临时将selinux关闭
确认selinux状态
/usr/sbin/sestatus -v
临时关闭
setenforce 0
永久关闭
修改/etc/selinux/config 设置为 SELINUX=disabled 访问测试，没有问题
http://192.168.163.146/examples/servlets/
http://192.168.163.146:8080/examples/servlets/
到了这里，完成了apache+单台tomcat的配置。
C:\Apache24\bin>ab-n 10000 -c 4http://192.168.163.146/examples/staticProtocal.html
This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.163.146 (be patient)
Completed 1000 requests
...
Completed 10000 requests
Finished 10000 requests
Server Software: Apache/2.2.15
Server Hostname: 192.168.163.146
Server Port: 80
Document Path: /examples/staticProtocal.html
Document Length: 50663 bytes
Concurrency Level: 4
Time taken for tests: 13.555 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 509320000 bytes
HTML transferred: 506630000 bytes
Requests per second: 737.73 [#/sec] (mean)
Time per request: 5.422 (mean)
Time per request: 1.356 (mean, across all concurrent requests)
Transfer rate: 36693.36 received 很遗憾,apache+单台tomcat组合的QPS:737.73
接下来，开始搭建apache+双节点环境。
3.搭建apache+mod_jk+2个tomcat集群环境
3.1复制tomcat环境为node2
# cd /u01/tomcat-sets/
# ls
apache-tomcat-node1
# cp -Rapache-tomcat-node1/ apache-tomcat-node2
# chown -R hadoop:hadoop /u01/tomcat-sets
# ll
total 8
drwxr-xr-x. 9 hadoop hadoop 4096 Aug 10 18:48 apache-tomcat-node1
drwxr-xr-x9 hadoop hadoop 4096 Aug 13 08:36 apache-tomcat-node2 3.2 修改apache-tomcat-node2节点的相关端口配置
配置好了简单测试下
访问以下均正常
http://192.168.163.146:8080/examples/staticProtocal.html
ps -ef |grep tomcat
# ps -ef |grep tomcat
root 2797 10 08:31 pts/0 00:00:03 /usr/bin/java -Djava.util.logging.config.file=/u01/tomcat-sets/apache-tomcat-node1/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/u01/tomcat-sets/apache-tomcat-node1/endorsed -classpath /u01/tomcat-sets/apache-tomcat-node1/bin/bootstrap.jar:/u01/tomcat-sets/apache-tomcat-node1/bin/tomcat-juli.jar -Dcatalina.base=/u01/tomcat-sets/apache-tomcat-node1 -Dcatalina.home=/u01/tomcat-sets/apache-tomcat-node1 -Djava.io.tmpdir=/u01/tomcat-sets/apache-tomcat-node1/temp org.apache.catalina.startup.Bootstrap start
root 2993 17 08:47 pts/0 00:00:00 /usr/bin/java -Djava.util.logging.config.file=/u01/tomcat-sets/apache-tomcat-node2/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/u01/tomcat-sets/apache-tomcat-node2/endorsed -classpath /u01/tomcat-sets/apache-tomcat-node2/bin/bootstrap.jar:/u01/tomcat-sets/apache-tomcat-node2/bin/tomcat-juli.jar -Dcatalina.base=/u01/tomcat-sets/apache-tomcat-node2 -Dcatalina.home=/u01/tomcat-sets/apache-tomcat-node2 -Djava.io.tmpdir=/u01/tomcat-sets/apache-tomcat-node2/temp org.apache.catalina.startup.Bootstrap start 关闭一个tomcat节点，再测。（以前遇到过两个节点shutdown端口一样，导致关闭异常。）
3.3 配置后端tomcat 主机及端口映射,编辑或新建文件，/etc/httpd/workers.properties。
与2.4步骤一样，但要求要做到支持多节点和负载均衡。
worker.list=tomcat1
worker.tomcat1.port=8009
worker.tomcat1.host=127.0.0.1
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=1
worker.tomcat2.port=8019
worker.tomcat2.host=127.0.0.1
worker.tomcat2.type=ajp13
worker.tomcat2.lbfactor=1
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=tomcat1,tomcat2
worker.loadbalancer.sticky_session=1 3.4 再次编辑节点服务器配置(server.xml)
node1节点server.xml最终信息
node2节点配置信息(server.xml)
值得一提的是：搭建过程中遇到一个问题。查看tomcat错误信息
java.net.BindException: Cannot assign requested address; No faulty members identified.
按照网上解决方案：主机配置错误。这才想起来，曾经切换过虚拟机的Network Adapter。
参照网址http://blog.csdn.net/ligang2585116/article/details/43796371。
解决方式，修改/etc/hosts中的IP，修改为现在的IP即可。
# ifconfig
eth0 Link encap:EthernetHWaddr 00:50:56:20:2B:84
inet addr:192.168.163.146Bcast:192.168.163.255Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe20:2b84/64 Scope:Link
# cat /etc/hosts
192.168.163.146 hadoop1
3.5测试切换效果。
为了区分不同tomcat下的应用，将staticProtocal.html的内容作了下调整。
结果
为了排除浏览器缓存干扰，最好使用强制刷新。（CTRL+F5）,这时候可以验证下权重。
3.6 测试下双节点集群的吞吐量。
C:\Apache24\bin>ab-n 10000 -c 4http://192.168.163.146/examples/staticProtocal.html
This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.163.146 (be patient)
Completed 1000 requests
...
Finished 10000 requests
Server Software: Apache/2.2.15
Server Hostname: 192.168.163.146
Server Port: 80
Document Path: /examples/staticProtocal.html
Document Length: 49598 bytes
Concurrency Level: 4
Time taken for tests: 13.996 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 498670000 bytes
HTML transferred: 495980000 bytes
Requests per second: 714.51 [#/sec] (mean)
Time per request: 5.598 (mean)
Time per request: 1.400 (mean, across all concurrent requests)
Transfer rate: 34795.47 received
Connection Times (ms)
minmean[+/-sd] median max
Connect: 0 0 0.4 1 5
Processing: 0 5 9.1 3 162
Waiting: 0 3 4.9 2 125
Total: 0 6 9.1 3 162
ERROR: The median and mean for the initial connection time are more than twice the standard
deviation apart. These results are NOT reliable.
吞吐量最差：714.51
为什么？
到了这里，还没有讲到session和memcached的影子。
结论（以1个50K的静态文本）
环境
吞吐量
1.单台tomcat
828.812.单个apache
840.173.apache+单tomcat节点
737.73apache+双tomcat节点714.51 通过对比，很直观得出结论。
[*] apache处理静态文件的能力，比tomcat的能力高一些.
[*] apache+单tomcat节点情况处理静态文件，纯属胡搞做实验，没有任何实际意义，很显然多了一层代理，当然吞吐量下来了。有点脱了裤子放屁的意思。
[*] 至于双节点tomcat的吞吐量下降的这么厉害，应该和我的环境有关系，我的CPU是单核，跑2个节点，想想发生了资源争用。
[*] 最后，这个实验不能掩盖tomcat的威力，它的强大在处理动态处理上，而这恰恰是apache做不了的。
------------------------
The End,下一篇，重点验证下集群间的session的共享，和memcached在集群中的应用。
页:
[1]