Tomcat+Nginx+Memcacheed集群部署

4343223 · 发表于 2016-9-30 10:11:24

主机环境 redhat6.5 64位
实验环境服务端1 ip172.25.29.1 nginx

服务端2 ip 172.25.29.2 tomcat+memcached

服务端3 ip 172.25.29.3 tomcat+memcaceed

安装包 jdk-7u79-linux-x64.tar.gz

apache-tomcat-7.0.37.tar.gz

nginx-goodies-nginx-sticky-module-ng-c78b7dd79d0d.tar.gz

asm-3.2.jar

kryo-1.04.jar

kryo-serializers-0.10.jar

memcached-session-manager-1.6.3.jar

memcached-session-manager-tc7-1.6.3.jar

minlog-1.2.jar

msm-kryo-serializer-1.6.3.jar

reflectasm-1.01.jar

spymemcached-2.7.3.jar

防火墙状态关闭

1.jdk安装、环境配置及测试服务端2
1.解压、作软链接
[iyunv@server2mnt]# tar zxf jdk-7u79-linux-x64.tar.gz -C /usr/local/  #解压jdk
[iyunv@server2mnt]# cd /usr/local/ #切换到解压目录
[iyunv@server2local]# ls
bin  etc games  include  jdk1.7.0_79 lib  lib64  libexec sbin  share  src
[iyunv@server2local]# ln -s jdk1.7.0_79/ java    #作软链接
[iyunv@server2local]# ll
total44
drwxr-xr-x.2 root root 4096 Jun 28  2011 bin
drwxr-xr-x.2 root root 4096 Jun 28  2011 etc
drwxr-xr-x.2 root root 4096 Jun 28  2011 games
drwxr-xr-x.2 root root 4096 Jun 28  2011 include
lrwxrwxrwx.1 root root 12 Sep 24 10:50 java ->jdk1.7.0_79/ #查看
drwxr-xr-x.8 uucp  143 4096 Apr 11  2015 jdk1.7.0_79
drwxr-xr-x.2 root root 4096 Jun 28  2011 lib
drwxr-xr-x.2 root root 4096 Jun 28  2011 lib64
drwxr-xr-x.2 root root 4096 Jun 28  2011 libexec
drwxr-xr-x.2 root root 4096 Jun 28  2011 sbin
drwxr-xr-x.5 root root 4096 Aug  8 21:38 share
drwxr-xr-x.2 root root 4096 Jun 28  2011 src
2.将jdk添加到环境变量
[iyunv@server2local]# vim /etc/profile
79 export JAVA_HOME=/usr/local/java
80 exportCLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
81 export PATH=$PATH:$JAVA_HOME/bin
[iyunv@server2local]# source /etc/profile
3.测试
[iyunv@server2local]# echo $JAVA_HOME #测试添加环境变量是否成功
/usr/local/java
[iyunv@server2local]# echo $CLASSPATH
.:/usr/local/java/lib:/usr/local/java/jre/lib
[iyunv@server2local]# echo $JAVA_HOME
/usr/local/java
[iyunv@server2local]# cd java    #测试java的环境是否配置成功
[iyunv@server2java]# vim test.java #写个简单的测试页
  1 public class test{
  2       public static void main(String[] args)
  3       {
  4                System.out.println("Helloworld!");
  5       }
  6 }
[iyunv@server2java]# javac test.java #链接生成test.class文件
[iyunv@server2mnt]# java test #执行
Helloworld!

2.tomcat的安装服务端2
1.解压、作软链接
[iyunv@server2mnt]# tar zxf apache-tomcat-7.0.8.tar.gz -C /usr/local/  #解压
[iyunv@server2mnt]# cd /usr/local/
[iyunv@server2local]# ls
apache-tomcat-7.0.8  etc include  jdk1.7.0_79  lib64 sbin src
bin                games  java lib       libexec  share
[iyunv@server2local]# ln -s apache-tomcat-7.0.8/ tomcat #作软链接
[iyunv@server2local]# ll
total48
drwxr-xr-x.9 root root 4096 Sep 24 11:11 apache-tomcat-7.0.8
drwxr-xr-x.2 root root 4096 Jun 28  2011 bin
drwxr-xr-x.2 root root 4096 Jun 28  2011 etc
drwxr-xr-x.2 root root 4096 Jun 28  2011 games
drwxr-xr-x.2 root root 4096 Jun 28  2011 include
lrwxrwxrwx.1 root root 12 Sep 24 10:50 java ->jdk1.7.0_79/
drwxr-xr-x.8 uucp  143 4096 Sep 24 11:02 jdk1.7.0_79
drwxr-xr-x.2 root root 4096 Jun 28  2011 lib
drwxr-xr-x.2 root root 4096 Jun 28  2011 lib64
drwxr-xr-x.2 root root 4096 Jun 28  2011 libexec
drwxr-xr-x.2 root root 4096 Jun 28  2011 sbin
drwxr-xr-x.5 root root 4096 Aug  8 21:38 share
drwxr-xr-x.2 root root 4096 Jun 28  2011 src
lrwxrwxrwx.1 root root 20 Sep 24 11:12 tomcat-> apache-tomcat-7.0.8/    #查看
[iyunv@server2local]# cd tomcat/bin
[iyunv@server2bin]# ./startup.sh    #开启tomcat

2.测试
[iyunv@server2bin]#cd ..
[iyunv@server2tomcat]# vim webapps/ROOT/test.jsp #写测试仪页
  1 server2-The Time is <%=newjava.util.Date()%>
#测试 172.25.29.2:8080

172.25.29.2:8080/test.jsp

在服务端3上进行同样的配置也可以用scp把服务端2上java和tomcat目录传过去如下
[iyunv@server2local]# scp -r java/ tomcat/ 172.25.29.3:/usr/local/
在进行系统环境里加上java的配置启动tomcat即可

3.Nginx添加sticky (服务端1)
1.nginx负载均衡已经配置好了

在前面的博客里已经写过nginx源码安装这里就不再重复了也可以参考前面的博客

[iyunv@server1~]# cd /usr/local/lnmp/nginx/conf
[iyunv@server1conf]# vim nginx.conf
20 upstream wen {
21                server 172.25.29.2:8080; #轮询机制
22                server 172.25.29.3:8080;
23       }
49       location / {
50          root html;
51          index  index.html index.jspindex.php index.htm; #默认发布目录
52       }

68       location ~ \.jsp$ {
69          proxy_pass http://wen;
70       }
[iyunv@server1conf]# nginx -t #检测
nginx:the configuration file /usr/local/lnmp/nginx/conf/nginx.conf syntax is ok
nginx:configuration file /usr/local/lnmp/nginx/conf/nginx.conf test is successful
[iyunv@server1conf]# nginx -s reload #刷新
测试 172.25.29.1

172.25.29.1/test.jsp

刷新之后

也就是server2 和server3相互交替

2.改变负载均衡机制为sticky

由于nginx是轮询机制如果在访问页面如填写信息时突然卡住刷新之后就会跳到另一个server上就得重新开始填写。但是Nginx里有很多算法其中的ip_hash也可以防止这些问题。使用ip_hash有一个问题是当客户端和服务器之间使用cdn内容分发系统高速缓存时客户端的访问到达cdn由cdn访问服务器识别的ip是cdn的ip那么将集中于访问后台的一台服务器(x相当于DDOS攻击)会加速服务器的损坏。那么为了防止这些问题就有用下面的算法sticky不是nginx里自带的nginx-sticky-module为 nginx 的第三方模块,使 nginx 支持 sticky 模式,所以需要将包加入配置、重新编译、安装nginx

1.重新源码安装nginx添加一个模块
[iyunv@server1local]# nginx -s stop #关闭nginx
[iyunv@server1mnt]# tar zxf nginx-goodies-nginx-sticky-module-ng-c78b7dd79d0d.tar.gz #解压
[iyunv@server1mnt]# ls
nginx-1.8.1.tar.gz
nginx-goodies-nginx-sticky-module-ng-c78b7dd79d0d
nginx-goodies-nginx-sticky-module-ng-c78b7dd79d0d.tar.gz

[root@server1nginx-1.8.1]# make clean #清除上一次的缓存文件
rm-rf Makefile objs
[root@server1nginx-1.10.1]# ./configure --prefix=/usr/local/lnmp/nginx #重新配置--with-http_ssl_module--with-http_stub_status_module--add-module=/mnt/nginx-goodies-nginx-sticky-module-ng-c78b7dd79d0d
[root@server1nginx-1.8.1]# make    #编译、链接
[root@server1nginx-1.8.1]# make install #安装
[root@server1nginx-1.8.1]#cd /usr/local/lnmp/nginx/conf
[iyunv@server1conf]# vim nginx.conf
18       upstream westos{
19                sticky;    #使用sticky
20                server 172.25.29.2:8080;
21                server 172.25.29.3:8080;
[iyunv@server1local]# nginx -t #检测nginx文件里是否有错误
nginx:the configuration file /usr/local/lnmp/nginx/conf/nginx.conf syntax is ok
nginx:configuration file /usr/local/lnmp/nginx/conf/nginx.conf test is successful
[iyunv@server1local]# nginx #启动
2.测试
测试 172.25.29.1/test.jsp

刷新之后结果不变还是

4交叉存储避免单点故障添加memcached服务端2

配置好之后如果一台服务器宕机了那么这台服务器正在运行的业务将直接结束正在存储的数据丢失。为了防止这些问题Tomcat1将session存储到Tomcat2的memcached中当Tomcat2的memcached不可用时Tomcat1将session存储到自己的memcached上Tomcat2则正好相反。使用这种配置就避免了单点故障。

1.安装、开启memcached、写测试页

[iyunv@server2ROOT]# yum install -y memcached    #安装memcached
[iyunv@server2ROOT]# /etc/init.d/memcached start    #开启memcached
Startingmemcached:                                     [  OK  ]
[iyunv@server2tomcat]# bin/shutdown.sh #关闭Tomcat
[iyunv@server2ROOT]# vim test.jsp #写一个jsp的测试页面
  1 <%@ page contentType="text/html;charset=GBK" %>
  2 <%@ page import="java.util.*"%>
  3<html><head><title>Cluster AppTest</title></head>
  4 <body>
  5 Server Info:
  6 <%
  7 out.println(request.getLocalAddr() + ": " + request.getLocalPort()+"<br>");% >
  8 <%
  9 out.println("<br> ID " + session.getId()+"<br>");
10 String dataName =request.getParameter("dataName");
11 if (dataName != null &&dataName.length() > 0) {
12 String dataValue =request.getParameter("dataValue");
13 session.setAttribute(dataName, dataValue);
14 }
15 out.print("<b>Sessionlist</b>");
16 Enumeration e =session.getAttributeNames();
17 while (e.hasMoreElements()) {
18 String name = (String)e.nextElement();
19 String value =session.getAttribute(name).toString();
20 out.println( name + " = " +value+"<br>");
21 System.out.println( name + " = "+ value);
22 }
23 %>
24 <form action="test.jsp"method="POST">
25 name:<input type=text size=20name="dataName">
26 <br>
27 key:<input type=text size=20name="dataValue">
28 <br>
29 <input type=submit>
30 </form>
31 </body>
32 </html>

[iyunv@server2ROOT]# cd /usr/local/tomcat/lib
[iyunv@server2mnt]# ls /mnt/update/    #先前下载好的包
asm-3.2.jar                            minlog-1.2.jar
kryo-1.04.jar                         msm-kryo-serializer-1.6.3.jar
kryo-serializers-0.10.jar             reflectasm-1.01.jar
memcached-session-manager-1.6.3.jar    spymemcached-2.7.3.jar
memcached-session-manager-tc7-1.6.3.jar
[iyunv@server2mnt]# cd /usr/local/tomcat/lib/
[iyunv@server2lib]# mv /mnt/update/* . #将包移动到指定路径
[iyunv@server2lib]# ls
annotations-api.jar                memcached-session-manager-tc7-1.6.3.jar
asm-3.2.jar                         minlog-1.2.jar
catalina-ant.jar                   msm-kryo-serializer-1.6.3.jar
catalina-ha.jar                   reflectasm-1.01.jar
catalina.jar                      servlet-api.jar
catalina-tribes.jar                spymemcached-2.7.3.jar
ecj-4.2.1.jar                      tomcat-api.jar
el-api.jar                         tomcat-coyote.jar
jasper-el.jar                      tomcat-dbcp.jar
jasper.jar                         tomcat-i18n-es.jar
jsp-api.jar                         tomcat-i18n-fr.jar
kryo-1.04.jar                      tomcat-i18n-ja.jar
kryo-serializers-0.10.jar          tomcat-jdbc.jar
memcached-session-manager-1.6.3.jar  tomcat-util.jar
[iyunv@server2lib]# cd ..
UsingCATALINA_BASE: /usr/local/tomcat
UsingCATALINA_HOME: /usr/local/tomcat
UsingCATALINA_TMPDIR: /usr/local/tomcat/temp
UsingJRE_HOME:       /usr/local/java
UsingCLASSPATH:    /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
[iyunv@server2tomcat]# vim conf/context.xml #添加节点及节点失效后该怎么存储
34 <ManagerclassName="de.javakaffee.web.msm.MemcachedBackupSessionManager"
35memcachedNodes="n1:172.25.29.2:11211,n2:172.25.29.3:11211"    #结点
36 failoverNodes="n1"          #当n2失效时存储到n1上
37requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
38transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderF actory"
39 />
[iyunv@server2tomcat]# bin/startup.sh #开启Tomcat
服务端3和服务端2的配置基本相同唯一不同的是上面提到的/usr/local/tomcat/conf/context.xml里的36行把“n1”改成“n2”可以用scp传过去再做修改。

2.测试
测试 172.25.29.1/test.jsp

添加用户

添加完成

添加3个用户

[iyunv@server2tomcat]# tail -f logs/catalina.out    #查看日志
Sep28, 2016 11:39:53 PM de.javakaffee.web.msm.serializer.kryo.KryoTranscoder<init>
INFO:Starting with initialBufferSize 102400 and maxBufferSize 2048000
Sep28, 2016 11:39:53 PM de.javakaffee.web.msm.MemcachedSessionServicestartInternal
INFO:MemcachedSessionService finished initialization, sticky true, operation timeout1000, with node ids [n2] and failover node ids [n1]
Sep28, 2016 11:39:53 PM org.apache.coyote.AbstractProtocol start
INFO:Starting ProtocolHandler ["http-bio-8080"]
Sep28, 2016 11:39:53 PM org.apache.coyote.AbstractProtocol start
INFO:Starting ProtocolHandler ["ajp-bio-8009"]
Sep28, 2016 11:39:53 PM org.apache.catalina.startup.Catalina start
INFO:Server startup in 1921 ms
user1= 111
user2= 222
user1= 111          #上面创建的用户
user2= 222
user1= 111
user3= 333
^C
[iyunv@server2tomcat]# telnet 172.25.29.3 11211  #远程登陆服务端3查看是否是否写进入11211是memcached的端口号
Trying172.25.29.3...
Connectedto 172.25.29.3.
Escapecharacter is '^]'.
get50B9BAB1CBB21C9BF884CC3613560752-n2 #get后面的是上面截图里的ID
VALUE50B9BAB1CBB21C9BF884CC3613560752-n2 2048 125
[1]WWqt2Wqt01WqtWqt#50B9BAB1CBB21C9BF884CC3613560752-n2
user2
222user1
111user3
333    #大概可以看出来添加的用户
END
quit
Connectionclosed by foreign host.

[iyunv@server2tomcat]# bin/shutdown.sh  #关闭Tomcat1
UsingCATALINA_BASE: /usr/local/tomcat
UsingCATALINA_HOME: /usr/local/tomcat
UsingCATALINA_TMPDIR: /usr/local/tomcat/temp
UsingJRE_HOME:       /usr/local/java
UsingCLASSPATH:    /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar

当关闭Tomcat1续输入数据时并没有中断而是跳到Tomcat2上且数据并未丢失

[iyunv@server2tomcat]# telnet 172.25.29.3 11211 #远程登陆服务端3
Trying172.25.29.3...
Connectedto 172.25.29.3.
Escapecharacter is '^]'.
get50B9BAB1CBB21C9BF884CC3613560752-n2
VALUE50B9BAB1CBB21C9BF884CC3613560752-n2 2048 137
[1]WWqt2Wqu01WquWqu4#50B9BAB1CBB21C9BF884CC3613560752-n2

user2 222user1 111user4 444user3 333 #大概可以看到数据还是写入服务端3
END
quit
Connectionclosed by foreign host.

sjwayrhz · 发表于 2016-10-28 12:25:41

这个真是非常好的。

herry_dlzeng · 发表于 2016-11-27 14:33:38

很经典的东西

账号		自动登录	找回密码
密码			立即注册

VMware vcenter+vSphere 6.5 U2共享

【跟谁学】韩宇极简英语课-技术人员不得不

用Zabbix通过JMX方式监控weblogic

winhex数据恢复教程（非常巨大，内容丰富）

Symantec Backup Exec 2015 2016/2012 BE20

NetScaler VPX部署之：NetScaler Gateway调

zabbix3.4.1安装部署+微信推送信息+大屏显

[经验分享] Tomcat+Nginx+Memcacheed集群部署

扫码加入运维网微信交流群