|
|
| 快速入门 |
要在Tomcat 5.5容器里进行session复制，必须完成下列步骤：
- 你的所有会话属性值必须实现java.io.Serializable
- 把server.xml文件里的群集(Cluster)元素的注释取消(Uncomment)
- 把server.xml文件里的Valve(ReplicationValve)元素注释取消(Uncomment)
- 如果有多个Tomcat实例在一台机器上运行，确保每个实例的tcpListenPort属性值是不冲突的。
- 确定 web.xml 中含有 <distributable/> 元素， 或者设置 <Context distributable="true" />
- 确定设置好Engine元素的 jvmRoute 的属性值： <Engine name="Catalina" jvmRoute="node01" >
- 确定所有节点拥有相同的时间， 并且通过网络时间服务(NTP)同步操作系统的时间！
- 确定负载均衡器已经配置为黏性会话模式.
负载均衡可以有很多种方式实现，请参考负载均衡章节。
注意：记住你的会话状态是被一个cookie跟踪的，所以你的URL从外面看必须是相同的，否则一个新的session就会被产生。
注意：集群支持目前需要JDK 1.4或其后版本。
|
| 概要 |
要在Tomcat里实现会话session复制，下面三种方法都可以实现相同效果：
- 使用session 持久化, 并将 session 保存到一个共享的文件系统(持久管理器 + 文件存储)(PersistenceManager + FileStore)
- 使用session 持久化, 并将 session 保存到一个共享的 数据库 (持久管理器 + JDBC存储) (PersistenceManager + JDBCStore)
- 使用in-memory-replication，使用和Tomcat 5自带的SimpleTcpCluster(server/lib/catalina-cluster.jar)
In this release of session replication, Tomcat performs an all-to-all replication of session state. This is an algorithm that is only efficient when the clusters are small. For large clusters, the next release will support a primary-secondary session replication where the session will only be stored at one or maybe two backup servers. Currently you can use the domain worker attribute (mod_jk > 1.2.8) to build cluster partitions with the potential of very scaleable cluster solution. In order to keep the network traffic down in an all-to-all environment, you can split your cluster into smaller groups. This can be easily achieved by using different multicast addresses for the different groups. A very simple setup would look like this:
What is important to mention here, is that session replication is only the beginning of clustering. Another popular concept used to implement clusters is farming, ie, you deploy your apps only to one server, and the cluster will distribute the deployments across the entire cluster. This is all capabilities that can go into with the FarmWarDeployer (s. cluster example at server.xml)
下一章将深入讨论session复制是怎样实现的，以及怎样配置它。
|
| 它是怎样工作的 |
为了便于理解集群是怎样工作的,我们将用一系列的场景来描述. 在这个场景中，我们只计划使用两个tomcat 实例 TomcatA 和 TomcatB. 我们将覆盖下面连续的事件:
TomcatA 启动
TomcatB 启动 (等待TomcatA 启动完成)
TomcatA 接受一个请求，session S1 被产生。
TomcatA 崩溃停止
TomcatB 接受一个session S1的请求
TomcatA 启动
TomcatA 接受一个请求，session ( S1 )失效。
TomcatB 接受一个新的session ( S2 )的请求。
TomcatA 因为没有活动session S2过期失效。
好了, 现在我们有了一个很好的顺序,我们将告诉你在session复制代码里发生的所有事情
TomcatA 启动 Tomcat 标准方式启动. 当 Host 对象被创建之后, 一个 cluster 对象与它关联. 当上下文被解析完, 如果 web.xml中有distributable元素， Tomcat 通过 Cluster class (in this case SimpleTcpCluster)为支持复制的上下文创建一个管理器. 因此通过设置web.xml中的distributable来启用集群。 Tomcat 将为上下文创建 DeltaManager莱取代 StandardManager. cluster class 将启动 a membership service (multicast) 和 a replication service (tcp unicast). More on the architecture further down in this document.
TomcatB 启动 当TomcatB启动时，除了一个例外以外，它按照TomcatA一样的顺序启动。群集被启用，并将建立一组会员(TomcatA,TomcatB)。TomcatB现在将要request已经存在于群集里的服务器的会话状态，在这里这个服务器是TomcatA。TomcatA回应TomcatB的请求，并且在TomcatB开始为HTTP requests监听之前，这个状态已经由TomcatA传递给TomcatB了。在TomcatA不回应的情况下，TomcatB在60秒后中断，并发出一个日志记录。对于每个已经分布在web.xml里的web应用程序，会话状态会被转移。注意：要有效地使用会话复制，你的所有tomcat实例要配置成一样的。
TomcatA 接受一个请求，session S1 被产生。 TomcatA对待传递来的请求与对待没有会话复制一样。当请求完成以后，活动就开始了，ReplicationValve在回应被返回到用户那里之前会截断这个请求。在这时，它发现会话被修改过，它就用TCP把这个会话复制给TomcatB。一旦这个分段的数据被递交给操作系统TCP logic，请求就通过valve pipeline返回给用户。对于每个请求，整个会话都被复制，这允许在不调用setAttribute 或 removeAttribute情况下，在会话中修改属性的代码被复制。配置参数useDirtyFlag可以被用来优化会话复制的次数。
TomcatA 崩溃停止 当TomcatA崩溃时，TomcatB会收到通知说TomcatA已经退出群集。TomcatB把TomcatA从它的会员列单中删除掉，TomcatB里有任何更改也不会再通知TomcatA。装载均衡器会把对TomcatA的请求引向TomcatB以及所有当前活动会话。
TomcatB 接受一个session S1的请求 TomcatB像处理其他任何请求一样处理这个请求。
TomcatA 启动 TomcatA启动时，在它接受新的请求以及让它自己可被使用之前，它要按照上面 1) 2)中所描述的顺序进行启动。它会加入群集，与TomcatB联系，了解所有会话的当前状态。一旦它接受到会话状态，它就完成装载并打开它的HTTP/mod_jk ports。所以在TomcatA接受到来自TomcatB的会话状态之前，不会对它发出请求。
TomcatA 接受一个请求，session ( S1 )失效。 失效的呼叫被拦截，这个会话就加入到失效的会话行列。在这个请求完成以后，它向TomcatB发出一个"expire"信息，TomcatB也把这个会话变为失效。
TomcatB 接受一个新的session ( S2 )的请求。 与步骤3)情形一样。
TomcatA 因为没有活动session S2过期失效。 就如一个会话由用户让它无效一样，invalidate呼叫被拦截到，这个会话就加入失效的会话行列。这一点上，除非另一个请求通过系统来检查失效行列，会话失效不会在其他地方重复。
Phuuuhh! :)
Membership Clustering membership可以通过使用非常简单的multicast pings被建立。每个Tomcat实例定期会发出multicast ping，在ping message里，这个实例会散布它的IP 及 TCP 监听端口以用于复制。如果一个实例在所给的时间范围内还没有收到这样的ping，这个会员就被认为是不存在了。很简单，也很有效。当然，你需要在你的系统上让multicasting可被使用。
TCP Replication，在收到一个multicast ping以后，会员就被添加到群集里。在下一个复制请求时，发出请求的实例将使用host和port信息来建立一个TCP socket。通过使用这个socket，它把分段的数据发送过去。我选择TCP sockets 的原因是它具备有流量控制，并确保发送到位。因此，我知道，当我发送数据时，这个数据就会被送到。
Distributed locking and pages using frames： Tomcat 并没有让会话实例在群集之间保持一致。这个逻辑的实现需要太大空间，也会带来很多问题。如果你的客户使用多个请求同时访问同一个会话，那么最后一个请求会覆盖群集里的其它会话。
|
| Cluster Architecture |
组件层次：
|
| Cluster Configuration |
群集配置在样品server.xml 文件里有描述。需要指出的是以mcastXXX开头的是关于会员成分multicast ping的属性，以tcpXXX开头的是关于TCP replication的属性。
会员成分是通过在相同的multicast IP 和 port上的所有tomcat实例建立的。TCP listen port是从其他会员那里来的会话复制在这它里被接收到的端口。
复制阀被用来发现请求什么时候被完成，并启动复制。
One of the most important performance considerations is the synchronous (pooled or not pooled) versus asynchronous replication mode. In a synchronous replication mode the request doesn't return until the replicated session has been sent over the wire and reinstantiated on all the other cluster nodes. There are two settings for synchronous replication. Pooled or not pooled. Not pooled (ie replicationMode="fastasnycqueue" or "synchronous") means that all the replication request are sent over a single socket. Using synchronous mode can potentially becomes a bottleneck when a lot of messages generated, You can overcome this bottleneck by setting replicationMode="pooled" but then you worker threads blocks with replication . What is recommended here is to increase the number of threads that handle incoming replication request. This is the tcpThreadCount property in the cluster section of server.xml. The pooled setting means that we are using multiple sockets, hence increases the performance. Asynchronous replication, should be used if you have sticky sessions until fail over, then your replicated data is not time crucial, but the request time is, at this time leave the tcpThreadCount to be number-of-nodes-1. During async replication, the request is returned before the data has been replicated. async replication yields shorter request times, and synchronous replication guarantees the session to be replicated before the request returns.
The parameter "replicationMode" has four different settings: "pooled", "synchronous", "asynchronous" and "fastasyncqueue"
| Simple Cluster Configuration |
Simple one line configuration
The default mode configuration setup a fastasyncqueue mode cluster configuration with following parameters:
- Open Membership receiver at 228.0.0.4 and send to multicast udp port 8012
- Send membership every 1 sec and drop member after 30sec.
- Open message receiver at default ip interface at first free port between 8015 and 8019.
- Receiver message with SocketReplicationListener
- Configure a ReplicationTransmitter with fastasyncqueue sender mode.
- Add ClusterSessionListener and ReplicationValve.
NOTE: Use this configuration when you need very quick a test cluster with at your developer machine. You can change the default attributes from cluster sub elements. Use following cluster attribute prefixes sender., receiver., service., manager., valve. and listener..
Example configure cluster at windows laptop with network connection and change receiver port range
WARNING: When you add you sub elements, there overwrite the defaults complete.
Example configure cluster with cluster failover jsessionid support. In this case you need also the defaultmode Cluster listener ClusterSessionListener and ReplicationValve.
|
| Simple Engine Cluster Configuration for all hosts |
Simple one line engine configuration
See default mode configuration description as simple host cluster example before.
|
| Complex Cluster Configuration |
Example Configure cluster with complete sub elements. Activate this node as master farm delopyer. Message receiver is NIO based ReplicationListener with six parallel worker threads.
| http://futureinhands.iteye.com/admin/blogs/images/void.gif | http://futureinhands.iteye.com/admin/blogs/images/void.gif | http://futureinhands.iteye.com/admin/blogs/images/void.gif | | http://futureinhands.iteye.com/admin/blogs/images/void.gif |
<Server port="8011"
shutdown="SHUTDOWN" >
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="9012"
protocol="AJP/1.3"
<Connector port="9013"
maxThreads="100"
minSpareThreads="4"
maxSpareThreads="4"
/>
<Engine name="Catalina"
defaultHost="localhost"
jvmRoute="node01">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase" />
<Host name="localhost"
appBase="webapps">
<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"
doClusterLog="true"
clusterLogName="clusterlog"
manager.className="org.apache.catalina.cluster.session.DeltaManager"
manager.expireSessionsOnShutdown="false"
manager.notifyListenersOnReplication="false"
manager.notifySessionListenersOnReplication="false"
manager.sendAllSessions="false"
manager.sendAllSessionsSize="500"
manager.sendAllSessionsWaitTime="20">
<Membership
className="org.apache.catalina.cluster.mcast.McastService"
mcastAddr="228.0.0.4"
mcastBindAddress="127.0.0.1"
mcastClusterDomain="d10"
mcastPort="45564"
mcastFrequency="1000"
mcastDropTime="30000"/>
<Receiver
className="org.apache.catalina.cluster.tcp.ReplicationListener"
tcpListenAddress="auto"
tcpListenPort="9015"
tcpSelectorTimeout="100"
tcpThreadCount="6"
<Sender
className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"
replicationMode="fastasyncqueue"
recoverTimeout="5000"
recoverCounter="6"
doTransmitterProcessingStats="true"
doProcessingStats="true"
doWaitAckStats="true"
queueTimeWait="true"
queueDoStats="true"
queueCheckLock="true"
ackTimeout="15000"
waitForAck="true"
keepAliveTimeout="80000"
keepAliveMaxRequestCount="-1"/>
<Valve className="org.apache.catalina.cluster.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.css;.*\.png;.*\.jpeg;.*\.jpg;.*\.htm;.*\.html;.*\.txt;"
primaryIndicator="true" />
<Valve className="org.apache.catalina.cluster.session.JvmRouteBinderValve"
enabled="true" />
<ClusterListener className="org.apache.catalina.cluster.session.ClusterSessionListener" />
<ClusterListener className="org.apache.catalina.cluster.session.JvmRouteSessionIDBinderListener" />
<Deployer className="org.apache.catalina.cluster.deploy.FarmWarDeployer"
tempDir="${catalina.base}/war-temp"
deployDir="${catalina.base}/war-deploy/"
watchDir="${catalina.base}/war-listen/"
watchEnabled="true"/>
</Cluster>
</Host>
</Engine>
</Service>
</Server>
| http://futureinhands.iteye.com/admin/blogs/images/void.gif | | http://futureinhands.iteye.com/admin/blogs/images/void.gif | http://futureinhands.iteye.com/admin/blogs/images/void.gif | http://futureinhands.iteye.com/admin/blogs/images/void.gif |
|
| Cluster Configuration for ReplicationTransmitter |
List of Attributes
属性
描述
Default value
| replicationMode | replication mode (synchronous, pooled, asynchronous or fastasyncqueue) | pooled | | processSenderFrequency | Control the sender keepalive status and drop sender socket connection after timeout is reached. Check every processSenderFrequency value engine background ticks. | 2 | | compress | compress bytes before sending (consume memory, but reduce network traffic - GZIP) | false | | ackTimeout | acknowledge timeout and only usefull it waitForAck is true | 15000 msec | | waitForAck | Wait for ack after data send | false | | autoConnect | is sender disabled, fork a new socket | false | | doTransmitterProcessingStats | create processing time stats | false | |
Example to get statistic information, wait for ack at every message send and transfer at compressed mode
| Cluster Configuration for ReplicationTransmitter (fastayncqueue - mode) |
List of Attributes
属性
描述
Default value
| keepAliveTimeout | active socket keep alive timeout | 60000 msec | | keepAliveMaxRequestCount | max request over this socket | -1 | | doProcessingStats | create Processing time stats | false | | doWaitAckStats | create waitAck time stats | false | | resend | resend message after failure, can overwrite at message | false | | recoverTimeout | recover Timeout after push message failure | 5000 msec | | recoverCounter | number of recover tries | 0 | | queueDoStats | activated queue stats | false | | queueCheckLock | check to lost locks | false | | queueAddWaitTimeout | queue add wait time (tomcat connector thread waits) | 10000 msec | | queueRemoveWaitTimeout | queue remove wait time (queue thread waits) | 30000 msec | | maxQueueLength | max queue length (default without limit) | -1 | | threadPriority | change queue thread priority (1-10 ; 5 is normal) | 5 | |
Example to get a lot of statistic information, wait for ACK and recover after connection failure (wait 5 secs and 6 trails (==30 secs Mcast Timeout)
| Cluster Configuration for ReplicationTransmitter ( asynchronous - mode) |
List of Attributes
属性
描述
Default value
| keepAliveTimeout | active socket keep alive timeout | 60000 msec | | keepAliveMaxRequestCount | max request over this socket | -1 | | doProcessingStats | create Processing time stats | false | | doWaitAckStats | create waitAck time stats | false | | resend | resend message after failure, can overwrite at message | false | |
Example to get a processing statistic information, resend after failure and wait for ACK
| Cluster Configuration for ReplicationTransmitter ( synchronous - mode) |
List of Attributes
属性
描述
Default value
| keepAliveTimeout | active socket keep alive timeout | 60000 msec | | keepAliveMaxRequestCount | max request over this socket | -1 | | doProcessingStats | create Processing time stats | false | | doWaitAckStats | create waitAck time stats | true | | resend | resend message after failure, can overwrite at message | false | |
Example to get a no processing statistic information, no wait for ACK, after 10000 request renew socket and autoconnect before first request is send.
| Cluster Configuration for ReplicationTransmitter ( pooled - mode) |
List of Attributes
属性
描述
Default value
| keepAliveTimeout | active socket keep alive timeout | 60000 msec | | keepAliveMaxRequestCount | max request over this socket | -1 | | maxPoolSocketLimit | max pooled sockets (Sender Sockets) | 25 | | resend | resend message after failure, can overwrite at message | false | |
Example to get a no processing statistic information, wait for ACK, after 10000 request renew socket, only 10 SockerSender available and autoconnect before first request is send.
| Cluster Configuration for ReplicationTransmitter ( DeltaManager Attribute) |
List of Attributes
属性
描述
Default value
| expireSessionsOnShutdown | When server stopped, expire all sessions also at backup nodes (only for testing) | false | | maxActiveSessions | Number of active sessions. (Default is no limit) | -1 | | notifyListenersOnReplication | Notify application session listener to session creation and expiring events at backup nodes | true | | notifySessionListenersOnReplication | Notify application session listener to attribute changes at backup nodes | true | | stateTransferTimeout | Timeout that session state transfer is complete. Is attribute stateTransferTimeout == -1 then application wait that other node send the complete session state | 60 sec | | sendAllSessions | Flag to send sessions as splited blocks | true | | sendAllSessionsSize | Number of serialize sessions inside a send block session message. Only useful when sendAllSessions==false
| 1000 | | sendAllSessionsWaitTime | wait time between two session send blocks. | 2000 msec | | sendClusterDomainOnly | Send all session messages only to member inside same cluster domain (value od Membership attribute mcastClusterDomain). Also don't handle session messages from other domains. | true | | stateTimestampDrop | DeltaManager queued Sessions messages when send GET_ALL_SESSION to other node. with stateTimestampDrop all messages before state transfer message creation date (find session) are dropped. Only other GET_ALL_SESSION events are handle with date before state transfer message. | true | |
Example send all sessions at separate blocks. Serialize and send 100 session inside one block. Wait maximale two minutes before the complete backup sessions are loaded inside tomcat boot process. Between send blocks wait 5 secs to transfers the session block to other node. This save memory when you use the async modes with queues.
注意
As Cluster.defaultMode=true you can configure the manager attributes with prefix manager..
注意
With Cluster.setProperty(<String>,<String>) you can modify attributes for all register managers. The method exists as MBeans operation.
| Bind session after crash to failover node |
As you configure more then two nodes at same cluster for backup, most loadbalancer send don't all your requests after failover to the same node.
The JvmRouteBinderValve handle tomcat jvmRoute takeover using mod_jk module after node failure. After a node crashed the next request going to other cluster node. The JvmRouteBinderValve now detect the takeover and rewrite the jsessionid information to the backup cluster node. After the next response all client request goes direct to the backup node. The change sessionid send also to all other cluster nodes. Well, now the session stickyness work directly to the backup node, but traffic don't go back too restarted cluster nodes!
As jsessionid was created by cookie, the change JSESSIONID cookie resend with next response.
You must add JvmRouteBinderValve and the corresponding cluster message listener JvmRouteSessionIDBinderListener. As you add the new listener you must also add the default ClusterSessionListener that receiver the normal cluster messages.
Hint:
With attribute sessionIdAttribute you can change the request attribute name that included the old session id. Default attribuite name is org.apache.catalina.cluster.session.JvmRouteOrignalSessionID.
Trick:
You can enable this mod_jk turnover mode via JMX before you drop a node to all backup nodes! Set enable true on all JvmRouteBinderValve backups, disable worker at mod_jk and then drop node and restart it! Then enable mod_jk Worker and disable JvmRouteBinderValves again. This use case means that only requested session are migrated.
|
|
[table=98%]
[tr]
[td]通过 JMX 监控集群[/td]
[/tr]
[tr]
[td]
Monitoring is a very important question when you use a cluster. Some of the cluster objects are JMX MBeans
Add the following parameter to your startup script with Java 5:
[table]
[tr]
[td=1,1,1]http://futureinhands.iteye.com/admin/blogs/images/void.gif[/td]
[td]http://futureinhands.iteye.com/admin/blogs/images/void.gif[/td]
[td=1,1,1]http://futureinhands.iteye.com/admin/blogs/images/void.gif[/td]
[/tr]
[tr]
[td=1,1,1]http://futureinhands.iteye.com/admin/blogs/images/void.gif[/td]
[td]
set CATALINA_OPTS=\
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=%my.jmx.port% \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false
[/td]
[td=1,1,1]http://futureinhands.iteye.com/admin/blogs/images/void.gif[/td]
[/tr]
[tr]
[td=1,1,1]http://futureinhands.iteye.com/admin/blogs/images/void.gif[/td]
[td]http://futureinhands.iteye.com/admin/blogs/images/void.gif[/td]
[td=1,1,1]http://futureinhands.iteye.com/admin/blogs/images/void.gif heigh |
|
QQ群⑧：
窥视卡
雷达卡
发表于 2017-1-5 10:57:14
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡