停止Tomcat webapp报错的尝试解决

tonwei139

　　ubuntu12下 tomcat的版本是7.0.40.0
　　状况:
　　发现某个web项目最近往往执行bin/shutdown.sh无法关闭tomcat服务器，必须手动kill进程才行。
　　该web项目主要执行定时执行大量的http请求 将返回结果处理后存入mongo数据库
　　因此查看catalina.out日志，发现停止tomcat时抛出很多内存泄漏的警告 以下:
　　INFO: Stopping service Catalina
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [AsyncAppender-Worker-Thread-2] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [ZkClient-EventThread-14-192.168.161.35:2181,192.168.161.36:2181,192.168.161.37:2181] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [localhost-startStop-1-SendThread(192.168.161.37:2181)] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [localhost-startStop-1-EventThread] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-1-thread-2] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-1-thread-3] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [MongoCleaner1479337147] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-2-thread-2] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-2-thread-3] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [MongoCleaner58644297] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [pool-4-thread-2] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [pool-4-thread-3] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [pool-4-thread-4] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

....

SEVERE: The web application [xxxxx] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5a2224fb]) and a value of type [aaaaa] (value [aaaaa@77aa08f7]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [xxxxx] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5a2224fb]) and a value of type [aaaaa] (value [aaaaa@3c32cd81]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [xxxxx] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5a2224fb]) and a value of type [aaaaa] (value [aaaaa@2b51ef07]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [xxxxx] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5a2224fb]) and a value of type [aaaaa] (value [aaaaa@1023cac9]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [xxxxx] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5a2224fb]) and a value of type [aaaaa] (value [aaaaa@1dd65a66]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
　　其中线程名pool-4-thread-xxx引起了注意，应该是一个线程池，查看代码，发现为批量发送HTTP请求有建了一个ThreadPoolExecutor

    private static ThreadPoolService threadPoolService = new ThreadPoolService();
private ThreadPoolService() {
this.threadPoolExecutor = new ThreadPoolExecutor(100, 150, 500, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), new ThreadPoolExecutor.CallerRunsPolicy());
// this.completionService = new
// ExecutorCompletionService<Object>(this.threadPoolExecutor);
}
　　日志中名叫pool-4-thread-xxx 有100个，很明显，tomcat没能关掉ThreadPoolExecutor的核心线程，因此需要在关闭tomcat前手动关闭，因此在web.xml中新增一个ServletContextListener的实现

@Override
public void contextDestroyed(ServletContextEvent sce) {
ThreadPoolService.getInstance().shutdownNow();
}
　　重新尝试 果然所有的
　　SEVERE: The web application [xxxxx] appears to have started a thread named [pool-4-thread-x] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
　　没有了，还剩下一些ThreadLocal无法释放以及剩下一些Thread无法释放的警告
　　查资料
　　http://stackoverflow.com/questions/5292349/is-this-very-likely-to-create-a-memory-leak-in-tomcat
　　回复中给出了ThreadLocal的解决方案

public Integer immolate() {
int count = 0;
try {
final Field threadLocalsField = Thread.class.getDeclaredField("threadLocals");
threadLocalsField.setAccessible(true);
final Field inheritableThreadLocalsField = Thread.class.getDeclaredField("inheritableThreadLocals");
inheritableThreadLocalsField.setAccessible(true);
for (final Thread thread : Thread.getAllStackTraces().keySet()) {
count += clear(threadLocalsField.get(thread));
count += clear(inheritableThreadLocalsField.get(thread));
}
logger.info("immolated " + count + " values in ThreadLocals");
} catch (Exception e) {
throw new Error("ThreadLocalImmolater.immolate()", e);
}
return count;
}
private int clear(final Object threadLocalMap) throws Exception {
if (threadLocalMap == null)
return 0;
int count = 0;
final Field tableField = threadLocalMap.getClass().getDeclaredField("table");
tableField.setAccessible(true);
final Object table = tableField.get(threadLocalMap);
for (int i = 0, length = Array.getLength(table); i < length; ++i) {
final Object entry = Array.get(table, i);
if (entry != null) {
final Object threadLocal = ((WeakReference)entry).get();
if (threadLocal != null) {
log(i, threadLocal);
Array.set(table, i, null);
++count;
}
}
}
return count;
}
　　根据答复中的代码猜想，ThreadLocal的报错 估计是某些线程的ThreadLocal无法释放，为什么无法释放，因为那些线程还没停掉，每个ThreadLocal都是被一个Thread的ThreadMap下以<ThreadLocalObject, Object>的entry形式维护着，
　　这些entry继承了WeakReference，以上代码应该是将每个thread的threadMap的entry设成null，这样原来的entry没有引用源，作为一个WeakReference会在GC中被清除掉。
　　事实上也是如此，在将ServletContextListener换成

public void contextDestroyed(ServletContextEvent sce) {
ThreadPoolService.getInstance().shutdownNow();
immolate();
}
　　后，tomcat关闭报错只剩下一些Thread无法停止了
　　INFO: Stopping service Catalina
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [AsyncAppender-Worker-Thread-2] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [ZkClient-EventThread-14-192.168.161.35:2181,192.168.161.36:2181,192.168.161.37:2181] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [localhost-startStop-1-SendThread(192.168.161.37:2181)] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [localhost-startStop-1-EventThread] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-1-thread-2] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-1-thread-3] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [MongoCleaner1479337147] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-2-thread-2] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [cluster-2-thread-3] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [xxxxx] appears to have started a thread named [MongoCleaner58644297] but has failed to stop it. This is very likely to create a memory leak.
Feb 28, 2015 9:45:04 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
　　这些线程可能来自于各种框架以及各种未知...很难获取到它们去挨个停止，如果根据名称去找然后停止的话，又比较傻...继续查资料
　　http://wiki.apache.org/tomcat/MemoryLeakProtection 了解到
If a webapp creates a thread, by default its context classloader is set to the one of the parent thread (the thread that created the new thread). In a webapp, this parent thread is one of tomcat worker threads, whose context classloader is set to the webapp classloader when it executes webapp code.
Furthermore, the spawned thread may be executing (or blocked in) some code that involves classes loaded by the webapp, thus preventing the webapp classloader from being collected.
So, if the spawned thread is not properly terminated when the application is stopped, the webapp classloader will leak because of the strong reference held by the spawned thread.
 
就是说 在tomcat容器里，每个线程的ClassLoader归根结底都被默认设置成webapp ClassLoader。当某些线程无法及时关闭时，webapp classloader就会因为这些线程拥有的强引用，无法正常gc，因此报警。
了解了这个，就有办法了，修改public Integer immolate() method 插入几行

for (final Thread thread : Thread.getAllStackTraces().keySet()) {
count += clear(threadLocalsField.get(thread));
count += clear(inheritableThreadLocalsField.get(thread));
if (thread != null) {
thread.setContextClassLoader(null);
}
}
 
这样这些还没停止的线程的上下文ClassLoader就和webapp ClassLoader无关了，webapp ClassLoader可以正常GC，报错消失。
　　问题是解决了，但其中很多细节半猜半试出来的，并没有吃透它们，这些还待自己慢慢学习琢磨。