线上tomcat 网络无法访问问题排查

赵小黑

线上某台机器tomcat启动无法访问,通过jstack报错。
Unable to open socket file: target process not responding or HotSpot VM not loaded
因为这个错误提示误导我以为JDK的问题，升级JDK到1.7问题依旧。
 
这个时候查看日志发现：
INFO: validateJarFile(/home/admin/tlog-proxy/target/tlog-proxy.war/WEB-INF/lib/jsp-api-2.1-glassfish-2.1.v20091210.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
2015-12-03 02:43:53,332 org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/home/admin/tlog-proxy/target/tlog-proxy.war/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
2015-12-03 02:44:03,987 org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /home/admin/tlog-proxy/.default/deploy/tlog-proxy.war has finished in 10,908 ms
2015-12-03 02:44:03,990 org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-7001"]
2015-12-03 02:44:04,051 org.apache.catalina.startup.Catalina start
INFO: Server startup in 11069 ms
2015-12-03 02:44:09,258 org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: 打开的文件过多
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
        at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:220)
        at java.lang.Thread.run(Thread.java:745)
 
lsof | grep java | wc -l发现超过1万，正常应该只有几百。然后查看发现是抓取任务太多导致。
 
事后排查抓取任务才几百个，为啥会创建出上万个连接呢，最后排查发现HttpClient没有调用 close
 
　　public void close() {
　　if (client != null) {
　　client.getConnectionManager().closeIdleConnections(0, TimeUnit.SECONDS);
　　}
　　}