flume运行错误收集

jqkyp

　　1）hostname错误：
　　

2011-11-14 11:44:55,497 ERROR com.cloudera.util.NetUtils: Unable to get canonical host name! test: test java.net.UnknownHostException: test: test
at java.net.InetAddress.getLocalHost(InetAddress.java:1354)
at com.cloudera.util.NetUtils.<clinit>(NetUtils.java:43)
at com.cloudera.flume.agent.FlumeNode.setup(FlumeNode.java:439)
at com.cloudera.flume.agent.FlumeNode.main(FlumeNode.java:665)     错误原因：不能从hostname获取ip地址　　
　　解决办法：在/etc/hosts文件中添加主机名到ip地址的映射
　　2）java没有按照
　　

line 234: exec: java: not found    错误原因：java命令不存在　　
　　解决办法：安装java包


　　3）flume node运行collector时报错：
　　

ERROR com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode vv_188_collector-20 in error state CollectorSource | Collector because Compression codec com.hadoop.compression.lzo.LzoCodec not found.   错误原因：flume找不到lzo的压缩库　　
　　解决办法：在/etc/profile.d/flume.sh中添加lzo压缩库的路径：
　　

export FLUME_CLASSPATH=/opt/hadoop-0.20.203.0/lib/hadoop-lzo-0.4.8.jar
　　重新启动flume node，再运行purge命令：perge vv_188_collector，重新加载配置
　　4）flume master报错：无法连接zookeeper服务
　　zk的错误日志：

2011-11-14 11:12:29,070 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server test/1.2.3.4:3181, sessionid = 0x133a00fbe560001, negotiated timeout = 5000
2011-11-14 15:30:31,436 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x133a00fbe560001, timeout of 5000ms exceeded
2011-11-14 15:31:16,735 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x133a00fbe560000, timeout of 5000ms exceeded
2011-11-14 15:31:16,853 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 13678ms for sessionid 0x133a00fbe560001, closing socket connection and attempting reconnect
2011-11-14 15:31:16,853 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 13323ms for sessionid 0x133a00fbe560000, closing socket connection and attempting reconnect
2011-11-14 15:31:16,853 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x133a00fbe560001, likely client has closed socket
2011-11-14 15:31:16,854 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x133a00fbe560001
2011-11-14 15:31:16,990 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x133a00fbe560000
2011-11-14 15:31:17,162 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /1.2.3.4:50430 which had sessionid 0x133a00fbe560001
2011-11-14 15:31:17,212 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /1.2.3.4:50429 which had sessionid 0x133a00fbe560000
2011-11-14 15:31:17,213 WARN org.apache.zookeeper.server.NIOServerCnxn: Ignoring unexpected runtime exception
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:69)
at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:241)
2011-11-14 15:31:18,332 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server test/1.2.3.4:3181
2011-11-14 15:31:18,333 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to test/1.2.3.4:3181, initiating session
2011-11-14 15:31:18,333 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /1.2.3.4:33044
2011-11-14 15:31:18,333 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to renew session 0x133a00fbe560001 at /1.2.3.4:33044
2011-11-14 15:31:18,334 INFO org.apache.zookeeper.server.NIOServerCnxn: Invalid session 0x133a00fbe560001 for client /1.2.3.4:33044, probably expired
2011-11-14 15:31:18,348 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2011-11-14 15:31:18,348 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x133a00fbe560001 has expired, closing socket connection
2011-11-14 15:31:18,500 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /1.2.3.4:33044 which had sessionid 0x133a00fbe560001
2011-11-14 15:31:18,510 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server test/1.2.3.4:3181
2011-11-14 15:31:18,511 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to test/1.2.3.4:3181, initiating session
2011-11-14 15:31:18,511 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /1.2.3.4:33045
2011-11-14 15:31:19,217 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to renew session 0x133a00fbe560000 at /1.2.3.4:33045
2011-11-14 15:31:19,218 INFO org.apache.zookeeper.server.NIOServerCnxn: Invalid session 0x133a00fbe560000 for client /1.2.3.4:33045, probably expired
2011-11-14 15:31:19,218 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /1.2.3.4:33045 which had sessionid 0x133a00fbe560000
2011-11-14 15:31:19,219 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x133a00fbe560000 has expired, closing socket connection
2011-11-14 15:31:19,219 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
　　从zk的日志来看，是flume的master连接zk的session超时，flume的master zk客户端尝试刷新获得的session，被zk的master拒绝
　　flume master的错误日志：
　　2011-11-14 11:12:29,070 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server test/1.2.3.4:3181, sessionid = 0x133a00fbe560001, negotiated timeout = 5000
2011-11-14 15:30:31,436 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x133a00fbe560001, timeout of 5000ms exceeded
2011-11-14 15:31:16,735 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x133a00fbe560000, timeout of 5000ms exceeded
2011-11-14 15:31:16,853 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 13678ms for sessionid 0x133a00fbe560001, closing socket connection and attempting reconnect
2011-11-14 15:31:16,853 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 13323ms for sessionid 0x133a00fbe560000, closing socket connection and attempting reconnect
2011-11-14 15:31:16,853 WARN org.apache.zookeeper.server.NIOServerCnxn: EndOfStreamException: Unable to read additional data from client sessionid 0x133a00fbe560001, likely client has closed socket
2011-11-14 15:31:16,854 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x133a00fbe560001
2011-11-14 15:31:16,990 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x133a00fbe560000
2011-11-14 15:31:17,162 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /1.2.3.4:50430 which had sessionid 0x133a00fbe560001
2011-11-14 15:31:17,212 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /1.2.3.4:50429 which had sessionid 0x133a00fbe560000
2011-11-14 15:31:17,213 WARN org.apache.zookeeper.server.NIOServerCnxn: Ignoring unexpected runtime exception
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:69)
at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:241)
2011-11-14 15:31:18,332 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server test/1.2.3.4:3181
2011-11-14 15:31:18,333 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to test/1.2.3.4:3181, initiating session
2011-11-14 15:31:18,333 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /1.2.3.4:33044
2011-11-14 15:31:18,333 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to renew session 0x133a00fbe560001 at /1.2.3.4:33044
2011-11-14 15:31:18,334 INFO org.apache.zookeeper.server.NIOServerCnxn: Invalid session 0x133a00fbe560001 for client /1.2.3.4:33044, probably expired
2011-11-14 15:31:18,348 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2011-11-14 15:31:18,348 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x133a00fbe560001 has expired, closing socket connection
2011-11-14 15:31:18,500 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /1.2.3.4:33044 which had sessionid 0x133a00fbe560001
2011-11-14 15:31:18,510 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server test/1.2.3.4:3181
2011-11-14 15:31:18,511 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to test/1.2.3.4:3181, initiating session
2011-11-14 15:31:18,511 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /1.2.3.4:33045
2011-11-14 15:31:19,217 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to renew session 0x133a00fbe560000 at /1.2.3.4:33045
2011-11-14 15:31:19,218 INFO org.apache.zookeeper.server.NIOServerCnxn: Invalid session 0x133a00fbe560000 for client /1.2.3.4:33045, probably expired
2011-11-14 15:31:19,218 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /1.2.3.4:33045 which had sessionid 0x133a00fbe560000
2011-11-14 15:31:19,219 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x133a00fbe560000 has expired, closing socket connection
2011-11-14 15:31:19,219 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down     解决办法：重启flume master节点