安装flume-1.6.0后,测试所遇bug

所谓覅破解

　　案例1：Avro
　　Avro可以发送一个给定的文件给Flume，Avro 源使用AVRO RPC机制。
　　　　　　a)创建agent配置文件
　　vi /usr/local/flume-1.6.0/conf/avro.conf
　　

a1.sources = r1a1.sinks = k1a1.channels = c1  # Describe/configure the sourcea1.sources.r1.type= avroa1.sources.r1.channels = c1a1.sources.r1.bind = 0.0.0.0a1.sources.r1.port = 4141  # Describe the sinka1.sinks.k1.type= logger  # Use a channel which buffers events in memorya1.channels.c1.type= memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100  # Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1　　
　　

　　启动 flume
　　./flume-ng agent -c . -f /usr/local/flume-1.6.0/conf/avro.conf -n a1 -Dflume.root.logger=INFO,console
　　打印如下日志
　　Info: Including Hive libraries found via () for Hive access
+ exec /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/usr/local/flume-1.6.0/bin:/usr/local/flume-1.6.0/lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /usr/local/flume-1.6.0/conf/avro.conf-n a1
log4j:WARN No appenders could be found for logger (org.apache.flume.lifecycle.LifecycleSupervisor).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
　　

　　创建指定文件
　　echo "hello world" > /usr/local/flume-1.6.0/test.log

　　使用avro-client发送文件
　　./flume-ng avro-client -c /usr/local/flume-1.6.0/conf -H m1 -p 4141 -F/usr/local/flume-1.6.0/test.log

　　控制台无反应,测试失败!

　　查询资料发现是因为使用 -c 指定的conf位置出错;
　　修改执行命令:   ./flume-ng agent -c /usr/local/flume-1.6.0/conf -f /usr/local/flume-1.6.0/conf/avro.conf -n a1 -Dflume.root.logger=INFO,console
　　控制台输出日志如下:
　　 ......
　　2015-10-21 15:45:13,739 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink k1
2015-10-21 15:45:13,744 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source r1
2015-10-21 15:45:13,745 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:228)] Starting Avro source r1: { bindAddress: 0.0.0.0, port: 4141 }...
2015-10-21 15:45:14,133 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
2015-10-21 15:45:14,133 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SOURCE, name: r1 started
2015-10-21 15:45:14,134 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:253)] Avro source r1 started.

启动成功!
　　

　　案例二:
　　使用flume,直接将文件压缩后,上传到hdfs上.
　　定义conf文件  vi hdfs.conf

　　#定义agent名， source、channel、sink的名称
a1.sources = r1
a1.channels = c1
a1.sinks = k1

#具体定义source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/hadoop/logs

#具体定义channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 100

#定义拦截器，为消息添加时间戳
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
a1.sources.r1.ignorePattern = ^(.)*\\.tmp$

#具体定义sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://nn1.hadoop:9000/user/flume/syslogtcp/%Y%m%d%H
a1.sinks.k1.hdfs.filePrefix = events-
#a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.fileType = CompressedStream
a1.sinks.k1.hdfs.codeC = bzip2
#不按照条数生成文件
a1.sinks.k1.hdfs.rollCount = 0
#HDFS上的文件达到128M时生成一个文件 134217728
a1.sinks.k1.hdfs.rollSize = 2097152
#HDFS上的文件达到60秒生成一个文件
a1.sinks.k1.hdfs.rollInterval = 60
　　

　　其中hdfs.conf中标红的是设置传输格式和压缩格式,其中a1.sinks.k1.hdfs.fileType这一配置项只有 SequenceFile, DataStream or CompressedStream三种配置,默认为SequenceFile,二进制不能直接查看.DataStream不压缩输出文件，不能设置codeC选项;CompressedStream为压缩输出文件,hdfs.codeC必须设置,不能为空;
　　hdfs.codeC具体可以支持的压缩类型,通过查看源码找到为(gzip, bzip2, lzo, snappy)

　　a1.sinks.k1.hdfs.fileType = CompressedStream
a1.sinks.k1.hdfs.codeC = bzip2
　　------------------------------------------------- 分割线

　　a1.sources.r1.ignorePattern = ^(.)*\\.tmp$
　　   此项设置是忽略监控目录下的临时文件,防止出现,边写边读的情况
　　  ERROR org.apache.flume.source.SpoolDirectorySource: FATAL: Spool Directory source source1: { spoolDir: /home/hadoop/logs}: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.java.nio.charset.MalformedInputException
　　此异常可以通过设置此参数解决:a1.sources.r1.ignorePattern = ^(.)*\\.tmp$