flume-ng1.4 安装配置

54432

flume-ng 1.4 安装配置
记录一下flume-ng 的安装配置


三个基本概念

source：数据源
channel：数据传输通道
sink：数据目的地，如存储在hdfs
配置：

flume.conf


#define
agent1.sources = source1
agent1.channels = channel1
agent1.sinks = sink1

#
agent1.sources.source1.type = spooldir
agent1.sources.source1.spoolDir = /tmp/test/
agent1.sources.source1.deletePolicy = immediate

#Describe the sink
agent1.sinks.sink1.type = logger

# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapacity = 100

#Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
##sink配置

sink配置为logger类型，存储在日志文件中，测试无问题后再改为存储在hdfs中。

##源配置

将源的类型设置为spooldir用于监视目录下的文件

指定spooDir为tmp下的test文件夹

删除策略设置为及时删除，默认是不删除。



启动
flume-ng agent --conf /usr/local/flume/conf/ --conf-file flume.conf --name agent1 -Dflume.root.logger=INFO,console

测试
cd /tmp/test/
echo 'good day' > 123
结果

2014-06-22 10:03:39,540 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.deleteCurrentFile(ReliableSpoolingFileEventReader.java:366)] Preparing to delete file /tmp/test/123
2014-06-22 10:03:40,934 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{file=/tmp/test/123} body: 67 6F 6F 64 20 64 61 79                         good day }