使用flume的一个例子
新项目中需要使用到hadoop和vertica,使用flume把数据加载到hadoop中，我做了一个例子，即监控一个sharefolder,如果里面有文件，则会文件load到hadoop.
开启Flume agent:
./flume-ng agent -n agent-1-c conf -f /home/yaxiaohu/flumeconf/evantest.conf
以下内容是evantest.conf
agent-1.channels = ch-1
agent-1.sources = src-1
agent-1.sinks =hdfs-sink
agent-1.channels.ch-1.capacity = 1000
agent-1.channels.ch-1.transactionCapacity = 100
agent-1.channels.ch-1.type = memory
agent-1.sources.src-1.channels=ch-1
agent-1.sources.hdfs-sink.channel=ch-1
agent-1.sources.src-1.type = spooldir
agent-1.sources.src-1.channels = ch-1
agent-1.sources.src-1.spoolDir = /home/yaxiaohu/flumetest
agent-1.sources.src-1.fileHeader = true
agent-1.sources.src-1.deletePolicy = immediate
agent-1.sinks.hdfs-sink.type = hdfs
agent-1.sinks.hdfs-sink.channel = ch-1
agent-1.sinks.hdfs-sink.hdfs.path = /FAA/flume/%y-%m-%d/%H%M
agent-1.sinks.hdfs-sink.hdfs.filePrefix = %{Evan}
agent-1.sinks.hdfs-sink.hdfs.round = true
agent-1.sinks.hdfs-sink.hdfs.roundValue = 10
agent-1.sinks.hdfs-sink.hdfs.roundUnit = minute
agent-1.sinks.hdfs-sink.hdfs.useLocalTimeStamp=True
页:
[1]