yiwi 发表于 2015-11-27 19:58:28

Flume例子:使用netcat和exec作为信源,logger和file作为信宿,并分别为每对信源和信宿实现一个信道

  
  一展身手:使用netcat和exec作为信源,logger和file作为信宿,并分别为每对信源和信宿实现一个信道:
  (1)程序如下:保存为agentx.conf文件:
  agentx.sources = netsource execsource
  agentx.sinks = logsink filesink
  agentx.channels = memorychannel1memorychanne2
  
  agentx.sources.netsource.type = netcat
  agentx.sources.netsource.bind = 172.16.2.17
  agentx.sources.netsource.port = 3000
  
  agentx.sinks.logsink.type = logger
  
  agentx.channels.memorychannel1.type = memory
  agentx.channels.memorychannel1.capacity = 1000
  agentx.channels.memorychannel1.transactionCapacity = 100
  
  agentx.sources.netsource.channels = memorychannel1
  agentx.sinks.logsink.channel = memorychannel1
  
  agentx.sources.execsource.type = exec
  agentx.sources.execsource.command = cat /home/hadoop/message
  
  agentx.sinks.filesink.type = FILE_ROLL
  agentx.sinks.filesink.sink.directory = /home/hadoop/flume/files
  agentx.sinks.filesink.sink.rollInterval = 0
  
  agentx.channels.memorychanne2.type = file
  agentx.channels.memorychanne2.checkpointDir = /home/hadoop/flume/fc/checkpoint
  agentx.channels.memorychanne2.dataDirs = /home/hadoop/flume/fc/data
  
  agentx.sources.execsource.channels = memorychanne2
  agentx.sinks.filesink.channel = memorychanne2
  (2)将agentx.conf放到flume工作的相应位置,我的是 /usr/hadoop/flume目录下,然后执行命令:
  $ ls
  agent1.confagent2.confagent3.confagentx.confconfderby.loglogs
  $ flume-ng agent --conf conf --conf-file agentx.conf --name agentx
  Info: Sourcing environment configuration script /usr/hadoop/flume/conf/flume-env.sh
  Info: Including Hadoop libraries found via (/usr/hadoop/bin/hadoop) for HDFS access
  Info: Excluding /usr/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from classpath
  Info: Excluding /usr/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from classpath
  中间省略。。。
  
  (3)然后在从机slave1中,通过telnet远程访问主机master,并输入命令(需要先安装telnet: 使用买了yum install -y telnet 进行安装即可):
  $ curl telnet://172.16.2.17:3000
  Test from slave1
  OK
  about netcat and
  OK
  exec!
  OK
  ^C
  $
  
  (4)查看log的输出:
  05 Jan 2015 10:32:58,302 INFO (org.apache.flume.sink.LoggerSink.process:70)- Event: { headers:{} body: 54 65 73 74 20 66 72 6F 6D 20 73 6C 61 76 65 31 Test from slave1 }
  05 Jan 2015 10:33:07,306 INFO (org.apache.flume.sink.LoggerSink.process:70)- Event: { headers:{} body: 61 62 6F 75 74 20 6E 65 74 63 61 74 20 61 6E 64 about netcat and }
  05 Jan 2015 10:33:23,936 INFO (org.apache.flume.sink.LoggerSink.process:70)- Event: { headers:{} body: 65 78 65 63 21                                  exec! }
  (5)查看files的输出:
  $ cd /home/hadoop/flume/files/
  $ ls
  1420384890083-11420386984869-11420388077115-11420423829240-1
  $ cat 1420423829240-1
  Hello netcat and exec Flume!
页: [1]
查看完整版本: Flume例子:使用netcat和exec作为信源,logger和file作为信宿,并分别为每对信源和信宿实现一个信道