Flume: avro source and sink
In order to flow the data across multiple agents or hops, the sink of the previous agent and source of the current hop need to be avro type with the sink pointing to the hostname (or IP address) and port of the source.http://flume.apache.org/_images/UserGuide_image03.png
Hop 1:
a1.channels.ch1.type = memory
a1.sources.avro-source1.channels = ch1
a1.sources.avro-source1.type = avro
a1.sources.avro-source1.bind = 0.0.0.0
a1.sources.avro-source1.port = 41414
a1.sinks.avro-sink1.channel = ch1
a1.sinks.avro-sink1.type = avro
a1.sinks.avro-sink1.hostname = 192.168.10.204
a1.sinks.avro-sink1.port = 5545
a1.sinks.avro-sink1.batch-size = 1
a1.channels = ch1
a1.sources = avro-source1
a1.sinks = avro-sink1
Hop2
a1.channels.ch1.type = memory
a1.sources.avro-source1.channels = ch1
a1.sources.avro-source1.type = avro
a1.sources.avro-source1.bind = 0.0.0.0
a1.sources.avro-source1.port = 5545
a1.sinks.log-sink1.channel = ch1
a1.sinks.log-sink1.type = logger
a1.channels = ch1
a1.sources = avro-source1
a1.sinks = log-sink1
test using avro-client
flume-ng avro-client -H dmining05 --port 41414 -F /etc/passwd
References
http://flume.apache.org/FlumeUserGuide.html
页:
[1]