Flume之ChannelSelector源码分析

disl

　　Flume中channel选择器（selector.type配置）必须实现ChannelSelector接口，实现了该接口的类主要作用是告诉Source中接收到的Event应该发送到哪些Channel，在Flume中主要由两个实现方式：
　　1，复用，实现类：MultiplexingChannelSelector
　　2，复制，实现类：ReplicatingChannelSelector

　　如下：

　　
　　ChannelSelector接口两个主要的方法是：


//获取必选的Channel列表
public List<Channel> getRequiredChannels(Event event);
//获取可选的Channel列表
public List<Channel> getOptionalChannels(Event event);

ReplicatingChannelSelector (所有Channel默认的方式)

属性名默认描述

selector.type


replicating


组件名：replicating




selector.optional


–


标记哪些Channels是可选的


以下例子将c3标记为可选，写入c3失败的话会被忽略，如果写入c1和c2失败的话，这个事务就会失败:

a1.sources = r1
a1.channels = c1 c2 c3
a1.source.r1.selector.type = replicating
a1.source.r1.channels = c1 c2 c3
a1.source.r1.selector.optional = c3
ReplicatingChannelSelector初始化过程：
public void configure(Context context) {
//获取哪些Channel标记为可选
String optionalList = context.getString(CONFIG_OPTIONAL);
//将所有Channel都方法必须的Channel列表中
requiredChannels = new ArrayList<Channel>(getAllChannels());
Map<String, Channel> channelNameMap = getChannelNameMap();
if(optionalList != null && !optionalList.isEmpty()) {
//下面的操作：如果channel属于可选的，则加入可选的列表中，并从必选的列表中删除
for(String optional : optionalList.split("\\s+")) {
Channel optionalChannel = channelNameMap.get(optional);
requiredChannels.remove(optionalChannel);
if (!optionalChannels.contains(optionalChannel)) {
optionalChannels.add(optionalChannel);
}
}
}
}
MultiplexingChannelSelector


属性名
默认
Description

selector.type
replicating
组件名：multiplexing


selector.optional
–

标记哪些Channels是可选的






selector.header




flume.selector.header








selector.default
–





selector.mapping.*
–




示例:

a1.sources = r1
a1.channels = c1 c2 c3 c4
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = state
a1.sources.r1.selector.mapping.CZ = c1
a1.sources.r1.selector.mapping.US = c2 c3
a1.sources.r1.selector.default = c4
根据header中key为state的值，决定将数据写入那个channel中，如上示例将state=CZ写入到c1中，将state=US写入到c2，c3中，默认情况下写入c4




MultiplexingChannelSelector的初始化过程：
  public void configure(Context context) {
//获取Header的值
this.headerName = context.getString(CONFIG_MULTIPLEX_HEADER_NAME,
DEFAULT_MULTIPLEX_HEADER);
Map<String, Channel> channelNameMap = getChannelNameMap();
//获取默认的Channel
defaultChannels = getChannelListFromNames(
context.getString(CONFIG_DEFAULT_CHANNEL), channelNameMap);
//获取Mapping的值
Map<String, String> mapConfig =
context.getSubProperties(CONFIG_PREFIX_MAPPING);
//channelMapping变量存放了header变量中必须的Channel列表
channelMapping = new HashMap<String, List<Channel>>();
//将header对应的Channels存放到channelMapping变量中。
for (String headerValue : mapConfig.keySet()) {
List<Channel> configuredChannels = getChannelListFromNames(
mapConfig.get(headerValue),
channelNameMap);
//This should not go to default channel(s)
//because this seems to be a bad way to configure.
if (configuredChannels.size() == 0) {
throw new FlumeException("No channel configured for when "
+ "header value is: " + headerValue);
}
if (channelMapping.put(headerValue, configuredChannels) != null) {
throw new FlumeException("Selector channel configured twice");
}
}
//If no mapping is configured, it is ok.
//All events will go to the default channel(s).
Map<String, String> optionalChannelsMapping =
context.getSubProperties(CONFIG_PREFIX_OPTIONAL + ".");
//以下这一整段代码中是赛选出Header对应那些可选Channel列表。
optionalChannels = new HashMap<String, List<Channel>>();
for (String hdr : optionalChannelsMapping.keySet()) {
List<Channel> confChannels = getChannelListFromNames(
optionalChannelsMapping.get(hdr), channelNameMap);
if (confChannels.isEmpty()) {
confChannels = EMPTY_LIST;
}
//Remove channels from optional channels, which are already
//configured to be required channels.
List<Channel> reqdChannels = channelMapping.get(hdr);
//Check if there are required channels, else defaults to default channels
if(reqdChannels == null || reqdChannels.isEmpty()) {
//如果header对应的必选Channel列表为空，那么deault就作为它的必选Channel
reqdChannels = defaultChannels;
}
for (Channel c : reqdChannels) {
//如果header对应的Channel是必选的，那么就在可选的列表中删除。
if (confChannels.contains(c)) {
confChannels.remove(c);
}
}
if (optionalChannels.put(hdr, confChannels) != null) {
throw new FlumeException("Selector channel configured twice");
}
}
}

在看看MultiplexingChannelSelector中getRequiredChannels和getOptionalChannels方法，这两个方法也是根据HeaderName来获取Channel列表的：  @Override
public List<Channel> getRequiredChannels(Event event) {
String headerValue = event.getHeaders().get(headerName);
//headerValue不存在，就获取默认
if (headerValue == null || headerValue.trim().length() == 0) {
return defaultChannels;
}
//根据headerName获取必选的Channel列表
List<Channel> channels = channelMapping.get(headerValue);
//This header value does not point to anything
//Return default channel(s) here.
//必选列表为null，则返回默认的Channel列表
if (channels == null) {
channels = defaultChannels;
}
return channels;
}
@Override
public List<Channel> getOptionalChannels(Event event) {
String hdr = event.getHeaders().get(headerName);
//根据HeaderValue获取可选的channel列表
List<Channel> channels = optionalChannels.get(hdr);
//可选列表为null，则返回空列表。
if(channels == null) {
channels = EMPTY_LIST;
}
return channels;
}
自定义 Channel Selector

自定义的Channel Selector必须实现ChannelSelector接口，配置如下（全类名）a1.sources.r1.selector.type = org.example.MyChannelSelector