agent.sources.execSrc.command= tail -F /var/log/flume/flume.log | grep "error: "
1、使用消息中间件JMS或者KAFKA
请参考: 基于Flume+Log4j+Kafka的日志采集架构方案
客户端直接发送至kafaka queue , 用 log4j KafkaAppender
2、使用Flume Appender
对于Java web 应用,我们就最简单直接采取这种方式。 Flume Appender 我们这里就直接采用log4j2 , 关于日志框架的这些说明,请看另一片博客 spring boot use log4j log4j 关于flume Appender 的配置
The Flume Appender supports three modes of operation.
1、It can act as a remote Flume client which sends Flume events via Avro to a Flume Agent configured with an Avro Source.(同 步,Avro协议)
2、It can act as an embedded Flume Agent where Flume events pass directly into Flume for processing.(异步,需要维护客户端 flume)
3、It can persist events to a local BerkeleyDB data store and then asynchronously send the events to Flume, similar to the embedded Flume Agent but without most of the Flume dependencies.(先写数据库,再异步发送)
Usage as an embedded agent will cause the messages to be directly passed to the Flume Channel and then control will be immediately returned to the application. All interaction with remote agents will occur asynchronously. Setting the "type" attribute to "Embedded" will force the use of the embedded agent. In addition, configuring agent properties in the appender configuration will also cause the embedded agent to be used.
我们下面就简单的用第一种方式
客户端配置
log4j.xml
A SolrCloud collection is the top-level object for indexing documents and providing a query interface. Each collection must be associated with an instance directory. Different collections can use the same instance directory. Each collection is typically replicated among several SolrCloud instances. Each replica is called a core and is assigned to an individual Solr service. The assignment process is managed automatically,> 成功创建之后如何修改和扩展,请参考这里solectl usage
Morphline (ETL)
创建好 collection 之后,我们就需要将日志解析存储到solr里,方便检索。Morphline 就是这个中间过程的ETL工具(extracting, transforming and loading data), Flume 提供了Morphlion Solr Sink, 从 log flume的source中读取event,经过ETL导入到solr中。
}
配置 schema.xml
在上一节,配置solr的时候,我们生成了默认的模版,我们需要根据实际的需求修改schema.xml,在$HOME/weblogs/conf 下
The schema.xml file contains all of the details about which fields your documents can contain, and how those fields should be dealt with when adding documents to the index, or when querying those fields.
schema.xml
solrconfig.xml