ELK之使用消息队列收取日志
一、ELKStack简介Elstaicsearch:存储和搜索
logstash：收集
kibana：展示.专门为ES设计的展示平台二、ELK之使用消息队列收取日志
1、环境准备
环境准备
IP主机名操作系统
192.168.56.11linux-node1centos7
192.168.56.12linux-node2centos72、流程分析
日志-->logstash(flume)-->MQ(redis,rabbitmq)-->logstash(python scripts)-->es官网参考链接
https://www.elastic.co/guide/en/logstash/current/input-plugins.html流程图如下：
http://s5.运维网.com/wyfs02/M00/86/D1/wKiom1fMBxfz3KPMAACNTnyIbZQ801.png-wh_500x0-wm_3-wmp_4-s_2506172430.png
三、安装配置redis
1、安装redis
编译安装或者yum安装均可，这里我们选择yum安装
在192.168.56.12上执行如下命令：
yum -y install redis2、配置redis
编辑redis配置文件，这里需要修改的有两处
vim /etc/redis.conf
daemonize yes #是否以后台daemon方式运行
bind 192.168.56.12 #绑定主机
保存退出3、启动redis并测试连接
启动redis:
systemctl start redis
测试连接:
# redis-cli -h 192.168.56.12 -p 6379
192.168.56.12:6379>四、分析利用redis收取日志的过程
1、实现数据能写入redis
在192.168.56.12上编写redis.conf配置文件，实现数据从标准输入写入redis
# cat /etc/logstash/conf.d/redis.conf
input{
stdin{}
}
output{
redis{
host => "192.168.56.12" #主机地址
port => "6379" #redis默认端口是6379
db => "6" #redis默认db是0，这里我们写入到db 6
data_type => "list" #我们选择list作为数据类型
key => "demo" #数据写入redis时所用的key
}
}启动logstash，并输入内容进行验证
# /opt/logstash/bin/logstash -f redis.conf
Settings: Default pipeline workers: 1
Pipeline main started
papapapa连接redis，选择db，查看数据是否写入redis
# redis-cli -h 192.168.56.12 -p 6379
192.168.56.12:6379> select 6
OK
192.168.56.12:6379> keys *
1) "demo"
192.168.56.12:6379> llen demo
(integer) 1
192.168.56.12:6379> lindex demo -1
"{\"message\":\"papapapa\",\"@version\":\"1\",\"@timestamp\":\"2016-09-04T08:08:11.998Z\",\"host\":\"linux-node2\"}"
192.168.56.12:6379>我们看到可以把数据写入redis
2、实现从文件中读取日志，并写入redis
在192.168.56.11上编写/etc/logstash/conf.d/apache.conf配置文件
# vim /etc/logstash/conf.d/apache.conf
input {
file {
path => "/var/log/httpd/access_log"
start_position => "beginning"
type => "apache-accesslog"
}
file{
path => "/var/log/elasticsearch/myes.log"
type => "es-log"
start_position => "beginning"
codec => multiline{
pattern => "^\["
negate => true
what => "previous"
}
}
}
output {
if == "apache-accesslog" {
redis {
host => "192.168.56.12"
port => "6379"
db => "6"
data_type => "list"
key => "apache-accesslog"
}
if == "es-log"{
redis {
host => "192.168.56.12"
port => "6379"
db => "6"
data_type => "list"
key => "es-log"
}
}
}启动logstash
# /opt/logstash/bin/logstash -f apache.conf
Settings: Default pipeline workers: 4
Pipeline main started在redis上查看是否已经写入数据，连接redis，选择db 6，执行keys *
192.168.56.12:6379> keys *
1) "es-log"
2) "demo"
3) "apache-accesslog"
192.168.56.12:6379> llen es-log
(integer) 44
192.168.56.12:6379> lindex es-log -1
"{\"@timestamp\":\"2016-09-04T20:34:23.717Z\",\"message\":\" Cluster health status changed from to (reason: ] ...]).\",\"@version\":\"1\",\"path\":\"/var/log/elasticsearch/myes.log\",\"host\":\"linux-node1\",\"type\":\"es-log\"}"
192.168.56.12:6379> llen apache-acccesslog
(integer) 0
192.168.56.12:6379> lindex apache-accesslog -1
"{\"message\":\"papapapa\",\"@version\":\"1\",\"@timestamp\":\"2016-09-04T20:46:03.164Z\",\"path\":\"/var/log/httpd/access_log\",\"host\":\"linux-node1\",\"type\":\"apache-accesslog\"}"可以看到不仅能收取apache-accesslog，还能收入es-log
3、实现从redis读取数据
在192.168.56.12上编写/etc/logstash/conf.d/input_redis.conf配置文件，并打印到标准输出
# cat input_redis.conf
input{
redis {
type => "apache-accesslog"
host => "192.168.56.12"
port => "6379"
db => "6"
data_type => "list"
key => "apache-accesslog"
}
redis {
type => "es-log"
host => "192.168.56.12"
port => "6379"
db => "6"
data_type => "list"
key => "es-log"
}
}
output{
stdout {
codec => rubydebug
}
}启动logstash，logstash启动后，从redis读取的日志内容会立即打印到标准输出
# /opt/logstash/bin/logstash -f input_redis.conf
Settings: Default pipeline workers: 1
Pipeline main started
{
"message" => "::1 - - \"OPTIONS * HTTP/1.0\" 200 - \"-\" \"Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)\"",
"@version" => "1",
"@timestamp" => "2016-09-04T20:34:22.928Z",
"path" => "/var/log/httpd/access_log",
"host" => "linux-node1",
"type" => "apache-accesslog"
}
{
"message" => "::1 - - \"OPTIONS * HTTP/1.0\" 200 - \"-\" \"Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5 (internal dummy connection)\"",
"@version" => "1",
"@timestamp" => "2016-09-04T20:34:23.266Z",
"path" => "/var/log/httpd/access_log",
"host" => "linux-node1",
"type" => "apache-accesslog"
}
{
"@timestamp" => "2016-09-04T20:34:22.942Z",
"message" => " duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}",
"@version" => "1",
"path" => "/var/log/elasticsearch/myes.log",
"host" => "linux-node1",
"type" => "es-log"
}
{
"@timestamp" => "2016-09-04T20:34:23.277Z",
"message" => " stopping ...",
"@version" => "1",
"path" => "/var/log/elasticsearch/myes.log",
"host" => "linux-node1",
"type" => "es-log"
}4、格式化apache的日志
使用filter插件中的grok插件，格式化apache日志
# cat input_redis.conf
input{
redis {
type => "apache-accesslog"
host => "192.168.56.12"
port => "6379"
db => "6"
data_type => "list"
key => "apache-accesslog"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
output{
stdout {
codec => rubydebug
}
}重新启动logstash，并查看标准输出,日志内容已变成json格式
# /opt/logstash/bin/logstash -f input_redis.conf
Settings: Default pipeline workers: 1
Pipeline main started
{
"message" => "192.168.56.1 - - \"GET / HTTP/1.1\" 200 67 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36\"",
"@version" => "1",
"@timestamp" => "2016-09-04T21:31:40.819Z",
"path" => "/var/log/httpd/access_log",
"host" => "linux-node1",
"type" => "apache-accesslog",
"clientip" => "192.168.56.1",
"ident" => "-",
"auth" => "-",
"timestamp" => "05/Sep/2016:05:31:39 +0800",
"verb" => "GET",
"request" => "/",
"httpversion" => "1.1",
"response" => "200",
"bytes" => "67",
"referrer" => "\"-\"",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36\""
}
{
"message" => "192.168.56.1 - - \"GET /favicon.ico HTTP/1.1\" 404 209 \"http://192.168.56.11:81/\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36\"",
"@version" => "1",
"@timestamp" => "2016-09-04T21:31:40.825Z",
"path" => "/var/log/httpd/access_log",
"host" => "linux-node1",
"type" => "apache-accesslog",
"clientip" => "192.168.56.1",
"ident" => "-",
"auth" => "-",
"timestamp" => "05/Sep/2016:05:31:40 +0800",
"verb" => "GET",
"request" => "/favicon.ico",
"httpversion" => "1.1",
"response" => "404",
"bytes" => "209",
"referrer" => "\"http://192.168.56.11:81/\"",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36\""
}5、实现通过redis收取日志
分别在192.168.56.11和192.168.56.12上面启动redis，在192.168.56.11上实现读取日志apache日志到redis，在192.168.56.12实现从redis读数据，并转为json格式，然后写入es
在192.168.56.11上的配置文件
# cat /etc/logstash/conf.d/apache.conf
input {
file {
path => "/var/log/httpd/access_log"
start_position => "beginning"
type => "apache-accesslog"
}
}
output {
if == "apache-accesslog" {
redis {
host => "192.168.56.12"
port => "6379"
db => "6"
data_type => "list"
key => "apache-accesslog"
}
}
}在192.168.56.12上的配置文件
# cat in_redis_out.conf
input{
redis {
type => "apache-accesslog"
host => "192.168.56.12"
port => "6379"
db => "6"
data_type => "list"
key => "apache-accesslog"
}
}
filter {
if == "apache-accesslog"{
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
}
output{
if == "apache-accesslog"{
elasticsearch {
hosts => ["192.168.56.11:9200"]
index => "apache-accesslog-%{+YYYY.MM.dd}"
}
}
}启动logstash
# /opt/logstash/bin/logstash -f apache.conf
Settings: Default pipeline workers: 4
Pipeline main started# /opt/logstash/bin/logstash -f in_redis_out.conf
Settings: Default pipeline workers: 1
Pipeline main started注意：为了便于测试，我都是在前台启动；如果执行/etc/init.d/logstash start命令，这样会把目录/etc/logstash/conf.d/下的所有配置文件都加载，会对测试造成干扰。
6、通过ES查看日志内容
访问192.168.56.11上面的apache，以便于产生访问日志。
接下来访问http://192.168.56.11:9200/_plugin/head
我们可以看到apache-accesslog-2016.09.04的索引已经产生，如图所示：
http://s1.运维网.com/wyfs02/M00/86/D1/wKioL1fMB1PCHHpRAAHMAaPfUCk949.png-wh_500x0-wm_3-wmp_4-s_2500366026.png
点击数据浏览
http://s3.运维网.com/wyfs02/M00/86/D1/wKioL1fMB1PC04h5AAGDuXvzlao483.png-wh_500x0-wm_3-wmp_4-s_2766782544.png
页:
[1]