ELK在生产环境的简单应用

scaoping

　　一、简析ELK
　　 ELK是三个开源软件的缩写，分别表示：Elasticsearch , Logstash, Kibana , 它们都是开源软件。
　　 Elasticsearch：可以理解成一个仓库
　　 Logstash：过滤器，分析器，重量级的日志收集工具
　　 Kibana：图形化的展示，日志的进一步分析
　　由于Logstash是个重量级的日志收集工具，会消耗大量的资源，所以我们在日常的生产环境通常会使用到filebeat这个轻量级的日志收集工具。
　　二、简析日志走向
　　
　　三、生产环境的使用

　　1.filebeat  (版本必须低于elk的版本，不然日志收集会报错）
　　 curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.2.0-x86_64.rpm
　　 vim /etc/filebeat/filebeat.yml

　　  ...
　　paths:
　　- /application/nginx/logs/*.log
　　...
　　output.logstash:
　　# The Logstash hosts
　　hosts: ["l0.0.0.10:5044"]
　　  ...
2.Elasticsearch
rpm -ivh  elasticsearch-6.0.0.rpm3. Logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
rpm -ivh  logstash-6.0.0.rpm
vim  log_nginx.conf
input {
beats {
port => 5044
}
}
filter {
if [logtype] == "Service" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:log_time}\|%{GREEDYDATA:level}\|%{GREEDYDATA:component}\|%{GREEDYDATA:log_message}" }
}
if [component] !~ "^Sangmado.Inka.MomBrokers" {
drop { }
}
}
else {
grok {
match => { "message" => "\[%{TIMESTAMP_ISO8601:time_iso8601}\] \[%{NUMBER:pid}\] \[%{IPORHOST:remote_addr}\] \[%{IPORHOST:http_host}(:%{NUMBER:http_host_port})?\] \[%{IPORHOST:upstream_addr}(:%{NUMBER:upstream_port})?\] \[%{WORD:verb} %{URIPATH:request_uri}(?:%{URIPARAM:request_parameter})? HTTP/%{NUMBER:httpversion}\] \[%{NUMBER:status}\] \[%{BASE10NUM:upstream_response_time} s\] \[%{BASE10NUM:request_time} s\] \[(?:%{NUMBER:bytes_sent}|-) bytes\] \[(?:%{NUMBER:body_bytes_sent}|-) bytes\] \[%{GREEDYDATA:http_user_agent}\]" }
}
date {
locale => "en"
match => ["time_iso8601", "ISO8601"]
}
mutate {
convert => {
"pid" => "integer"
"upstream_response_time" => "float"
"request_time" => "float"
"status" => "integer"
"bytes_sent" => "integer"
"body_bytes_sent" => "integer"
   }
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout {
codec => rubydebug
}
}
3.kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpm
rpm -ivh kibana-6.0.0-x86_64.rpm
4.启动
systemctl restart elasticsearch.service
logstash/bin/logstash  -f   log_nginx.conf
systemctl start kibana
/etc/init.d/filebeat start

5.验证