logstash&Kibana杂记

liujjun

一、logstash基础

master1作为logstash_agent端，master1运行WEB网站。master2为logstash服务端，master3为Elasticsearch

1、安装logstash

1.1 yum安装

设置java环境变量：
[root@master1 ~]# vim /etc/profile.d/java.sh
export JAVA_HOME=/usr
logstash已经被Elasticsearch收购，直接在ES官网下载即可
[root@master1 ~]# ls
logstash-1.5.4-1.noarch.rpm
安装
[root@master1 ~]# yum install logstash-1.5.4-1.noarch.rpm
设置环境变量：
[root@master1 ~]# vim /etc/profile.d/logstash.sh
export PATH=/opt/logstash/bin:$PATH
重新加载：   
[root@master1 ~]# source /etc/profile.d/logstash.sh
1.2 创建配置文件

[root@master1 ~]# vim /etc/logstash/conf.d/sample.conf
input {
stdin {}
}
output {
stdout {
codec   => rubydebug
}
}
语法测试：
[root@master1 ~]# logstash -f /etc/logstash/conf.d/sample.conf --configtest
Configuration OK
1.3 运行logstash

[root@master1 ~]# logstash -f /etc/logstash/conf.d/sample.conf
Logstash startup completed
测试：
Logstash startup completed
Hello Logstash
{
"message" => "Hello Logstash",
"@version" => "1",
"@timestamp" => "2018-04-15T16:59:04.136Z",
"host" => "master1.com"
}
2、示例（input、filter插件）

2.1 系统日志文件简单示例

[root@master1 ~]# vim /etc/logstash/conf.d/filesample.conf
input {
file {
path => ["/var/log/messages"]
type => "system"
start_position => "beginning"
}
}
output {
stdout {
codec   => rubydebug
}
}
语法测试：
[root@master1 ~]# logstash -f /etc/logstash/conf.d/filesample.conf --configtest
Configuration OK
运行：
[root@master1 ~]# logstash -f /etc/logstash/conf.d/filesample.conf

　　file插件官网链接：
https://www.elastic.co/guide/en/logstash/1.5/plugins-inputs-file.html
　　结束：Ctrl+c

2.2 udp

master2安装 collectd，配置其network插件，向外发送数据。
[root@master2 ~]# yum install collectd
配置collectd
[root@master2 ~]# vim /etc/collectd.conf
#定义主机名
Hostname    "master2.com"
#打开几个监控项
LoadPlugin cpu
LoadPlugin df
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin network
定义发送到logstash端的监听端口




启动服务
[root@master2 ~]# systemctl start collectd.service
logstash端配置：
[root@master1 ~]# vim /etc/logstash/conf.d/udpsample.conf
input {
udp {
port    => 25826
codec   => collectd {}
type    => "collectd"
}
}
output {
stdout {
codec   => rubydebug
语法检测：
[root@master1 ~]# logstash -f /etc/logstash/conf.d/udpsample.conf --configtest
Configuration OK
启动：
[root@master1 ~]# logstash -f /etc/logstash/conf.d/udpsample.conf
Logstash startup completed

2.3 httpd

[root@master1 ~]# yum install httpd
[root@master1 ~]# systemctl start http
结构化文本数据
[root@master1 ~]# rpm -ql logstash | grep "patterns$"
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/grok-patterns
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/mcollective-patterns
[root@master1 ~]# vim /etc/logstash/conf.d/groksample.conf
input {
stdin {}
}
filter {
grok {        match   => { "message" => "%{IP:clientip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
}
output {
stdout {
codec   => rubydebug
}
}
语法测试：
Configuration OK
运行测试：
[root@master1 ~]# logstash -f /etc/logstash/conf.d/groksample.conf
Logstash startup completed
1.1.1.1 GET /index.html 30 0.23
{
"message" => "1.1.1.1 GET /index.html 30 0.23",
"@version" => "1",
"@timestamp" => "2018-04-17T01:41:09.951Z",
"host" => "master1.com",
"clientip" => "1.1.1.1",
"method" => "GET",
"request" => "/index.html",
"bytes" => "30",
"duration" => "0.23"
}
2.4 apachelogs

[root@master1 ~]# vim /etc/logstash/conf.d/apachelogssample.conf
input {
file {
path    => ["/var/log/httpd/access_log"]
type    => "apachelog"
start_position  => "beginning"
}
}
filter {
grok {
match   => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
output {
stdout {
codec   => rubydebug
}
}
[root@master1 ~]# logstash -f /etc/logstash/conf.d/apachelogssample.conf --configtest
Configuration OK
运行测试：
[root@master1 ~]# logstash -f /etc/logstash/conf.d/apachelogssample.conf
访问apache主页：http://10.201.106.131

2.5 nginxlog

编辑pattern
[root@master1 ~]# vim /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/grok-patterns
# Nginx Logs
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request} (?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes} |-) %{QS:referrer} %{QS:agent} %{NOTSPACE:http_x_forwarded_for}
安装启动nginx：
[root@master1 ~]# systemctl stop httpd.service
[root@master1 ~]# yum install nginx
[root@master1 ~]# systemctl start nginx.service
logstash配置：
[root@master1 ~]# cd /etc/logstash/conf.d/
[root@master1 conf.d]# cp apachelogssample.conf nginxlogsample.conf
[root@master1 conf.d]# vim nginxlogsample.conf
input {
file {
path    => ["/var/log/nginx/access.log"]
type    => "nginxlog"
start_position  => "beginning"
}
}
filter {
grok {
match   => { "message" => "%{NGINXACCESS}" }      
}
}
output {
stdout {
codec   => rubydebug
}
}
运行测试：
[root@master1 ~]# logstash -f /etc/logstash/conf.d/nginxlogsample.conf
Logstash startup completed

3、output插件

3.1 redis存入数据

安装redis
[root@master1 ~]# yum install redis
配置
[root@master1 ~]# vim /etc/redis.conf
#修改其监听在0.0.0.0（监听本机所有IP）即可。
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bind 0.0.0.0
启动：
[root@master1 ~]# systemctl start redis.service
logstash配置：
[root@master1 ~]# cd /etc/logstash/conf.d/
[root@master1 conf.d]# cp nginxlogsample.conf nglogredissample.conf
[root@master1 conf.d]# vim nglogredissample.conf
input {
file {
path    => ["/var/log/nginx/access.log"]
type    => "nginxlog"
start_position  => "beginning"
}
}
filter {
grok {
match   => { "message" => "%{NGINXACCESS}" }
}
}
output {
redis {
port    => "6379"
host    => ["127.0.0.1"]
data_type   => "list"
key     => "logstash-%{type}"   
}
}
语法测试：
[root@master1 conf.d]# logstash -f ./nglogredissample.conf --configtest
Configuration OK
运行测试：
[root@master1 ~]# logstash -f /etc/logstash/conf.d/nglogredissample.conf
Logstash startup completed
再次访问nginx主页，http://10.201.106.131
查看redis：
[root@master1 ~]# redis-cli
127.0.0.1:6379> LLEN logstash-nginxlog
(integer) 20
查看索引的第一个元素：
127.0.0.1:6379> LINDEX logstash-nginxlog 1
"{\"message\":\"10.201.106.1 - - [17/Apr/2018:13:51:38 +0800] \\\"GET /nginx-logo.png HTTP/1.1\\\" 200 368 \\\"http://10.201.106.131/\\\" \\\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36\\\" \\\"-\\\"\",\"@version\":\"1\",\"@timestamp\":\"2018-04-17T05:51:39.579Z\",\"host\":\"master1.com\",\"path\":\"/var/log/nginx/access.log\",\"type\":\"nginxlog\",\"clientip\":\"10.201.106.1\",\"remote_user\":\"-\",\"timestamp\":\"17/Apr/2018:13:51:38 +0800\",\"verb\":\"GET\",\"request\":\"/nginx-logo.png\",\"httpversion\":\"1.1\",\"response\":\"200\",\"bytes\":\"368\",\"referrer\":\"\\\"http://10.201.106.131/\\\"\",\"agent\":\"\\\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36\\\"\",\"http_x_forwarded_for\":\"\\\"-\\\"\"}"
3.2 redis读出数据到标准输出

master1为logstash-agent端
master2为logstash服务端
同步时间
master2的java环境变量设置
[root@master2 ~]# vim /etc/profile.d/java.sh
export JAVA_HOME=/usr
安装logstash
[root@master2 ~]# yum install logstash-1.5.4-1.noarch.rpm
定义环境变量：
[root@master2 ~]# vim /etc/profile.d/logstash.sh
export PATH=/opt/logstash/bin:$PATH
加载环境变量设置：
[root@master2 ~]# source /etc/profile.d/logstash.sh
配置接收redis数据
[root@master2 ~]# vim /etc/logstash/conf.d/server.conf
input {
redis {
port    => "6379"
host    => "10.201.106.131"
data_type   => "list"
key     => "logstash-nginxlog"
}
}
output {
stdout {
codec   => rubydebug
}
}
语法测试：
[root@master2 ~]# logstash -f /etc/logstash/conf.d/server.conf --configtest
Configuration OK
运行测试：
[root@master2 ~]# logstash -f /etc/logstash/conf.d/server.conf
Logstash startup completed

3.3 redis读出数据到Elasticsearch

3.3.1 Elasticsearch配置

　　master3为Elasticsearch

使用系统自带JDK环境，并安装java扩展
[root@master3 ~]# yum install java-1.7.0-openjdk-devel
设置java环境变量：
[root@master3 ~]# vim /etc/profile.d/java.sh
export JAVA_HOME=/usr
安装Elasticsearch：
[root@master3 ~]# yum install elasticsearch-1.7.2.noarch.rpm
配置：
[root@master3 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: loges
node.name: "master3.com"
启动：
[root@master3 ~]# systemctl daemon-reload
[root@master3 ~]# systemctl start elasticsearch
安装插件（方便查看状态）：
[root@master3 ~]# /usr/share/elasticsearch/bin/plugin -i bigdesk -u file:///root/bigdesk-latest.zip
[root@master3 ~]# /usr/share/elasticsearch/bin/plugin -l
Installed plugins:
- bigdesk
测试访问插件：

3.3.2 Kibana（前端展示）
　　下载链接：https://www.elastic.co/downloads/past-releases

[root@master3 ~]# ls
kibana-4.1.2-linux-x64.tar.gz
解压到/usr/local
[root@master3 ~]# tar xf kibana-4.1.2-linux-x64.tar.gz -C /usr/local/
[root@master3 local]# ln -sv kibana-4.1.2-linux-x64 kibana
‘kibana’ -> ‘kibana-4.1.2-linux-x64’
配置：
[root@master3 config]# pwd
/usr/local/kibana/config
[root@master3 config]# vim kibana.yml
#修改其中一个节点的IP或者node名字，如果是本机直接localhost
elasticsearch_url: "http://10.201.106.133:9200"
运行（如需运行后台，在命令后门加&即可）：
[root@master3 ~]# /usr/local/kibana/bin/kibana
访问：
http://10.201.106.133:5601

3.3.3 配置logstash输出到Elasticsearch

[root@master2 ~]# vim /etc/logstash/conf.d/server.conf
input {
redis {
port    => "6379"
host    => "10.201.106.131"
data_type   => "list"
key     => "logstash-nginxlog"
}
}
output {
elasticsearch {
cluster => "loges"
index   => "logstash-%{+YYYY.MM.dd}"
}
}
语法测试（尽量使用java8）
[root@master2 ~]# logstash -f /etc/logstash/conf.d/server.conf --configtest
[2018-04-18 01:42:55.146]  WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
Java 7 support is only best effort, it may not work. It will be removed in next release (1.0).
Configuration OK
启动（会自动发现Elasticsearch节点）：
[root@master2 ~]# logstash -f /etc/logstash/conf.d/server.conf
[2018-04-18 01:44:19.274]  WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
Java 7 support is only best effort, it may not work. It will be removed in next release (1.0).
Apr 18, 2018 1:44:21 AM org.elasticsearch.node.internal.InternalNode
INFO: [logstash-master2.com-2679-11622] version[1.7.0], pid[2679], build[929b973/2015-07-16T14:31:07Z]
Apr 18, 2018 1:44:21 AM org.elasticsearch.node.internal.InternalNode
INFO: [logstash-master2.com-2679-11622] initializing ...
Apr 18, 2018 1:44:22 AM org.elasticsearch.plugins.PluginsService
INFO: [logstash-master2.com-2679-11622] loaded [], sites []
Apr 18, 2018 1:44:27 AM org.elasticsearch.bootstrap.Natives
WARNING: JNA not found. native methods will be disabled.
Apr 18, 2018 1:44:29 AM org.elasticsearch.node.internal.InternalNode
INFO: [logstash-master2.com-2679-11622] initialized
Apr 18, 2018 1:44:29 AM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-master2.com-2679-11622] starting ...
Apr 18, 2018 1:44:30 AM org.elasticsearch.transport.TransportService doStart
INFO: [logstash-master2.com-2679-11622] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.201.106.132:9300]}
Apr 18, 2018 1:44:30 AM org.elasticsearch.discovery.DiscoveryService doStart
INFO: [logstash-master2.com-2679-11622] loges/xZYxFmKDSu6ziX8wtt2TSQ
Apr 18, 2018 1:44:33 AM org.elasticsearch.cluster.service.InternalClusterService$UpdateTask run
INFO: [logstash-master2.com-2679-11622] detected_master [master3.com][89ejQ2cHQzC-RlTMCRnd3g][master3.com][inet[/10.201.106.133:9300]], added {[master3.com][89ejQ2cHQzC-RlTMCRnd3g][master3.com][inet[/10.201.106.133:9300]],}, reason: zen-disco-receive(from master [[master3.com][89ejQ2cHQzC-RlTMCRnd3g][master3.com][inet[/10.201.106.133:9300]]])
Apr 18, 2018 1:44:33 AM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-master2.com-2679-11622] started
Logstash startup completed
查看master3的Elasticsearch索引：
[root@master3 ~]# curl -XGET 'localhost:9200/_cat/indices'
yellow open .kibana             1 1 1 0 2.5kb 2.5kb
yellow open logstash-2018.04.17 5 1 0 0  575b  575b
查看索引上的文档
[root@master3 ~]# curl -XGET 'localhost:9200/_search?pretty'
3.3.4 配置Kibana

　　可进行搜索：

3.3.5 服务后台运行

logstash：
/etc/logstash/conf.d将无用的配置文件清除后可通过守护进程启动。
service start logstash
kibana:
[root@master3 ~]# /usr/local/kibana/bin/kibana -l /var/log/kibina.log &
3.6

由于logstash较为重量级，agent侧可以使用lumberjack代替获取数据，减少对WEB服务器的资源占用 。