初探ELK-filebeat使用小结
初探ELK-filebeat使用小结2016/9/18
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
一、安装
1、下载
有2种方式下载，推荐缓存rpm包到本地yum源
1）直接使用rpm
# curl -L -O https://download.elastic.co/beats/filebeat/filebeat-1.3.1-x86_64.rpm
2）使用yum源
# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
# vim /etc/yum.repos.d/beats.repo
name=Elastic Beats Repository
baseurl=https://packages.elastic.co/beats/yum/el/$basearch
enabled=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
gpgcheck=1
# yum install logstash
# chkconfig filebeat on
2、配置
【默认的配置】
# cat /etc/filebeat/filebeat.yml |grep -Ev '^(#|#| #| #| #|$)'
filebeat:
prospectors:
-
paths:
- /var/log/*.log
input_type: log
registry_file: /var/lib/filebeat/registry
output:
elasticsearch:
hosts: ["localhost:9200"]
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB
二、使用
1、测试环境（已经部署了服务）
客户端：10.50.200.49 nginx(www.test.com, www.work.com)
服务端：10.50.200.220 logstash, elasticsearch, kibana
2、场景1：只有1个域名/模糊匹配N个域名
目的：将匹配的 access 日志收集起来集中展示。
【客户端】
输入：filebeat
输出：logstash
# cat /etc/filebeat/filebeat.yml |grep -Ev '^(#|#| #| #| #|$)'
filebeat:
prospectors:
-
paths:
- /var/log/nginx/access_*.log
input_type: log
document_type: NginxAccess
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["10.50.200.220:5044"]
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB
# service filebeat restart
【服务端】
输入：logstash
输出：elasticsearch
配置自定义的 pattern
# mkdir -p /etc/logstash/patterns.d
# vim /etc/logstash/patterns.d/extra_patterns
NGINXACCESS %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" (?:%{QS:content_type}|-) (?:%{QS:request_body}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{NUMBER:response} %{BASE16FLOAT:request_time} (?:%{NUMBER:bytes}|-)
调整 logstash 的配置，启用 filebeat 插件。
# cat /etc/logstash/conf.d/filebeat.conf
input {
beats {
port => "5044"
}
}
filter {
if =~ "NginxAccess-" {
grok {
patterns_dir => ["/etc/logstash/patterns.d"]
match => {
"message" => "%{NGINXACCESS}"
}
}
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
}
}
}
output {
if =~ "NginxAccess" {
elasticsearch {
hosts => "127.0.0.1:9200"
manage_template => false
index => "%{[@metadata]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata]}"
}
}
}
# service logstash restart
回到 kibana 界面，使用 index 名称为：
filebeat-*
来获取数据。
结果：符合预期。
3、场景2：N个域名分开收集
目的：将 www.test.com 和 www.work.com 的 access 日志收集起来分开展示
【客户端】
输入：filebeat
输出：logstash
# cat /etc/filebeat/filebeat.yml |grep -Ev '^(#|#| #| #| #|$)'
filebeat:
prospectors:
-
paths:
- /var/log/nginx/access_www.test.com*.log
input_type: log
document_type: NginxAccess-www.test.com
-
paths:
- /var/log/nginx/access_www.work.com*.log
input_type: log
document_type: NginxAccess-www.work.com
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["10.50.200.220:5044"]
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB
# service filebeat restart
【服务端】
输入：logstash
输出：elasticsearch
配置自定义的 pattern
# mkdir -p /etc/logstash/patterns.d
# vim /etc/logstash/patterns.d/extra_patterns
NGINXACCESS %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" (?:%{QS:content_type}|-) (?:%{QS:request_body}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{NUMBER:response} %{BASE16FLOAT:request_time} (?:%{NUMBER:bytes}|-)
调整 logstash 的配置，启用 filebeat 插件。
# cat /etc/logstash/conf.d/filebeat.conf
input {
beats {
port => "5044"
}
}
filter {
if =~ "NginxAccess-" {
grok {
patterns_dir => ["/etc/logstash/patterns.d"]
match => {
"message" => "%{NGINXACCESS}"
}
}
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
}
}
}
output {
if == "NginxAccess-www.test.com" {
elasticsearch {
hosts => "127.0.0.1:9200"
manage_template => false
index => "%{[@metadata]}-nginxaccess-www.test.com-%{+YYYY.MM.dd}"
document_type => "%{[@metadata]}"
}
} else if == "NginxAccess-www.work.com" {
elasticsearch {
hosts => "127.0.0.1:9200"
manage_template => false
index => "%{[@metadata]}-nginxaccess-www.work.com-%{+YYYY.MM.dd}"
document_type => "%{[@metadata]}"
}
}
}
# service logstash restart
回到 kibana 界面，使用 index 名称为：
nginxaccess-www.test.com-*
nginxaccess-www.work.com-*
来获取数据。
结果：符合预期。
三、小结FAQ
1、数据流向
-----------------------------------------------------------------------------
|---------client-------|----------server------------------------|
log_files ->filebeat ->logstash ->elasticsearch ->kibana
-----------------------------------------------------------------------------
2、关于模版
1）关闭logstash自动管理模板功能
manage_template => false
2）手动导入模版
# curl -XPUT 'http://localhost:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json
3）删除模版
# curl -XDELETE 'http://localhost:9200/filebeat-*'
ZYXW、参考
1、官网
https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html
https://www.elastic.co/guide/en/beats/libbeat/1.3/logstash-installation.html#logstash-setup
https://www.elastic.co/guide/en/beats/libbeat/1.3/setup-repositories.html
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-shell
页:
[1]