logstash解析haproxy的haproxy.log日志字段
在工作中需要在elk中展示haproxy的日志，用于分析访问客户IP、URI、PV等；日志样例：我只取出了2种类型的日志，在logstash解析时同时解析2种日志格式；
Sep5 10:36:55 localhost haproxy: 221.238.230.162:49225 SBD-Security SBD-Reglog/shvnginx02 0/0/896 5489 -- 1401/2/2/0/0 0/0
Sep5 10:36:54 localhost haproxy: 58.220.76.39:59768 SBD-Nginx imageservers/image01 0/0/0/2/2 200 5126 - - ---- 1353/1351/0/1/0 0/0 {|s06.abc001.cn} "GET /ftp_product_img/cn1100017322EA_1_thb.jpg?t=201709101850 HTTP/1.1" filebeat先读取haproxy.log的日志，写入redis中：
filebeat.inputs:
- type: log
paths:
- /var/log/haproxy/haproxy.log
tags: ["sbd_haproxy"]
fields:
type: sbd_haproxy
fields_under_root: true
output.redis:
hosts: ["10.78.1.181"]
key: "sbd_haproxy"
type: list logstash向redis读取数据，解析过滤之后写入elastic中：
input {
redis {
host => "10.78.1.181"
port => 6379
data_type => list
key => "sbd_haproxy"
}
}
filter {
grok {
match => ["message","%{HAPROXYHTTP}","message","%{HAPROXYTCP}"]
}
mutate {
remove_field => ["host","captured_response_cookie","haproxy_time","haproxy_month","haproxy_hour","srvconn","backend_queue","retries","termination_state"]
remove_field => ["haproxy_monthday","syslog_server","time_backend_connect","pid","srv_queue","beat","beconn","client_port","haproxy_milliseconds","@version"]
remove_field => ["haproxy_minute","offset","haproxy_second","actconn","source","program","haproxy_year","feconn","http_version"]
remove_field => ["message","prospector","time_duration","time_queue","syslog_timestamp","captured_request_cookie","prospector"]
}
date {
match => ["accept_date","dd/MMM/yyyy:HH:mm:ss.SSS"]
}
mutate {
remove_field => ["accept_date"]
}
}
output {
if == "sbd_haproxy" {
if == "sbd_haproxy" {
elasticsearch {
hosts=> ["10.78.1.184:9200","10.78.1.185:9200","10.78.1.188:9200"]
index=> "%{type}-%{+YYYY.MM.dd}"
}
#stdout { codec => rubydebug }
}
}
} kibana展示：
http://s1.运维网.com/images/20180906/1536215626562709.png
页:
[1]