Elastic+logstash+filebeat做Nginx日志分析
一、Elasticserach安装1、Installation(elastic 6.3.2 版本 依赖 javaJDK8)
下载合适的版本：
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
tar -xvf elasticsearch-6.4.0.tar.gz
cd elasticsearch-6.4.0/bin
./elasticsearch
2、集群健康检查
curl-XGEThttp://127.0.0.1:9200/_cat/health?v集群健康查看
curl-XGEThttp://127.0.0.1:9200/_cat/nodes?v 节点状态查看
curl-XGEThttp://127.0.0.1:9200/_cat/indices?v查看索引
curl-XPUThttp://127.0.0.1:9200/customer?pretty添加customer索引
curl-XDELETEhttp://127.0.0.1:9200/customer?pretty删除索引
3、Config
cluster.name:集群名
node.name: 节点名
path.data: 数据存储路径
path.logs:日志存储路径
network.host:监听地址
http.port:监听端口
JVM配置、logging配置参考官方文档
重要参数：
Path settings
Cluster name
Node name
Network host
Discovery settings
Heap size
Heap dump path
GC logging
Temp directory
重要的系统参数：
Disable swapping
Increase file descriptors
Ensure sufficient virtual memory
Ensure sufficient threads
JVM DNS cache settings
4、Running as daemon
./bin/elasticsearch -d -p pid 启动
kill cat pid 停止
5、Set upX-Pack
参考x-pack破解方法
二、kibana安装及配置
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz
tar -xzf kibana-6.4.0-linux-x86_64.tar.gz
cd kibana-6.4.0-linux-x86_64/
1、启动./bin/kibana
2、config：
server.port: 5601
server.host: "192.168.12.81"
kibana.index: ".kibana"
elasticsearch.username: "elastic"
elasticsearch.password: "dinpay"
三、logstash安装及配置
wgethttps://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz
配置 filebeat 收集Nginx日志： （Nginx.conf）
input {
beats {
port => 5044
codec => "json"
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "test1-nginx-access-%{+YYYY.MM.dd}"
template_overwrite => true
user => elastic
password => dinpay
}
}
配置logstash.ymlelastic安装在本机（没有配置https）
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: dinpay
xpack.monitoring.elasticsearch.url: ["http://127.0.0.1:9200"]
四、filebeat安装及配置
wget https://artifacts.elastic.co/downloads/beats/filebeat/https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz
tar -xffilebeat-6.4.0-linux-x86_64.tar.gz
cd filebeat-6.4.0-linux-x86_64
配置 filebeat.yml
[*] type:log
paths:
[*]/var/log/nginx/access.log
output.logstash:
hosts: ["192.168.12.81:5044"]
xpack.monitoring:
enabled: true
elasticsearch:
hosts: ["http://192.168.12.81:9200", "http://192.168.12.81:9200"]
username: elastic
password: dinpay
五、Nginx 日志格式配置：
log_format json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$clientRealIp",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"http_user_agent":"$http_user_agent",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status",'
'"ss":"$upstream_status"}';
access_log/var/log/nginx/access.logjson;
map $http_x_forwarded_for $clientRealIp {
""$remote_addr;
~^(?P+),?.*$ $firstAddr;
}
页:
[1]