ELK（elasticsearch+logstash+kibana）+redis实现nginx 日志的分析
一、使用软件wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.5/elasticsearch-2.3.5.tar.gz
wget https://download.elastic.co/logstash/logstash/logstash-2.3.4.tar.gz
wget https://download.elastic.co/kibana/kibana/kibana-4.5.4-linux-x64.tar.gz
wget http://download.redis.io/releases/redis-3.2.3.tar.gz
wget http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz?AuthParam=1472034436_3bb80b8600d4bb6c93f7c3d00a508578 -O jdk-7u79-linux-x64.tar.gz
二、安装(普通用户)
192.168.224.30(master） Elasticsearch,logstash,kibaa,Nginx,redis
192.168.224.20(minion)Logstash,nginx
1、jdk
export JAVA_HOME=/home/liuly/jdk
export PATH=$PATH:$JAVA_HOME/bin
export> 2、elasticsearch
tar xf elasticsearch-2.3.5.tar.gz
ln -sf elasticsearch-2.3.5 elasticsearch
cd elasticsearch
vim config/elasticsearch.yml
cluster.name: liuly
node.name: master
network.host: 192.168.224.30
http.port: 9200
./bin/elasticsearch &
安装插件
./bin/plugin install mobz/elasticsearch-head
./bin/plugin install lmenezes/elasticsearch-kopf
2、kibana
tar xf kibana-4.5.4-linux-x64.tar.gz
ln -sf kibana-4.5.4-linux-x64 kibana
cd kibana
vim config/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.224.30:9200"
kibana.index: ".kibana"
./bin/kibana &
nginx 的配置（访问方便点）
http {
include /etc/nginx/mime.types;
default_typeapplication/octet-stream;
log_format json '{"@timestamp":"$time_iso8601",'
'"@version":"1",'
'"host":"$server_addr",'
'"client":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"domain":"$host",'
'"url":"$uri",'
'"status":"$status"}';
access_log/var/log/nginx/access.logjson;
server {
listen 80;
server_name192.168.224.30;
include /etc/nginx/default.d/*.conf;
location / {
proxy_pass http://192.168.224.30:5601;
#proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
}
3、redis安装(2.4以上版本)
bind 0.0.0.0
（略）
低版本会出现这个错误
Redis connection problem {:exception=>#, :level=>:warn}
4、logstash
tar xf logstash-2.3.4.tar.gz
ln -sf logstash-2.3.4 logstash
cd logstash
mkdir config;cd config
vim log_server.conf
input {
redis {
host => '192.168.224.30'
data_type => 'list'
port => "6379"
key => 'nginx-log'
type => 'nginx-log'
}
redis {
host => '192.168.224.30'
data_type => 'list'
port => '6379'
key => 'nginx22'
type => 'nginx22'
}
}
output {
if == "nginx-log" {
elasticsearch {
hosts => ["192.168.224.30:9200"]
index => "nginx-log-%{+YYYY-MM}"
}
}
if == "nginx22" {
elasticsearch {
hosts => ["192.168.224.30:9200"]
index => "nginx22-%{+YYYY-MM}"
}
}
}
vim log_client.conf
input {
file {
add_field => {"Local_Host" => "master"}
path => ["/var/log/nginx/access.log"]
codec => json {
charset => "UTF-8"
}
}
}
output {
redis {
host => "192.168.224.30"
data_type => "list"
key => "nginx-log"
}
}
cd ..
./bin/logstash -f config/log_client.conf &
./bin/logstash -f config/log_server.conf &
minion
vim log_agent.conf
input {
file {
add_field => {"Local_Host" => "minion"}
path => ["/var/log/nginx/access.log"]
codec => json {
charset => "UTF-8"
}
}
}
output {
redis {
host => "192.168.224.30"
data_type => "list"
key => "nginx22"
}
}
页:
[1]