ELK（elasticsearch+logstash+kibana）+redis实现nginx 日志的分析

34yhtee

                      一、使用软件

wget https://download.elastic.co/elas ... search-2.3.5.tar.gz

wget https://download.elastic.co/kiba ... .4-linux-x64.tar.gz

wget https://download.elastic.co/kiba ... .4-linux-x64.tar.gz

wget http://download.redis.io/releases/redis-3.2.3.tar.gz

wget http://download.oracle.com/otn-p ... b6c93f7c3d00a508578 -O jdk-7u79-linux-x64.tar.gz
二、安装(普通用户)
192.168.224.30(master） Elasticsearch,logstash,kibaa,Nginx,redis
192.168.224.20(minion)  Logstash,nginx

1、jdk
export JAVA_HOME=/home/liuly/jdk
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
2、elasticsearch
tar xf elasticsearch-2.3.5.tar.gz
ln -sf elasticsearch-2.3.5 elasticsearch
cd elasticsearch
vim config/elasticsearch.yml
cluster.name: liuly
node.name: master
network.host: 192.168.224.30
http.port: 9200

./bin/elasticsearch &
安装插件
./bin/plugin install mobz/elasticsearch-head
./bin/plugin install lmenezes/elasticsearch-kopf
2、kibana
tar xf kibana-4.5.4-linux-x64.tar.gz
ln -sf kibana-4.5.4-linux-x64 kibana
cd kibana
vim config/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.224.30:9200"
kibana.index: ".kibana"

./bin/kibana &

nginx 的配置（访问方便点）
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format json '{"@timestamp":"$time_iso8601",'
               '"@version":"1",'
               '"host":"$server_addr",'
               '"client":"$remote_addr",'
               '"size":$body_bytes_sent,'
               '"responsetime":$request_time,'
               '"domain":"$host",'
               '"url":"$uri",'
               '"status":"$status"}';
    access_log  /var/log/nginx/access.log  json;

    server {
    listen       80;
    server_name  192.168.224.30;

    include /etc/nginx/default.d/*.conf;
    location / {
         proxy_pass http://192.168.224.30:5601;
         #proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection 'upgrade';
         proxy_set_header Host $host;
         proxy_cache_bypass $http_upgrade;

    }
}


}

3、redis安装(2.4以上版本)
bind 0.0.0.0
（略）
低版本会出现这个错误
Redis connection problem {:exception=>#<Redis::CommandError: ERR unknown command 'script'>, :level=>:warn}

4、logstash
tar xf logstash-2.3.4.tar.gz
ln -sf logstash-2.3.4 logstash
cd logstash
mkdir config;cd config
vim log_server.conf
input {
   redis {
    host => '192.168.224.30'
    data_type => 'list'
    port => "6379"
    key => 'nginx-log'
    type => 'nginx-log'
       }
   redis {
    host => '192.168.224.30'   
    data_type => 'list'
    port => '6379'
    key => 'nginx22'
    type => 'nginx22'
       }
}

output {
  if [type] == "nginx-log" {
    elasticsearch {
        hosts => ["192.168.224.30:9200"]
        index => "nginx-log-%{+YYYY-MM}"
}
}
  if [type] == "nginx22" {
    elasticsearch {
        hosts => ["192.168.224.30:9200"]
        index => "nginx22-%{+YYYY-MM}"
  }
}
}

vim log_client.conf
input {
    file {
    add_field => {"Local_Host" => "master"}
    path => ["/var/log/nginx/access.log"]
    codec => json {
    charset => "UTF-8"
}
    }
  }
output {
        redis {
                host => "192.168.224.30"
                data_type => "list"
                key => "nginx-log"
        }
}

cd ..
./bin/logstash -f config/log_client.conf &
./bin/logstash -f config/log_server.conf &

minion
vim log_agent.conf
input {
    file {
        add_field => {"Local_Host" => "minion"}
        path => ["/var/log/nginx/access.log"]
        codec => json {
        charset => "UTF-8"
                  }
       }
  }
output {
        redis {
                host => "192.168.224.30"
                data_type => "list"
                key => "nginx22"
        }
}






                   

知呼

Ethan

顶顶顶顶顶顶顶顶顶顶顶顶顶顶