elk+redis centos6.6安装与配置

Izhuceul

本人安装配置ELK走了不少弯路，初学者按以下步骤安装及可使用； 建议安装配置之前先对阅读相关架构原理。

logstash,elasticsearch,kibana
怎么进行nginx的日志分析呢？首先，架构方面，nginx是有日志文件的，它的每个请求的状态等都有日志文件进行记录。其次，需要有个队
列，redis的list结构正好可以作为队列使用。然后分析使用elasticsearch就可以进行分析和查询了。

我们需要的是一个分布式的，日志收集和分析系统。logstash有agent和indexer两个角色。对于agent角色，放在单独的web机
器上面，然后这个agent不断地读取nginx的日志文件，每当它读到新的日志信息以后，就将日志传送到网络上的一台redis队列上。对于队列上的这
些未处理的日志，有不同的几台logstash
indexer进行接收和分析。分析之后存储到elasticsearch进行搜索分析。再由统一的kibana进行日志web界面的展示。

　　

相关环境：Centos_x64_6.6  redis2.8.23  Elasticsearch 2.0.0, Logstash 2.0.0,  Kibana4.2.0

软件下载地址：

redis： http://redis.io/download
Elasticsearch: wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.0.0/elasticsearch-2.0.0.rpm
Logstash: wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.0.0-1.noarch.rpmKibana: wget  https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz
Kibana: wget https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz
jdk：  wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpmyum localinstall jdk-8u65-linux-x64.rpm　　

Install Java 8
[root@localhost src]# rpm -ivh jdk-8u65-linux-x64.rpm
Preparing...                ########################################### [100%]
   1:jdk1.8.0_65            ########################################### [100%]
Unpacking JAR files...
        tools.jar...
        plugin.jar...
        javaws.jar...
        deploy.jar...
        rt.jar...
        jsse.jar...
        charsets.jar...
        localedata.jar...
        jfxrt.jar...
[root@localhost src]# ln -s  /usr/java/jdk1.8.0_65/jre/bin/java /usr/bin/javaInstall redis
先安装依赖tcl
[root@localhost src]# yum install tcl
安装redis
[root@localhost src]#tar zxvf redis-2.8.23.tar.gz
[root@localhost src]# cd redis-2.8.23
[root@localhost redis-2.8.23]#
[root@localhost redis-2.8.23]# make
[root@localhost redis-2.8.23]# make install
redis安装后配置[root@localhost redis-2.8.23]# cd utils/
[root@localhost utils]# ./install_server.shWelcome to the redis service installer
This script will help you easily set up a running redis server
Please select the redis port for this instance: [6379]
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf]
Selected default - /etc/redis/6379.conf
Please select the redis log file name [/var/log/redis_6379.log]
Selected default - /var/log/redis_6379.log
Please select the data directory for this instance [/var/lib/redis/6379]
Selected default - /var/lib/redis/6379
Please select the redis executable path [/usr/local/bin/redis-server]
Selected config:
Port           : 6379
Config file    : /etc/redis/6379.conf
Log file       : /var/log/redis_6379.log
Data dir       : /var/lib/redis/6379
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!Install Elasticsearch
[root@localhost src]#rpm -ivh elasticsearch-2.0.0.rpm
warning: elasticsearch-2.0.0.rpm: Header V4 RSA/SHA1 Signature, key ID d88e42b4: NOKEY
Preparing...                ########################################### [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
   1:elasticsearch          ########################################### [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
sudo service elasticsearch start
[root@localhost src]# /etc/init.d/elasticsearch start
  正在启动 elasticsearch：                                   [确定]
[root@localhost src]# chkconfig elasticsearch onInstall Kibana
[root@localhost opt]#groupadd -g 1005 kibana
[root@localhost opt]#useradd -u 1005 -g 1005 kibana
[root@localhost opt]# tar zxvf kibana-4.2.0-linux-x64.tar.gz  -C /opt/
[root@localhost ~]# cd /opt[root@localhost opt]# mv kibana-4.2.0-linux-x64/ kibana
[root@localhost opt]#chown -R kibana: /opt/kibana
[root@localhost opt]#cd /etc/init.d &&  curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-init
[root@localhost opt]#cd /etc/init.d &&  curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-initcd /etc/default &&  curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-default
[root@localhost opt]#service kibana start
[root@localhost opt]#chkconfig kibana on
如果启动不成功请手动启动 nohup  /opt/kibana/bin/kibana -l /var/log/kibana.log &
检查5601端口是否启动 netstat -ntlup |grep ":5601"Install Logstash
[root@localhost src]rpm -ivh logstash-2.0.0-1.noarch.rpm
Preparing...                ########################################### [100%]
   1:logstash               ########################################### [100%]
server端配置
[root@localhost conf.d]# cat /etc/logstash/conf.d/index.conf
input {
  redis {
        host => "127.0.0.1"
        port => 6379
        data_type => "list"
        key => "logstash:redis"
        type => "redis-input"
}
}
filter {
    grok {   
        match => [ "message", "%{WORD:http_host} %{URIHOST:api_domain} %{IP:inner_ip} %{IP:lvs_ip} \[%{HTTPDATE:timestamp}\] \"%{WORD:http_verb} %{URIPATH:baseurl}(?:\?%{NOTSPACE:request}|) HTTP/%{NUMBER:http_version}\" (?:-|%{NOTSPACE:request}) %{NUMBER:http_status_code} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{NUMBER:time_duration:float} (?:%{NUMBER:time_backend_response:float}|-)"]
    kv {
        prefix => "request."
        field_split => "&"
        source => "request"
    }
    urldecode {
        all_fields => true
    }
    date { type => "log-date"
        match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]
    }
}
output {
  elasticsearch { hosts => ["localhost:9200"] }
}
本机日志配置
[root@localhost conf.d]# cat /etc/logstash/conf.d/agent.conf
input {
  file { path => "日志路径" }  #logstash用户需要对文件有读权限
}
output {
     redis {
          host => "127.0.0.1"  #redis IP地址
          data_type => "list"
          key => "logstash:redis"
        }
}[root@localhost src]# /etc/init.d/logstash start[root@localhost src]# chkconfig logstash on　　

客房端安装配置

　　

在nginx.conf 中设置日志格式：logstash

log_format logstash '$http_host $server_addr $remote_addr [$time_local] "$request" '
                    '$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" '
                    '$request_time $upstream_response_time';[root@localhost src]rpm -ivh logstash-2.0.0-1.noarch.rpm
Preparing...                ########################################### [100%]
   1:logstash               ########################################### [100%] [root@localhost conf.d]# cat /etc/logstash/conf.d/agent.conf
input {
  file { path => "日志路径" }  #logstash用户需要对文件有读权限
}
output {
     redis {
          host => "x.x.x.x"  #redis IP地址
          data_type => "list"
          key => "logstash:redis"
        }
}