ELK日志集中化管理系统

45332

ELK Stack 是 Elasticsearch、Logstash、Kibana 三个开源软件的组合。在实时数据检索和分析场合，三者通常是配合共用，而且又都先后归于 Elastic.co 公司名下，故有此简称。     ELK Stack 在最近两年迅速崛起，成为机器数据分析，或者说实时日志处理领域，开源界的第一选择。
ELK由三个组建构成，分别是
Elasticsearch，负责数据的索引和存储
Logstash ，负责日志的采集和格式化
Kibana，负责前端统计的展示
大致的架构如下：

redis提供队列来作为broker负责log传输过程中的缓冲，也可以由kafka等来代替。


一、logstansh安装

1、JDK安装  

1 2 3 4 5

yum -y install java-1.8.0 [iyunv@ELKServer ~]# java -version openjdk version "1.8.0_101" OpenJDK Runtime Environment (build 1.8.0_101-b13) OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)

2、logstansh安装

1 2 3 4 5

wget https://download.elastic.co/logstash/logstash/logstash-2.3.4.zip unzip logstash-2.3.4.zip mv logstash-2.3.4 /usr/local/ echo "PATH=$PATH:/usr/local/logstash-2.3.4/bin" >> /etc/profile source /etc/profile

3、新建logstansh配置文件目录

1	mkdir /usr/local/logstash-2.3.4/conf

4、测试logstansh

1 2 3 4 5

[iyunv@ELKServer logstash-2.3.4]#  logstash -e "input {stdin{}} output {stdout{}}" hello Settings: Default pipeline workers: 1 Pipeline main started 2016-07-28T12:47:41.597Z 0.0.0.0 hello

二、redis安装
1、redis安装

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

wget http://download.redis.io/releases/redis-2.8.20.tar.gz yum install tcl -y tar -zxvf redis-2.8.20.tar.gz mv redis-2.8.20/ /usr/local/ [iyunv@ELKServer logstash-2.3.4]# cd /usr/local/redis-2.8.20/ [iyunv@ELKServer redis-2.8.20]# make MALLOC=libc [iyunv@ELKServer redis-2.8.20]# make install [iyunv@ELKServer redis-2.8.20]# cd utils/ [iyunv@ELKServer utils]# ./install_server.sh   #所有选项默认 Welcome to the redis service installer This script will help you easily set up a running redis server Please select the redis port for this instance: [6379] Selecting default: 6379 Please select the redis config file name [/etc/redis/6379.conf] Selected default - /etc/redis/6379.conf Please select the redis log file name [/var/log/redis_6379.log] Selected default - /var/log/redis_6379.log Please select the data directory for this instance [/var/lib/redis/6379] Selected default - /var/lib/redis/6379 Please select the redis executable path [/usr/local/bin/redis-server] Selected config: Port           : 6379 Config file    : /etc/redis/6379.conf Log file       : /var/log/redis_6379.log Data dir       : /var/lib/redis/6379 Executable     : /usr/local/bin/redis-server Cli Executable : /usr/local/bin/redis-cli Is this ok? Then press ENTER to go on or Ctrl-C to abort. Copied /tmp/6379.conf => /etc/init.d/redis_6379 Installing service... Successfully added to chkconfig! Successfully added to runlevels 345! Starting Redis server... Installation successful!

2、查看redis的监控端口

1 2 3

[iyunv@ELKServer utils]# netstat -tnlp |grep redis tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      2085/redis-server * tcp        0      0 :::6379                     :::*                        LISTEN      2085/redis-server *

3、测试redis是否缓存数据
a、新建logstansh配置文件如下：

1 2 3 4 5 6 7 8 9 10

[iyunv@ELKServer conf]# cat output_redis.conf input { stdin { } }    #手动输入数据 output {                    stdout { codec => rubydebug }  #页面debug信息     redis {         host => '192.168.10.49'         data_type => 'list'         key => 'redis'     } }

b、启动logstansh

1 2 3 4 5 6 7 8 9 10 11 12 13

[iyunv@ELKServer conf]# logstash -f output_redis.conf --verbose starting agent {:level=>:info} starting pipeline {:id=>"main", :level=>:info} Settings: Default pipeline workers: 1 Starting pipeline {:id=>"main", :pipeline_workers=>1, :batch_size=>125, :batch_delay=>5, :max_inflight=>125, :level=>:info} Pipeline main started hello   #手动输入 {        "message" => "hello",       "@version" => "1",     "@timestamp" => "2016-07-28T13:09:42.651Z",           "host" => "0.0.0.0" }

c、查看redis中是否存在数据。

1	cd /usr/local/redis-2.8.20/src/

执行下面这条命令后再执行logstash -f output_redis.conf(最好分两个ssh会话测试)。

1 2 3

[iyunv@ELKServer src]# ./redis-cli monitor OK 1469711599.811344 [0 192.168.10.49:56179] "rpush" "redis" "{\"message\":\"hello\",\"@version\":\"1\",\"@timestamp\":\"2016-07-28T13:13:18.446Z\",\"host\":\"0.0.0.0\"}"

三、elasticsearch安装
1、elasticsearch安装配置

1 2 3

wget https://download.elastic.co/elas ... ticsearch-2.3.4.zip unzip elasticsearch-2.3.4.zip mv elasticsearch-2.3.4 /usr/local/

修改elasticsearch配置文件

1	vim /usr/local/elasticsearch-2.3.4/config/elasticsearch.yml

把下面参数的注释去掉并改成服务器IP。这里只做简单安装，优化及集群后面再介绍

1	network.host: 192.168.10.49

2、elasticsearch启动

1 2	[iyunv@ELKServer bin]# /usr/local/elasticsearch-2.3.4/bin/elasticsearch  -d [iyunv@ELKServer bin]# Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.'

elasticsearch启动不能使用root用户。新建elk用户，并把elasticsearch目录用户所有者改成elk

1	[elk@ELKServer bin]$ ./elasticsearch -d

查看elasticsearch是否启动

1 2 3

[elk@ELKServer bin]$ netstat -tnlp |grep java tcp        0      0 ::ffff:192.168.10.49:9200   :::*                        LISTEN      2476/java            tcp        0      0 ::ffff:192.168.10.49:9300   :::*                        LISTEN      2476/java

3、测试logstansh和elasticsearch是否能结合使用
a、新建logstansh配置文件elasticsearch.conf

1 2 3 4 5 6 7

[iyunv@ELKServer bin]# cd /usr/local/logstash-2.3.4/conf [iyunv@ELKServer conf]# cat elasticsearch.conf input { stdin {} }    #手动输入 output {     elasticsearch { hosts => "192.168.10.49" }        stdout { codec=> rubydebug }   #页面debug信息 }

b、启动elasticsearch.conf配置文件

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

[iyunv@ELKServer conf]# logstash -f elasticsearch.conf --verbose starting agent {:level=>:info} starting pipeline {:id=>"main", :level=>:info} Settings: Default pipeline workers: 1 Using mapping template from {:path=>nil, :level=>:info} Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"float"}, "longitude"=>{"type"=>"float"}}}}}}}, :level=>:info} New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["192.168.10.49"], :level=>:info} Starting pipeline {:id=>"main", :pipeline_workers=>1, :batch_size=>125, :batch_delay=>5, :max_inflight=>125, :level=>:info} Pipeline main started hello elasticsearch    #这行是手动输入的 {        "message" => "hello elasticsearch",       "@version" => "1",     "@timestamp" => "2016-07-28T14:20:09.460Z",           "host" => "0.0.0.0" }

c、查看elasticsearch是否获取到了"hello elasticsearch"

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

[iyunv@ELKServer conf]# curl http://localhost:9200/_search?pretty {   "took" : 45,   "timed_out" : false,   "_shards" : {     "total" : 5,     "successful" : 5,     "failed" : 0   },   "hits" : {     "total" : 1,     "max_score" : 1.0,     "hits" : [ {       "_index" : "logstash-2016.07.28",       "_type" : "logs",       "_id" : "AVYx4H2BdePMZ0WyBMGl",       "_score" : 1.0,       "_source" : {         "message" : "hello elasticsearch",         "@version" : "1",         "@timestamp" : "2016-07-28T14:20:09.460Z",         "host" : "0.0.0.0"       }     } ]   } }

4、安装elasticsearch插件
elasticsearch有很多插件：http://www.searchtech.pro/elasticsearch-plugins
elasticsearch-head插件安装

1 2 3 4 5 6 7

[iyunv@ELKServer lang-expression]# cd /usr/local/elasticsearch-2.3.4/bin/ [iyunv@ELKServer bin]# ./plugin install mobz/elasticsearch-head -> Installing mobz/elasticsearch-head... Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ... Downloading .................DONE Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ... Installed head into /usr/local/elasticsearch-2.3.4/plugins/head

查看elasticsearch-head插件显示的页面
http://192.168.10.49:9200/_plugin/head/


四、kibana安装
1、安装kibana

1 2 3 4

[iyunv@ELKServer ~]# tar -zxvf kibana-4.5.2-linux-x64.tar.gz   [iyunv@ELKServer ~]# mv kibana-4.5.2-linux-x64 /usr/local/ [iyunv@ELKServer ~]# cd /usr/local/kibana-4.5.2-linux-x64/config/ [iyunv@ELKServer config]# vim kibana.yml

修改kibana配置文件，把下面这行改成elasticsearc的访问路径

1	elasticsearch.url: "http://192.168.10.49:9200"

2、启动kibana

1 2 3 4 5 6 7 8 9 10 11 12

[iyunv@ELKServer bin]# /usr/local/kibana-4.5.2-linux-x64/bin/kibana&   log   [23:10:23.215] [info][status][plugin:kibana] Status changed from uninitialized to green - Ready   log   [23:10:23.288] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch   log   [23:10:23.344] [info][status][plugin:kbn_vislib_vis_types] Status changed from uninitialized to green - Ready   log   [23:10:23.377] [info][status][plugin:markdown_vis] Status changed from uninitialized to green - Ready   log   [23:10:23.415] [info][status][plugin:metric_vis] Status changed from uninitialized to green - Ready   log   [23:10:23.445] [info][status][plugin:spyModes] Status changed from uninitialized to green - Ready   log   [23:10:23.464] [info][status][plugin:statusPage] Status changed from uninitialized to green - Ready   log   [23:10:23.485] [info][status][plugin:table_vis] Status changed from uninitialized to green - Ready   log   [23:10:23.510] [info][listening] Server running at http://0.0.0.0:5601   log   [23:10:28.470] [info][status][plugin:elasticsearch] Status changed from yellow to yellow - No existing Kibana index found   log   [23:10:31.442] [info][status][plugin:elasticsearch] Status changed from yellow to green - Kibana index ready

3、测试kibana
访问页面：http://192.168.10.49:5601/

五、ELK实例测试
1、简单介绍下ELK直接的数据传输
a、logstansh根据日志存放路径采集日志（input），然后转给redis缓存（output）
b、将消息从redis拿出来（input），然后用filter对日志分割处理
c、最后将处理好的日志转给ealsticsearch（output）
在测试中我们没有对日志做filter处理
2、新建logstansh配置文件
logstash_redis.conf  logstansh客户端配置文件

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

input {         file {                 path => "/var/log/messages"                 start_position => beginning                 sincedb_write_interval => 0                 add_field => {"Host"=>"192.168.10.47"}                 type => "SYSLOG_LOG"         } } output {             redis {                 host => "192.168.10.47:6379"                 data_type => "list"                 key => "logstash:syslog_log"             } }

redis_elasticserach.conf   logstansh服务端配置文件

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

input {     redis {         host => '192.168.10.47'           data_type => 'list'         port => "6379"         key => 'logstash:syslog_log'         type => 'redis-input'       } } output {     elasticsearch {         hosts => "192.168.10.47" index => "logstash-%{+YYYY.MM.dd}"     } }

2、启动ELK各项服务

1 2 3 4 5

logstash -f logstash_redis.conf & logstash -f redis_elasticsearch.conf & /etc/init.d/redis_6379 start /usr/local/elasticsearch-2.3.4/bin/elasticsearch  -d(elk用户启动) /usr/local/kibana-4.5.2-linux-x64/bin/kibana&

3、检查日志传输是否正常
a、查看elasticsearch数据信息
http://192.168.10.49:9200/_plugin/head/

b、查看kibana是否有数据显示
http://192.168.10.49:5601/
注意：因为上面从message获取的日志是日志文件首行开始获取，时间可能是几天前的。
在kibana查看数据是注意选择时间范围