Kibana+Logstash+Elasticsearch+Redis安装部署

trt4324

最近做日志分析，发现logstash较符合自己的需求，

• Logstash：做系统log收集，转载的工具。同时集成各类日志插件,对日志查询和分析的效率有很大的帮助.一般使用shipper作为log收集、indexer作为log转载.  
• Logstash shipper收集log 并将log转发给redis 存储  
• Logstash indexer从redis中读取数据并转发给elasticsearch  
• redis：是一个db，logstash shipper将log转发到redis数据库中存储。Logstash indexer从redis中读取数据并转发给elasticsearch。  
• Elasticsearch：elasticsearch是基于lucene的开源搜索引擎,用来做索引。
• Kibana： 开源web展现,界面很漂亮,是一个功能强大的elasticsearch数据显示客户端，logstash已经内置了kibana，你也可以单独部署kibana，最新版的kibana3是纯html+js客户端.
  
  

软件下载目录

http://www.elasticsearch.org/downloads/

我的环境如下

os：centos6.3_x86-64

redis-2.8.7.tar.gz   

kibana-3.0.0

java version "1.7.0_51"

elasticsearch-0.90.12

一，安装java

yum -y install java

二,安装redis

cd ~/src

wget http://download.redis.io/releases/redis-2.8.7.tar.gz

tar -zxf redis-2.8.7.tar.gz

cd redis-2.8.7.tar.gz

make

sudo make install

安装完毕后

/etc/init.d/redis_6379 start

测试是否正常

[iyunv@file1 ~]# redis-cli ping

PONG

[iyunv@file1 ~]#

[iyunv@file1 ~]# netstat -tanpu|grep redis

tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      1391/redis-server *

三，安装Elasticsearch

cd /search

sudo mkdir elasticsearch

cd elasticsearch

sudo wget https://download.elasticsearch.o ... csearch-0.90.12.zip

sudo unzip elasticsearch-0.90.12.zip

备注：当开始使用的是1.x.x java报错，后来用的0.9.。

https://groups.google.com/forum/#!topic/logstash-users/fvFT7pgQTEM

Are you using elasticsearch_http for your output?  If not, 1.3.3 is based on 0.90.x elasticsearch, and won’t play nice with 1.0.x elasticsearch with just the “elasticsearch” output.

启动ES服务器

切换到elasticsearch目录运行

bin/elasticsearch -f

默认端口是9200

curl -X GET http://localhost:9200

[iyunv@file1 ~]# curl -X GET http://localhost:9200

{

  "ok" : true,

  "status" : 200,

  "name" : "Master Pandemonium",

  "version" : {

    "number" : "0.90.12",

    "build_hash" : "26feed79983063ae83bfa11bd4ce214b1f45c884",

    "build_timestamp" : "2014-02-25T15:38:23Z",

    "build_snapshot" : false,

    "lucene_version" : "4.6"

  },

  "tagline" : "You Know, for Search"

}

四.安装logstash

cd /search

sudo mkdir logstash

cd logstash

sudo wget http://download.elasticsearch.or ... h-1.2.1-flatjar.jar

新建配置文件index.conf

# This is the logstash server index configuration.

# This file will be put in the same folder with logtash.jar file in the

# /etc/logtash/

# This takes information straight from redis and loads it into elasticsearch.

input {

  redis {

    host => "127.0.0.1"

    type => "syslog"

    threads => 4

    # these settings should match the output of the agent

    data_type => "list"

    key => "logstash"

    # We use json_event here since the sender is a logstash agent

    format => "json_event"

    }

}

output {

  elasticsearch {

    host => "127.0.0.1"

  }

}

新建shiper.conf

input {

  stdin {

    type => "test"

  }

}

output {

  stdout { codec => rubydebug }

  redis { host => "127.0.0.1" data_type => "list" key => "logstash" }

}

运行配置

java -jar logstash.jar agent -f shipper.conf

java -jar logstash.jar agent -f index.conf

五，配置kibana

logstash的最新版已经内置kibana，你也可以单独部署kibana。kibana3是纯粹JavaScript+html的客户端，所以可以部署到任意http服务器上。

https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.zip

解压到web目录

http://127.0.0.1/kibana/index.html