（实际应用）ELK环境搭建
今天给大家带来的是开源实时日志分析 ELK , ELK 由 ElasticSearch 、 Logstash 和 Kiabana 三个开源工具组成。官方网站：https://www.elastic.co其中的3个软件是：
Elasticsearch 是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制， restful 风格接口，多数据源，自动搜索负载等。
Logstash 是一个完全开源的工具，他可以对你的日志进行收集、分析，并将其存储供以后使用（如，搜索）。
kibana 也是一个开源和免费的工具，他 Kibana 可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志
系统系统需要安装的软件ip描述centos6.5Elasticsearch/test5192.168.253.210搜索存储日志centos6.5Elasticsearch/test4192.168.253.200搜索存储日志centos6.5Logstash/nginx/test1192.168.253.150用来收集日志给上面centos6.5Kibana/nginx/test2192.168.253.100用来后端的展示架构原理图：
https://s5.运维网.com/wyfs02/M02/99/A1/wKioL1lKVxzCfxdiAAFNzZZlHKs211.png
一、先安装elasticsearch集群，并测试通过再进行其他软件安装。
在test5，test4上安装分别安装elasticsearch-2.3.3.rpm前提要安装java1.8 步骤如下：
yum remove java-1.7.0-openjdk
rpm -ivh jdk-8u51-linux-x64.rpm
java -version
yum localinstall elasticsearch-2.3.3.rpm -y
service elasticsearch start
cd /etc/elasticsearch/
vim elasticsearch.yml
https://s1.运维网.com/wyfs02/M00/99/A1/wKiom1lKVx-xNboCAANLPQoz6w4738.png
https://s2.运维网.com/wyfs02/M00/99/A1/wKiom1lKVyKTTA7bAAPGFRJjA-o790.png
修改如下配置
cluster.name: myelk#设置集群的名称,在一个集群里面都是这个名称，必须相同
node.name: test5 #设置每一个节点的名，每个节点的名称必须不一样。
path.data: /path/to/data #指定数据的存放位置，线上的机器这个要放到单一的大分区里面。
path.logs: /path/to/logs #日志的目录
bootstrap.mlockall: true#启动最优内存配置，启动就分配了足够的内存，性能会好很多，测试我就不启动了。
network.host: 0.0.0.0#监听的ip地址，这个表示所有的地址。
http.port: 9200 #监听的端口号
discovery.zen.ping.unicast.hosts: ["hostip", "hostip"]#知道集群的ip有那些，没有集群就会出现就一台工作
mkdir -pv /path/to/{data,logs}
chown elasticsearch.elasticsearch /path -R
启动服务器service elasticsearch start并查看监控端口启动
# ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 :::54411 :::*
LISTEN 0 128 :::111 :::*
LISTEN 0 128 *:111 *:*
LISTEN 0 50 :::9200 :::*
LISTEN 0 50 :::9300 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 *:22 *:*
LISTEN 0 128 *:51574 *:*
LISTEN 0 128 127.0.0.1:631 *:*
LISTEN 0 128 ::1:631 :::*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 100 127.0.0.1:25 *:*
https://s3.运维网.com/wyfs02/M00/99/A1/wKioL1lKVyOhBbEzAACoXbhIE_c104.png
两台的配置都一样就是上面的IP和note名称要配置不一样就行
https://s3.运维网.com/wyfs02/M01/99/A1/wKiom1lKVyaDSZVhAANPyt2qmco961.png
https://s2.运维网.com/wyfs02/M00/99/A1/wKioL1lKVyeD4qCJAACn6hwvQvE344.png
# ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:45822 *:*
LISTEN 0 128 :::39620 :::*
LISTEN 0 128 :::111 :::*
LISTEN 0 128 *:111 *:*
LISTEN 0 50 :::9200 :::*
LISTEN 0 50 :::9300 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 *:22 *:*
LISTEN 0 128 127.0.0.1:631 *:*
LISTEN 0 128 ::1:631 :::*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 100 127.0.0.1:25 *:*
安装插件 head和kopf 之后访问 ip:9200/_plugin/head和ip:9200/_plugin/kopf（插件可以图形查看elasticsearch的状态和删除创建索引）
/usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
-> Installing lmenezes/elasticsearch-kopf...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...
Downloading ......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed kopf into /usr/share/elasticsearch/plugins/kopf
# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed head into /usr/share/elasticsearch/plugins/head
https://s2.运维网.com/wyfs02/M01/99/A1/wKiom1lKVyiBj93YAAEUyCNbmUs526.png
https://s2.运维网.com/wyfs02/M01/99/A1/wKioL1lKVynCOTBmAAGLpzLmIzQ080.png
二、安装nginx和logstash软件
yum -y install zlib zlib-devel openssl openssl--devel pcre pcre-devel
nginx-1.8.1-1.el6.ngx.x86_64.rpm
logstash-2.3.3-1.noarch.rpm
jdk-8u51-linux-x64.rpm
在test1上安装好nginx服务 就是收集它的日志呢
日志在/var/log/nginx/access.log
然后在test1上安装logstash-2.3.3-1.noarch.rpm
yum remove java-1.7.0-openjdk
rpm -ivh jdk-8u91-linux-x64.rpm
rpm -ivh logstash-2.3.3-1.noarch.rpm
/etc/init.d/logstash start#启动服务
/opt/logstash/bin/logstash -e "input {stdin{}} output{stdout{ codec=>"rubydebug"}}"#检测环境执行这个命令检测环境正常否，启动完成后 直接输入东西就会出现
Settings: Default pipeline workers: 1
Pipeline main started
hello world
{
"message" => "hello world",
"@version" => "1",
"@timestamp" => "2017-05-24T08:04:46.993Z",
"host" => "0.0.0.0"
}
之后输入/opt/logstash/bin/logstash -e 'input {stdin{}} output{ elasticsearch { hosts => ["192.168.253.200:9200"] index => "test"}}'
就是输入东西到253.200的elasticsearch上 会在/path/to/data/myelk/nodes/0/indices 生成你名称test索引文件目录 可以多输入几个到253.200的目录看看有没有文件有就证明正常。
# ls /path/to/data/myelk/nodes/0/indices/
test
之后在test1的/etc/logstash/conf.d 建立以.conf结尾的配置文件，我收集nginx就叫nginx.conf了内容如下
# cd /etc/logstash/conf.d/
# ls
nginx.conf
# cat nginx.conf
input {
file {
type => "accesslog"
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
output {
if == "accesslog" {
elasticsearch{
hosts => ["192.168.253.200"]
index => "nginx-access-%{+YYYY.MM.dd}"
}
}
}
/etc/init.d/logstash configtest
ps -ef |grep java
/opt/logstash/bin/logstash -f nginx.conf
之后在elasticearch查看有没有索引生成。多访问下nginx服务
如果没有就修改这个文件
vi /etc/init.d/logstash
LS_USER=root ###把这里换成root或者把访问的日志加个权限可以让logstash可以读取它 重启服务就会生成索引了
LS_GROUP=root
LS_HOME=/var/lib/logstash
LS_HEAP_SIZE="1g"
LS_LOG_DIR=/var/log/logstash
LS_LOG_FILE="${LS_LOG_DIR}/$name.log"
LS_CONF_DIR=/etc/logstash/conf.d
LS_OPEN_FILES=16384
LS_NICE=19
KILL_ON_STOP_TIMEOUT=${KILL_ON_STOP_TIMEOUT-0} #default value is zero to this variable but could be updated by user request
LS_OPTS=""
test4查看：
# ls /path/to/data/myelk/nodes/0/indices/
nginx-access-2017.05.23test
# cat logstash.log
{:timestamp=>"2017-05-24T16:05:19.659000+0800", :message=>"Pipeline main started"}
三、安装kibana软件
上面的都安装完成后在test2上面安装kibana
rpm -ivh kibana-4.5.1-1.x86_64.rpm
编辑配置文件在这里/opt/kibana/config/kibana.yml 就修改下面几项就行
server.port: 5601端口
server.host: "0.0.0.0"监听
elasticsearch.url: "http://192.168.48.200:9200" elasticsearch地址
https://s5.运维网.com/wyfs02/M02/99/A1/wKiom1lKVy3Rg2wyAASXgVhxpAc105.png
/etc/init.d/kibana start 启动服务
访问kibanahttp://ip:5601
https://s4.运维网.com/wyfs02/M02/99/A1/wKioL1lKVy_CE48rAAFZIxOCkTA703.png
添加展示的索引，就是在上面定义的nginx-access-2016.07.03
https://s5.运维网.com/wyfs02/M00/99/A1/wKiom1lKVzDxtwCPAAGf0QTti0M211.png
https://s2.运维网.com/wyfs02/M01/99/A1/wKioL1lKVzGTz54CAAH4Fr2Gdfo086.png
https://s2.运维网.com/wyfs02/M00/99/A1/wKioL1lKVzPCCqmYAAIgCspbUBw762.png
https://s5.运维网.com/wyfs02/M01/99/A1/wKiom1lKVzSA1GLqAAD-UanfhyI701.png
配置kibana上面的收集Nginx日志的logstash
在kibana那台服务器上面安装logstash（按照之前的步骤安装）
然后再logstash的/etc/logstash/conf.d/下面
写一个配置文件：
# vim nginx.conf
input {
file {
type => "accesslog"
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
output {
if == "accesslog" {
elasticsearch{
hosts => ["192.168.253.200"]
index => "nginx-access-%{+MM.dd.YYYY}"
}
}
}
/opt/logstash/bin/logstash -f nginx.conf
查看Elasticsearch中多出了一个以月-日-年的Nginx访问日志索引
# ls /path/to/data/myelk/nodes/0/indices/
nginx-access-05.23.2017nginx-access-2017.05.23test
然后在kibana，浏览器上面按照之前的创建，生成一个新的日志文件
https://s5.运维网.com/wyfs02/M01/99/A1/wKioL1lKVzWCwN3HAAD2hwV-VTw111.png
四、其他的一些配置。
kibana是直接访问的比较不安全，我们需要用nginx访问代理，并设置权限用户名和密码访问
先在kibana服务器上安装nginx 不介绍了
在nginx里面配置
#############################################################################
server
{
listen 80;
server_name localhost;
auth_basic "Restricted Access";
auth_basic_user_file /usr/local/nginx/conf/htpasswd.users;#密码和用户
location / {
proxy_pass http://localhost:5601;#代理kibana的5601之后就可以直接80访问了
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header REMOTE-HOST $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
#############################################################################
创建密码和用户文件：htpasswd.users
需要安装httpd-tool包先安装它
htpasswd -bc /usr/local/nginx/conf/htpasswd.users admin paswdadmin#前面是用户后面是密码
https://s2.运维网.com/wyfs02/M02/99/A1/wKioL1lKWF2h9B9GAAARyQRhdto321.png
之后通过访问需要密码和用户并且是80端口了
https://s1.运维网.com/wyfs02/M02/99/A1/wKiom1lKVzeRdjmxAAGtdvk60ww913.png
页:
[1]