ELK部署logstash安装部署及应用（二）

scaoping

Logstash 安装部署注意事项：

• Logstash基本概念:
  
• logstash收集日志基本流程: input-->codec-->filter-->codec-->output
  
• input:从哪里收集日志。
  
• filter:发出去前进行过滤
  
• output:输出至Elasticsearch或Redis消息队列
  
• codec:输出至前台，方便边实践边测试
  
• 数据量不大日志按照月来进行收集
  
• 如果通过logstash来采集日志，那么每个客户端都需要安装logstash
  
• 安装需要前置系统环境
  
• 安装java
  
• 下载并安装GPG key [root@master_agent yum.repos.d]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
  
• 配置yum仓库
  

[root@master_agent yum.repos.d]# echo '[logstash-2.3]> name=Logstash repository for 2.3.x packages
> baseurl=https://packages.elastic.co/logstash/2.3/centos> gpgcheck=1> gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
> enabled=1' >>/etc/yum.repos.d/logstash.repo

• 　　安装logstash
  

[root@master_agent yum.repos.d]# yum install -y logstash

• 　　查看多种输出输出方式
  

实例1
[root@master_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'

• 　　输出内容：
  

[root@master_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hello2017-06-27T04:10:21.010Z master_agent hello
hello word            2017-06-27T04:10:48.513Z master_agent hello wordquit2017-06-27T04:11:12.959Z master_agent quitexit2017-06-27T04:11:19.064Z master_agent exit实例2
[root@master_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug} }'
[root@master_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug} }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hello word
{       "message" => "hello word",      "@version" => "1",    "@timestamp" => "2017-06-27T04:13:14.560Z",          "host" => "master_agent"}实例3 不输出在屏幕上，直接记录在es里
[root@master_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"]  } }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hello word!!
Pipeline main has been shutdown实例5

• 　　记录两份，屏幕输出一份、es记录一份 [root@master_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"] } stdout{ codec => rubydebug } }'
  

[root@master_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"]  } stdout{ codec => rubydebug  }  }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hello fly salt
{       "message" => "hello fly salt",      "@version" => "1",    "@timestamp" => "2017-06-27T04:32:50.297Z",          "host" => "master_agent"}查看web页面记录的内容

Logstash配置文件

• 配置Logstash,将创建一个配置文件,指定您想要使用的插件和设置为每个插件。 你可以参考事件字段在配置和使用条件符合一定时处理事件标准；当您运行logstash,你使用-f指定您的配置文件。 参考链接
  
• 实例如下：
  
• [root@master_agent yum.repos.d]# vim /etc/logstash/conf.d/logstash.conf ##新建一个配置语法文件，让输入输出标准化
  

input { stdin { } }output {
  elasticsearch { hosts => ["localhost:9200"] }  stdout { codec => rubydebug }
}

• 　　[root@master_agent yum.repos.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf ### 执行命令配置语法文件 手动输入hao123.com，查看结果
  

[root@master_agent yum.repos.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hao123.com
{       "message" => "hao123.com",      "@version" => "1",    "@timestamp" => "2017-06-27T05:06:02.695Z",          "host" => "master_agent"}input插件
官方链接

• 　　创建一个文件夹用于专门存放logfile配置文件 存放的路径
  
• 　　[root@master_agent yum.repos.d]# mkdir -p /home/elk/logfile/
  
• 　　Input plugins  file 收集实例：
  
• 　　编辑一个新的配置文件
  
• 　　
  
  

[root@master_agent yum.repos.d]# mkdir -p /home/elk/logfile/[root@master_agent yum.repos.d]# cd /home/elk/logfile/[root@master_agent logfile]# ls[root@master_agent logfile]# vim file.conf

• 　　编写配置文件内容
  

input {
        file {
                path => "/var/log/messages"   ###读取log日志文件路径
                type => "system"                ### 设置个类型
                start_position => "beginning"   ### 从日志最开始地方读取
        }
}output {
        elasticsearch {
                hosts => ["172.16.1.201:9200"]
                index => "system-%{+YYYY.MM.dd}"   ##设置个索引每天创建
        }
}注意：如果采集的日志文件日志量不是很大的情况下，采用每月建一从索引，日志量很大的情况下可以按照每天来建立索引