ELK部署logstash安装部署及应用（二）
Logstash 安装部署注意事项：[*] Logstash基本概念:
[*]logstash收集日志基本流程: input-->codec-->filter-->codec-->output
[*]input:从哪里收集日志。
[*]filter:发出去前进行过滤
[*]output:输出至Elasticsearch或Redis消息队列
[*]codec:输出至前台，方便边实践边测试
[*]数据量不大日志按照月来进行收集
[*]如果通过logstash来采集日志，那么每个客户端都需要安装logstash
[*] 安装需要前置系统环境
[*]安装java
[*] 下载并安装GPG key # rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
[*] 配置yum仓库
# echo '> name=Logstash repository for 2.3.x packages
> baseurl=https://packages.elastic.co/logstash/2.3/centos> gpgcheck=1> gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
> enabled=1' >>/etc/yum.repos.d/logstash.repo
[*] 安装logstash
# yum install -y logstash
[*] 查看多种输出输出方式
实例1
# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
[*] 输出内容：
# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hello2017-06-27T04:10:21.010Z master_agent hello
hello word 2017-06-27T04:10:48.513Z master_agent hello wordquit2017-06-27T04:11:12.959Z master_agent quitexit2017-06-27T04:11:19.064Z master_agent exit实例2
# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug} }'
# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug} }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hello word
{ "message" => "hello word", "@version" => "1", "@timestamp" => "2017-06-27T04:13:14.560Z", "host" => "master_agent"}实例3 不输出在屏幕上，直接记录在es里
# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"]} }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hello word!!
Pipeline main has been shutdown实例5
[*] 记录两份，屏幕输出一份、es记录一份 # /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"] } stdout{ codec => rubydebug } }'
# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"]} stdout{ codec => rubydebug}}'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hello fly salt
{ "message" => "hello fly salt", "@version" => "1", "@timestamp" => "2017-06-27T04:32:50.297Z", "host" => "master_agent"}查看web页面记录的内容
https://s3.运维网.com/wyfs02/M01/9D/A4/wKiom1mDClOB3FSMAAD-BxJZcCE498.png-wh_500x0-wm_3-wmp_4-s_3637999389.png
Logstash配置文件
[*] 配置Logstash,将创建一个配置文件,指定您想要使用的插件和设置为每个插件。 你可以参考事件字段在配置和使用条件符合一定时处理事件标准；当您运行logstash,你使用-f指定您的配置文件。 参考链接
[*] 实例如下：
[*] # vim /etc/logstash/conf.d/logstash.conf ##新建一个配置语法文件，让输入输出标准化
input { stdin { } }output {
elasticsearch { hosts => ["localhost:9200"] }stdout { codec => rubydebug }
}
[*] # /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf ### 执行命令配置语法文件 手动输入hao123.com，查看结果
# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default pipeline workers: 1Pipeline main started
hao123.com
{ "message" => "hao123.com", "@version" => "1", "@timestamp" => "2017-06-27T05:06:02.695Z", "host" => "master_agent"}input插件
官方链接
[*] 创建一个文件夹用于专门存放logfile配置文件 存放的路径
[*] # mkdir -p /home/elk/logfile/
[*] Input pluginsfile 收集实例：
[*] 编辑一个新的配置文件
[*]
# mkdir -p /home/elk/logfile/# cd /home/elk/logfile/# ls# vim file.conf
[*] 编写配置文件内容
input {
file {
path => "/var/log/messages" ###读取log日志文件路径
type => "system" ### 设置个类型
start_position => "beginning" ### 从日志最开始地方读取
}
}output {
elasticsearch {
hosts => ["172.16.1.201:9200"]
index => "system-%{+YYYY.MM.dd}" ##设置个索引每天创建
}
}注意：如果采集的日志文件日志量不是很大的情况下，采用每月建一从索引，日志量很大的情况下可以按照每天来建立索引
https://s2.运维网.com/wyfs02/M00/9D/A4/wKioL1mDCt3i_OPSAAK_uPi7jSU724.png
https://s1.运维网.com/wyfs02/M01/9D/A4/wKiom1mDCt3Ds9EPAABvwAUE5iI280.png
页:
[1]