CentOS7.2部署ELK6

ddsdjn · 发表于 2019-4-17 16:03:51

　　需求背景：
业务发展越来越庞大，服务器越来越多
各种访问日志、应用日志、错误日志量越来越多，导致运维人员无法很好的去管理日志
开发人员排查问题，需要到服务器上查日志，不方便
运营人员需要一些数据，需要我们运维到服务器上分析日志　　为什么要用到ELK：
一般我们需要进行日志分析场景：直接在日志文件中 grep、awk 就可以获得自己想要的信息。但在规模较大也就是日志量多而复杂的场景中，
此方法效率低下，面临问题包括日志量太大如何归档、文本搜索太慢怎么办、如何多维度查询。需要集中化的日志管理，所有服务器上的日志收集汇总。
常见解决思路是建立集中式日志收集系统，将所有节点上的日志统一收集，管理，访问。
大型系统通常都是一个分布式部署的架构，不同的服务模块部署在不同的服务器上，问题出现时，大部分情况需要根据问题暴露的关键信息，
定位到具体的服务器和服务模块，构建一套集中式日志系统，可以提高定位问题的效率。
一个完整的集中式日志系统，需要包含以下几个主要特点：
收集－能够采集多种来源的日志数据
传输－能够稳定的把日志数据传输到中央系统
存储－如何存储日志数据
分析－可以支持 UI 分析
警告－能够提供错误报告，监控机制
而ELK则提供了一整套解决方案，并且都是开源软件，之间互相配合使用，完美衔接，高效的满足了很多场合的应用。是目前主流的一种日志系统。　　ELK简介：
ELK是三个开源软件的缩写，分别为：Elasticsearch 、 Logstash以及Kibana , 它们都是开源软件。不过现在还新增了一个Beats，
它是一个轻量级的日志收集处理工具(Agent)，Beats占用资源少，适合于在各个服务器上搜集日志后传输给Logstash，官方也推荐此工具，
目前由于原本的ELK Stack成员中加入了 Beats 工具所以已改名为Elastic Stack。　　Elastic Stack包含：
Elasticsearch是个开源分布式搜索引擎，提供搜集、分析、存储数据三大功能。它的特点有：分布式，零配置，自动发现，索引自动分片，
索引副本机制，restful风格接口，多数据源，自动搜索负载等。详细可参考Elasticsearch权威指南。
Logstash 主要是用来日志的搜集、分析、过滤日志的工具，支持大量的数据获取方式。一般工作方式为c/s架构，client端安装在需要收集日志的主机上，
server端负责将收到的各节点日志进行过滤、修改等操作在一并发往elasticsearch上去。
Kibana 也是一个开源和免费的工具，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助汇总、分析和搜索重要数据日志。
Beats在这里是一个轻量级日志采集器，其实Beats家族有6个成员，早期的ELK架构中使用Logstash收集、解析日志，但是Logstash对内存、cpu、io等资源消耗比较高。
相比 Logstash，Beats所占系统的CPU和内存几乎可以忽略不计　　ELK一些组件：
openjdk 1.8.0_191
ElasticSearch
Logstash
Kibana
Filebeat　　ELK Stack （5.0版本之后）--> Elastic Stack == （ELK Stack + Beats）。目前Beats包含六种工具：
Packetbeat：网络数据（收集网络流量数据）
Metricbeat：指标（收集系统、进程和文件系统级别的 CPU 和内存使用情况等数据）
Filebeat：日志文件（收集文件数据）
Winlogbeat： windows事件日志（收集 Windows 事件日志数据）
Auditbeat：审计数据（收集审计日志）
Heartbeat：运行时间监控（收集系统运行时的数据）　　关于x-pack工具：
x-pack对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包，是收费的，所以本文不涉及x-pack的安装。　　

　　ELK相关链接：
ELK官网：https://www.elastic.co/cn/
中文指南：https://www.gitbook.com/book/chenryn/elk-stack-guide-cn/details　　https://s1.运维网.com/images/20190330/1553946970246931.png
　　备注：以上内容来自网络。

　　一、准备
　　
　　ELK注意点：
1、主要掌握logstash的原理、ElasticSearch原理、Kibana可视化技术（这里我采用Grafana可视化）
2、logstash的时候output先输出到命令控制台，如果没问题再写入配置文件。
3、要引入redis或者kanfa，有很多优点。一般情况下redis都能胜任，kanfa太重型。　　1、关闭防火墙和selinux
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl stop firewalld
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl disable firewalld
[root@iZj6c7bp1a289r04izstw4Z ~]# vi /etc/sysconfig/selinux
SELINUX=disabled　　2、安装常用工具、同步时间
[root@iZj6c7bp1a289r04izstw4Z ~]# yum -y install lrzsz rdate wget vim net-tools ntpdate
[root@iZj6c7bp1a289r04izstw4Z ~]# rdate -s time-b.nist.gov　　3、更改最大文件描述符、打开数、进程数
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/security/limits.conf
* hard nofile 65536
* soft nofile 65536
* soft nproc 65536
* hard nproc 65536　　4、修改最大打开进程数，最低2048
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/security/limits.d/20-nproc.conf
*       soft nproc    4096　　

　　5、其他设置
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/sysctl.conf
vm.max_map_count=655360
net.core.somaxconn=65535
net.ipv4.ip_forward = 1
[root@iZj6c7bp1a289r04izstw4Z ~]# sysctl -p　　二、安装JDK环境
可以yum、rpm、tar.gz等方式安装。为了简单和省事，我直接yum方式安装（推荐yum方式安装，因为它会将java程序放在/usr/bin等目录下。
其它两种方式会造成服务启动不了，需要设置环境变量、elk配置文件或做软链接）。
[root@iZj6c7bp1a289r04izstw4Z ~]# yum install java-1.8.0-openjdk -y
[root@localhost ~]# java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)　　三、安装ELK
　　1、配置yum源
[root@iZj6c7bp1a289r04izstw4Z ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/yum.repos.d/elk.repo
[elk-6]
name=elk-6
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
[root@iZj6c7bp1a289r04izstw4Z ~]# yum clean all
[root@iZj6c7bp1a289r04izstw4Z ~]# yum makecache　　2、安装ELK
[root@iZj6c7bp1a289r04izstw4Z ~]# yum install elasticsearch logstash kibana filebeat -y
备注：当然elk除了采用yum方式安装，还可以rpm、tar.gz等方式安装。由于默认yum方式安装和其它方式安装的版本是一样，所以我选择采用yum方式。
rpm包链接如下：
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.1.rpm
https://artifacts.elastic.co/downloads/logstash/logstash-6.6.1.rpm
https://artifacts.elastic.co/downloads/kibana/kibana-6.6.1-x86_64.rpm
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.6.1-x86_64.rpm　　四、配置ELK
　　1、配置elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# rpm -qc elasticsearch
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/jvm.options
/etc/elasticsearch/log4j2.properties
/etc/elasticsearch/role_mapping.yml
/etc/elasticsearch/roles.yml
/etc/elasticsearch/users
/etc/elasticsearch/users_roles
/etc/init.d/elasticsearch
/etc/sysconfig/elasticsearch
/usr/lib/sysctl.d/elasticsearch.conf
/usr/lib/systemd/system/elasticsearch.service
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: ELK
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.1.104:9300"]
discovery.zen.minimum_master_nodes: 1
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl daemon-reload
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl enable elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl restart elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl status elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# ps aux |grep elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -nltp | grep java
tcp       0    0 0.0.0.0:9200          0.0.0.0:*             LISTEN    1776/java
tcp       0    0 0.0.0.0:9300          0.0.0.0:*             LISTEN    1776/java
[root@iZj6c7bp1a289r04izstw4Z ~]# curl -X GET http://localhost:9200
{
  "name" : "node-1",
  "cluster_name" : "ELK",
  "cluster_uuid" : "egGlhWnESzeeUalJ2FXUuA",
  "version" : {
"number" : "6.6.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1fd8f69",
"build_date" : "2019-02-13T17:10:04.160291Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}　　

　　2、配置kibana
[root@iZj6c7bp1a289r04izstw4Z ~]# rpm -qc kibana
/etc/kibana/kibana.yml
[root@iZj6c7bp1a289r04izstw4Z ~]# vi /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"
logging.dest: /var/log/kibana.log
#创建日志文件：
[root@master-node ~]# touch /var/log/kibana.log; chmod 777 /var/log/kibana.log
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl start kibana
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl status kibana
[root@iZj6c7bp1a289r04izstw4Z ~]# ps aux |grep kibana
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 5601
tcp       0    0 0.0.0.0:5601          0.0.0.0:*             LISTEN    6030/node
注：由于kibana是使用node.js开发的，所以进程名称为node
然后在浏览器里进行访问，如：http://172.31.116.244:5601/ ，由于我们并没有安装x-pack，所以此时是没有用户名和密码的，可以直接访问的：　　https://s1.运维网.com/images/20190330/1553947119800697.png
　　
如果需要账号密码认证的，需要安装插件，但是插件好像是收费的。
不过可以安装nginx和认证模块，然后代理kibana就可以，具体步骤如下：
[root@iZj6c7bp1a289r04izstw4Z ~]# yum -y install epel-release 　　#安装epel源
[root@iZj6c7bp1a289r04izstw4Z ~]# yum -y install nginx httpd-tools 　　#安装Nginx、httpd用户认证工具
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/nginx/nginx.conf #注释如下图
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/nginx/conf.d/kibana.conf
server {
      listen  5666;
      server_name    kibana;
      auth_basic    "Restricted Access";
      auth_basic_user_file /etc/nginx/kibana-user;
      location / {
         proxy_pass http://localhost:5601;
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection 'upgrade';
         proxy_set_header Host $host;
         proxy_cache_bypass $http_upgrade;
      }
}
[root@iZj6c7bp1a289r04izstw4Z ~]# htpasswd -cm /etc/nginx/kibana-user gxm
New password:
Re-type new password:
Adding password for user gxm
[root@iZj6c7bp1a289r04izstw4Z ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl start nginx
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl status nginx　　https://s1.运维网.com/images/20190330/1553947153533997.png
　　3、配置logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# rpm -qc logstash
/etc/logstash/jvm.options
/etc/logstash/log4j2.properties
/etc/logstash/logstash-sample.conf
/etc/logstash/logstash.yml
/etc/logstash/pipelines.yml
/etc/logstash/startup.options
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/logstash.yml
path.config: /etc/logstash/conf.d　　4、测试logstash场景1（系统日志作为输入，直接输出到终端）。
#设置syslog日志配置文件
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/rsyslog.conf
#### RULES ####
*.* @172.31.116.244:10514
[root@data-node1 ~]# systemctl restart rsyslog#设置logstash配置文件
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/conf.d/syslog-01.conf
input {  # 定义日志源
  syslog {
type => "system-syslog"  # 定义类型
port => 10514 # 定义监听端口
  }
}
output {  # 定义日志输出
  stdout {
codec => rubydebug  # 将日志输出到当前的终端上显示
  }
}
备注：安装完之后，先不要启动服务，先配置logstash收集syslog日志（只做测试，后期这个文件会改输出）。# 检测配置文件是否有错，为ok则代表配置文件没有问题。
# 命令说明：
--path.settings 用于指定logstash的配置文件所在的目录
-f 指定需要被检测的配置文件的路径
--config.test_and_exit 指定检测完之后就退出，不然就会直接启动了
[root@iZj6c7bp1a289r04izstw4Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog-01.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-02T23:54:57,560][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-02T23:55:06,025][INFO ][logstash.runner       ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
#指定刚才的配置文件，启动logstash：
[root@iZj6c7bp1a289r04izstw4Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog-01.conf
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
#这时终端会停留在这里，因为我们在配置文件中定义的是将信息输出到当前终端
#打开新终端检查一下10514端口是否已被监听（logstash配置文件定义的监听端口）：
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 10514
tcp6    0    0 :::10514             :::*                   LISTEN    4312/java
#远程登录到服务器，让系统产生日志，观察上述命令输出：
可以看到终端中以JSON的格式打印了收集到的日志，测试成功。
[root@iZj6c7bp1a289r04izstw4Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog-01.conf
{
      "@timestamp" => 2019-03-02T16:14:30.000Z,
      "@version" => "1",
            "host" => "172.31.116.243",
      "severity" => 6,
      "facility" => 3,
"facility_label" => "system",
"severity_label" => "Informational",
         "message" => "Starting logstash...",
      "timestamp" => "Mar  3 00:14:30",
            "type" => "system-syslog",
      "priority" => 30,
      "logsource" => "iZj6c7bp1a289r04izstw4Z",
         "program" => "systemd"
}　　5、测试logstash场景2（系统日志作为输入，输出到es服务器）
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/conf.d/syslog-02.conf
input {
  syslog {
type => "system-syslog-to-es"
port => 10514
  }
}
output {
  elasticsearch {
hosts => ["localhost:9200"]  # 定义es服务器的ip
index => "system-syslog-toes-%{+YYYY.MM}" # 定义索引，传递给es，在kibana展示
  }
}# 检测配置文件是否有错，为ok则代表配置文件没有问题。
# 命令说明：
--path.settings 用于指定logstash的配置文件所在的目录
-f 指定需要被检测的配置文件的路径
--config.test_and_exit 指定检测完之后就退出，不然就会直接启动了
[root@iZj6c7bp1a289r04izstw4Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog-02.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-02T23:54:57,560][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-02T23:55:06,025][INFO ][logstash.runner       ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl start logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl enable logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# ps aux |grep logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 10514
#我这里启动logstash后，进程是正常存在的，但是9600以及10514端口却没有被监听。于是查看logstash的日志看看有没有错误信息的输出，但是发现没有记录日志信息，那就只能转而去查看messages的日志，发现错误信息如下报错1：
[root@iZj6c7bp1a289r04izstw4Z ~]# cat /var/log/messages | grep "ERROR"
Mar  3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,731 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
Mar  3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,755 main ERROR RollingFileManager (/var/log/logstash/logstash-slowlog-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-slowlog-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-slowlog-plain.log (Permission denied)
Mar  3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,778 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@43fa1646] unable to create manager for [/var/log/logstash/logstash-slowlog-plain.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@53460794[pattern=/var/log/logstash/logstash-slowlog-plain-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)]), strategy=DefaultRolloverStrategy(min=1, max=7, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c] %.10000m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@43fa1646] unable to create manager for [/var/log/logstash/logstash-slowlog-plain.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@53460794[pattern=/var/log/logstash/logstash-slowlog-plain-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)]), strategy=DefaultRolloverStrategy(min=1, max=7, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c] %.10000m%n, filePermissions=null, fileOwner=null]]
Mar  3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,801 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
Mar  3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,825 main ERROR RollingFileManager (/var/log/logstash/logstash-slowlog-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-slowlog-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-slowlog-plain.log (Permission denied)
报错2：
Mar  3 00:34:36 iZj6c7bp1a289r04izstw4Z logstash: [2019-03-03T00:34:36,440][FATAL][logstash.runner       ] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:447:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:229:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in `block in validate_all'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:139:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:278:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `'"]}
Mar  3 00:34:36 iZj6c7bp1a289r04izstw4Z logstash: [2019-03-03T00:34:36,462][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Mar  3 00:34:36 iZj6c7bp1a289r04izstw4Z systemd: logstash.service: main process exited, code=exited, status=1/FAILURE
解决办法：
[root@iZj6c7bp1a289r04izstw4Z ~]# ll /var/log/logstash/
total 8
-rw-r--r-- 1 root    root 1756 Mar  3 00:19 logstash-plain-2019-03-02-1.log.gz
-rw-r--r-- 1 root root 1706 Mar  3 00:31 logstash-plain.log
-rw-r--r-- 1 root root 0 Mar  2 23:36 logstash-slowlog-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# chown logstash /var/log/logstash/logstash-slowlog-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# chown logstash /var/log/logstash/logstash-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# ll /var/log/logstash/
total 8
-rw-r--r-- 1 root    root 1756 Mar  3 00:19 logstash-plain-2019-03-02-1.log.gz
-rw-r--r-- 1 logstash root 1706 Mar  3 00:31 logstash-plain.log
-rw-r--r-- 1 logstash root 0 Mar  2 23:36 logstash-slowlog-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# ll /var/lib/logstash/
total 16
drwxr-xr-x 2 root root 4096 Mar  2 23:36 dead_letter_queue
drwxr-xr-x 3 root root 4096 Mar  2 23:41 plugins
drwxr-xr-x 2 root root 4096 Mar  2 23:36 queue
-rw-r--r-- 1 root root 36 Mar  2 23:40 uuid
[root@iZj6c7bp1a289r04izstw4Z ~]# chown -R logstash /var/lib/logstash/
[root@iZj6c7bp1a289r04izstw4Z ~]# ll /var/lib/logstash/
total 16
drwxr-xr-x 2 logstash root 4096 Mar  2 23:36 dead_letter_queue
drwxr-xr-x 3 logstash root 4096 Mar  2 23:41 plugins
drwxr-xr-x 2 logstash root 4096 Mar  2 23:36 queue
-rw-r--r-- 1 logstash root 36 Mar  2 23:40 uuid
端口监听正常：
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl restart logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 10514
tcp       0    0 0.0.0.0:10514          0.0.0.0:*             LISTEN    5945/java
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 9600
tcp       0    0 127.0.0.1:9600       0.0.0.0:*             LISTEN    5945/java
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/logstash.yml
http.host: "172.31.116.244"
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl restart logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 9600
tcp       0    0 172.31.116.244:9600    0.0.0.0:*             LISTEN    6132/java
#yellow状态解决（不建议按照如下方法解决，建议esasticsearch做集群）
[root@iZj6c7bp1a289r04izstw4Z ~]# curl 'localhost:9200/_cat/indices?v'
health status index                   uuid                pri rep docs.count docs.deleted store.size pri.store.size
green  open .kibana_1                iWQ-ElLnRXaq0mJtsCXe-Q 1 0       2          0    8.6kb       8.6kb
yellow open system-syslog-toes-2019.03 ZBfgIsFQRnuXIDU1PKYkkg 5 1       451          0 202.5kb       202.5kb
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl 'http://localhost:9200/_cat/health?v'
epoch    timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1551578321 01:58:41  ELK    yellow       1       1    6 6 0 0       5          0                -                54.5%
#正常情况下，集群（（单机版称为空集群））得健康状态分为三种：
green：最健康得状态，说明所有的分片包括备份都可用
yellow：基本的分片可用，但是备份不可用（或者是没有备份）
red：部分的分片可用，表明分片有一部分损坏。此时执行查询部分数据仍然可以查到，遇到这种情况，还是赶快解决比较好
从上边查看的集群状态中还包括一些其他信息：集群名字时ELK，集群的节点个数为1，分片数量为5，活动分片百分比为 54.5%
#索引的健康状态也为yellow，索引的健康状态也有三种，yellow、green、red与集群的健康状态解释是一样的。
#单节点部署elasticsearch，而默认的分片副本数目配置为1，而相同的分片不能在一个节点上，所以就存在副本分片指定不明确的问题，
所以显示为yellow，我们可以通过在elasticsearch集群上添加一个节点来解决问题，如果你不想这么做，
你可以删除那些指定不明确的副本分片（当然这不是一个好办法）但是作为测试和解决办法还是可以尝试的，下面我们试一下删除副本分片的办法。
[root@iZj6c7bp1a289r04izstw4Z ~]# curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 6,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 54.54545454545454
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/_settings' -d '
> {
>    "index" : {
>       "number_of_replicas" : 0
>    }
> }'
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 6,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl 'localhost:9200/_cat/indices?v'
health status index                uuid                pri rep docs.count docs.deleted store.size pri.store.size
green  open system-syslog-toes-2019.03 NMhafFEzT9-w6RWFCP0flw 5 0    13251          0    2.4mb       2.4mb
green  open .kibana_1          iWQ-ElLnRXaq0mJtsCXe-Q 1 0       2          0    8.6kb       8.6kb
如上，可以看到，在logstash配置文件中定义的system-syslog索引成功获取到了，证明配置没问题，logstash与es通信正常。
获取指定索引详细信息，有数据表示正常：
[root@iZj6c7bp1a289r04izstw4Z ~]# curl -XGET 'localhost:9200/system-syslog-toes-2019.03?pretty'
如果日后需要删除索引的话，使用以下命令可以删除指定索引：
[root@iZj6c7bp1a289r04izstw4Z ~]# curl -XDELETE 'localhost:9200/.kibana_1'　　6、测试logstash场景3（nginx日志文件作为输入，经过过滤后，输出到es服务器）
#安装nginx
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# yum install -y nginx
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start nginx
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl http://127.0.0.1
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# tail -f /var/log/nginx/access.log#nginx日志有main和log_json两种，默认为main普通文本格式
ELK存储为json格式，文本格式华，拆分出如ip地址、访问agent等，便于后续使用
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/nginx/nginx.conf
log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';
access_log  /var/log/nginx/access.log  main;
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# tail -f /var/log/nginx/access.log
127.0.0.1 - - [03/Mar/2019:10:22:10 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"#logstash配置
#创建nginx正则表达式，引用grok正则
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/logstash/conf.d/nginx-access
#nginx-access
WZ ([^ ]*)
NGINXACCESS %{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}
备注：grok使用的正则表达式在grok-patterns文件
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# find / -name grok-patterns
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# cat /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/logstash/conf.d/nginx.conf
input {
  file {  # 指定一个文件作为输入源
path => "/var/log/nginx/access.log"  # 指定文件的路径
start_position => "beginning"  # 指定何时开始收集
type => "nginx"  # 定义日志类型，可自定义
  }
}
# #筛选
filter {
# 如果是nginx访问日志
  if ( [type] == "nginx" ) {
#按字段切割
grok {
   patterns_dir=>"/etc/logstash/conf.d/nginx-access"
   match => { "message" => "%{NGINXACCESS}" }
   }
# 时间格式转换
date {
   match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
   }
# 删除不需要的字段
mutate {
   remove_field => [ "offset", "@version", "beat", "input_type", "tags","id"]
   }
}
}
output {
#标准输出，调试使用#
  stdout {
codec => rubydebug { }
  }
# 输出到es#
  elasticsearch {
hosts => ["localhost:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
  }
}[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-03T10:27:56,895][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-03T10:28:04,969][INFO ][logstash.runner       ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl stop logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf
然后访问nginx网站，如下表示成功。
{
   "xforward" => "\"-\"",
      "message" => "127.0.0.1 - - [03/Mar/2019:10:50:55 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.29.0\" \"-\"",
         "host" => "iZj6c23v8zk8gxxvjdylt8Z",
         "type" => "nginx",
      "agent" => "\"curl/7.29.0\"",
   "timestamp" => "03/Mar/2019:10:50:55 +0800",
"httpversion" => "1.1",
   "remote_ip" => "127.0.0.1",
      "referer" => "\"-\"",
         "path" => "/var/log/nginx/access.log",
      "method" => "GET",
      "request" => "/",
      "status" => "200",
      "bytes" => "612",
   "@timestamp" => 2019-03-03T02:50:55.000Z
}
{
   "xforward" => "\"-\"",
      "message" => "127.0.0.1 - - [03/Mar/2019:10:22:10 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.29.0\" \"-\"",
         "host" => "iZj6c23v8zk8gxxvjdylt8Z",
         "type" => "nginx",
      "agent" => "\"curl/7.29.0\"",
   "timestamp" => "03/Mar/2019:10:22:10 +0800",
"httpversion" => "1.1",
   "remote_ip" => "127.0.0.1",
      "referer" => "\"-\"",
         "path" => "/var/log/nginx/access.log",
      "method" => "GET",
      "request" => "/",
      "status" => "200",
      "bytes" => "612",
   "@timestamp" => 2019-03-03T02:22:10.000Z
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl 'localhost:9200/_cat/indices?v'
health status index                   uuid                pri rep docs.count docs.deleted store.size pri.store.size
green  open .kibana_1                iWQ-ElLnRXaq0mJtsCXe-Q 1 0       2          0    8.6kb       8.6kb
yellow open system-syslog-toes-2019.03 ZBfgIsFQRnuXIDU1PKYkkg 5 1    68178          0    5.6mb       5.6mb
yellow open nginx-2019.03.03          lQdtD7pqRFyerEWHfMapvQ 5 1       2          0    24.3kb       24.3kb　　7、测试logstash场景4（filebeat收集nginx日志并作为输入，经过过滤后，输出到es服务器）
#filebeat收集nginx日志，然后传递到logstash，logstash过滤后输出到es服务器。filebeat比logstash直接收取要占用资源少。
#安装nginx
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# yum install -y nginx
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start nginx
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl http://127.0.0.1
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# tail -f /var/log/nginx/access.log#nginx日志有main和log_json两种，默认为main普通文本格式
ELK存储为json格式，文本格式华，拆分出如ip地址、访问agent等，便于后续使用
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/nginx/nginx.conf
log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';
access_log  /var/log/nginx/access.log  main;
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# tail -f /var/log/nginx/access.log
127.0.0.1 - - [03/Mar/2019:10:22:10 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"filebeat配置：
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/filebeat/filebeat.yml
- input_type: log
  enable: yes
  #tags: nginx-access
  paths:
- /var/log/nginx/access.log
  exclude_lines: ["^$"]
#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]
output.logstash:
  hosts: ["localhost:5044"]
  #hosts: ["172.16.50.32:5044"]
  #index: filebeat
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl enable  filebeat
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl restart filebeat
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl status  filebeat#logstash配置
#创建nginx正则表达式，引用grok正则
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/logstash/conf.d/nginx-access
#nginx-access
WZ ([^ ]*)
NGINXACCESS %{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}
备注：grok使用的正则表达式在grok-patterns文件
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# find / -name grok-patterns
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# cat /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/logstash/conf.d/nginx-filebeat.conf
input {
  #filebeat客户端#
  beats {
   port => 5044
  }
}
# #筛选
filter {
# 如果是nginx访问日志
  if ( [type] == "nginx-access-test" ) {
#按字段切割
grok {
   patterns_dir=>"/etc/logstash/conf.d/nginx-access"
   match => { "message" => "%{NGINXACCESS}" }
   }
# 时间格式转换
date {
   match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
   }
# 删除不需要的字段
mutate {
   remove_field => [ "offset", "@version", "beat", "input_type", "tags","id"]
   }
}
}
output {
#标准输出，调试使用#
  stdout {
codec => rubydebug { }
  }
# 输出到es#
  elasticsearch {
hosts => ["localhost:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
  }
}[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx-filebeat.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-03T10:27:56,895][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-03T10:28:04,969][INFO ][logstash.runner       ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl stop logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx-filebeat.conf
然后访问nginx网站，如下表示成功。
{
      "source" => "/var/log/nginx/access.log",
   "remote_ip" => "127.0.0.1",
"httpversion" => "1.1",
      "bytes" => "612",
         "meta" => {
      "cloud" => {
               "instance_id" => "i-j6c23v8zk8gxxvjdylt8",
         "availability_zone" => "cn-hongkong-b",
                  "provider" => "ecs",
                     "region" => "cn-hongkong"
      }
},
      "method" => "GET",
   "xforward" => "\"-\"",
      "agent" => "\"curl/7.29.0\"",
      "referer" => "\"-\"",
   "timestamp" => "03/Mar/2019:11:11:03 +0800",
      "message" => "127.0.0.1 - - [03/Mar/2019:11:11:03 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.29.0\" \"-\"",
   "prospector" => {
      "type" => "log"
},
         "type" => "nginx-access-test",
      "status" => "200",
      "input" => {
      "type" => "log"
},
         "host" => {
               "os" => {
         "platform" => "centos",
         "version" => "7 (Core)",
            "family" => "redhat",
         "codename" => "Core",
            "name" => "CentOS Linux"
      },
               "id" => "20181129113200424400422638950048",
               "name" => "iZj6c23v8zk8gxxvjdylt8Z",
      "architecture" => "x86_64",
      "containerized" => true
},
      "request" => "/",
   "@timestamp" => 2019-03-03T03:11:03.000Z,
         "log" => {
      "file" => {
         "path" => "/var/log/nginx/access.log"
      }
}
}[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl 'localhost:9200/_cat/indices?v'
health status index                      uuid                pri rep docs.count docs.deleted store.size pri.store.size
green  open .kibana_1                   iWQ-ElLnRXaq0mJtsCXe-Q 1 0       2          0    8.6kb       8.6kb
yellow open system-syslog-toes-2019.03 ZBfgIsFQRnuXIDU1PKYkkg 5 1    68178          0    5.6mb       5.6mb
yellow open nginx-access-test-2019.03.03 YCOqQhpjSMKkqHmvDzxfOA 5 1       8          0    87.4kb       87.4kb
yellow open nginx-2019.03.03          lQdtD7pqRFyerEWHfMapvQ 5 1       2          0    24.6kb       24.6kb
备注：上述logstash配置里面的调试输出在调试完成后需要关掉，logstash可以根据input、filter、output和优先级进行设计子文件。　　8、配置kibana

http://en-resource://database/5710:0 https://s1.运维网.com/images/20190330/1553947209359738.png

https://s1.运维网.com/images/20190330/1553947218891729.png

https://s1.运维网.com/images/20190330/1553947228316770.png

https://s1.运维网.com/images/20190330/1553947247124516.png

　　9、收集腾讯云CDN日志，安装grafana5.3可视化日志
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.3.2-1.x86_64.rpm
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# yum install fontconfig freetype* urw-fonts -y
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl enable grafana-server
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start grafana-server
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl status grafana-server
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# grafana-cli plugins install grafana-piechart-panel
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# grafana-cli plugins install grafana-clock-panel
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# grafana-cli plugins uninstall blackmirror1-singlestat-math-panel
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# service grafana-server restart
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# rpm -qc grafana
/etc/init.d/grafana-server
/etc/sysconfig/grafana-server
/usr/lib/systemd/system/grafana-server.service
备注：这里介绍安装其它插件，本教程用不到。
#列出可安装查看
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# grafana-cli plugins list-remote
#安装完成重启服务
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# service grafana-server restart
#默认插件安装目录
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# ll /var/lib/grafana/plugins/添加如下两行：
[root@iZj6cdt6ht4gae9swlcd87Z cdn]# vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: "*"
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# mkdir -p /data/logs/cdn/
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
- /data/logs/cdn/*.com
  exclude_lines: ["^$"]
  fields:
type: "cdn-log"
  fields_under_root: true
#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl enable  filebeat
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl restart filebeat
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl status  filebeat
#filter中alter插件logstash6版本默认没安装，如果安装请按照如下方法：
#我们需要用到alter插件。
参考链接：https://www.elastic.co/guide/en/logstash/current/plugins-filters-alter.html
[root@iZj6c23v8zk8gxxvjdylt8Z cdn]# cd /usr/share/logstash/bin/
[root@iZj6c23v8zk8gxxvjdylt8Z bin]# ./logstash-plugin install logstash-filter-alter
Validating logstash-filter-alter
Installing logstash-filter-alter
Installation successful[root@iZj6c23v8zk8gxxvjdylt8Z Kibana_Hanization]# vim /etc/logstash/conf.d/cdn.conf
input {
beats {
      port => 5044
}
}
filter {
if [type] == "cdn-log" {
      grok {
         patterns_dir => ["./patterns"]
         match => { "message" => "%{DATESTAMP_EVENTLOG:timestamp} %{IPORHOST:client_ip} %{IPORHOST:server_name} %{NOTSPACE:request} %{NUMBER:bytes} %{NUMBER:province} %{NUMBER:operator} %{NUMBER:status} (?:%{URI:referrer}|%{WORD:referrer}) %{NUMBER:request_time} %{QS:agent} \"$%{WORD:range}$\" %{WORD:method} (?:HTTP/%{NUMBER:protocol}|HTTPS) %{WORD:cache}" }
      }
      date {
         match => [ "timestamp", "yyyyMMddHHmmss"]
         target => "@timestamp"
      }
      alter {
         condrewrite => [
            "province", "22", "北京",
            "province", "86", "内蒙古",
            "province", "146", "山西",
            "province", "1069", "河北",
            "province", "1177", "天津",
            "province", "119", "宁夏",
            "province", "152", "陕西",
            "province", "1208", "甘肃",
            "province", "1467", "青海",
            "province", "1468", "新疆",
            "province", "145", "黑龙江",
            "province", "1445", "吉林",
            "province", "1464", "辽宁",
            "province", "2", "福建",
            "province", "120", "江苏",
            "province", "121", "安徽",
            "province", "122", "山东",
            "province", "1050", "上海",
            "province", "1442", "浙江",
            "province", "182", "河南",
            "province", "1135", "湖北",
            "province", "1465", "江西",
            "province", "1466", "湖南",
            "province", "118", "贵州",
            "province", "153", "云南",
            "province", "1051", "重庆",
            "province", "1068", "四川",
            "province", "1155", "西藏",
            "province", "4", "广东",
            "province", "173", "广西",
            "province", "1441", "海南",
            "province", "0", "其他",
            "province", "1", "港澳台",
            "province", "-1", "境外",
            "operator", "2", "中国电信",
            "operator", "26", "中国联通",
            "operator", "38", "教育网",
            "operator", "43", "长城宽带",
            "operator", "1046", "中国移动",
            "operator", "3947", "中国铁通",
            "operator", "-1", "海外运营商",
            "operator", "0", "其他运营商"
         ]
      }
}
}
output {
stdout {
      codec => rubydebug { }
}
elasticsearch {
   hosts => ["localhost:9200"]
   index => "%{[type]}-%{+YYYY.MM.dd}"
}
}[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/cdn.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-03T10:27:56,895][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-03T10:28:04,969][INFO ][logstash.runner       ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# chown logstash /var/log/logstash/logstash-slowlog-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# chown logstash /var/log/logstash/logstash-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# chown -R logstash /var/lib/logstash/
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/logstash.yml
http.host: "172.31.116.244"
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl restart logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl enable logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 5044
tcp       0    0 0.0.0.0:10514          0.0.0.0:*             LISTEN    5945/java
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 9600
tcp       0    0 172.31.116.244:9600    0.0.0.0:*             LISTEN    6132/java
[root@iZj6cdt6ht4gae9swlcd87Z ~]# curl 'localhost:9200/_cat/indices?v'
health status index             uuid                pri rep docs.count docs.deleted store.size pri.store.size
yellow open cdn-log-2019.03.02 VGN_ljUjRIyerCZu8HHilA 5 1       584          0    1.4mb       1.4mb
yellow open cdn-log-2019.03.03 BokIxrVKQpOsDnlgilS8XQ 5 1    1052          0    1.4mb       1.4mb
green  open .kibana_1       TiwdgoXhSsK6mcUZEJVmMw 1 0       2          0    8.6kb       8.6kb
#新日志在目录后，index这里也更新了，而且docs.count数据和日志行数是对的。
[root@iZj6cdt6ht4gae9swlcd87Z cdn]# curl 'localhost:9200/_cat/indices?v'
health status index             uuid                pri rep docs.count docs.deleted store.size pri.store.size
yellow open cdn-log-2019.03.02 VGN_ljUjRIyerCZu8HHilA 5 1    1720          0       2mb          2mb
green  open .kibana_1       TiwdgoXhSsK6mcUZEJVmMw 1 0       2          0    8.6kb       8.6kb
yellow open cdn-log-2019.03.03 BokIxrVKQpOsDnlgilS8XQ 5 1    8534          0    6.4mb       6.4mb访问grafana：

https://s1.运维网.com/images/20190330/1553947418968661.png

https://s1.运维网.com/images/20190330/1553947428820878.png

https://s1.运维网.com/images/20190330/1553947435503260.png

https://s1.运维网.com/images/20190330/1553947446278728.png

https://s1.运维网.com/images/20190330/1553947496988054.png

　　备注：注意时间要选择正确，要不没数据显示。