GCE 部署 ELK 7.1可视化分析 nginx
一、准备我这边有一个网站放在了 Google VM 上面，所以打算在购买一台服务器安装 ELK 来处理分析 nginx 的访问日志。[*]操作系统版本：CentOS 7
[*]ELK版本：7.1
1.1、服务器环境准备我们这里还是采用官方的 yum 源进行安装，简单省事，首先配置官方 yum 仓库。
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearchcat >> /etc/yum.repos.d/elasticsearch.repo <<EOFname=Elasticsearch repository for 7.x packagesbaseurl=https://artifacts.elastic.co/packages/7.x/yumgpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-mdEOF关闭防火墙。
systemctl stop firewalld.servicesystemctl disable firewalld.service二、安装 ESElasticsearch 的软件包自带一个版本的 OpenJDK，所以我们这里不需要单独去安装 JDK 了，直接使用官方的就行。
yum install elasticsearch -y因为默认是监听 IP 是127.0.0.1，我们需要开启配置文件/etc/elasticsearch/elasticsearch.yml的一个参数network.host，写上自己的本地IP即可。
2.1、遇到小问题如此修改了一下配置文件之后，我发现无法启动了，日志也没有看到什么错误，我就手动执行查看问题。
# su - elasticsearch -s /bin/sh -c "/usr/share/elasticsearch/bin/elasticsearch"su: warning: cannot change directory to /nonexistent: No such file or directoryOpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. using data paths, mounts [[/ (rootfs)]], net usable_space , net total_space , types heap size , compressed ordinary object pointers ................................. parsed roles from file controller (64 bit): Version 7.1.1 (Build fd619a36eb77df) Copyright (c) 2019 Elasticsearch BV Using REST wrapper from plugin org.elasticsearch.xpack.security.Security using discovery type and seed hosts providers initialized starting ... publish_address {10.154.0.3:9300}, bound_addresses {10.154.0.3:9300} bound or publishing to a non-loopback address, enforcing bootstrap checksERROR: bootstrap checks failed: max file descriptors for elasticsearch process is too low, increase to at least : the default discovery settings are unsuitable for production use; at least one of must be configured stopping ... stopped closing ... closed Native controller process has stopped - no new native processes can be started我们可以看到下面的两个错误，一个是说最大文件打开数比较小，一个说还必须开启一个 discovery 设置，那我们分别操作一下。
#临时修改，立刻生效ulimit -n 655350 #永久修改echo "* soft nofile 655360" >> /etc/security/limits.confecho "* hard nofile 655360" >> /etc/security/limits.conf我修改后的配置文件如下：
node.name: node-1path.data: /var/lib/elasticsearchpath.logs: /var/log/elasticsearchnetwork.host: 10.154.0.3cluster.initial_master_nodes: ["node-1", "node-2"]然后我们就可以启动了。
systemctl daemon-reloadsystemctl enable elasticsearch.servicesystemctl start elasticsearch.service访问一下页面。
{"name" : "node-1","cluster_name" : "elasticsearch","cluster_uuid" : "P0GGZU9pRzutp-zHDZvYuQ","version" : { "number" : "7.1.1", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "7a013de", "build_date" : "2019-05-23T14:04:00.380842Z", "build_snapshot" : false, "lucene_version" : "8.0.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1"},"tagline" : "You Know, for Search"}三、安装 Kibana为了节省机器资源，我们这里使用安装在一台机器上面，因为环境上面已经配置好，我们这里直接进行安装。
yum install kibana -y编辑配置文件/etc/kibana/kibana.yml，修改如下内容：
server.port: 5601server.host: "10.154.0.3"elasticsearch.hosts: ["http://10.154.0.3:9200"]启动服务。
systemctl daemon-reloadsystemctl enable kibana.servicesystemctl start kibana.service访问画面正常显示即我们安装完成。
四、安装 Logstashyum install logstash -y创建配置文件/etc/logstash/conf.d/logstash.conf，增加如下内容：
input {beats { port => 5044}}filter {if == "nginx_access" { grok { match => { "message" => "%{NGINXACCESS}" } }}if == "apache_access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }} mutate { remove_field =>["message"] remove_field =>["host"] remove_field =>["input"] remove_field =>["prospector"] remove_field =>["beat"]}geoip { source => "clientip"}date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]}}output {if == "nginx_access" { elasticsearch { hosts => "172.18.8.200:9200" index => "%{}-%{+YYYY.MM.dd}" }}if == "apache_access" { elasticsearch { hosts => "172.18.8.200:9200" index => "%{}-%{+YYYY.MM.dd}" }}}
大哥 你QQ是多少 我最近也在搞elk+filebeat想咨询你一下
页:
[1]