ELK实战之Tomcat的json日志收集

吸毒的虫子

一、安装tomcat

1、下载tomcat

[root@linux-node2 ~]# wget http://apache.fayea.com/tomcat/tomcat-8/v8.5.20/bin/apache-tomcat-8.5.20.tar.gz
[root@linux-node2 ~]# tar -zxf apache-tomcat-8.5.24.tar.gz
[root@linux-node2 ~]# mv apache-tomcat-8.5.24 /usr/local/tomcat

2、修改tomcat日志格式

[root@linux-node2 ~]# cd /usr/local/tomcat/conf
[root@linux-node2 conf ]# cp server.xml{,.bak}
[root@linux-node2 conf ]# vim server.xml

3、增加tomcat网页

[root@linux-node2 conf ]# cd ../webapps/
[root@linux-node2 webapps ]# mkdir webdir && cd webdir
[root@linux-node2 webdir ]# echo "welcome to use tomcat" > index.html
[root@linux-node2 conf ]# ../bin/catalina.sh start
[root@linux-node2 conf ]# netstat -tulnp |grep 8080
tcp6       0      0 :::8080                 :::*                    LISTEN      2362/java           
4、压测页面，生成tomcat的访问日志

[root@linux-node2 conf ]# ab -n1000 -c100 http://192.168.56.12:8080/webdir/index.html
[root@linux-node2 ~]# tailf /usr/local/tomcat/logs/tomcat_access_log.2017-12-28.log
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:56 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}

二、配置logstash

1、配置logstash

[root@linux-node2 ~]# vim /etc/logstash/conf.d/tomcat_accesslog.conf
input {
file {
path => "/usr/local/tomcat/logs/tomcat_access_log.*.log"
type => "tomcat-accesslog"
start_position => "beginning"
stat_interval => "2"
}
}
output {
elasticsearch {
hosts => ["192.168.56.11:9200"]
index => "logstash-tomcat5612-accesslog-%{+YYYY.MM.dd}"
}
file {
path => "/tmp/logstash-tomcat5612-accesslog-%{+YYYY.MM.dd}"
}
}

2、检查logstash的配置语法并重启logstash

[root@linux-node2 tomcat]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat_accesslog.conf -t
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@linux-node2 tomcat]# systemctl restart logstash
[root@linux-node2 tomcat]# ps -ef |grep logstash
logstash  2527     1 98 09:33 ?        00:00:28 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
root      2572 15939  0 09:34 pts/3    00:00:00 grep --color=auto logstash

3、elasticsearch的head插件查看

http://i2.运维网.com/images/blog/201712/28/993e93a1831faab9d9c63bc0aa70355f.png
数据浏览：
http://i2.运维网.com/images/blog/201712/28/cd5f3e7cf22e3f9757c2f466513bf7b1.png

　　注：此处如果无法出现tomcat的数据索引，查看一下是否有权限访问日志。

[root@linux-node2 logs]# ll /usr/local/tomcat/logs/ -d
drwxr-x--- 2 root root 4096 12月 28 09:29 /usr/local/tomcat/logs/
[root@linux-node2 logs]# chmod 755 /usr/local/tomcat/logs
[root@linux-node2 logs]# ll /usr/local/tomcat/logs/
总用量 512
-rw-r----- 1 root root   7140 12月 28 09:29 catalina.2017-12-28.log
-rw-r----- 1 root root   7140 12月 28 09:29 catalina.out
-rw-r----- 1 root root      0 12月 28 09:29 host-manager.2017-12-28.log
-rw-r----- 1 root root    284 12月 28 09:29 localhost.2017-12-28.log
-rw-r----- 1 root root      0 12月 28 09:29 manager.2017-12-28.log
-rw-r----- 1 root root 502039 12月 28 09:47 tomcat_access_log.2017-12-28.log
[root@linux-node2 logs]# chmod 644 /usr/local/tomcat/logs/
4、添加到Kibana

http://i2.运维网.com/images/blog/201712/28/d451036ea0fbaedd8a781a37b402d1ae.png