ELK实战之Tomcat的json日志收集
一、安装tomcat1、下载tomcat
# wget http://apache.fayea.com/tomcat/tomcat-8/v8.5.20/bin/apache-tomcat-8.5.20.tar.gz
# tar -zxf apache-tomcat-8.5.24.tar.gz
# mv apache-tomcat-8.5.24 /usr/local/tomcat
2、修改tomcat日志格式
# cd /usr/local/tomcat/conf
# cp server.xml{,.bak}
# vim server.xml
3、增加tomcat网页
# cd ../webapps/
# mkdir webdir && cd webdir
# echo "welcome to use tomcat" > index.html
# ../bin/catalina.sh start
# netstat -tulnp |grep 8080
tcp6 0 0 :::8080 :::* LISTEN 2362/java
4、压测页面，生成tomcat的访问日志
# ab -n1000 -c100 http://192.168.56.12:8080/webdir/index.html
# tailf /usr/local/tomcat/logs/tomcat_access_log.2017-12-28.log
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
二、配置logstash
1、配置logstash
# vim /etc/logstash/conf.d/tomcat_accesslog.conf
input {
file {
path => "/usr/local/tomcat/logs/tomcat_access_log.*.log"
type => "tomcat-accesslog"
start_position => "beginning"
stat_interval => "2"
}
}
output {
elasticsearch {
hosts => ["192.168.56.11:9200"]
index => "logstash-tomcat5612-accesslog-%{+YYYY.MM.dd}"
}
file {
path => "/tmp/logstash-tomcat5612-accesslog-%{+YYYY.MM.dd}"
}
}
2、检查logstash的配置语法并重启logstash
# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat_accesslog.conf -t
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
# systemctl restart logstash
# ps -ef |grep logstash
logstash2527 1 98 09:33 ? 00:00:28 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
root 2572 159390 09:34 pts/3 00:00:00 grep --color=auto logstash
3、elasticsearch的head插件查看
http://i2.运维网.com/images/blog/201712/28/993e93a1831faab9d9c63bc0aa70355f.png
数据浏览：
http://i2.运维网.com/images/blog/201712/28/cd5f3e7cf22e3f9757c2f466513bf7b1.png
注：此处如果无法出现tomcat的数据索引，查看一下是否有权限访问日志。
# ll /usr/local/tomcat/logs/ -d
drwxr-x--- 2 root root 4096 12月 28 09:29 /usr/local/tomcat/logs/
# chmod 755 /usr/local/tomcat/logs
# ll /usr/local/tomcat/logs/
总用量 512
-rw-r----- 1 root root 7140 12月 28 09:29 catalina.2017-12-28.log
-rw-r----- 1 root root 7140 12月 28 09:29 catalina.out
-rw-r----- 1 root root 0 12月 28 09:29 host-manager.2017-12-28.log
-rw-r----- 1 root root 284 12月 28 09:29 localhost.2017-12-28.log
-rw-r----- 1 root root 0 12月 28 09:29 manager.2017-12-28.log
-rw-r----- 1 root root 502039 12月 28 09:47 tomcat_access_log.2017-12-28.log
# chmod 644 /usr/local/tomcat/logs/
4、添加到Kibana
http://i2.运维网.com/images/blog/201712/28/d451036ea0fbaedd8a781a37b402d1ae.png
页:
[1]