使用logstash分析Apache日志

Cnbaby

转自：http://blog.sina.com.cn/s/blog_a84e73f70101ck5r.html

网站刚刚上线，就发现一些扫描信息，因此，有必要搭一个日志分析系统，对web访问日 志进行分析，或许能够发现一些有意思的东西。logstash是一个非常灵活的日志分析引擎， 使用强大的elasticsearch作为数据存储、索引的数据库。

118.26.224.17 - - [10/Jun/2012:18:26:46 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 490 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:47 +0000] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 404 479 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:47 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 479 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:48 +0000] "GET /pma/scripts/setup.php HTTP/1.1" 404 475 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:48 +0000] "GET /myadmin/scripts/setup.php HTTP/1.1" 404 478 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:49 +0000] "GET /MyAdmin/scripts/setup.php HTTP/1.1" 404 478 "-" "ZmEu"



创建日志分析环境

#!/bin/bash
# 创建日志环境的目录
mkdir loganalysis
cd loganalysis
# 下载logstash，目前是1.1.0版本
mkdir bin
cd bin
wget http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
# 创建启动脚本
cat << EOF >>run.sh
#!/bin/bash
java -jar $LOGSTASH_HOME/bin/logstash-1.1.0-monolithic.jar agent -f $LOGSTASH_HOME/conf/apache-parse.conf -- web --backend elasticsearch:///?local
EOF
# 环境变量设置
cd ..
cat << EOF >>env.sh
#!/bin/bash
export LOGSTASH_HOME=`pwd`
EOF
# 下载Apache日志解析配置文件
mkdir conf
cd conf
wget http://logstash.net/docs/1.1.0/tutorials/10-minute-walkthrough/apache-parse.conf




修改apache-parse.conf


apache-parse.conf默认不使用elasticsearch，所以需要加上 elasticsearch
{ embedded => true }

input {
tcp {
type => &quot;apache&quot;
port => 3333
}
}
filter {
grok {
type => &quot;apache&quot;
# See the following URL for a complete list of named patterns
# logstash/grok ships with by default:
# https://github.com/logstash/logstash/tree/master/patterns
#
# The grok filter will use the below pattern and on successful match use
# any captured values as new fields in the event.
pattern => &quot;%{COMBINEDAPACHELOG}&quot;
}
date {
type => &quot;apache&quot;
# Try to pull the timestamp from the 'timestamp' field (parsed above with
# grok). The apache time format looks like: &quot;18/Aug/2011:05:44:34 -0700&quot;
timestamp => &quot;dd/MMM/yyyy:HH:mm:ss Z&quot;
}
}
output {
# Use stdout in debug mode again to see what logstash makes of the event.
stdout {
debug => true
}
elasticsearch { embedded => true }
}



安装grok


apache-parse.conf使用grok作为输入的接口，但是grok与ubuntu主流版本(只测试过 11.10)并不兼容，按照INSTALL说明的安装无法成功。幸好，我们只要libgrok.so即可。

cd /tmp
git clone https://github.com/jordansissel/grok.git
cd grok
make libgrok.so
sudo cp libgrok.so /usr/local/lib
sudo ldconfig -v



运行logstash

$ . env.sh   #导入环境变量
$ bin/run.sh


启动完成之后，将apache的访问日志导入logstash：

nc localhost 3333
最后使用浏览器打开http://localhost:9292/ 访问logstash的web界面。