用goaccess每天自动分析nginx日志

342132123

上次用awstats来自动分析nginx的日志，不过发现它只是累计的不能看一天或某个时间段内情况，于是就加了个goaccess来分析日志，
思路：每天将原来awstats集中过来的日志文件处理一下，生成符合goaccess的新的日志文件，然后用goaccess来生产静态页，nginx将这些静态页展示出来。
一、软件
    goaccess:
    ncurses(goaccess的依赖)

1	ubuntu：apt-get install libncursesw5-dev

二、安装

1 2 3 4 5 6

wget http://tar.goaccess.io/goaccess-1.1.1.tar.gz     tar -zxvf goaccess-1.1.1.tar.gz     cd goaccess-1.1.1     ./configure —enable-geoip —enable-utf8     make     make install

三、配置goaccess.conf
    默认goaccess的log-format是比较简单的，我这自定义了

1 2 3

time-format %H:%S:%M     date-format %d/%b/%Y     log-format %d:%t %^ %h %s %T  %m %U %H %b %R %u

官方给了各日志段的代码，可以参考着修改

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

SPECIFIERS %x A date and time field matching the time-format and date-format variables. This is used when a timestamp is given instead of the date and time being in two separate variables. %ttime field matching the time-format variable. %ddate field matching the date-format variable. %vThe server name according to the canonical name setting (Server Blocks or Virtual Host). %eThis is the userid of the person requesting the document as determined by HTTP authentication. %hhost (the client IP address, either IPv4 or IPv6) %rThe request line from the client. This requires specific delimiters around the request (as single quotes, double quotes, or anything else) to be parsable. If not, we have to use a combination of special format specifiers as %m %U %H. %qThe query string. %mThe request method. %UThe URL path requested. Note: If the query string is in %U, there is no need to use %q. However, if the URL path, does not include any query string, you may use %q and the query string will be appended to the request. %HThe request protocol. %sThe status code that the server sends back to the client. %bThe size of the object returned to the client. %RThe "Referer" HTTP request header. %uThe user-agent HTTP request header. %DThe time taken to serve the request, in microseconds. %TThe time taken to serve the request, in seconds with milliseconds resolution. %L The time taken to serve the request, in milliseconds as a decimal number. %^Ignore this field. %~Move forward through the log string until a non-space (!isspace) char is found. |Vertical pipe or bar is used for either the character prior to the vertical pipe or followed by the vertical pipe.

• 
  

四、编写脚本
    我这nginx得日志格式是自定义的，所有做了处理。不同线路的访问日志也合并到一起来处理的。

1 2 3 4 5 6 7 8 9 10 11 12

#!/bin/bash export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" DATE=`/bin/date -d "1 days ago" +%Y%m%d` PREFIXS='www p m news' DOMS='xxx.com' BASIC_DIR="/data/awstats" for PREFIX in $PREFIXS do cat $BASIC_DIR/logs/$PREFIX/11/access_$DATE.log $BASIC_DIR/logs/$PREFIX/13/access_$DATE.log >$BASIC_DIR/goaccess/orig_logs/$PREFIX.$DOMS/access_$DATE.log awk -F "|" '{print $2,$3,$5,$6,$12,$14,$11,$13}' $BASIC_DIR/goaccess/orig_logs/$PREFIX.$DOMS/access_$DATE.log >$BASIC_DIR/goaccess/logs/$PREFIX.$DOMS/access_$DATE.log /usr/local/bin/goaccess -f $BASIC_DIR/goaccess/logs/$PREFIX.$DOMS/access_$DATE.log -p $BASIC_DIR/goaccess/scripts/goaccess.conf >$BASIC_DIR/goaccess/html/$PREFIX.$DOMS/access_$DATE.html done

五、crontab自动处理
每天早上6点开始处理

1	0 6 * * * (bash /data/awstats/goaccess/scripts/goaccess.sh)

六、nginx展示goaccess的结果

cat goaccess.conf

1 2 3 4 5 6 7

server{         listen 88;         server_name 11.1.2.252;         root /data/awstats/goaccess/html/;         autoindex on;         access_log logs/access.ga.log;         }