配置apache日志，分别记录不同搜索引擎爬虫的访问记录
编辑/usr/local/apache/conf/extra/http_vhost.conf，在添加的虚拟主机中配置日志如下：SetEnvIfNoCase User-Agent Baiduspider baidu_robot #百度访问日志
SetEnvIfNoCase User-Agent Googlebot google_robot #谷歌访问日志
SetEnvIfNoCase User-Agent 360Spider 360__robot
SetEnvIfNoCase User-Agent Iaskspiderxinglang_robot
SetEnvIfNoCase User-Agent Sogou sogou_robot
SetEnvIfNoCase User-Agent YodaoBot wangyi_robot
CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.site_%Y%m%d.log 86400" combined env=!image-request
CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.google_%Y%m%d.log 86400" combined env=google_robot
CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.baidu_%Y%m%d.log 86400" combined env=baidu_robot
CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.360_%Y%m%d.log 86400" combined env=360_robot
CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.xinglang_%Y%m%d.log 86400" combined env=xinglang_robot
CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.sougou_%Y%m%d.log 86400" combined env=sougou_robot
CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.wangyi_%Y%m%d.log 86400" combined env=wangyi_robot
然后每天生成不同的日志来记录，实现不同的访问日志记录不同搜索引擎爬虫的访问记录。
页:
[1]