python处理nginx日志，并统计分析
要求：原始日志的倒数第二个字段不为空且不为'-'的情况下，统计倒数第四个字段不为空且不为'-'的且不重复的个数。
python脚本如下：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#!/usr/bin/envpython
#encoding=utf-8
# nginx_log_analysis.py
FileHd = open('aaa.com_access.log-20160506','r')
FileText = FileHd.readlines()
FileTextTemp = []
FileTextTempSplit = []
AAA_UID = []
FileHd.close()
for i in range(len(FileText)):
FileTextTemp.append(FileText)
FileTextTempSplit.append(FileTextTemp.split(' '))
for i in range(len(FileTextTempSplit)):
for j in range(len(FileTextTempSplit)):
length = len(FileTextTempSplit)
if FileTextTempSplit != '-'\
and len(FileTextTempSplit) != 0\
and FileTextTempSplit != '-'\
and len(FileTextTempSplit) != 0:
AAA_UID.append(FileTextTempSplit)
'''
这样的aaa_uid统计是未去重的
STATS_FD = open('stats.txt','w')
for AAA_uid in AAA_UID:
STATS_FD.writelines(aaa_uid+'\n')
STATS_FD.close()
'''
'''
这是aaa_uid去重统计
'''
count = 0
STATS_FD = open('stats_uniq.txt','w')
AAA_UID_UNIQ = list(set(AAA_UID))
for aaa_uid in AAA_UID_UNIQ:
STATS_FD.writelines(aaa_uid+'\n')
count += 1
STATS_FD.close()
print count
这样处理一个不到280MB的日志，time运行脚本:
1
timenginx_log_analysis.py
需要14秒多(一台资源是2核，4GB内存的虚拟机上运行）
页:
[1]