HAproxy增加日志记录功能和自定义日志输出内容、格式

8700 · 发表于 2016-9-21 08:50:28

一.增加haproxy日志记录功能

1.1 由于数据分析的需要，我们必须打开haproxy日志，记录相关信息。
  在配置前，我们先来了解一下日志的level:local0～local7 16～23保留为本地使用
  emerg 0 系统不可用
   alert 1 必须马上采取行动的事件
   crit 2 关键的事件
   err 3 错误事件
   warning 4 警告事件
   notice 5 普通但重要的事件
   info 6 有用的信息
   debug 7 调试信息

1.2 实际配置：
########################################

vi haproxy.conf(在default处添加如下信息)
defaults
log global
option httplog
log 127.0.0.1 local3
########################################

#######################################

vi /etc/rsyslog.conf(添加如下内容)
local3.* /data/logs/haproxy.log
#######################################

#######################################

vi /etc/sysconfig/rsyslog

把SYSLOGD_OPTIONS="-m 0"
改成 SYSLOGD_OPTIONS="-r -m 0 -c 2"
#######################################

相关解释说明:
-r:打开接受外来日志消息的功能,其监控514 UDP端口;
-x:关闭自动解析对方日志服务器的FQDN信息,这能避免DNS不完整所带来的麻烦;
-m:修改syslog的内部mark消息写入间隔时间(0为关闭),例如240为每隔240分钟写入一次"--MARK--"信息;
-h:默认情况下,syslog不会发送从远端接受过来的消息到其他主机,而使用该选项,则把该开关打开,所有
接受到的信息都可根据syslog.conf中定义的@主机转发过去

配置完毕后重启rsyslogd和haproxy即可.
日志内容如下：

1
2
3
4
5

Sep 18 15:48:11 localhost haproxy[12124]: Proxy html-server started.
Sep 18 15:48:11 localhost haproxy[12124]: Proxy html-server started.
Sep 18 15:48:29 localhost haproxy[12125]: 10.69.112.139:59244 [18/Sep/2016:15:48:29.027] http-in html-server/html-A 0/0/0/0/0 200 254 - - --NI 0/0/0/0/0 0/0 "GET / HTTP/1.1"
Sep 18 15:48:29 localhost haproxy[12125]: 10.69.112.139:59244 [18/Sep/2016:15:48:29.027] http-in html-server/html-A 0/0/0/0/0 200 254 - - --NI 0/0/0/0/0 0/0 "GET / HTTP/1.1"
Sep 18 15:48:29 localhost haproxy[12125]: 10.69.112.139:59244 [18/Sep/2016:15:48:29.027] http-in html-server/html-A 0/0/0/0/0 200 254 - - --NI 0/0/0/0/0 0/0 "GET / HTTP/1.1"

但日志可读性很差，我们需要显示更多的信息，但默认的都捕获不到，只能自定义需要显示的内容了。

二.定义haproxy日志输出内容和格式

2.1自定义haproxy日志。直接上配置文件。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

global
log 127.0.0.1 local3 info
maxconn 4096
chroot /gomeo2o/haproxy
uid haproxy
gid haproxy
daemon
nbproc 1
description haproxy
pidfile /var/run/haproxy.pid
defaults
log global
# option httplog
balance roundrobin
mode http
maxconn 4096
retries 3
option redispatch
option forceclose
option dontlognull
option abortonclose
stats uri /haproxy
frontend http-in
bind 0.0.0.0:80
mode http
log global
option httplog
option httpclose
acl html url_reg -i \.html$
use_backend html-server if html
default_backend html-server

#((
capture request header Host len 64
capture request header User-Agent len 128
capture request header X-Forwarded-For len 100
capture request header Referer len 200
capture response header Server len 40
capture response header Server-ID len 40
#capture捕获信息
log-format %ci:%cp\ %si:%sp\ %B\ %U\ %ST\ %r\ %b\ %f\ %bi\ %hrl\ %hsl\
      #log-forcat定义日志需显示内容(变量)
      #利用capture捕获信息，log-forcat定义变量
      #详解请见2.1和2.2
#))

backend html-server
mode http
balance roundrobin
option httpchk GET /index.html
cookie SERVERID insert indirect nocache
server html-A 10.69.112.150:80 weight 1 cookie 3 check inter 2000 rise 2 fall 5
server html-B 10.69.112.148:80 weight 1 cookie 4 check inter 2000 rise 2 fall 5

其他参数在此不做过多解释，主要查看两个中括号之间的内容。

2.2 capture捕获

haproxy利用capture捕获请求(request)和响应(response)信息，具体需捕获内容可以做如下操作：
用浏览器访问一个网站(我用的google),比如www.gomeplus.com,按F12，内容如下(注意我红色框起来的部分)：
QQ截图20160921085016.png

找到response headers项，这里就是你可以捕获的信息。比如捕获server和server-id：

1 2	capture response header Server len 40 capture response header Server-ID len 40

备注：2.1配置文件当中的capture基本的信息都能捕获到了，如有特别需求，请查阅官网(capture到底能捕获多少信息，多详细，我也在查询当中)。

2.3 定义log-format变量
定义示例：

1 2	log-format %T\ %t\ Some\ Text log-format %{+Q}o\ %t\ %s\ %{-Q}r

请参阅下面的表为定义的变量(转自官方文档)：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

+---+------+-----------------------------------------------+-------------+
  | R | var  | field name (8.2.2 and 8.2.3 for description)  | type       |
  +---+------+-----------------------------------------------+-------------+
  | | %o | special variable, apply flags on all next var |          |
  +---+------+-----------------------------------------------+-------------+
  | | %B | bytes_read          (from server to client)  | numeric    |
  | H | %CC  | captured_request_cookie                      | string    |
  | H | %CS  | captured_response_cookie                   | string    |
  | | %H | hostname                                     | string    |
  | H | %HM  | HTTP method (ex: POST)                      | string    |
  | H | %HP  | HTTP request URI without query string (path)  | string    |
  | H | %HQ  | HTTP request URI query string (ex: ?bar=baz)  | string    |
  | H | %HU  | HTTP request URI (ex: /foo?bar=baz)          | string    |
  | H | %HV  | HTTP version (ex: HTTP/1.0)                | string    |
  | | %ID  | unique-id                                  | string    |
  | | %ST  | status_code                                  | numeric    |
  | | %T | gmt_date_time                               | date       |
  | | %Tc  | Tc                                           | numeric    |
  | | %Td  | Td = Tt - (Tq + Tw + Tc + Tr)                | numeric    |
  | | %Tl  | local_date_time                            | date       |
  | H | %Tq  | Tq                                           | numeric    |
  | H | %Tr  | Tr                                           | numeric    |
  | | %Ts  | timestamp                                  | numeric    |
  | | %Tt  | Tt                                           | numeric    |
  | | %Tw  | Tw                                           | numeric    |
  | | %U | bytes_uploaded    (from client to server)  | numeric    |
  | | %ac  | actconn                                     | numeric    |
  | | %b | backend_name                               | string    |
  | | %bc  | beconn    (backend concurrent connections)  | numeric    |
  | | %bi  | backend_source_ip    (connecting address)  | IP       |
  | | %bp  | backend_source_port    (connecting address)  | numeric    |
  | | %bq  | backend_queue                               | numeric    |
  | | %ci  | client_ip                (accepted address)  | IP       |
  | | %cp  | client_port             (accepted address)  | numeric    |
  | | %f | frontend_name                               | string    |
  | | %fc  | feconn    (frontend concurrent connections)  | numeric    |
  | | %fi  | frontend_ip             (accepting address)  | IP       |
  | | %fp  | frontend_port          (accepting address)  | numeric    |
  | | %ft  | frontend_name_transport ('~' suffix for SSL)  | string    |
  | | %lc  | frontend_log_counter                         | numeric    |
  | | %hr  | captured_request_headers default style       | string    |
  | | %hrl | captured_request_headers CLF style          | string list |
  | | %hs  | captured_response_headers default style    | string    |
  | | %hsl | captured_response_headers CLF style          | string list |
  | | %ms  | accept date milliseconds (left-padded with 0) | numeric    |
  | | %pid | PID                                        | numeric    |
  | H | %r | http_request                               | string    |
  | | %rc  | retries                                     | numeric    |
  | | %rt  | request_counter (HTTP req or TCP session)    | numeric    |
  | | %s | server_name                                  | string    |
  | | %sc  | srv_conn    (server concurrent connections)  | numeric    |
  | | %si  | server_IP                (target address)  | IP       |
  | | %sp  | server_port                (target address)  | numeric    |
  | | %sq  | srv_queue                                  | numeric    |
  | S | %sslc| ssl_ciphers (ex: AES-SHA)                   | string    |
  | S | %sslv| ssl_version (ex: TLSv1)                      | string    |
  | | %t | date_time    (with millisecond resolution)  | date       |
  | | %ts  | termination_state                            | string    |
  | H | %tsc | termination_state with cookie status       | string    |
  +---+------+-----------------------------------------------+-------------+

最后查看定义之后的日志内容输出：

1
2
3
4
5

Sep 20 17:06:12 localhost haproxy[16372]: 10.69.112.139:39957 10.69.112.148:80 2614 179 200 GET / HTTP/1.1 html-server http-in 10.69.112.151 www.qiang999.com curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - - nginx/1.9.8 -\
Sep 20 17:06:12 localhost haproxy[16372]: 10.69.112.139:39957 10.69.112.148:80 2614 179 200 GET / HTTP/1.1 html-server http-in 10.69.112.151 www.qiang999.com curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - - nginx/1.9.8 -\
Sep 20 17:06:12 localhost haproxy[16372]: 10.69.112.139:39966 10.69.112.150:80 254 179 200 GET / HTTP/1.1 html-server http-in 10.69.112.151 www.qiang999.com curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - - nginx/1.9.8 -\
Sep 20 17:06:12 localhost haproxy[16372]: 10.69.112.139:39966 10.69.112.150:80 254 179 200 GET / HTTP/1.1 html-server http-in 10.69.112.151 www.qiang999.com curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - - nginx/1.9.8 -\
Sep 20 17:06:13 localhost haproxy[16372]: 10.69.112.139:39974 10.69.112.148:80 2614 179 200 GET / HTTP/1.1 html-server http-in 10.69.112.151 www.qiang999.com curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - - nginx/1.9.8 -\

账号		自动登录	找回密码
密码			立即注册

VMware vcenter+vSphere 6.5 U2共享

【跟谁学】韩宇极简英语课-技术人员不得不

用Zabbix通过JMX方式监控weblogic

winhex数据恢复教程（非常巨大，内容丰富）

Symantec Backup Exec 2015 2016/2012 BE20

NetScaler VPX部署之：NetScaler Gateway调

zabbix3.4.1安装部署+微信推送信息+大屏显

[经验分享] HAproxy增加日志记录功能和自定义日志输出内容、格式

相关帖子

扫码加入运维网微信交流群