升级Zabbix proxy到3.0.2后无法启动报错

1232121

一 问题描述
将一台zabbix proxy由2.4.5升级到3.0.2后启动了就直接崩溃了。
错误信息如下：

1	2367:20160508:153246.830 One child process died (PID:42385,exitcode/signal:11). Exiting ...

1	42367:20160508:153248.904 Zabbix Proxy stopped. Zabbix 3.0.2 (revision 59540).

将日志级别调高点，设置DebugLevel=4，查看报错

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

42629:20160508:153529.004 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42628:20160508:153529.004 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42634:20160508:153529.004 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42651:20160508:153529.004 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42659:20160508:153529.004 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42661:20160508:153529.005 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42655:20160508:153529.005 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42663:20160508:153529.005 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42664:20160508:153529.005 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42666:20160508:153529.005 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42665:20160508:153529.006 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42667:20160508:153529.006 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42630:20160508:153529.006 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42668:20160508:153529.006 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42671:20160508:153529.006 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42670:20160508:153529.007 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42647:20160508:153529.007 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42674:20160508:153529.007 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42680:20160508:153529.007 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ... 42679:20160508:153529.008 Got signal [signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0]. Exiting ...

1 2 3 4 5 6 7 8 9 10

9102:20160508:170455.283 End of zbx_strpool_destroy()   9102:20160508:170455.283 End of free_configuration_cache()   9102:20160508:170455.283 In free_ipmi_handler()   9102:20160508:170455.283 End of free_ipmi_handler()   9102:20160508:170455.284 In free_selfmon_collector() collector:0x7fa793e23000   9102:20160508:170455.284 End of free_selfmon_collector()   9102:20160508:170455.284 In unload_modules()   9102:20160508:170455.284 End of unload_modules()   9102:20160508:170455.284 Zabbix Proxy stopped. Zabbix 3.0.2 (revision 59540). zabbix_proxy [9152]: [file:'selfmon.c',line:375] lock failed: [22] Invalid argument

总共有4台proxy，每台proxy的操作系统版本都是CentOS6。就单独这一台zabbix proxy启动有问题，刚启动就有进程死掉。


二 问题解决
查看这台proxy服务器的内核版本是3.9.8-1.el6.elrepo.x86_64，而其他几台proxy的内核版本是2.6.32-358.el6.x86_64 刚开始也没有朝着内核版本的问题方向思考。以为还是proxy的版本或者系统参数设置有问题。
1.排查Linux下的信号量
查看Zabbix的官方BUG追踪平台Invalid argument可能是信号量达到了限制

可以尝试增大信号量的值

echo 256 40000 32 32000 > /proc/sys/kernel/sem这个四个参数分别是
SEMMSL 每个信号量集的含有的信号量的最大值

SEMMNS 系统层面的信号量数量最大限制
SEMOPM semop(2)调用可以指定的最大操作数量

SEMMNI 系统层面的信号数量最小限制


通过echo方式更改重启后会失效，可以更改/etc/sysctl.conf
kernel.sem = 250 32000 100 10000然后执行sysctl -p
更改参数后proxy启动仍然会崩溃


2.排查内核版本原因

Updating to latest kernel (3.10.0-327.10.1.el7.x86_64) + reboot solved it for me
The error reported originally may be something else than what affected me.

查看帖子怀疑是内核版本造成的，这台proxy以前有人升级过内核，更改linux启动项设置默认启动内核为2.6.32.重启后问题得到解决，proxy不再崩溃。但是之前proxy 2.4.5在3.9.8的内核上可以正常运行。
遇到这个BUG的人挺多的，有好几个帖子都是反馈这个问题。

三 补充知识
这个BUG涉及到Linux下的信号量相关的知识。信号量和共享内存有关。