zabbix 突然间报:Zabbix agent on {HOST.NAME} is unreachable for 5 minutes
N多机器都有这样的报警;
登陆查看被监控机,一切正常,网络一切正常;
查看zabbix agent 日志,没有异常;
查看zabbix server 日志 ,大部分提示信息为:
21567:20131203:141448.893 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update ids set nextid=nextid+1 where nodeid=0 and table_name='events' and field_name='eventid']
zabbix_server [21567]: ERROR [file:db.c,line:1501] Something impossible has just happened.
update triggers set lastchange=1386049597,value=1 where triggerid=14912;
update ids set nextid=nextid+1 where nodeid=0 and table_name='events' and field_name='eventid'
delete from escalations where escalationid between 364655 and 364665
登陆MySQL查看,错误日志中的SQL都处在 lock wait 状态,磁盘IO利用率100%,
应该是IO导致的问题,可以从这条语句判断:update triggers set lastchange=1386049597,value=1 where triggerid=14912; 触发器的状态没有更新过来;