ceph同步数据过程OSD进程异常退出记录
做的操作: ceph集群扩容了几个节点。异常现象: ceph集群同步时，总是OSD进程异常的宕掉(同步一段时间数据后)。
ceph版本:9.2.1
日志:
7月 25 09:25:57 ceph6 ceph-osd: 0> 2017-07-25 09:25:57.471502 7f46fe478700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_ch
7月 25 09:25:57 ceph6 ceph-osd: common/HeartbeatMap.cc: 81: FAILED assert(0 == "hit suicide timeout")
7月 25 09:25:57 ceph6 ceph-osd: ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
7月 25 09:25:57 ceph6 ceph-osd: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85)
7月 25 09:25:57 ceph6 ceph-osd: 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, long)+0x2d9)
7月 25 09:25:57 ceph6 ceph-osd: 3: (ceph::HeartbeatMap::is_healthy()+0xd6)
7月 25 09:25:57 ceph6 ceph-osd: 4: (ceph::HeartbeatMap::check_touch_file()+0x2c)
7月 25 09:25:57 ceph6 ceph-osd: 5: (CephContextServiceThread::entry()+0x15b)
7月 25 09:25:57 ceph6 ceph-osd: 6: (()+0x7dc5)
7月 25 09:25:57 ceph6 ceph-osd: 7: (clone()+0x6d)
7月 25 09:25:57 ceph6 ceph-osd: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this.
7月 25 09:25:57 ceph6 ceph-osd: terminate called after throwing an instance of 'ceph::FailedAssertion'
7月 25 09:25:57 ceph6 ceph-osd: *** Caught signal (Aborted) **
7月 25 09:25:57 ceph6 ceph-osd: in thread 7f46fe478700
7月 25 09:25:57 ceph6 ceph-osd: ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
7月 25 09:25:57 ceph6 ceph-osd: 1: (()+0x7e6fe2)
7月 25 09:25:57 ceph6 ceph-osd: 2: (()+0xf370)
7月 25 09:25:57 ceph6 ceph-osd: 3: (gsignal()+0x37)
7月 25 09:25:57 ceph6 ceph-osd: 4: (abort()+0x148)
7月 25 09:25:57 ceph6 ceph-osd: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165)
7月 25 09:25:57 ceph6 ceph-osd: 6: (()+0x5e946)
7月 25 09:25:57 ceph6 ceph-osd: 7: (()+0x5e973)
7月 25 09:25:57 ceph6 ceph-osd: 8: (()+0x5eb9f)
7月 25 09:25:57 ceph6 ceph-osd: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a)
7月 25 09:25:57 ceph6 ceph-osd: 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, long)+0x2d9)
7月 25 09:25:57 ceph6 ceph-osd: 11: (ceph::HeartbeatMap::is_healthy()+0xd6)
7月 25 09:25:57 ceph6 ceph-osd: 12: (ceph::HeartbeatMap::check_touch_file()+0x2c)
7月 25 09:25:57 ceph6 ceph-osd: 13: (CephContextServiceThread::entry()+0x15b)
7月 25 09:25:57 ceph6 ceph-osd: 14: (()+0x7dc5)
7月 25 09:25:57 ceph6 ceph-osd: 15: (clone()+0x6d)
7月 25 09:25:57 ceph6 ceph-osd: 2017-07-25 09:25:57.525027 7f46fe478700 -1 *** Caught signal (Aborted) **
7月 25 09:25:57 ceph6 ceph-osd: in thread 7f46fe478700
7月 25 09:25:57 ceph6 ceph-osd: ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
7月 25 09:25:57 ceph6 ceph-osd: 1: (()+0x7e6fe2)
7月 25 09:25:57 ceph6 ceph-osd: 2: (()+0xf370)
7月 25 09:25:57 ceph6 ceph-osd: 3: (gsignal()+0x37)
7月 25 09:25:57 ceph6 ceph-osd: 4: (abort()+0x148)
7月 25 09:25:57 ceph6 ceph-osd: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165)
7月 25 09:25:57 ceph6 ceph-osd: 6: (()+0x5e946)
7月 25 09:25:57 ceph6 ceph-osd: 7: (()+0x5e973)
7月 25 09:25:57 ceph6 ceph-osd: 8: (()+0x5eb9f)
7月 25 09:25:57 ceph6 ceph-osd: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a)
7月 25 09:25:57 ceph6 ceph-osd: 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, long)+0x2d9)
7月 25 09:25:57 ceph6 ceph-osd: 11: (ceph::HeartbeatMap::is_healthy()+0xd6)
7月 25 09:25:57 ceph6 ceph-osd: 12: (ceph::HeartbeatMap::check_touch_file()+0x2c)
7月 25 09:25:57 ceph6 ceph-osd: 13: (CephContextServiceThread::entry()+0x15b)
7月 25 09:25:57 ceph6 ceph-osd: 14: (()+0x7dc5)
7月 25 09:25:57 ceph6 ceph-osd: 15: (clone()+0x6d)
7月 25 09:25:57 ceph6 ceph-osd: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this.
7月 25 09:25:57 ceph6 ceph-osd: 0> 2017-07-25 09:25:57.525027 7f46fe478700 -1 *** Caught signal (Aborted) **
7月 25 09:25:57 ceph6 ceph-osd: in thread 7f46fe478700
7月 25 09:25:57 ceph6 ceph-osd: ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
7月 25 09:25:57 ceph6 ceph-osd: 1: (()+0x7e6fe2)
7月 25 09:25:57 ceph6 ceph-osd: 2: (()+0xf370)
7月 25 09:25:57 ceph6 ceph-osd: 3: (gsignal()+0x37)
7月 25 09:25:57 ceph6 ceph-osd: 4: (abort()+0x148)
7月 25 09:25:57 ceph6 ceph-osd: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165)
7月 25 09:25:57 ceph6 ceph-osd: 6: (()+0x5e946)
7月 25 09:25:57 ceph6 ceph-osd: 7: (()+0x5e973)
7月 25 09:25:57 ceph6 ceph-osd: 8: (()+0x5eb9f)
7月 25 09:25:57 ceph6 ceph-osd: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a)
7月 25 09:25:57 ceph6 ceph-osd: 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, long)+0x2d9)
7月 25 09:25:57 ceph6 ceph-osd: 11: (ceph::HeartbeatMap::is_healthy()+0xd6)
7月 25 09:25:57 ceph6 ceph-osd: 12: (ceph::HeartbeatMap::check_touch_file()+0x2c)
7月 25 09:25:57 ceph6 ceph-osd: 13: (CephContextServiceThread::entry()+0x15b)
7月 25 09:25:57 ceph6 ceph-osd: 14: (()+0x7dc5)
7月 25 09:25:57 ceph6 ceph-osd: 15: (clone()+0x6d)
7月 25 09:25:57 ceph6 ceph-osd: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this.
用关键词"FAILED assert(0 == "hit suicide timeout")"搜索时，很多都为“common/HeartbeatMap.cc: 79”类开头的异常。
基本没有“common/HeartbeatMap.cc: 81”开头的。
期间找了各种尝试，巴拉巴拉好多。最后发现问题出现在新节点与老节点内核版本不一致的情况。更新后恢复正常。
页:
[1]