heartbeat + pacemaker实现pg流复制自动切换（二）
五、测试5.1 备节点失效
在node2上杀死postgres数据库进程，模拟备节点上数据库崩溃：
# killall -9 postgres 查看此时集群状态：
# crm_mon -Afr1
============
Last updated: Mon Jan 27 08:36:49 2014
Stack: Heartbeat
Current DC: node1 (30b7dc95-25c5-40d7-b1e4-7eaf2d5cdf07) - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
4 Resources configured.
============
Online: [ node1 node2 ]
Full list of resources:
vip-slave (ocf::heartbeat:IPaddr2): Started node1
Resource Group: master-group
vip-master (ocf::heartbeat:IPaddr2): Started node1
vip-rep (ocf::heartbeat:IPaddr2): Started node1
Master/Slave Set: msPostgresql
Masters: [ node1 ]
Stopped: [ pgsql:1 ]
Clone Set: clnPingCheck
Started: [ node1 node2 ]
Node Attributes:
* Node node1:
+ default_ping_set : 100
+ master-pgsql:0 : 1000
+ pgsql-data-status : LATEST
+ pgsql-master-baseline : 0000000010000000
+ pgsql-status : PRI
* Node node2:
+ default_ping_set : 100
+ master-pgsql:1 : -INFINITY
+ pgsql-data-status : DISCONNECT
+ pgsql-status : STOP
Migration summary:
* Node node1:
* Node node2:
pgsql:1: migration-threshold=1 fail-count=1
Failed actions:
pgsql:1_monitor_7000 (node=node2, call=11, rc=7, status=complete): not running
{vip-slave资源已成功切换到了node1上}
重启node2上的heartbeat，数据库将重新伴随启动：
# service heartbeat restart 过段时间后查看状态：
# crm_mon -Afr1
============
Last updated: Mon Jan 27 08:39:16 2014
Stack: Heartbeat
Current DC: node1 (30b7dc95-25c5-40d7-b1e4-7eaf2d5cdf07) - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
4 Resources configured.
============
Online: [ node1 node2 ]
Full list of resources:
vip-slave (ocf::heartbeat:IPaddr2): Started node2
Resource Group: master-group
vip-master (ocf::heartbeat:IPaddr2): Started node1
vip-rep (ocf::heartbeat:IPaddr2): Started node1
Master/Slave Set: msPostgresql
Masters: [ node1 ]
Slaves: [ node2 ]
Clone Set: clnPingCheck
Started: [ node1 node2 ]
Node Attributes:
* Node node1:
+ default_ping_set : 100
+ master-pgsql:0 : 1000
+ pgsql-data-status : LATEST
+ pgsql-master-baseline : 0000000010000000
+ pgsql-status : PRI
* Node node2:
+ default_ping_set : 100
+ master-pgsql:1 : 100
+ pgsql-data-status : STREAMING|SYNC
+ pgsql-status : HS:sync
Migration summary:
* Node node1:
* Node node2:
{vip-slave又重新回到了nod2上，且流复制重新建立}
5.2 主节点失效切换
在node1上杀死postgres数据库进程，模拟备节点上数据库崩溃：
# killall -9 postgres 等会查看集群状态：
# crm_mon -Afr -1
============
Last updated: Mon Jan 27 08:43:03 2014
Stack: Heartbeat
Current DC: node1 (30b7dc95-25c5-40d7-b1e4-7eaf2d5cdf07) - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
4 Resources configured.
============
Online: [ node1 node2 ]
Full list of resources:
vip-slave (ocf::heartbeat:IPaddr2): Started node2
Resource Group: master-group
vip-master (ocf::heartbeat:IPaddr2): Started node2
vip-rep (ocf::heartbeat:IPaddr2): Started node2
Master/Slave Set: msPostgresql
Masters: [ node2 ]
Stopped: [ pgsql:0 ]
Clone Set: clnPingCheck
Started: [ node1 node2 ]
Node Attributes:
* Node node1:
+ default_ping_set : 100
+ master-pgsql:0 : -INFINITY
+ pgsql-data-status : DISCONNECT
+ pgsql-status : STOP
* Node node2:
+ default_ping_set : 100
+ master-pgsql:1 : 1000
+ pgsql-data-status : LATEST
+ pgsql-master-baseline : 00000000120000B0
+ pgsql-status : PRI
Migration summary:
* Node node1:
pgsql:0: migration-threshold=1 fail-count=1
* Node node2:
Failed actions:
pgsql:0_monitor_2000 (node=node1, call=25, rc=7, status=complete): not running
{vip-master/vip-rep都已成功切换到node2上，且node2已变为master，node2上pg数据库状态已切换为PRI}
5.3 主节点恢复
修复原主节点后将其恢复为当前备节点
在node1上执行一次基础同步：
$ pwd
/opt/pgsql/data
$ rm -rf *
$ pg_basebackup -h 192.168.2.3 -U postgres -D /opt/pgsql/data/ -P
19172/19172 kB (100%), 1/1 tablespace
NOTICE:pg_stop_backup complete, all required WAL segments have been archived
$ ls
backup_label base pg_clog pg_ident.confpg_notifypg_stat_tmppg_tblspc PG_VERSIONpostgresql.conf
backup_label.oldglobalpg_hba.confpg_multixact pg_serialpg_subtranspg_twophasepg_xlog recovery.done
启动heartbeat之前必须删除资锁，不然资源将不会伴随heartbeat启动：
# rm -rf /var/lib/pgsql/tmp/PGSQL.lock {该锁文件在当节点为主节点时创建，但不会因为heartbeat的异常停止或数据库/系统的异常终止而自动删除，所以在恢复一个节点的时候只要该节点充当过主节点就需要手动清理该锁文件}
重启node1上的heartbeat：
# service heartbeat restart 过段时间后查看集群状态：
# crm_mon -Afr1
============
Last updated: Mon Jan 27 08:50:43 2014
Stack: Heartbeat
Current DC: node2 (f2dcd1df-7429-42f5-82e9-b73921f97cab) - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
4 Resources configured.
============
Online: [ node1 node2 ]
Full list of resources:
vip-slave (ocf::heartbeat:IPaddr2): Started node1
Resource Group: master-group
vip-master (ocf::heartbeat:IPaddr2): Started node2
vip-rep (ocf::heartbeat:IPaddr2): Started node2
Master/Slave Set: msPostgresql
Masters: [ node2 ]
Slaves: [ node1 ]
Clone Set: clnPingCheck
Started: [ node1 node2 ]
Node Attributes:
* Node node1:
+ default_ping_set : 100
+ master-pgsql:0 : 100
+ pgsql-data-status : STREAMING|SYNC
+ pgsql-status : HS:sync
* Node node2:
+ default_ping_set : 100
+ master-pgsql:1 : 1000
+ pgsql-data-status : LATEST
+ pgsql-master-baseline : 00000000120000B0
+ pgsql-status : PRI
Migration summary:
* Node node1:
* Node node2:
{vip-slave已成功切到node1上，node1成功成为流复制备节点}
六、管理
6.1 启动关闭heartbeat
# service heartbeat start
# service heartbeat stop
6.2 查看HA状态
# crm status6.3 查看资源状态及节点属性
# crm_mon -Afr -16.4 查看配置
# crm configure show6.5 实时监控HA
# crm_mon -Afr6.6 crm_resource命令
资源启动/关闭：
# crm_resource -r vip-master -v started
# crm_resource -r vip-master -v stoped
列举资源：
# crm_resource -L
vip-slave (ocf::heartbeat:IPaddr2): Started
Resource Group: master-group
vip-master (ocf::heartbeat:IPaddr2): Started
vip-rep (ocf::heartbeat:IPaddr2): Started
Master/Slave Set: msPostgresql
Masters: [ node1 ]
Slaves: [ node2 ]
Clone Set: clnPingCheck
Started: [ node1 node2 ]
查看资源位置：
# crm_resource -W -r pgsql
resource pgsql is running on: node2
迁移资源：
# crm_resource -M -r vip-slave -N node2删除资源：
# crm_resource -D -r vip-slave -t primitive6.7 crm命令
列举指定的RA：
# crm ra list ocf pacemaker
ClusterMon Dummy HealthCPU HealthSMART Stateful SysInfo SystemHealth controld ping pingd
remote
删除节点：
# crm node delete node2停用节点：
# crm node standby node2启用节点：
# crm node online node2配置pacemaker：
# crm configure
crm(live)configure#
……
……
crm(live)configure# commit
crm(live)configure# quit
6.8 重置failcount
# crm resource
crm(live)resource# failcount pgsql set node1 0
crm(live)resource# failcount pgsql show node1
scope=statusname=fail-count-pgsql value=0
# crm resource cleanup pgsql
Cleaning up pgsql:0 on node1
Waiting for 1 replies from the CRMd. OK
# crm_failcount -G -U node1 -r pgsql
scope=statusname=fail-count-pgsql value=INFINITY
# crm_failcount -D -U node1 -r pgsql
七、问题记录
7.1 Q1
问题现象：
heartbeat日志中报如下错误：
Jan 24 07:47:36 node1 heartbeat: : WARN: nodename node1 uuid changed to node2
Jan 24 07:47:38 node1 heartbeat: : WARN: nodename node2 uuid changed to node1
Jan 24 07:47:38 node1 heartbeat: : WARN: nodename node1 uuid changed to node2
Jan 24 07:47:40 node1 heartbeat: : WARN: nodename node2 uuid changed to node1
Jan 24 07:47:40 node1 heartbeat: : WARN: nodename node1 uuid changed to node2
Jan 24 07:47:42 node1 heartbeat: : WARN: nodename node2 uuid changed to node1
Jan 24 07:47:42 node1 heartbeat: : WARN: nodename node1 uuid changed to node2
解决方式：
因为是通过虚拟机克隆生成的node2,所以hb_uuid相同，需要删除后重新生成，如下：
# rm -rf /var/lib/heartbeat/hb_uuid
# service heartbeat restart
重启之后将会生成新的hb_uuid
7.2 Q2
问题现象：
加载配置时报错：
# crm configure load update pgsql.crm
ERROR: pgsql: parameter rep_mode does not exist
ERROR: pgsql: parameter node_list does not exist
ERROR: pgsql: parameter master_ip does not exist
ERROR: pgsql: parameter restore_command does not exist
ERROR: pgsql: parameter primary_conninfo_opt does not exist
WARNING: pgsql: specified timeout 60s for stop is smaller than the advised 120
WARNING: pgsql: action monitor_Master not advertised in meta-data, it may not be supported by the RA
WARNING: pgsql: specified timeout 60s for start is smaller than the advised 120
WARNING: pgsql: action notify not advertised in meta-data, it may not be supported by the RA
WARNING: pgsql: action demote not advertised in meta-data, it may not be supported by the RA
WARNING: pgsql: action promote not advertised in meta-data, it may not be supported by the RA
WARNING: pingCheck: specified timeout 60s for start is smaller than the advised 90
WARNING: pingCheck: specified timeout 60s for stop is smaller than the advised 100
Do you still want to commit?
解决方式：
原因是pgsql脚本过旧，不支持配置pgsql.crm中设置的一些参数，需要从网上下载并替换pgsql
https://raw.github.com/ClusterLabs/resource-agents
7.3 Q3
问题现象：
加载配置时报错：
# crm configure load update pgsql.crm
lrmadmin: 2014/01/24_09:18:44 ERROR: lrm_get_rsc_type_metadata(578): got a return code HA_FAIL from a reply message of rmetadata with function get_ret_from_msg.
ERROR: ocf:heartbeat:pgsql: could not parse meta-data:
ERROR: ocf:heartbeat:pgsql: could not parse meta-data:
ERROR: ocf:heartbeat:pgsql: no such resource agent
WARNING: pingCheck: specified timeout 60s for start is smaller than the advised 90
WARNING: pingCheck: specified timeout 60s for stop is smaller than the advised 100
Do you still want to commit?
解决方式：
原因是pgsql脚本权限不正确，使用下面命令修改即可：
# chmod 755 /usr/lib/ocf/resource.d/heartbeat/pgsql
7.4 Q4
问题现象：
启动heartbeat时报错：
# service heartbeat start
/usr/lib/ocf/lib//heartbeat/ocf-shellfuncs: line 56: @OCF_ROOT_DIR@/lib/heartbeat/ocf-binaries: No such file or directory
解决方式：
因为在CentOS5.5中@OCF_ROOT_DIR@变量无法替换为正确的路径导致，可通过修改脚本实现，如下：
编辑ocf-shellfuncs修改如下内容：
if [ -z "$OCF_ROOT" ]; then
# : ${OCF_ROOT=@OCF_ROOT_DIR@}
: ${OCF_ROOT=/usr/lib/ocf}
fi
7.5 Q5
问题现象：
启动heartbeat时报错：
# service heartbeat start
/usr/lib/ocf/lib//heartbeat/ocf-shellfuncs: line 60: /usr/lib/ocf/lib/heartbeat/ocf-rarun: No such file or directory
解决方式：
因为缺少ocf-rarun脚本导致
下载放入相应路径即可：
下载地址https://raw.github.com/ClusterLabs/resource-agents
7.6 Q6
问题现象：
启动heartbeat时因找不到启动脚本而报错：
# service heartbeat start
Starting High-Availability services:Heartbeat failure . Failed.
heartbeat: 2014/01/23_09:06:59 info: Pacemaker support: yes
heartbeat: 2014/01/23_09:06:59 ERROR: Client child command is not executable
heartbeat: 2014/01/23_09:06:59 ERROR: Directive failfasthacluster /usr/lib64/heartbeat/cib failed
heartbeat: 2014/01/23_09:06:59 ERROR: Client child command is not executable
heartbeat: 2014/01/23_09:06:59 ERROR: Directive respawn root /usr/lib64/heartbeat/stonithd failed
heartbeat: 2014/01/23_09:06:59 ERROR: Client child command is not executable
heartbeat: 2014/01/23_09:06:59 ERROR: Directive respawnhacluster /usr/lib64/heartbeat/attrd failed
heartbeat: 2014/01/23_09:06:59 ERROR: Client child command is not executable
heartbeat: 2014/01/23_09:06:59 ERROR: Directive failfasthacluster /usr/lib64/heartbeat/crmd failed
heartbeat: 2014/01/23_09:06:59 ERROR: Heartbeat not started: configuration error.
heartbeat: 2014/01/23_09:06:59 ERROR: Configuration error, heartbeat not started.
解决方式：
ln -s /usr/libexec/pacemaker/cib /usr/lib64/heartbeat/cib
ln -s /usr/libexec/pacemaker/stonithd /usr/lib64/heartbeat/stonithd
ln -s /usr/libexec/pacemaker/attrd /usr/lib64/heartbeat/attrd
ln -s /usr/libexec/pacemaker/crmd /usr/lib64/heartbeat/crmd
7.7 Q7
问题现象：
启动heartbeat时报错：
Jan 23 09:10:15 db1 heartbeat: : info: Heartbeat generation: 1390439416
Jan 23 09:10:15 db1 heartbeat: : info: No uuid found for current node - generating a new uuid.
Jan 23 09:10:15 db1 heartbeat: : info: Creating FIFO /var/lib/heartbeat/fifo.
Jan 23 09:10:15 db1 heartbeat: : info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth1
Jan 23 09:10:15 db1 heartbeat: : info: glib: ucast: bound send socket to device: eth1
Jan 23 09:10:15 db1 heartbeat: : ERROR: glib: ucast: error setting option SO_REUSEPORT(w): Protocol not available
Jan 23 09:10:15 db1 heartbeat: : ERROR: make_io_childpair: cannot open ucast eth1
Jan 23 09:10:16 db1 heartbeat: : CRIT: Emergency Shutdown: Master Control process died.
Jan 23 09:10:16 db1 heartbeat: : CRIT: Killing pid 2129 with SIGTERM
Jan 23 09:10:16 db1 heartbeat: : CRIT: Emergency Shutdown(MCP dead): Killing ourselves.
解决方式：
1.升级内核版本，当前内核版本不支持ucast;
2.换用其它的检测方式，如mcast/bcast。
7.8 Q8
问题现象：
使用bcast心跳检测方式时报错：
Jan 24 01:30:20 db2 heartbeat: : ERROR: glib: Error binding socket (Address already in use). Retrying.
Jan 24 01:30:21 db2 heartbeat: : ERROR: glib: Error binding socket (Address already in use). Retrying.
Jan 24 01:30:22 db2 heartbeat: : ERROR: glib: Error binding socket (Address already in use). Retrying.
Jan 24 01:30:23 db2 heartbeat: : ERROR: glib: Error binding socket (Address already in use). Retrying.
Jan 24 01:30:24 db2 heartbeat: : ERROR: glib: Unable to bind socket (Address already in use). Giving up.
Jan 24 01:30:24 db2 heartbeat: : info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1
Jan 24 01:30:24 db2 heartbeat: : ERROR: make_io_childpair: cannot open bcast eth1
Jan 24 01:30:25 db2 heartbeat: : CRIT: Emergency Shutdown: Master Control process died.
Jan 24 01:30:25 db2 heartbeat: : CRIT: Killing pid 29856 with SIGTERM
Jan 24 01:30:25 db2 heartbeat: : CRIT: Emergency Shutdown(MCP dead): Killing ourselves.
解决方式：
说明694端口已经被占用，查看
# netstat -nlp | grep 694
udp 0 0 0.0.0.0:694 0.0.0.0:* 1367/rpcbind
udp 0 0 :::694 :::* 1367/rpcbind
换个UDP端口，如在ha.cf中指定udpport 692
八、参考资源
脚本：
https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/pgsql
脚本使用说明：
https://github.com/t-matsuo/resource-agents/wiki/Resource-Agent-for-PostgreSQL-9.1-streaming-replication
crm_resouce命令：
http://www.novell.com/zh-cn/documentation/sle_ha/book_sleha/data/man_crmresource.html
crm_failcount命令：
http://www.novell.com/zh-cn/documentation/sle_ha/book_sleha/data/man_crmfailcount.html
支持一下
页:
[1]