corosync集群安装配置及crmsh使用详解
corosync+pacemaker的简单实现前提：
1）本配置共有两个测试节点，分别lidefu1和lidefu2，相的IP地址分别为172.16.21.3和172.16.21.4；
2）集群服务为apache的httpd服务；
3）提供web服务的地址为172.16.21.21，即vip；
4）系统为CentOS 6.5 64bits
1、准备工作
为了配置一台Linux主机成为HA的节点，通常需要做出如下的准备工作：
1）所有节点的主机名称和对应的IP地址解析服务可以正常工作，且每个节点的主机名称需要跟"uname -n“命令的结果保持一致；因此，需要保证两个节点上的/etc/hosts文件均为下面的内容：
172.16.21.3 lidefu3
172.16.21.4 lidefu4
为了使得重新启动系统后仍能保持如上的主机名称，还分别需要在各节点执行类似如下的命令：
Node1:
# sed -i 's@\(HOSTNAME=\).*@\1lidefu3@g'/etc/sysconfig/network
# hostname lidefu3
Node2：
# sed -i 's@\(HOSTNAME=\).*@\1lidefu4 @g' /etc/sysconfig/network
# hostname lidefu4
2）设定两个节点可以基于密钥进行ssh通信，这可以通过类似如下的命令实现：
Node1:
# ssh-keygen -t rsa –P ‘’
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@lidefu4
Node2:
# ssh-keygen -t rsa –P ‘’
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@lidefu3
安装corosync ,两个节点都要安装
1: yum install –y corosync
查看安装生成文件
1: rpm -ql corosync
配置corosync,两个节点的配置需要一样
1: cd /etc/corosync/
2: cp corosync.conf.example corosync.conf
3: vim corosync.conf
4: compatibility: whitetank #是否兼容0.8以前的版本
5: totem { #corosync的协议
6: version: 2 #协议的版本号
7: secauth: on #是否开启安全认证功能,这个如果关闭,别人知道我们的多播地址,就很容易加入到集群中,开启会消耗更多的cpu,但是还是建议开启
8: threads: 0 #实现认证时的并行线程数,0表示使用默认配置
9: interface { #子模块
10: ringnumber: 0 #环号,当主机上有多块网卡,为了避免自己主机上的其他网卡收到自己主机上发出的心跳信息,所使用的环号
11: bindnetaddr: 172.16.21.0#集群工作的网络地址,
12: mcastaddr: 226.16.21.1 #多播地址
13: mcastport: 5405 #多播端口
14: ttl: 1 #至广播一次
15: }
16: }
17:
18: logging { #日志相关
19: fileline: off
20: to_stderr: no #日志信息是否发往标准错误输出
21: to_logfile: yes #是否送给日志文件
22: to_syslog: no #是否发往syslog
23: logfile: /var/log/cluster/corosync.log #日志文件
24: debug: off #是否开启debug
25: timestamp: on #是否记录日志的时间戳,消耗I/O
26: logger_subsys { #日志的子系统
27: subsys: AMF #下面的AMF
28: debug: off #debug
29: }
30: }
31:
32: amf { #与amf的编程接口相关
33: mode: disabled
34: }
安装pacemaker,两个节点都要安装
1: yum install pacemaker -y#注:好像不能与heartbeat共存
2: rpm -q pacemaker
3: rpm -ql pacemaker
配置corosync.conf,让pacemaker可以与corosync共同启动,添加早corosync.conf的末行
1: servcie { #定义的服务
2: ver: 0 #版本号
3: name: pacemaker #服务名,表示当corosync启动的时候,pacemaker同时启动
4: }
5: aisexec { #表示启动上面的service时以谁的身份,默认也是root,所以其实可以省略
6: user: root #以root的身份运行
7: group: root #以root组的身份运行
8: }
9: 实际上pacemaker可以自己当成一个服务来启动,只是这里的corosync版本比较老,所以需要把他和corosync一起启动
生成密钥文件,为了通信的安全
1: corosync-keygen #生成认证文件,请使劲敲键盘以获取随机数
安装crmsh和pssh,两个节点都要安装,crmsh作为pacemaker的接口,而安装crmsh需要依赖pssh
1: yum install crmsh-1.2.6-4.el6.x86_64.rpm pssh-2.3.1-2.el6.x86_64.rpm
复制authkey和corosync文件到节点1
1: scp -p authkey corosync lidefu1:/etc/corosync
启动两个节点的corosync
1: service corosync start
查看日志,看是否启动成功
1: less /var/log/cluster/corosync.log
查看corosync引擎是否正常启动：
1: grep -e "Corosync Cluster Engine" -e "configuration file" /var/log/cluster/corosync.log
2: Apr 20 17:39:36 corosync Corosync Cluster Engine ('1.4.1'): started and ready to provide service.
3: Apr 20 17:39:36 corosync Successfully read main configuration file '/etc/corosync/corosync.conf'
查看初始化成员节点通知是否正常发出：
1: grepTOTEM/var/log/cluster/corosync.log
2: Apr 20 17:39:36 corosync Initializing transport (UDP/IP Multicast).
3: Apr 20 17:39:36 corosync Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
4: Apr 20 17:39:36 corosync The network interface is now up.
5: Apr 20 17:39:37 corosync Type of received message is wrong...ignoring 25.
6: Apr 20 17:39:37 corosync A processor joined or left the membership and a new membership was formed.
7: ...
检查启动过程中是否有错误产生。下面的错误信息表示packmaker不久之后将不再作为corosync的插件运行，因此，建议使用cman作为集群基础架构服务；此处可安全忽略。
1: grep ERROR: /var/log/cluster/corosync.log | grep -v unpack_resources
2: Apr 20 17:39:37 corosync ERROR: process_ais_conf: You have configured a cluster using the Pacemaker plugin for Corosync. The plugin is not supported in this environment and will be removed very soon.
3: Apr 20 17:39:37 corosync ERROR: process_ais_conf:Please see Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) for details on using Pacemaker with CMAN
查看pacemaker是否正常启动：
1: grep pcmk_startup /var/log/cluster/corosync.log
2: Apr 20 17:39:37 corosync info: pcmk_startup: CRM: Initialized
3: Apr 20 17:39:37 corosync Logging: Initialized pcmk_startup
4: Apr 20 17:39:37 corosync info: pcmk_startup: Maximum core file size is: 18446744073709551615
5: Apr 20 17:39:37 corosync info: pcmk_startup: Service: 9
6: Apr 20 17:39:37 corosync info: pcmk_startup: Local hostname: lidefu3
如果安装了crmsh，可使用如下命令查看集群节点的启动状态
1: crm status #查看状态
2: Last updated: Sun Apr 20 18:14:01 2014
3: Last change: Sun Apr 20 18:00:25 2014 via crmd on lidefu3
4: Stack: classic openais (with plugin)
5: Current DC: NONE
6: 1 Nodes configured, 2 expected votes
7: 0 Resources configured
8:
9:
10: Node lidefu3: UNCLEAN (offline)
查看crm命令的帮助,crm的命令有两种模式,一种是批处理模式,一种是交互式
1: crm #直接执行crm命令
2: crm(live)# help #live表示直接修改在执行的配置信息,即修改即生效
3:
4: This is crm shell, a Pacemaker command line interface.
5:
6: Available commands:
7:
8: cib manage shadow CIBs
9: resource resources management
10: configure CRM cluster configuration
11: node nodes management
12: options user preferences
13: history CRM cluster history
14: site Geo-cluster support
15: ra resource agents information center
16: status show cluster status
17: help,? show help (help topics for list of topics)
18: end,cd,up go back one level
19: quit,bye,exit exit the program
首先禁用stonith,因为没有stonith设备
1: property stonith-enabled=false #关闭stonith
2: verify #查看命令是否错误
3: commit #提交
4: show #查看集群所有的配置信息
注:当一个集群没有法定票数,节点故障资源也不会转移
查看primitive的帮助
1: help primitive
2: primitive <rsc> {[<class>:[<provider>:]]<type>|@<template>}
3:
4:
5:
6:
7: ...]
8:
9: attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
10: id_spec :: $id=<id> | $id-ref=<id>
11: op_type :: start | stop | monitor
查看各资源代理的可代理类型classes和list
1: classes #列出资源类型
2: lsb
3: o / heartbeat pacemaker
4: service
5: stonith
6: crm(live)ra# list lsb#查看该资源代理可代理的资源
7: NetworkManager abrt-ccpp abrt-oops abrtd acpid atd
8: auditd autofs blk-availability bluetooth corosync corosync-notifyd
9: ...
查看资源可使用的参数meta
1: classes
2: list o
3: meta o:IPaddr2
4:
5: Parameters (* denotes required, [] the default):
6:
7: ip* (string): IPv4 or IPv6 address
8: nic (string): Network interface
9: broadcast (string): Broadcast address
10: iflabel (string): Interface label
11: lvs_support (boolean, ): Enable support for LVS DR
定义资源primitive
1: primitive webip o:IPaddr2 params ip=172.16.21.21
2: primitive webserver lsb:httpd
3: verify
4: commit
5: show
定义组,对于corosync是后定义组的,不像heartbeat只能先定义组
1: group webservice webip webserver
2: verify
3: commit
4: show
访问网页
1: curl http://172.16.21.21 #也可以通过windows的浏览器
2: this is fours page33333333333333
让节点3停止页
1: node
2: standby lidefu3
3: cd ..
4: status
再次访问网页
1: curl http://172.16.21.21
2: this is four page444444444444
让节点3上线,看资源是否转回来
1: crm node online lidefu3
2: crm status #结果是没转回来,因为前面的步骤并没有定义资源的倾向性
删除资源的一种方式
1: crm
2: configure
3: edit#可以直接编辑删掉,但是不建议这么做
停止资源stop,而后才做删除操作
1: crm
2: resource
3: stop webservice
4: status webservice
删除资源delete
1: crm
2: configure
3: delete webservice #注意:删除的仅仅是webservice这个组,并没有删除组内的资源
定义排列约束colocation,两个资源在一起,类似分组
1: Usage:
2: ...............
3: colocation <id> <score>: <rsc>[:<role>] <rsc>[:<role>] ...
4:
5: ...............
6: Example:
7: ...............
8: colocation dummy_and_apache -inf: apache dummy
9: colocation c1 inf: A ( B C )
10: ...............
1: configure
2: colocation webserver_and_webip inf: webserver webip #前面的是随后面的,webserver是随着webip的
3: verify
4: commit
查看所有资源的详细信息
1: configure
2: show xml
定义顺序约束,假定启动是线性的
1: Usage:
2: ...............
3: order <id> {kind|<score>}: <rsc>[:<action>] <rsc>[:<action>] ...
4:
5:
6: kind :: Mandatory(强制) | Optional(可选) | Serialize(强制)
7: ...............
8: Example:
9: ...............
10: order c_apache_1 Mandatory: apache:start ip_1
11: order o1 Serialize: A ( B C )
12: order order_2 Mandatory: [ A B ] C
13: ...............
1: order webip_before_webserver mandatory: webip webserver
位置约束,location,让资源更倾向于运行在哪个节点上,资源的倾向性是各个资源的和
1:location <id> <rsc> {node_pref|rules}
2:
3: node_pref :: <score>: <node>
4:
5: rules ::
6: rule [$role=<role>] <score>: <expression>
7: [$role=<role>] <score>: <expression> ...]
8:
9: id_spec :: $id=<id> | $id-ref=<id>
10: score :: <number> | <attribute> | [-]inf
11: expression :: <simple_exp>
12: bool_op :: or | and
13: simple_exp :: <attribute> <binary_op> <value>
14: | <unary_op> <attribute>
15: | date <date_expr>
16: type :: string | version | number
17: binary_op :: lt | gt | lte | gte | eq | ne
18: unary_op :: defined | not_defined
19:
20: date_expr :: lt <end>
21: | gt <start>
22: | in_range start=<start> end=<end>
23: | in_range start=<start> <duration>
24: | date_spec <date_spec>
25: duration|date_spec ::
26: hours=<value>
27: | monthdays=<value>
28: | weekdays=<value>
29: | yearsdays=<value>
30: | months=<value>
31: | weeks=<value>
32: | years=<value>
33: | weekyears=<value>
34: | moon=<value>
35: ...............
36: Examples:
37: ...............
38: location conn_1 internal_www 100: node1
39:
40: location conn_1 internal_www \
41: rule 50: #uname eq node1 \
42: rule pingd: defined pingd
43:
44: location conn_2 dummy_float \
45: rule -inf: not_defined pingd or pingd number:lte 0
1: location webip_on_lidefu4 webip 200: lidefu4
配置全局属性,当一台主机挂了的时候,不具有法定票数时,剩下的一台主机同样可以运行资源
1: configure
2: property no-quorum-policy=ignore
3: verify
4: commit
定义资源的默认属性
1: crm(live)configure# rsc_defaults
2: allow-migrate= interval-origin= multiple-active= restart-type=
3: description= is-managed= priority= target-role=定义资源就启动?
4: failure-timeout= migration-threshold=resource-stickiness=资源粘性,只对当前节点生效,还是不太熟
以上的定义只是节点出故障才会转移资源,并没有定义资源出故障怎么办.所以下面定义资源故障
1: killall httpd #不正常终止运行的服务
2: crm
3: status #查看节点状态,发现依然显示正常运行
4: stop webserver#停止服务
5: stop webip #停止服务
6: resource #进去咯
7: cleanup webserver#对不正常关闭的资源做清理
8: cleanup webip #对不正常关闭的资源做清理
定义监控,当资源出故障的时候,会尝试重启服务,如果重启不了就会转换节点
1: Usage:
2: ...............
3: monitor <rsc>[:<role>] <interval>[:<timeout>]
4: ...............
5: Example:
6: ...............
7: monitor apence 60m:60s
8: ...............
1: monitor webserver 20s:15s
2: verify
3: commit
1: crm(live)configure# monitor webserver 20s:15s
定义好了,可以尝试在当前运行httpd节点的服务停止,然后等待15秒以上,查看他是否启动成功,笔者这里是启动成功的.
页:
[1]