Corosync+Pacemaker+Ldirectord+Lvs+Httpd
Corosync+Pacemaker+Ldirectord+Lvs+Httpd一、硬件环境
4台虚拟机在同一网段
操作系统：centos6.3
关闭系统不必要的服务脚本
1
2
3
4
5
6
7
8
9
10
11
12
#!/bin/bash
services=`chkconfig --list|cut -f1|cut -d" " -f1`
for ser in $services
do
if [ "$ser" == "network" ] || [ "$ser" == "rsyslog" ] || [ "$ser" == "sshd" ] || [ "$ser" == "crond" ] || [ "$ser" == "atd" ];
then
chkconfig "$ser" on
else
chkconfig "$ser" off
fi
done
reboot
二、ip地址规划
1
2
3
4
5
master 172.30.82.45
slave 172.30.82.58
node1 172.30.82.3
node2 172.30.82.11
VIP 172.30.82.61
三、注意：
1、设置各个节点间的时间同步
ntpdate 172.30.82.254 &>/dev/null
2、基于hosts文件实现能够互相用主机名访问,修改/etc/hosts文件
3、使用uname -n执行结果要和主机名相同
4、确保ldirectord服务关闭开机启动
chkconfig ldirectord off
5、关闭selinux
setenfroce 0
四、相关软件下载及安装
从pacemaker1.1.8开始，crm发展成了一个独立项目，叫crmsh。也就是说，我们安装了pacemaker后，并没有crm这个命令，我们要实现对集群资源管理，还需要独立安装crmsh
1
2
3
4
5
6
7
8
9
pssh-2.3.1-4.1.x86_64.rpm crmsh-2.1-1.6.x86_64.rpmpython-pssh-2.3.1-4.1.x86_64.rpm下载地址
http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/x86_64/
libdnet 下载地址：
http://dl.fedoraproject.org/pub/epel/6/x86_64/repoview/letter_l.group.html
ldirectord-3.9.6-0rc1.1.1.x86_64.rpm 下载地址：
http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/x86_64/
yum install corosync pacemakerlibesmtp –y
yum install -y python-dateutil python-lxml redhat-rpm-config cluster-glue cluster-glue-libs resource-agents
yum --nogpgcheck localinstall pssh-2.3.1-4.1.x86_64.rpm crmsh-2.1-1.6.x86_64.rpmpython-pssh-2.3.1-4.1.x86_64.rpm ldirectord-3.9.6-0rc1.1.1.x86_64.rpm
五、配置director节点的高可用
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
1、拷贝配置文件
cp corosync.conf.example corosync.conf
cp /usr/share/doc/ldirectord-3.9.6/ldirectord.cf /etc/ha.d/
2、生成autokeys文件
corosync-keygen
3、修改corosync.conf
totem {
version: 2
secauth: off #是否开启秘钥认证
threads: 0 #发送集群节点认证信息使用的进程数
interface {
ringnumber: 0 #为避免冗余环路设定的所在的网络接口
bindnetaddr: 172.30.82.0 #集群所在网络
mcastaddr: 239.238.16.1 #集群通告组播地址
mcastport: 5405 #服务端口
ttl: 1
}
}
logging {
fileline: off #日志是否打印行号
to_stderr: no #是否输出标准错误（到显示器）
to_logfile: yes #定义日志
logfile: /var/log/corosync.log
to_syslog: no #是否开启系统日志
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
service { #服务启动时启动pacemaker
ver: 0
name: pacemaker
}
4、修改ldirectord配置文件ldirectord.cf
checktimeout=3 # 检测超时
checkinterval=1 # 检测间隔
autoreload=yes # 从新载入客户机
logfile="/var/log/ldirectord.log" # 日志路径
logfile="local0"
quiescent=no # realserver 宕机后从lvs列表中删除，恢复后自动添加进列表
virtual=172.30.82.61:80 # 监听VIP地址80端口
real=172.30.82.3:80 gate # 真机IP地址和端口 路由模式
real=172.30.82.11:80 gate
fallback=127.0.0.1:80 gate # 如果real节点都宕机，则回切到环回地址
service=http # 服务是http
request=".text.html" # 保存在real的web根目录并且可以访问,通过它来判断real是否存活
receive="OK" # 检测文件内容
scheduler=rr # 调度算法
protocol=tcp # 检测协议
checktype=negotiate # 检测类型
checkport=80 # 检测端口
5、复制配置文件到备用节点：
scp -P authkeyscorosync.confldirectord.cfslave:/etc/ha.d/
六、DR模型下配置realserver脚本：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#!/bin/bash
VIP=172.30.82.61
host=`/bin/hostname`
case "$1" in
start)
# Start LVS-DR real server on this machine.
/sbin/ifconfig lo down
/sbin/ifconfig lo up
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
/sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
;;
stop)
# Stop LVS-DR real server loopback device(s).
/sbin/ifconfig lo:0 down
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
;;
status)
# Status of LVS-DR real server.
islothere=`/sbin/ifconfig lo:0 | grep $VIP`
isrothere=`netstat -rn | grep "lo" | grep $VIP`
if [ ! "$islothere" -o ! "$isrothere" ];then
# Either the route or the lo:0 device
# not found.
echo "LVS-DR real server is stopped."
else
echo "LVS-DR real server is running."
fi
;;
*)
# Invalid entry.
echo "$0: Usage: $0 {start|status|stop}"
exit 1
;;
esac
七、real上安装httpd服务并添加测试页面
1
2
3
4
5
6
7
8
9
10
11
1、node1
yum install -y httpd
echo "Welcome to realserver 1" >/var/www/html/index.html
echo "OK" >/var/www/html/.text.html
service httpd start
2、node2
yum install -y httpd
echo "Welcome to realserver 2" >/var/www/html/index.html
echo "OK" >/var/www/html/.text.html
service httpd start
八、开启、配置并测试高可用集群服务
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
1、在master上执行
service corosync start
ssh slave 'service corosync start'
注意：启动node2需要在node1上使用如上命令进行，不要在node2节点上直接启动；
查看corosync引擎是否正常启动
#grep -e "Corosync Cluster Engine" -e "configuration file" /var/log/corosync.log
May 19 23:11:05 corosync Corosync Cluster Engine exiting with status 0 at main.c:2055.
May 19 23:11:46 corosync Corosync Cluster Engine ('1.4.7'): started and ready to provide service.
May 19 23:11:46 corosync Successfully read main configuration file '/etc/corosync/corosync.conf'
查看初始化成员节点通知是否正常发出：
# grepTOTEM/var/log/corosync.log
May 19 19:59:44 corosync Initializing transport (UDP/IP Multicast).
May 19 19:59:44 corosync Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
May 19 19:59:44 corosync The network interface is now up.
May 19 19:59:44 corosync A processor joined or left the membership and a new membership was formed.
检查启动过程中是否有错误产生：
# # grep ERROR: /var/log/corosync.log
查看pacemaker是否正常启动：
May 19 23:11:46 corosync info: pcmk_startup: CRM: Initialized
May 19 23:11:46 corosync Logging: Initialized pcmk_startup
May 19 23:11:46 corosync info: pcmk_startup: Maximum core file size is: 18446744073709551615
May 19 23:11:46 corosync info: pcmk_startup: Service: 9
May 19 23:11:46 corosync info: pcmk_startup: Local hostname: master
使用如下命令查看集群节点的启动状态：
# crm status
Last updated: Wed May 20 00:10:38 2015
Last change: Tue May 19 22:49:50 2015
Stack: classic openais (with plugin)
Current DC: slave - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ master slave ]
2、配置集群资源，这里需要配置2个基本资源1个组资源
a、配置vip
crm(live)configure# primitivevip ocf:heartbeat:IPaddr params ip=172.30.82.61 nic=eth0 cidr_netmask=24
b、配置ldirectord服务资源
crm(live)configure#priimitive ldir lsb:ldirectord
c、配置组资源,组资源将基本资源定义在同一台服务器上运行，默认情况集群资源会均衡运行在集群中各个节点
crm(live)configure#group lvsserver vip ldir
d、不用定义组，可以通过资源粘性及资源约束来也可定义资源的倾向性，这里只是举例：
顺序约束：资源的启动顺序
crm(live)configure# order vip_before_ldir mandatory: vip ldir
排列约束：哪些资源运行在一起
crm(live)configure# colocation ldir_with_vip inf: vip ldir
位置约束：资源更倾向运行在那个节点上
crm(live)configure# location vip_on_mater vip rule 100: #uname eq node1
e、其他的一些配置
禁用stonith设备
crm(live)configure# property stonith-enabled=false
设定集群未到达法定票数的工作机制为忽略，因为只有两台服务器只能选此项
crm(live)configure#no-quorum-policy=ignore
corosync的框架、运行原理、配置命令说明需自行研究，这里倾向于环境搭建及测试
查看集群配置信息库：
crm(live)configure#show
node master
node slave
primitive ldir lsb:ldirectord
primitive vip IPaddr \
params ip=172.30.82.61 nic=eth0 cidr_netmask=24
group lvsserver vip ldir
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
验证配置语法：
crm(live)configure# verify
不报错即提交固化配置：
crm(live)configure# commit
3、测试集群服务，客户端访问172.30.82.61
a、master 上执行：
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP172.30.82.61:80 rr
-> 172.30.82.3:80 Route 1 0 13
-> 172.30.82.11:80 Route 1 0 14
b、slave 上执行：
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
说明集群资源只运行在master上
4、集群资源转移测试
a、master上执行
service corosync stop
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
b、在slave上执行
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP172.30.82.61:80 rr
-> 172.30.82.3:80 Route 1 1 17
-> 172.30.82.11:80 Route 1 0 18
说明集群资源转移成功
c、后端服务故障检测node1上执行
service httpd stop
查看master集群服务
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP172.30.82.61:80 rr
-> 172.30.82.11:80 Route 1 0 0
恢复node1服务
service httpd start
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP172.30.82.61:80 rr
-> 172.30.82.11:80 Route 1 0
-> 172.30.82.3:80 Route 1
No package ldirectord available.
Error: Nothing to do 这个怎么解决
页:
[1]