Heartbeat+Drbd+NFS高可用实现
继续之前的操作，来完成heartbeat+drbd+nfs高可用的实现。heartbeat+drbd还是使用原来的环境即可，然而需要添加一台nfs客户端，客户端的信息如下：
主机名IP地址角色
server136.contoso.com192.168.49.136nfs客户端
一、环境准备
1）注意：因为我的客户端是192.168.49.0/24端，所以这里需要把VIP改为这个网段，所以先修改两个节点的haresources文件，将VIP修改为192.168.49.100。可以根据实际情况来看是否需要这一步操作。
# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.comIPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4
# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.comIPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4
2）两个drbd节点要安装nfs（以下操作需要在两个节点上操作）
1
2
3
4
5
6
7
8
yum -y install rpcbind nfs-utils
vi /etc/exports
/data192.168.49.0/255.255.255.0(rw,no_root_squash,no_all_squash,sync)
/etc/init.d/rpcbind start
chkconfig rpcbind on
/etc/init.d/nfs start
chkconfig nfs off
chmod -R 777 /data
3)客户端上安装nfs
1
2
3
yum -y install rpcbind nfs-utils
/etc/init.d/rpcbind start
chkconfig rpcbind on
4)客户端上测试nfs挂载
1
2
3
4
showmount -e 192.168.49.100
mkdir /data
mount -t nfs 192.168.49.100:/data /data
echo "mount -t nfs 192.168.49.100:/data /data" >>/etc/rc.local
# showmount -e 172.16.49.100
Export list for 172.16.49.100:
/data 192.168.49.0/255.255.255.0
# mount -t nfs 192.168.49.100:/data /data
# cd /data
# mkdir nfsclient
# touch client.txt
# ls
10.txt20.txt30.txt40.txt50.txt60.txt70.txt80.txt90.txtclient.txt
11.txt21.txt31.txt41.txt51.txt61.txt71.txt81.txt91.txtlost+found
12.txt22.txt32.txt42.txt52.txt62.txt72.txt82.txt92.txtnfsclient
13.txt23.txt33.txt43.txt53.txt63.txt73.txt83.txt93.txt
14.txt24.txt34.txt44.txt54.txt64.txt74.txt84.txt94.txt
15.txt25.txt35.txt45.txt55.txt65.txt75.txt85.txt95.txt
16.txt26.txt36.txt46.txt56.txt66.txt76.txt86.txt96.txt
17.txt27.txt37.txt47.txt57.txt67.txt77.txt87.txt97.txt
18.txt28.txt38.txt48.txt58.txt68.txt78.txt88.txt98.txt
19.txt29.txt39.txt49.txt59.txt69.txt79.txt89.txt99.txt
1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt
二、修改heartbeat资源配置
1）添加nfs服务
因为两个节点都没有设置nfs开机启动，而是让heartbeat接管nfs服务，在主节点获取VIP之后，然后启动nfs服务，所以需要将nfs服务加入heartbeat资源中
2）添加脚本文件
添加一个脚本文件killnfsd，目的是在发生故障切换时，能保障客户端挂载nfs正常。因为如果没有该脚本，在NFS服务切换后，客户端需要重新mount一下nfs共享目录，才可以正常使用，否则会出现“stale NFS file handle”的错误。
1
2
3
4
5
6
7
vi /etc/ha.d/resource.d/killnfsd
#!/bin/bash
#
killall -9 nfsd
/etc/init.d/nfs restart
exit 0
chmod +x /etc/ha.d/resource.d/killnfsd
3）编辑/etc/ha.d/haresources文件
vi /etc/ha.d/haresources
# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.comIPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 nfs killnfsd
# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.comIPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 nfs killnfsd
三、配置nfs共享并手动挂载测试
1、正常启动heartbeat
依次启动heartbeat01和heartbeat02上的heartbeat服务，然后分别到两个节点上查看：
heartbeat01：
# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:4 nr:35740 dw:35744 dr:709 al:1 bm:9 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# ls /data
10.txt20.txt30.txt40.txt50.txt60.txt70.txt80.txt90.txtclient.txt
11.txt21.txt31.txt41.txt51.txt61.txt71.txt81.txt91.txtlost+found
12.txt22.txt32.txt42.txt52.txt62.txt72.txt82.txt92.txtnfsclient
13.txt23.txt33.txt43.txt53.txt63.txt73.txt83.txt93.txt
14.txt24.txt34.txt44.txt54.txt64.txt74.txt84.txt94.txt
15.txt25.txt35.txt45.txt55.txt65.txt75.txt85.txt95.txt
16.txt26.txt36.txt46.txt56.txt66.txt76.txt86.txt96.txt
17.txt27.txt37.txt47.txt57.txt67.txt77.txt87.txt97.txt
18.txt28.txt38.txt48.txt58.txt68.txt78.txt88.txt98.txt
19.txt29.txt39.txt49.txt59.txt69.txt79.txt89.txt99.txt
1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 2528) is running...
nfsd (pid 2544 2543 2542 2541 2540 2539 2538 2537) is running...
rpc.rquotad (pid 2523) is running...
heartbeat02：
# ip a |grep 49.100
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:12288 nr:4 dw:4 dr:12288 al:0 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# ls /data
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
2、在客户端上手动挂载nfs共享目录
# showmount -e 192.168.49.100
Export list for 192.168.49.100:
/data 192.168.49.0/255.255.255.0
# mount -t nfs 192.168.49.100:/data /data
# ls /data
10.txt20.txt30.txt40.txt50.txt60.txt70.txt80.txt90.txtclient.txt
11.txt21.txt31.txt41.txt51.txt61.txt71.txt81.txt91.txtlost+found
12.txt22.txt32.txt42.txt52.txt62.txt72.txt82.txt92.txtnfsclient
13.txt23.txt33.txt43.txt53.txt63.txt73.txt83.txt93.txt
14.txt24.txt34.txt44.txt54.txt64.txt74.txt84.txt94.txt
15.txt25.txt35.txt45.txt55.txt65.txt75.txt85.txt95.txt
16.txt26.txt36.txt46.txt56.txt66.txt76.txt86.txt96.txt
17.txt27.txt37.txt47.txt57.txt67.txt77.txt87.txt97.txt
18.txt28.txt38.txt48.txt58.txt68.txt78.txt88.txt98.txt
19.txt29.txt39.txt49.txt59.txt69.txt79.txt89.txt99.txt
1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt
# touch test1.txt
此时，可以正常写入。
这里我添加一个脚本，来执行一个持续写入的任务，用来测试在故障切换过程中，向NFS共享目录/data中写入数据是否受影响。脚本内容如下：
1
2
3
4
5
6
7
8
9
#!/bin/bash
#
while true;do
for i in {1..1000}
do
touch /data/action$i.txt
sleep 2
done
done
然后后台执行该脚本：
1
2
# sh wrnfs.sh &
5272
四、故障切换测试
1、手工停止heartbeat01上的heartbeat服务，观察一下两个节点以及nfs客户单的状态：
1）heartbeat01:
# /etc/init.d/heartbeat stop
Stopping High-Availability services: Done.
2）heartbeat02:
# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:12292 nr:1600 dw:1604 dr:13001 al:1 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# ls /data
10.txt21.txt32.txt43.txt54.txt65.txt76.txt87.txt98.txt action5.txt
11.txt22.txt33.txt44.txt55.txt66.txt77.txt88.txt99.txt action6.txt
12.txt23.txt34.txt45.txt56.txt67.txt78.txt89.txt9.txt action7.txt
13.txt24.txt35.txt46.txt57.txt68.txt79.txt8.txt action10.txtaction8.txt
14.txt25.txt36.txt47.txt58.txt69.txt7.txt 90.txtaction11.txtaction9.txt
15.txt26.txt37.txt48.txt59.txt6.txt 80.txt91.txtaction12.txtclient.txt
16.txt27.txt38.txt49.txt5.txt 70.txt81.txt92.txtaction13.txtlost+found
17.txt28.txt39.txt4.txt 60.txt71.txt82.txt93.txtaction14.txtnfsclient
18.txt29.txt3.txt 50.txt61.txt72.txt83.txt94.txtaction1.txt test1.txt
19.txt2.txt 40.txt51.txt62.txt73.txt84.txt95.txtaction2.txt
1.txt 30.txt41.txt52.txt63.txt74.txt85.txt96.txtaction3.txt
20.txt31.txt42.txt53.txt64.txt75.txt86.txt97.txtaction4.txt
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 2571) is running...
nfsd (pid 2587 2586 2585 2584 2583 2582 2581 2580) is running...
rpc.rquotad (pid 2566) is running...
3）nfs客户端
# ls /data
10.txt24.txt38.txt51.txt65.txt79.txt92.txt action16.txtaction7.txt
11.txt25.txt39.txt52.txt66.txt7.txt 93.txt action17.txtaction8.txt
12.txt26.txt3.txt 53.txt67.txt80.txt94.txt action18.txtaction9.txt
13.txt27.txt40.txt54.txt68.txt81.txt95.txt action19.txtclient.txt
14.txt28.txt41.txt55.txt69.txt82.txt96.txt action1.txt lost+found
15.txt29.txt42.txt56.txt6.txt 83.txt97.txt action20.txtnfsclient
16.txt2.txt 43.txt57.txt70.txt84.txt98.txt action21.txttest1.txt
17.txt30.txt44.txt58.txt71.txt85.txt99.txt action22.txt
18.txt31.txt45.txt59.txt72.txt86.txt9.txt action23.txt
19.txt32.txt46.txt5.txt 73.txt87.txtaction10.txtaction24.txt
1.txt 33.txt47.txt60.txt74.txt88.txtaction11.txtaction2.txt
20.txt34.txt48.txt61.txt75.txt89.txtaction12.txtaction3.txt
21.txt35.txt49.txt62.txt76.txt8.txt action13.txtaction4.txt
22.txt36.txt4.txt 63.txt77.txt90.txtaction14.txtaction5.txt
23.txt37.txt50.txt64.txt78.txt91.txtaction15.txtaction6.txt
2、然后再启动heartbeat01的heartbeat服务，再次查看故障切换的情况
1）heartbeat01
# /etc/init.d/heartbeat start
Starting High-Availability services: INFO:Resource is stopped
Done.
# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1604 nr:38732 dw:40336 dr:1766 al:3 bm:9 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# ls /data
10.txt39.txt67.txt95.txt action21.txtaction4.txt action78.txt
11.txt3.txt 68.txt96.txt action22.txtaction50.txtaction79.txt
12.txt40.txt69.txt97.txt action23.txtaction51.txtaction7.txt
13.txt41.txt6.txt 98.txt action24.txtaction52.txtaction80.txt
14.txt42.txt70.txt99.txt action25.txtaction53.txtaction81.txt
15.txt43.txt71.txt9.txt action26.txtaction54.txtaction82.txt
16.txt44.txt72.txtaction100.txtaction27.txtaction55.txtaction83.txt
17.txt45.txt73.txtaction101.txtaction28.txtaction56.txtaction84.txt
18.txt46.txt74.txtaction102.txtaction29.txtaction57.txtaction85.txt
19.txt47.txt75.txtaction103.txtaction2.txt action58.txtaction86.txt
1.txt 48.txt76.txtaction104.txtaction30.txtaction59.txtaction87.txt
20.txt49.txt77.txtaction105.txtaction31.txtaction5.txt action88.txt
21.txt4.txt 78.txtaction106.txtaction32.txtaction60.txtaction89.txt
22.txt50.txt79.txtaction107.txtaction33.txtaction61.txtaction8.txt
23.txt51.txt7.txt action108.txtaction34.txtaction62.txtaction90.txt
24.txt52.txt80.txtaction109.txtaction35.txtaction63.txtaction91.txt
25.txt53.txt81.txtaction10.txt action36.txtaction64.txtaction92.txt
26.txt54.txt82.txtaction110.txtaction37.txtaction65.txtaction93.txt
27.txt55.txt83.txtaction111.txtaction38.txtaction66.txtaction94.txt
28.txt56.txt84.txtaction112.txtaction39.txtaction67.txtaction95.txt
29.txt57.txt85.txtaction11.txt action3.txt action68.txtaction96.txt
2.txt 58.txt86.txtaction12.txt action40.txtaction69.txtaction97.txt
30.txt59.txt87.txtaction13.txt action41.txtaction6.txt action98.txt
31.txt5.txt 88.txtaction14.txt action42.txtaction70.txtaction99.txt
32.txt60.txt89.txtaction15.txt action43.txtaction71.txtaction9.txt
33.txt61.txt8.txt action16.txt action44.txtaction72.txtclient.txt
34.txt62.txt90.txtaction17.txt action45.txtaction73.txtlost+found
35.txt63.txt91.txtaction18.txt action46.txtaction74.txtnfsclient
36.txt64.txt92.txtaction19.txt action47.txtaction75.txttest1.txt
37.txt65.txt93.txtaction1.txt action48.txtaction76.txt
38.txt66.txt94.txtaction20.txt action49.txtaction77.txt
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 4151) is running...
nfsd (pid 4168 4167 4166 4165 4164 4163 4162 4161) is running...
rpc.rquotad (pid 4146) is running...
2）heartbeat02
# ip a |grep 49.100
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:15280 nr:1624 dw:4616 dr:13321 al:3 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# ls /data
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
3）nfs客户端
# ls /data
10.txt44.txt78.txt action111.txtaction145.txtaction39.txtaction72.txt
11.txt45.txt79.txt action112.txtaction146.txtaction3.txt action73.txt
12.txt46.txt7.txt action113.txtaction147.txtaction40.txtaction74.txt
13.txt47.txt80.txt action114.txtaction148.txtaction41.txtaction75.txt
14.txt48.txt81.txt action115.txtaction149.txtaction42.txtaction76.txt
15.txt49.txt82.txt action116.txtaction14.txt action43.txtaction77.txt
16.txt4.txt 83.txt action117.txtaction150.txtaction44.txtaction78.txt
17.txt50.txt84.txt action118.txtaction151.txtaction45.txtaction79.txt
18.txt51.txt85.txt action119.txtaction152.txtaction46.txtaction7.txt
19.txt52.txt86.txt action11.txt action153.txtaction47.txtaction80.txt
1.txt 53.txt87.txt action120.txtaction154.txtaction48.txtaction81.txt
20.txt54.txt88.txt action121.txtaction15.txt action49.txtaction82.txt
21.txt55.txt89.txt action122.txtaction16.txt action4.txt action83.txt
22.txt56.txt8.txt action123.txtaction17.txt action50.txtaction84.txt
23.txt57.txt90.txt action124.txtaction18.txt action51.txtaction85.txt
24.txt58.txt91.txt action125.txtaction19.txt action52.txtaction86.txt
25.txt59.txt92.txt action126.txtaction1.txt action53.txtaction87.txt
26.txt5.txt 93.txt action127.txtaction20.txt action54.txtaction88.txt
27.txt60.txt94.txt action128.txtaction21.txt action55.txtaction89.txt
28.txt61.txt95.txt action129.txtaction22.txt action56.txtaction8.txt
29.txt62.txt96.txt action12.txt action23.txt action57.txtaction90.txt
2.txt 63.txt97.txt action130.txtaction24.txt action58.txtaction91.txt
30.txt64.txt98.txt action131.txtaction25.txt action59.txtaction92.txt
31.txt65.txt99.txt action132.txtaction26.txt action5.txt action93.txt
32.txt66.txt9.txt action133.txtaction27.txt action60.txtaction94.txt
33.txt67.txtaction100.txtaction134.txtaction28.txt action61.txtaction95.txt
34.txt68.txtaction101.txtaction135.txtaction29.txt action62.txtaction96.txt
35.txt69.txtaction102.txtaction136.txtaction2.txt action63.txtaction97.txt
36.txt6.txt action103.txtaction137.txtaction30.txt action64.txtaction98.txt
37.txt70.txtaction104.txtaction138.txtaction31.txt action65.txtaction99.txt
38.txt71.txtaction105.txtaction139.txtaction32.txt action66.txtaction9.txt
39.txt72.txtaction106.txtaction13.txt action33.txt action67.txtclient.txt
3.txt 73.txtaction107.txtaction140.txtaction34.txt action68.txtlost+found
40.txt74.txtaction108.txtaction141.txtaction35.txt action69.txtnfsclient
41.txt75.txtaction109.txtaction142.txtaction36.txt action6.txt test1.txt
42.txt76.txtaction10.txt action143.txtaction37.txt action70.txt
43.txt77.txtaction110.txtaction144.txtaction38.txt action71.txt
3、手动重启heartbeat01，查看故障切换的情况
1）heartbeat01
# reboot
#
Broadcast message from root@heartbeat01.contoso.com
(/dev/pts/0) at 4:23 ...
The system is going down for reboot NOW!
2）heartbeat02
# ip a|grep 49.100
inet 192.168.49.100/24 scope global eth1
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:15284 nr:3924 dw:6920 dr:14074 al:3 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# ls /data
10.txt49.txt87.txt action125.txtaction163.txtaction32.txtaction70.txt
11.txt4.txt 88.txt action126.txtaction164.txtaction33.txtaction71.txt
12.txt50.txt89.txt action127.txtaction165.txtaction34.txtaction72.txt
13.txt51.txt8.txt action128.txtaction166.txtaction35.txtaction73.txt
14.txt52.txt90.txt action129.txtaction167.txtaction36.txtaction74.txt
15.txt53.txt91.txt action12.txt action168.txtaction37.txtaction75.txt
16.txt54.txt92.txt action130.txtaction169.txtaction38.txtaction76.txt
17.txt55.txt93.txt action131.txtaction16.txt action39.txtaction77.txt
18.txt56.txt94.txt action132.txtaction170.txtaction3.txt action78.txt
19.txt57.txt95.txt action133.txtaction171.txtaction40.txtaction79.txt
1.txt 58.txt96.txt action134.txtaction172.txtaction41.txtaction7.txt
20.txt59.txt97.txt action135.txtaction173.txtaction42.txtaction80.txt
21.txt5.txt 98.txt action136.txtaction174.txtaction43.txtaction81.txt
22.txt60.txt99.txt action137.txtaction175.txtaction44.txtaction82.txt
23.txt61.txt9.txt action138.txtaction176.txtaction45.txtaction83.txt
24.txt62.txtaction100.txtaction139.txtaction177.txtaction46.txtaction84.txt
25.txt63.txtaction101.txtaction13.txt action178.txtaction47.txtaction85.txt
26.txt64.txtaction102.txtaction140.txtaction179.txtaction48.txtaction86.txt
27.txt65.txtaction103.txtaction141.txtaction17.txt action49.txtaction87.txt
28.txt66.txtaction104.txtaction142.txtaction180.txtaction4.txt action88.txt
29.txt67.txtaction105.txtaction143.txtaction181.txtaction50.txtaction89.txt
2.txt 68.txtaction106.txtaction144.txtaction182.txtaction51.txtaction8.txt
30.txt69.txtaction107.txtaction145.txtaction183.txtaction52.txtaction90.txt
31.txt6.txt action108.txtaction146.txtaction184.txtaction53.txtaction91.txt
32.txt70.txtaction109.txtaction147.txtaction185.txtaction54.txtaction92.txt
33.txt71.txtaction10.txt action148.txtaction186.txtaction55.txtaction93.txt
34.txt72.txtaction110.txtaction149.txtaction18.txt action56.txtaction94.txt
35.txt73.txtaction111.txtaction14.txt action19.txt action57.txtaction95.txt
36.txt74.txtaction112.txtaction150.txtaction1.txt action58.txtaction96.txt
37.txt75.txtaction113.txtaction151.txtaction20.txt action59.txtaction97.txt
38.txt76.txtaction114.txtaction152.txtaction21.txt action5.txt action98.txt
39.txt77.txtaction115.txtaction153.txtaction22.txt action60.txtaction99.txt
3.txt 78.txtaction116.txtaction154.txtaction23.txt action61.txtaction9.txt
40.txt79.txtaction117.txtaction155.txtaction24.txt action62.txtclient.txt
41.txt7.txt action118.txtaction156.txtaction25.txt action63.txtlost+found
42.txt80.txtaction119.txtaction157.txtaction26.txt action64.txtnfsclient
43.txt81.txtaction11.txt action158.txtaction27.txt action65.txttest1.txt
44.txt82.txtaction120.txtaction159.txtaction28.txt action66.txt
45.txt83.txtaction121.txtaction15.txt action29.txt action67.txt
46.txt84.txtaction122.txtaction160.txtaction2.txt action68.txt
47.txt85.txtaction123.txtaction161.txtaction30.txt action69.txt
48.txt86.txtaction124.txtaction162.txtaction31.txt action6.txt
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 4160) is running...
nfsd (pid 4176 4175 4174 4173 4172 4171 4170 4169) is running...
rpc.rquotad (pid 4155) is running...
3)nfs客户端
# ls /data
10.txt50.txt90.txt action130.txtaction170.txtaction2.txt action6.txt
11.txt51.txt91.txt action131.txtaction171.txtaction30.txtaction70.txt
12.txt52.txt92.txt action132.txtaction172.txtaction31.txtaction71.txt
13.txt53.txt93.txt action133.txtaction173.txtaction32.txtaction72.txt
14.txt54.txt94.txt action134.txtaction174.txtaction33.txtaction73.txt
15.txt55.txt95.txt action135.txtaction175.txtaction34.txtaction74.txt
16.txt56.txt96.txt action136.txtaction176.txtaction35.txtaction75.txt
17.txt57.txt97.txt action137.txtaction177.txtaction36.txtaction76.txt
18.txt58.txt98.txt action138.txtaction178.txtaction37.txtaction77.txt
19.txt59.txt99.txt action139.txtaction179.txtaction38.txtaction78.txt
1.txt 5.txt 9.txt action13.txt action17.txt action39.txtaction79.txt
20.txt60.txtaction100.txtaction140.txtaction180.txtaction3.txt action7.txt
21.txt61.txtaction101.txtaction141.txtaction181.txtaction40.txtaction80.txt
22.txt62.txtaction102.txtaction142.txtaction182.txtaction41.txtaction81.txt
23.txt63.txtaction103.txtaction143.txtaction183.txtaction42.txtaction82.txt
24.txt64.txtaction104.txtaction144.txtaction184.txtaction43.txtaction83.txt
25.txt65.txtaction105.txtaction145.txtaction185.txtaction44.txtaction84.txt
26.txt66.txtaction106.txtaction146.txtaction186.txtaction45.txtaction85.txt
27.txt67.txtaction107.txtaction147.txtaction187.txtaction46.txtaction86.txt
28.txt68.txtaction108.txtaction148.txtaction188.txtaction47.txtaction87.txt
29.txt69.txtaction109.txtaction149.txtaction189.txtaction48.txtaction88.txt
2.txt 6.txt action10.txt action14.txt action18.txt action49.txtaction89.txt
30.txt70.txtaction110.txtaction150.txtaction190.txtaction4.txt action8.txt
31.txt71.txtaction111.txtaction151.txtaction191.txtaction50.txtaction90.txt
32.txt72.txtaction112.txtaction152.txtaction192.txtaction51.txtaction91.txt
33.txt73.txtaction113.txtaction153.txtaction193.txtaction52.txtaction92.txt
34.txt74.txtaction114.txtaction154.txtaction194.txtaction53.txtaction93.txt
35.txt75.txtaction115.txtaction155.txtaction195.txtaction54.txtaction94.txt
36.txt76.txtaction116.txtaction156.txtaction196.txtaction55.txtaction95.txt
37.txt77.txtaction117.txtaction157.txtaction197.txtaction56.txtaction96.txt
38.txt78.txtaction118.txtaction158.txtaction198.txtaction57.txtaction97.txt
39.txt79.txtaction119.txtaction159.txtaction199.txtaction58.txtaction98.txt
3.txt 7.txt action11.txt action15.txt action19.txt action59.txtaction99.txt
40.txt80.txtaction120.txtaction160.txtaction1.txt action5.txt action9.txt
41.txt81.txtaction121.txtaction161.txtaction20.txt action60.txtclient.txt
42.txt82.txtaction122.txtaction162.txtaction21.txt action61.txtlost+found
43.txt83.txtaction123.txtaction163.txtaction22.txt action62.txtnfsclient
44.txt84.txtaction124.txtaction164.txtaction23.txt action63.txttest1.txt
45.txt85.txtaction125.txtaction165.txtaction24.txt action64.txt
46.txt86.txtaction126.txtaction166.txtaction25.txt action65.txt
47.txt87.txtaction127.txtaction167.txtaction26.txt action66.txt
48.txt88.txtaction128.txtaction168.txtaction27.txt action67.txt
49.txt89.txtaction129.txtaction169.txtaction28.txt action68.txt
4.txt 8.txt action12.txt action16.txt action29.txt action69.txt
4、将heartbeat01上的eth1（心跳网卡）down掉，查看故障切换状态
# ifdown eth1
# ifconfig
eth0 Link encap:EthernetHWaddr 00:0C:29:B3:83:3F
inet addr:192.168.49.133Bcast:192.168.49.255Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feb3:833f/64 Scope:Link
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
RX packets:4180 errors:0 dropped:0 overruns:0 frame:0
TX packets:3877 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3130356 (2.9 MiB)TX bytes:2748373 (2.6 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNINGMTU:65536Metric:1
RX packets:28 errors:0 dropped:0 overruns:0 frame:0
TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1960 (1.9 KiB)TX bytes:1960 (1.9 KiB)
1）heartbeat01
# ip a|grep 49.100
inet 192.168.49.100/24 scope global eth1
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:2068 nr:2612 dw:4680 dr:1105 al:3 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# ls /data
10.txt77.txt action143.txtaction209.txtaction275.txtaction40.txt
11.txt78.txt action144.txtaction20.txt action276.txtaction41.txt
12.txt79.txt action145.txtaction210.txtaction277.txtaction42.txt
13.txt7.txt action146.txtaction211.txtaction278.txtaction43.txt
14.txt80.txt action147.txtaction212.txtaction279.txtaction44.txt
15.txt81.txt action148.txtaction213.txtaction27.txt action45.txt
16.txt82.txt action149.txtaction214.txtaction280.txtaction46.txt
17.txt83.txt action14.txt action215.txtaction281.txtaction47.txt
18.txt84.txt action150.txtaction216.txtaction282.txtaction48.txt
19.txt85.txt action151.txtaction217.txtaction283.txtaction49.txt
1.txt 86.txt action152.txtaction218.txtaction284.txtaction4.txt
20.txt87.txt action153.txtaction219.txtaction285.txtaction50.txt
21.txt88.txt action154.txtaction21.txt action286.txtaction51.txt
22.txt89.txt action155.txtaction220.txtaction287.txtaction52.txt
23.txt8.txt action156.txtaction221.txtaction288.txtaction53.txt
24.txt90.txt action157.txtaction222.txtaction289.txtaction54.txt
25.txt91.txt action158.txtaction223.txtaction28.txt action55.txt
26.txt92.txt action159.txtaction224.txtaction290.txtaction56.txt
27.txt93.txt action15.txt action225.txtaction291.txtaction57.txt
28.txt94.txt action160.txtaction226.txtaction292.txtaction58.txt
29.txt95.txt action161.txtaction227.txtaction293.txtaction59.txt
2.txt 96.txt action162.txtaction228.txtaction294.txtaction5.txt
30.txt97.txt action163.txtaction229.txtaction295.txtaction60.txt
31.txt98.txt action164.txtaction22.txt action296.txtaction61.txt
32.txt99.txt action165.txtaction230.txtaction297.txtaction62.txt
33.txt9.txt action166.txtaction231.txtaction298.txtaction63.txt
34.txtaction100.txtaction167.txtaction232.txtaction299.txtaction64.txt
35.txtaction101.txtaction168.txtaction233.txtaction29.txt action65.txt
36.txtaction102.txtaction169.txtaction234.txtaction2.txt action66.txt
37.txtaction103.txtaction16.txt action235.txtaction300.txtaction67.txt
38.txtaction104.txtaction170.txtaction236.txtaction301.txtaction68.txt
39.txtaction105.txtaction171.txtaction237.txtaction302.txtaction69.txt
3.txt action106.txtaction172.txtaction238.txtaction303.txtaction6.txt
40.txtaction107.txtaction173.txtaction239.txtaction304.txtaction70.txt
41.txtaction108.txtaction174.txtaction23.txt action305.txtaction71.txt
42.txtaction109.txtaction175.txtaction240.txtaction306.txtaction72.txt
43.txtaction10.txt action176.txtaction241.txtaction307.txtaction73.txt
44.txtaction110.txtaction177.txtaction242.txtaction308.txtaction74.txt
45.txtaction111.txtaction178.txtaction243.txtaction309.txtaction75.txt
46.txtaction112.txtaction179.txtaction244.txtaction30.txt action76.txt
47.txtaction113.txtaction17.txt action245.txtaction310.txtaction77.txt
48.txtaction114.txtaction180.txtaction246.txtaction311.txtaction78.txt
49.txtaction115.txtaction181.txtaction247.txtaction312.txtaction79.txt
4.txt action116.txtaction182.txtaction248.txtaction313.txtaction7.txt
50.txtaction117.txtaction183.txtaction249.txtaction314.txtaction80.txt
51.txtaction118.txtaction184.txtaction24.txt action315.txtaction81.txt
52.txtaction119.txtaction185.txtaction250.txtaction316.txtaction82.txt
53.txtaction11.txt action186.txtaction251.txtaction317.txtaction83.txt
54.txtaction120.txtaction187.txtaction252.txtaction318.txtaction84.txt
55.txtaction121.txtaction188.txtaction253.txtaction319.txtaction85.txt
56.txtaction122.txtaction189.txtaction254.txtaction31.txt action86.txt
57.txtaction123.txtaction18.txt action255.txtaction320.txtaction87.txt
58.txtaction124.txtaction190.txtaction256.txtaction321.txtaction88.txt
59.txtaction125.txtaction191.txtaction257.txtaction322.txtaction89.txt
5.txt action126.txtaction192.txtaction258.txtaction323.txtaction8.txt
60.txtaction127.txtaction193.txtaction259.txtaction324.txtaction90.txt
61.txtaction128.txtaction194.txtaction25.txt action325.txtaction91.txt
62.txtaction129.txtaction195.txtaction260.txtaction326.txtaction92.txt
63.txtaction12.txt action196.txtaction261.txtaction327.txtaction93.txt
64.txtaction130.txtaction197.txtaction262.txtaction328.txtaction94.txt
65.txtaction131.txtaction198.txtaction263.txtaction329.txtaction95.txt
66.txtaction132.txtaction199.txtaction264.txtaction32.txt action96.txt
67.txtaction133.txtaction19.txt action265.txtaction330.txtaction97.txt
68.txtaction134.txtaction1.txt action266.txtaction331.txtaction98.txt
69.txtaction135.txtaction200.txtaction267.txtaction332.txtaction99.txt
6.txt action136.txtaction201.txtaction268.txtaction33.txt action9.txt
70.txtaction137.txtaction202.txtaction269.txtaction34.txt client.txt
71.txtaction138.txtaction203.txtaction26.txt action35.txt lost+found
72.txtaction139.txtaction204.txtaction270.txtaction36.txt nfsclient
73.txtaction13.txt action205.txtaction271.txtaction37.txt test1.txt
74.txtaction140.txtaction206.txtaction272.txtaction38.txt
75.txtaction141.txtaction207.txtaction273.txtaction39.txt
76.txtaction142.txtaction208.txtaction274.txtaction3.txt
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 3192) is running...
nfsd (pid 3208 3207 3206 3205 3204 3203 3202 3201) is running...
rpc.rquotad (pid 3187) is running...
# ifconfig
eth0 Link encap:EthernetHWaddr 00:0C:29:B3:83:3F
inet addr:192.168.49.133Bcast:192.168.49.255Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feb3:833f/64 Scope:Link
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
RX packets:6166 errors:0 dropped:0 overruns:0 frame:0
TX packets:7300 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3365517 (3.2 MiB)TX bytes:6246399 (5.9 MiB)
eth1 Link encap:EthernetHWaddr 00:0C:29:B3:83:49
inet addr:192.168.49.100Bcast:0.0.0.0Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feb3:8349/64 Scope:Link
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
RX packets:447 errors:0 dropped:0 overruns:0 frame:0
TX packets:183 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:119694 (116.8 KiB)TX bytes:43324 (42.3 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNINGMTU:65536Metric:1
RX packets:44 errors:0 dropped:0 overruns:0 frame:0
TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3080 (3.0 KiB)TX bytes:3080 (3.0 KiB)
2）heartbeat02
# ip a|grep 49.100
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:17896 nr:10696 dw:16304 dr:14402 al:3 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# ls /data
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
3）nfs客户端
# ls /data
10.txt action102.txtaction194.txtaction285.txtaction376.txtaction467.txt
11.txt action103.txtaction195.txtaction286.txtaction377.txtaction468.txt
12.txt action104.txtaction196.txtaction287.txtaction378.txtaction469.txt
13.txt action105.txtaction197.txtaction288.txtaction379.txtaction46.txt
14.txt action106.txtaction198.txtaction289.txtaction37.txt action470.txt
15.txt action107.txtaction199.txtaction28.txt action380.txtaction471.txt
16.txt action108.txtaction19.txt action290.txtaction381.txtaction472.txt
17.txt action109.txtaction1.txt action291.txtaction382.txtaction473.txt
18.txt action10.txt action200.txtaction292.txtaction383.txtaction474.txt
19.txt action110.txtaction201.txtaction293.txtaction384.txtaction475.txt
1.txt action111.txtaction202.txtaction294.txtaction385.txtaction476.txt
20.txt action112.txtaction203.txtaction295.txtaction386.txtaction477.txt
21.txt action113.txtaction204.txtaction296.txtaction387.txtaction478.txt
22.txt action114.txtaction205.txtaction297.txtaction388.txtaction479.txt
23.txt action115.txtaction206.txtaction298.txtaction389.txtaction47.txt
24.txt action116.txtaction207.txtaction299.txtaction38.txt action480.txt
25.txt action117.txtaction208.txtaction29.txt action390.txtaction481.txt
26.txt action118.txtaction209.txtaction2.txt action391.txtaction482.txt
27.txt action119.txtaction20.txt action300.txtaction392.txtaction483.txt
28.txt action11.txt action210.txtaction301.txtaction393.txtaction484.txt
29.txt action120.txtaction211.txtaction302.txtaction394.txtaction485.txt
2.txt action121.txtaction212.txtaction303.txtaction395.txtaction486.txt
30.txt action122.txtaction213.txtaction304.txtaction396.txtaction487.txt
31.txt action123.txtaction214.txtaction305.txtaction397.txtaction488.txt
32.txt action124.txtaction215.txtaction306.txtaction398.txtaction489.txt
33.txt action125.txtaction216.txtaction307.txtaction399.txtaction48.txt
34.txt action126.txtaction217.txtaction308.txtaction39.txt action490.txt
35.txt action127.txtaction218.txtaction309.txtaction3.txt action491.txt
36.txt action128.txtaction219.txtaction30.txt action400.txtaction492.txt
37.txt action129.txtaction21.txt action310.txtaction401.txtaction493.txt
38.txt action12.txt action220.txtaction311.txtaction402.txtaction494.txt
39.txt action130.txtaction221.txtaction312.txtaction403.txtaction495.txt
3.txt action131.txtaction222.txtaction313.txtaction404.txtaction496.txt
40.txt action132.txtaction223.txtaction314.txtaction405.txtaction497.txt
41.txt action133.txtaction224.txtaction315.txtaction406.txtaction498.txt
42.txt action134.txtaction225.txtaction316.txtaction407.txtaction499.txt
43.txt action135.txtaction226.txtaction317.txtaction408.txtaction49.txt
44.txt action136.txtaction227.txtaction318.txtaction409.txtaction4.txt
45.txt action137.txtaction228.txtaction319.txtaction40.txt action500.txt
46.txt action138.txtaction229.txtaction31.txt action410.txtaction501.txt
47.txt action139.txtaction22.txt action320.txtaction411.txtaction502.txt
48.txt action13.txt action230.txtaction321.txtaction412.txtaction503.txt
49.txt action140.txtaction231.txtaction322.txtaction413.txtaction50.txt
4.txt action141.txtaction232.txtaction323.txtaction414.txtaction51.txt
50.txt action142.txtaction233.txtaction324.txtaction415.txtaction52.txt
51.txt action143.txtaction234.txtaction325.txtaction416.txtaction53.txt
52.txt action144.txtaction235.txtaction326.txtaction417.txtaction54.txt
53.txt action145.txtaction236.txtaction327.txtaction418.txtaction55.txt
54.txt action146.txtaction237.txtaction328.txtaction419.txtaction56.txt
55.txt action147.txtaction238.txtaction329.txtaction41.txt action57.txt
56.txt action148.txtaction239.txtaction32.txt action420.txtaction58.txt
57.txt action149.txtaction23.txt action330.txtaction421.txtaction59.txt
58.txt action14.txt action240.txtaction331.txtaction422.txtaction5.txt
59.txt action150.txtaction241.txtaction332.txtaction423.txtaction60.txt
5.txt action151.txtaction242.txtaction333.txtaction424.txtaction61.txt
60.txt action152.txtaction243.txtaction334.txtaction425.txtaction62.txt
61.txt action153.txtaction244.txtaction335.txtaction426.txtaction63.txt
62.txt action154.txtaction245.txtaction336.txtaction427.txtaction64.txt
63.txt action155.txtaction246.txtaction337.txtaction428.txtaction65.txt
64.txt action156.txtaction247.txtaction338.txtaction429.txtaction66.txt
65.txt action157.txtaction248.txtaction339.txtaction42.txt action67.txt
66.txt action158.txtaction249.txtaction33.txt action430.txtaction68.txt
67.txt action159.txtaction24.txt action340.txtaction431.txtaction69.txt
68.txt action15.txt action250.txtaction341.txtaction432.txtaction6.txt
69.txt action160.txtaction251.txtaction342.txtaction433.txtaction70.txt
6.txt action161.txtaction252.txtaction343.txtaction434.txtaction71.txt
70.txt action162.txtaction253.txtaction344.txtaction435.txtaction72.txt
71.txt action163.txtaction254.txtaction345.txtaction436.txtaction73.txt
72.txt action164.txtaction255.txtaction346.txtaction437.txtaction74.txt
73.txt action165.txtaction256.txtaction347.txtaction438.txtaction75.txt
74.txt action166.txtaction257.txtaction348.txtaction439.txtaction76.txt
75.txt action167.txtaction258.txtaction349.txtaction43.txt action77.txt
76.txt action168.txtaction259.txtaction34.txt action440.txtaction78.txt
77.txt action169.txtaction25.txt action350.txtaction441.txtaction79.txt
78.txt action16.txt action260.txtaction351.txtaction442.txtaction7.txt
79.txt action170.txtaction261.txtaction352.txtaction443.txtaction80.txt
7.txt action171.txtaction262.txtaction353.txtaction444.txtaction81.txt
80.txt action172.txtaction263.txtaction354.txtaction445.txtaction82.txt
81.txt action173.txtaction264.txtaction355.txtaction446.txtaction83.txt
82.txt action174.txtaction265.txtaction356.txtaction447.txtaction84.txt
83.txt action175.txtaction266.txtaction357.txtaction448.txtaction85.txt
84.txt action176.txtaction267.txtaction358.txtaction449.txtaction86.txt
85.txt action177.txtaction268.txtaction359.txtaction44.txt action87.txt
86.txt action178.txtaction269.txtaction35.txt action450.txtaction88.txt
87.txt action179.txtaction26.txt action360.txtaction451.txtaction89.txt
88.txt action17.txt action270.txtaction361.txtaction452.txtaction8.txt
89.txt action180.txtaction271.txtaction362.txtaction453.txtaction90.txt
8.txt action181.txtaction272.txtaction363.txtaction454.txtaction91.txt
90.txt action182.txtaction273.txtaction364.txtaction455.txtaction92.txt
91.txt action183.txtaction274.txtaction365.txtaction456.txtaction93.txt
92.txt action184.txtaction275.txtaction366.txtaction457.txtaction94.txt
93.txt action185.txtaction276.txtaction367.txtaction458.txtaction95.txt
94.txt action186.txtaction277.txtaction368.txtaction459.txtaction96.txt
95.txt action187.txtaction278.txtaction369.txtaction45.txt action97.txt
96.txt action188.txtaction279.txtaction36.txt action460.txtaction98.txt
97.txt action189.txtaction27.txt action370.txtaction461.txtaction99.txt
98.txt action18.txt action280.txtaction371.txtaction462.txtaction9.txt
99.txt action190.txtaction281.txtaction372.txtaction463.txtclient.txt
9.txt action191.txtaction282.txtaction373.txtaction464.txtlost+found
action100.txtaction192.txtaction283.txtaction374.txtaction465.txtnfsclient
action101.txtaction193.txtaction284.txtaction375.txtaction466.txttest1.txt
4）查看heartbeat01上的日志
# tail -f /var/log/ha-log
Sep 26 05:01:00 heartbeat01.contoso.com heartbeat: : ERROR: glib: mcast_write: Unable to send HBcomm packet eth1 225.0.0.1:694 len=225 [-1]: Invalid argument
Sep 26 05:01:00 heartbeat01.contoso.com heartbeat: : ERROR: write_child: write failure on mcast eth1.: Invalid argument
Sep 26 05:01:02 heartbeat01.contoso.com heartbeat: : ERROR: glib: mcast_write: Unable to send HBcomm packet eth1 225.0.0.1:694 len=225 [-1]: Invalid argument
Sep 26 05:01:02 heartbeat01.contoso.com heartbeat: : ERROR: write_child: write failure on mcast eth1.: Invalid argument
Sep 26 05:01:04 heartbeat01.contoso.com heartbeat: : ERROR: glib: mcast_write: Unable to send HBcomm packet eth1 225.0.0.1:694 len=225 [-1]: Invalid argument
Sep 26 05:01:04 heartbeat01.contoso.com heartbeat: : ERROR: write_child: write failure on mcast eth1.: Invalid argument
Sep 26 05:01:06 heartbeat01.contoso.com heartbeat: : ERROR: glib: mcast_write: Unable to send HBcomm packet eth1 225.0.0.1:694 len=225 [-1]: Invalid argument
Sep 26 05:01:06 heartbeat01.contoso.com heartbeat: : ERROR: write_child: write failure on mcast eth1.: Invalid argument
Sep 26 05:01:06 heartbeat01.contoso.com heartbeat: : WARN: Temporarily Suppressing write error messages
Sep 26 05:01:06 heartbeat01.contoso.com heartbeat: : WARN: Is a cable unplugged on mcast eth1?
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: : WARN: node heartbeat02.contoso.com: is dead
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: : WARN: No STONITH device configured.
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: : WARN: Shared disks are not protected.
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: : info: Resources being acquired from heartbeat02.contoso.com.
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: : info: Link heartbeat02.contoso.com:eth1 dead.
Sep 26 05:01:17 heartbeat01.contoso.com ipfail: : info: Status update: Node heartbeat02.contoso.com now has status dead
harc(default): 2016/09/26_05:01:17 info: Running /etc/ha.d//rc.d/status status
mach_down(default): 2016/09/26_05:01:17 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default): 2016/09/26_05:01:17 info: mach_down takeover complete for node heartbeat02.contoso.com.
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: : info: mach_down takeover complete.
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.49.100): 2016/09/26_05:01:17 INFO:Resource is stopped
Sep 26 05:01:17 heartbeat01.contoso.com heartbeat: : info: Local Resource acquisition completed.
harc(default): 2016/09/26_05:01:17 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default): 2016/09/26_05:01:17 received ip-request-resp IPaddr::192.168.49.100/24/eth1 OK yes
ResourceManager(default): 2016/09/26_05:01:17 info: Acquiring resource group: heartbeat01.contoso.com IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 nfs killnfsd
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.49.100): 2016/09/26_05:01:17 INFO:Resource is stopped
ResourceManager(default): 2016/09/26_05:01:17 info: Running /etc/ha.d/resource.d/IPaddr 192.168.49.100/24/eth1 start
IPaddr(IPaddr_192.168.49.100): 2016/09/26_05:01:17 INFO: Adding inet address 192.168.49.100/24 to device eth1
IPaddr(IPaddr_192.168.49.100): 2016/09/26_05:01:17 INFO: Bringing device eth1 up
IPaddr(IPaddr_192.168.49.100): 2016/09/26_05:01:17 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.49.100 eth1 192.168.49.100 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.49.100): 2016/09/26_05:01:17 INFO:Success
Sep 26 05:01:17 heartbeat01.contoso.com ipfail: : info: NS: We are dead. :<
Sep 26 05:01:17 heartbeat01.contoso.com ipfail: : info: Link Status update: Link heartbeat02.contoso.com/eth1 now has status dead
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0): 2016/09/26_05:01:17 INFO:Running OK
Sep 26 05:01:18 heartbeat01.contoso.com ipfail: : info: We are dead. :<
Sep 26 05:01:18 heartbeat01.contoso.com ipfail: : info: Asking other side for ping node count.
注意：上面将heartbeat01的网卡eth1停掉，之所以没有发生脑裂是因为我在/etc/drbd.d/test.res中配置文件中的节点地址是192.168.49.0/24段，也就是eth0的网段，所以没有发生裂脑。
5、下面将该配置修改为172.16.49.0/24段，再进行测试：
on heartbeat01.contoso.com {
device /dev/drbd0;
disk /dev/sdb1;
address 172.16.49.133:7788;
flexible-meta-disk internal;
}
on heartbeat02.contoso.com {
device /dev/drbd0;
disk /dev/sdb1;
address 172.16.49.134:7788;
meta-disk internal;
1）heartbeat01
# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:2324 nr:0 dw:9888 dr:3707 al:4 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:40
# ls -l /data|wc -l
942
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 9672) is running...
nfsd (pid 9688 9687 9686 9685 9684 9683 9682 9681) is running...
rpc.rquotad (pid 9667) is running...
2)heartbeat02
# ip a |grep 49.100
inet 192.168.49.100/24 scope global eth1
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:0 nr:2324 dw:2420 dr:3093 al:3 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:88
# ls -l /data |wc -l
942
# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 6280) is running...
nfsd (pid 6296 6295 6294 6293 6292 6291 6290 6289) is running...
rpc.rquotad (pid 6275) is running...
3)nfs客户端
# ls -l /data|wc -l
968
# ls -l /data|wc -l
993
# ls -l /data |wc -l
942
此时，客户端虽然仍在进行数据写入，但是数据是写入到heartbeat01中，而且此时两个节点都有192.168.49.100这个IP地址，drbd中每个节点对端的状态都是Unknown状态，这就是典型的脑裂状况。
# ifup eth1
Determining if ip address 172.16.49.133 is already in use for device eth1...
将heartbeat01的eth1启动后，发现heartbeat的裂脑状态消除，然而，drbd两个节点的状态依然不对，如下：
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:2324 nr:0 dw:16888 dr:5012 al:4 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:6552
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----
ns:0 nr:2324 dw:2436 dr:3093 al:3 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:88
五、发生脑裂的处理
1）在secondary主机上执行如下命令：
1
2
3
drbdadm secondary test
drbdadm disconnect test
drbdadm -- --discard-my-data connect test
2）在Primary主机上执行如下命令：
1
2
drbdadm disconnect all
drbdadm connect test
# drbdadm disconnect test
# drbdadm -- --discard-my-data connect test
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----
ns:0 nr:0 dw:2436 dr:3093 al:3 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:88
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:2324 nr:0 dw:16888 dr:5012 al:4 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:6552
# drbdadm disconnect all
# drbdadm connect test
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:7312 nr:0 dw:17644 dr:12312 al:4 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:7336 dw:9772 dr:3093 al:3 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
此时，两台drbd主机的状态已经恢复到一致状态，脑裂的问题解决完毕。
页:
[1]