DRBD+heartbeat 解决nfs单点故障
1.1 环境准备1.1.1 添加新磁盘，分区并格式化fdisk -uc /dev/sdbmkfs.ext4 /dev/sdb1dd if=/dev/zero of=/dev/sdb1bs=1M count=11.1.2 在hosts文件中添加以下记录cat /etc/host172.16.1.111 test01172.16.1.112 test02172.16.1.113 web1.1.3 关闭selinux和iptablessetenforce 0/etc/init.d/iptables stop1.1.4 测试约定主机名，系统版本及内核版本：
# uname -nr
test01 2.6.32-573.el6.x86_64
# cat /etc/redhat-release
CentOS release 6.7 (Final)
# uname -nrtest02 2.6.32-573.el6.x86_64# cat /etc/redhat-release CentOS release 6.7 (Final)
# uname -nrweb 2.6.32-573.el6.x86_64# cat /etc/redhat-release CentOS release 6.7 (Final)
测试IP：test01 172.16.1.111test02 172.16.1.112web 172.16.1.113虚拟IP 172.16.1.100
1.2 DRBD的安装1.2.1 配置DRBD默认官方源中没有drbd软件，所以使用elrepo源
# rpm --importhttps://www.elrepo.org/RPM-GPG-KEY-elrepo.org# rpm -Uvhhttp://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpmRetrieving http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpmPreparing... ########################################### 1:elrepo-release ########################################### # yum installdrbd84-utils kmod-drbd84 -y
# modprobe drbd# lsmod|grepdrbddrbd 3659310libcrc32c 12461 drbd
编辑配置文件
# vim/etc/drbd.conf# You can find an examplein/usr/share/doc/drbd.../drbd.conf.example#include"drbd.d/*.res";global{ usage-count no;}common{ syncer{ rate 200M; }}resource r1 { protocol C; startup{ wfc-timeout 120; degr-wfc-timeout 120; } disk { on-io-error detach; } net { timeout 60; connect-int 10; ping-int 10; max-buffers 2048; max-epoch-size 2048; cram-hmac-alg"sha1"; shared-secret"Mysql-abcD"; } on test01 { device /dev/drbd0; disk /dev/sdb1; address 172.16.1.31:6666; meta-disk internal; } on test02 { device /dev/drbd0; disk /dev/sdb1; address 172.16.1.41:6666; meta-disk internal; }}1.2.3 初始化设备# drbdadmcreate-md r1initializing activity logNOT initializing bitmapWriting meta data...New drbd meta data blocksuccessfully created.
# drbdadm up r1# cat /proc/drbdversion: 8.4.6(api:1/proto:86-101)GIT-hash:833d830e0152d1e457fa7856e71e11248ccf3f70 build by phil@Build64R6, 2015-04-0914:35:00 0: cs:Connected ro:Secondary/Secondaryds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0ua:0 ap:0 ep:1 wo:f oos:5241660现在两个节点都处于Secondary状态
将test01手动提升至primary。并将/dev/drbd0进行格式化以供挂在使用
# drbdadmprimary --force r1
# cat/proc/drbdversion: 8.4.6(api:1/proto:86-101)GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70build by root@test01, 2015-12-07 10:40:31 0: cs:Connected ro:Primary/Secondaryds:UpToDate/UpToDate C r----- ns:1047484 nr:0 dw:0 dr:1048148 al:0 bm:0lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
1.2.4 格式化虚拟分区# mkfs.ext4/dev/drbd0mke2fs 1.41.12 (17-May-2010)文件系统标签=操作系统:Linux.....Writing superblocks andfilesystem accounting information: 完成 This filesystem will beautomatically checked every 33 mounts or180 days, whichever comesfirst.Use tune2fs -c or -i to override.
# tune2fs -c -1/dev/drbd0tune2fs 1.41.12 (17-May-2010)Setting maximal mount count to-1
1.2.5 挂载测试# mkdir /data# mount -t ext4/dev/drbd0 /data# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda2 9.4G 1.5G7.4G17% /tmpfs 495M 0495M 0% /dev/shm/dev/sda1 190M36M145M20% /boot/dev/drbd0 991M 1.3M939M 1% /data
# touch/data/drbd.test# ls /data/drbd.test lost+found手动切换连个节点状态，并进行挂载测试
# umount /data# drbdadmsecondary r1# cat/proc/drbdversion: 8.4.6(api:1/proto:86-101)GIT-hash:833d830e0152d1e457fa7856e71e11248ccf3f70 build by root@test01, 2015-12-0710:40:31 0: cs:Connected ro:Secondary/Secondaryds:UpToDate/UpToDate C r----- ns:1080828 nr:0 dw:33344 dr:1048921 al:10bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
将test01手动提升至primary。并将/dev/drbd0进行格式化以供挂在使用
[root@test02drbd-utils-8.9.2]# drbdadm primary r1[root@test02drbd-utils-8.9.2]# cat /proc/drbdversion: 8.4.6(api:1/proto:86-101)GIT-hash:833d830e0152d1e457fa7856e71e11248ccf3f70 build by root@test02, 2015-12-0710:40:31 0: cs:Connected ro:Primary/Secondaryds:UpToDate/UpToDate C r----- ns:0 nr:1080828 dw:1080828 dr:664 al:0 bm:0lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# mkdir /data# mount -t ext4/dev/drbd0 /data# ls /data/drbd.test lost+found
# umount /data# drbdadmsecondary r1现在drbd两个节点已经配置成功，并手动测试成功。drbd的两个节点，同时时间只有primary状态的节点提供服务。
1.3 配置nfs服务test01 test02 web配置nfs的共享目录为/data ，test01 test02同时/dev/drbd0的挂载目录也是/data
1.3.1 将test01 drbd状态提升为primary并将/dev/drbd0挂载至/data,编辑/etc/exports文件，并本地测试。# cat/proc/drbd version: 8.4.6(api:1/proto:86-101)GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70build by root@test01, 2015-12-07 10:40:31 0: cs:Connected ro:Secondary/Secondaryds:UpToDate/UpToDate C r----- ns:1080828 nr:8 dw:33352 dr:1048921 al:10bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# drbdadmprimary r1# mount -t ext4/dev/drbd0 /data# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda2 9.4G 1.5G7.4G17% /tmpfs 495M 0 495M 0% /dev/shm/dev/sda1 190M36M145M20% /boot/dev/drbd0 991M 1.3M939M 1% /data
# cat/etc/exports /data172.16.1.0/24(rw,sync,all_squash)
#/etc/init.d/rpcbind start#/etc/init.d/nfs start# chown -R nfsnobody.nfsnobody/data# showmount -e172.16.1.111Export list for 172.16.1.111:/data 172.16.1.0/24
# mount -t nfs172.16.1.111:/data /mnt# ls /mntdrbd.test lost+found# touch/mnt/test01.test# ls /mntdrbd.test lost+foundtest01.test
1.3.2 将test01状态降为secondary，将test02状态提升为primary，并将/dev/drbd0挂载至/data，编辑/etc/exports文件，并本地测试# umount /mnt#/etc/init.d/nfs stop# umount /data# drbdadmsecondary r1# cat/proc/drbdversion: 8.4.6(api:1/proto:86-101)GIT-hash:833d830e0152d1e457fa7856e71e11248ccf3f70 build by root@test01, 2015-12-0710:40:31 0: cs:Connected ro:Secondary/Secondaryds:UpToDate/UpToDate C r----- ns:1080952 nr:8 dw:33476 dr:1049979 al:10bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# drbdadm primary r1# mount -t ext4/dev/drbd0 /data# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda2 9.4G 1.5G7.4G17% /tmpfs 495M 0 495M 0% /dev/shm/dev/sda1 190M36M145M20% /boot/dev/drbd0 991M 1.3M939M 1% /data
# cat/etc/exports /data172.1.6.0/24(rw,sync,all_squash)
#/etc/init.d/rpcbind start#/etc/init.d/nfs start# chown -Rnfsnobody.nfsnobody /data# showmount -e172.16.1.112Export list for 172.16.1.112:/data 172.16.1.0/24
# mount -t nfs172.16.1.112:/data /mnt# ls /mntdrbd.test lost+foundtest01.test# touch/mnt/test02.test# ls /mntdrbd.test lost+foundtest01.testtest02.test
# umount /mnt#/etc/init.d/nfs stop# umount /data# drbdadmsecondary r1version: 8.4.6(api:1/proto:86-101)GIT-hash:833d830e0152d1e457fa7856e71e11248ccf3f70 build by root@test02, 2015-12-0710:40:31 0: cs:Connected ro:Secondary/Secondaryds:UpToDate/UpToDate C r----- ns:124 nr:1080952 dw:1081076 dr:1726 al:2bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0至此：drbd nfs手动挂载测试成功
1.4 配置heartbeat配置heartbeat以达到nfs主节点失效自动将所有资源切换到备用节点
1.4.1 两台nfs服务器上安装heartbeat添加epel扩展源。官方yum源没有heartbeatyum -y install epel-releaseyum install heartbeat -y
1.4.2 将两台服务器默认配置文件拷贝到/etc/ha.d/并做更改# rpm -qdheartbeat|grep doc/usr/share/doc/heartbeat-3.0.4/AUTHORS/usr/share/doc/heartbeat-3.0.4/COPYING/usr/share/doc/heartbeat-3.0.4/COPYING.LGPL/usr/share/doc/heartbeat-3.0.4/ChangeLog/usr/share/doc/heartbeat-3.0.4/README/usr/share/doc/heartbeat-3.0.4/apphbd.cf/usr/share/doc/heartbeat-3.0.4/authkeys/usr/share/doc/heartbeat-3.0.4/ha.cf/usr/share/doc/heartbeat-3.0.4/haresources
# cp/usr/share/doc/heartbeat-3.0.4/authkeys /usr/share/doc/heartbeat-3.0.4/ha.cf/usr/share/doc/heartbeat-3.0.4/haresources /etc/ha.d/
# egrep -v"#|^$" /etc/ha.d/ha.cflogfile /var/log/ha-loglogfacility local0keepalive 2deadtime 30warntime 10initdead 60mcast eth0 225.0.0.1 694 1 0auto_failback onnode test01node test02
# egrep -v"#|^$" /etc/ha.d/authkeysauth 11 sha1 liyanan# chmod600/etc/ha.d/authkeys# ll/etc/ha.d/authkeys -rw------- 1 root root 647 12月7 15:13 /etc/ha.d/authkeys #echo " test01drbddisk::r1 Filesystem::/dev/drbd0::/data::ext4 nfsIPaddr::172.16.1.100/24/eth1" >>/etc/ha.d/haresources将附件的脚本加入到/etc/ha.d/resource.d/ 下并赋予可执行权限。1.4.3 启动两台服务器heartbeat#/etc/init.d/heartbeat start#/etc/init.d/heartbeat statusheartbeat OK is running on test01 ...
1.4.4 测试heartbeat
全部启动heartbeat服务
#/etc/init.d/heartbeat statusheartbeat OK is running on test01 ...#/etc/init.d/heartbeat statusheartbeat OK is running on test02 ...
检查test01挂载
# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda2 9.4G 1.6G7.3G18% /tmpfs 495M 0 495M 0% /dev/shm/dev/sda1 190M36M145M20% /boot/dev/drbd0 991M 1.3M939M 1% /data检查test02挂载
# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda2 9.4G 1.6G7.3G18% /tmpfs 495M 0 495M 0% /dev/shm/dev/sda1 190M36M145M20% /boot关闭test01的heartbeat服务后再检查挂载
#/etc/init.d/heartbeat stopStopping High-Availabilityservices: Done.
# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda2 9.4G 1.6G7.3G18% /tmpfs 495M 0495M 0% /dev/shm/dev/sda1 190M36M145M20% /boot
再次检查test02挂载
# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda2 9.4G 1.6G7.3G18% /tmpfs 495M 0 495M 0% /dev/shm/dev/sda1 190M36M145M20% /boot/dev/drbd0 991M 1.3M939M 1% /data再次启动test01的heartbeat服务，并检查挂载
#/etc/init.d/heartbeat startStarting High-Availabilityservices: Done.
# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda2 9.4G 1.6G7.3G18% /tmpfs 495M 0 495M 0% /dev/shm/dev/sda1 190M36M145M20% /boot/dev/drbd0 991M 1.3M939M 1% /data
再次检查test02挂载
# df -hFilesystem SizeUsed Avail Use% Mounted on/dev/sda2 9.4G 1.6G7.3G18% /tmpfs 495M 0 495M 0% /dev/shm/dev/sda1 190M36M145M20% /boot
经过循环测试，证明heartbeat服务已经成功接管drbd nfs VIP服务
可通过VIP向外部提供nfs挂载解决nfs单点故障。
终极测试，循环写入数据测试丢包率
测试方式：由test01为主，提供nfs服务、test02为备，接管nfs服务。web挂载test01提供的nfs共享目录。通过间隔 1S 连续在web的挂载点上写入数据，期间手动关闭test01的heartbeat服务，造成意外宕机的实验效果，等脚本执行完毕后，在web端查看写入的数据，以便检测丢包率。
检查挂载情况及虚拟IP
# df -h
Filesystem SizeUsed Avail Use% Mounted on
/dev/sda2 9.4G1.6G7.4G18% /
tmpfs 242M 0242M 0% /dev/shm
/dev/sda1 190M 36M145M20% /boot
/dev/drbd0 4.8G 11M4.6G 1% /data
# ip add |grep 172.16
inet 172.16.1.111/24 brd 172.16.1.255 scope global eth1
inet 172.16.1.100/24 brd 172.16.1.255 scope global secondary eth1
# df -h
Filesystem SizeUsed Avail Use% Mounted on
/dev/sda2 9.4G1.6G7.4G18% /
tmpfs 242M 0242M 0% /dev/shm
/dev/sda1 190M 36M145M20% /boot
# ip add |grep 172.16
inet 172.16.1.112/24 brd 172.16.1.255 scope global eth1
# df -h
Filesystem SizeUsed Avail Use% Mounted on
/dev/sda2 9.4G2.1G6.9G24% /
tmpfs 242M 0242M 0% /dev/shm
/dev/sda1 190M 36M145M20% /boot
172.16.1.100:/data4.8G 11M4.6G 1% /data/www/
测试脚本：
for ((n=1;n<30;n++))
do
touch /data/www/$n
sleep 1
done
期间关闭test01的heartbeat服务，并检查最终测试结果
# ls /data/www/
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 18 19 20 21 22 23 24 25 26 27 28 29
测试结果表明数据丢包率为0，试验成功
页:
[1]