DRBD数据镜像主备节点同步数据

231321

一、DRBD的安装

1、从DRBD官网下载源码包http://drbd.linbit.com/en/
2、编译安装
[iyunv@web1 drbd-8.4.6]# yum install kernel kernel-devel kernel-headers flex
[iyunv@web1 ~]# reboot
[iyunv@web1 ~]# ll -d /usr/src/kernels/2.6.32-573.18.1.el6.x86_64
drwxr-xr-x 22 root root 4096 Mar 19 03:53 /usr/src/kernels/2.6.32-573.18.1.el6.x86_64

[iyunv@web1 ~]# tar zxf drbd-8.4.6.tar.gz
[iyunv@web1 ~]# cd drbd-8.4.6
[iyunv@web1 drbd-8.4.6]# make KDIR=/usr/src/kernels/2.6.32-573.18.1.el6.x86_64
[iyunv@web1 drbd-8.4.6]# make install

3、查看drbd.ko的路径

[iyunv@web1 drbd-8.4.6]# modprobe -l |grep -i drbd
updates/drbd.ko
[iyunv@web1 drbd-8.4.6]# ls /lib/modules/2.6.32-573.18.1.el6.x86_64/updates/
drbd.ko

4、加载drbd模块到内核
[iyunv@web1 drbd-8.4.6]# modprobe drbd

5、检查drbd是否安装成功
[iyunv@web1 drbd-8.4.6]# lsmod | grep -i drbd
drbd                  365259  0
libcrc32c               1246  2 drbd,ip_vs

二、安装drbd-utils
1、下载源码包http://www.drbd.org/download/drbd/utils/drbd-utils-8.9.6.tar.gz，并编译安装
[iyunv@web1 ~]# tar zxf drbd-utils-8.9.6.tar.gz
[iyunv@web1 ~]# cd drbd-utils-8.9.6
[iyunv@web1 drbd-utils-8.9.6]# ./configure --prefix=/usr/local/drbd-utils-8.9.6 --without-83support
[iyunv@web1 drbd-utils-8.9.6]# make

执行make的时候报了一个错：

1 2 3 4 5 6 7 8

--xinclude --stringparam variablelist.term.break.after 1 \     http://docbook.sourceforge.net/r ... anpages/docbook.xsl drbdsetup.xml error : Operation in progress warning: failed to load external entity "http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl" cannot parse http://docbook.sourceforge.net/r ... anpages/docbook.xsl make[1]: *** [drbdsetup.8] Error 4 make[1]: Leaving directory `/root/drbd-utils-8.9.6/documentation/v9' make: *** [doc] Error 2

2、安装docbook-style-xsl

1	[iyunv@web1 drbd-utils-8.9.6]# yum -y install docbook-style-xsl

重新make

1 2	[iyunv@web1 drbd-utils-8.9.6]# make [iyunv@web1 drbd-utils-8.9.6]# make install

1 2	[iyunv@web1 drbd-utils-8.9.6]# ls /usr/local/drbd-utils/ etc  lib  sbin  share  var

3、链接启动程序，并设置drbd开机启动

1 2 3

[iyunv@web1 drbd-utils-8.9.6]# ln -s /usr/local/drbd-utils/etc/rc.d/init.d/drbd /etc/init.d/ [iyunv@web1 drbd-utils-8.9.6]# chkconfig --add drbd [iyunv@web1 drbd-utils-8.9.6]# chkconfig drbd on

4、查看drbd.conf配置文件及包含的配置文件

1 2 3 4 5

[iyunv@web1 drbd-utils-8.9.6]# cat /usr/local/drbd-utils/etc/drbd.conf # You can find an example in  /usr/share/doc/drbd.../drbd.conf.example include "drbd.d/global_common.conf"; include "drbd.d/*.res";

5、查看global_common.conf配置文件默认配置

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

[iyunv@web1 drbd-utils-8.9.6]# cat /usr/local/drbd-utils/etc/drbd.d/global_common.confnf # DRBD is the result of over a decade of development by LINBIT. # In case you need professional services for DRBD or have # feature requests visit http://www.linbit.com global {     usage-count yes;     # minor-count dialog-refresh disable-ip-verification     # cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600; } common {     handlers {         # These are EXAMPLE handlers only.         # They may have severe implications,         # like hard resetting the node under certain circumstances.         # Be careful when chosing your poison.         # pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";         # pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";         # local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";         # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";         # split-brain "/usr/lib/drbd/notify-split-brain.sh root";         # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";         # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";         # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;     }     startup {         # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb     }     options {         # cpu-mask on-no-data-accessible     }     disk {         # size on-io-error fencing disk-barrier disk-flushes         # disk-drain md-flushes resync-rate resync-after al-extents                 # c-plan-ahead c-delay-target c-fill-target c-max-rate                 # c-min-rate disk-timeout     }     net {         # protocol timeout max-epoch-size max-buffers unplug-watermark         # connect-int ping-int sndbuf-size rcvbuf-size ko-count         # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri         # after-sb-1pri after-sb-2pri always-asbp rr-conflict         # ping-timeout data-integrity-alg tcp-cork on-congestion         # congestion-fill congestion-extents csums-alg verify-alg         # use-rle     } }

    至此，DRBD安装完成


三、主备配置实例
1、环境说明

主机角色	IP地址	镜像磁盘分区	操作系统
master（主用节点）-web1	192.168.1.250	/dev/sdb1	CentOS6.7_64
slave（备用节点） -web2	192.168.1.209	/dev/sdb1	CentOS6.7_64

2、分别在主备节点安装drbd
3、配置drbd.conf文件
global_common.conf配置文件包含DRBD的global和common配置部分，下面将其全部写到
/usr/local/drbd-utils/etc/drbd.conf配置文件中

1	[iyunv@web1 ~]# cat /usr/local/drbd-utils/etc/drbd.d/global_common.conf >> /usr/local/drbd-utils/etc/drbd.conf

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57

[iyunv@web1 ~]# vim /usr/local/drbd-utils/etc/drbd.conf global {         usage-count no; } common {         syncer { rate 200M; } } resource r0 {         protocol C;         handlers {                 pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";                 pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";                 local-io-error "echo o > /proc/sysrq-trigger ; halt -f";                 fence-peer "/usr/lib64/heartbeat/drbd-peer-outdater -t 5";                 pri-lost "echo pri-lost. Have a look at the log files. | mail -s 'DRBD Alert' reboot";                 split-brain "/usr/lib/drbd/notify-split-brain.sh root";                 out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";                 before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";                 after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;         }         startup {         wfc-timeout 120;         degr-wfc-timeout 120;         }         disk {         on-io-error detach;         fencing resource-only;         }         net {         cram-hmac-alg "sha1";         shared-secret "MySQL-HA";         }         device /dev/drbd0;         on master {         disk /dev/sdb1;         address 192.168.1.250:7788;         meta-disk internal;         }         on slave {         disk /dev/sdb1;         address 192.168.1.209:7788;         meta-disk internal;         } }

4、将drbd.conf 配置文件同时复制到slave节点上

5、在主备节点上的sdb1分区上创建提供DRBD记录信息的数据块

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

[iyunv@web1 ~]# drbdadm create-md r0 WARN:   You are using the 'drbd-peer-outdater' as fence-peer program.   If you use that mechanism the dopd heartbeat plugin program needs   to be able to call drbdsetup and drbdmeta with root privileges.   You need to fix this with these commands:   chgrp haclient /lib/drbd/drbdsetup-84   chmod o-x /lib/drbd/drbdsetup-84   chmod u+s /lib/drbd/drbdsetup-84   chgrp haclient /sbin/drbdmeta   chmod o-x /sbin/drbdmeta   chmod u+s /sbin/drbdmeta initializing activity log NOT initializing bitmap Writing meta data... New drbd meta data block successfully created.

在slave上执行drbdadm create-md r0时报错

1 2 3 4 5 6 7 8 9 10

Device size would be truncated, which would corrupt data and result in 'access beyond end of device' errors. You need to either    * use external meta data (recommended)    * shrink that filesystem first    * zero out the device (destroy the filesystem) Operation refused. Command 'drbdmeta 0 v08 /dev/sdb1 internal create-md' terminated with exit code 40

解决：在/dev/sdb1写入一些数据
[iyunv@web2 ~]# dd if=/dev/zero of=/dev/sdb1 bs=1M count=100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

[iyunv@web2 ~]# drbdadm create-md r0 WARN:   You are using the 'drbd-peer-outdater' as fence-peer program.   If you use that mechanism the dopd heartbeat plugin program needs   to be able to call drbdsetup and drbdmeta with root privileges.   You need to fix this with these commands:   chgrp haclient /lib/drbd/drbdsetup-84   chmod o-x /lib/drbd/drbdsetup-84   chmod u+s /lib/drbd/drbdsetup-84   chgrp haclient /sbin/drbdmeta   chmod o-x /sbin/drbdmeta   chmod u+s /sbin/drbdmeta initializing activity log NOT initializing bitmap Writing meta data... New drbd meta data block successfully created.

6、在主备节点上分别启动DRBD

1 2	[iyunv@web1 ~]# service drbd start [iyunv@web2 ~]# service drbd start

1 2 3 4 5 6 7 8

[iyunv@web2 ~]# ps -ef |grep drbd root       1720      2  0 17:33 ?        00:00:00 [drbd-reissue] root       1727      2  0 17:33 ?        00:00:00 [drbd_submit] root       1994      2  0 17:47 ?        00:00:00 [drbd_w_r0] root       2007      2  0 17:47 ?        00:00:00 [drbd_r_r0] root       2012      2  0 17:47 ?        00:00:00 [drbd_a_r0] root       2013      2  0 17:47 ?        00:00:00 [drbd_ack_sender] root       2018   1868  0 17:47 pts/0    00:00:00 grep drbd

7、在任意节点查看节点状态

1 2 3 4 5 6

[iyunv@web2 ~]# service drbd status drbd driver loaded OK; device status: version: 8.4.6 (api:1/proto:86-101) GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70 build by root@web2, 2016-03-19 06:03:56 m:res  cs         ro                   ds                         p  mounted  fstype 0:r0   Connected  Secondary/Secondary  Inconsistent/Inconsistent  C

1 2 3 4 5

[iyunv@web1 ~]# cat /proc/drbd version: 8.4.6 (api:1/proto:86-101) GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70 build by root@web1, 2016-03-19 04:35:38 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----     ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:2096348

• ro表示角色信息，第一次启动drbd时，两个节点默认都处于Secondary状态
  
• ds表示磁盘状态信息，Inconsistent/Inconsistent表示两个节点的磁盘数据处于不一致状态
  
• ns表示网络发送的数据包信息
  
• dw表示磁盘写信息
  
• dr表示磁盘读信息
  
  
  

8、设置主节点，在master上执行drbdadm -- --overwrite-data-of-peer primary all

1 2 3 4 5 6 7 8 9 10 11 12 13 14

[iyunv@web1 ~]# drbdadm -- --overwrite-data-of-peer primary all WARN:   You are using the 'drbd-peer-outdater' as fence-peer program.   If you use that mechanism the dopd heartbeat plugin program needs   to be able to call drbdsetup and drbdmeta with root privileges.   You need to fix this with these commands:   chgrp haclient /lib/drbd/drbdsetup-84   chmod o-x /lib/drbd/drbdsetup-84   chmod u+s /lib/drbd/drbdsetup-84   chgrp haclient /sbin/drbdmeta   chmod o-x /sbin/drbdmeta   chmod u+s /sbin/drbdmeta

9、设置过程中，需要重启drbd服务

1 2 3 4 5 6

[iyunv@web1 ~]# /etc/init.d/drbd restart [iyunv@web2 ~]# /etc/init.d/drbd restart ##将web1设置为主节点 [iyunv@web1 ~]# drbdadm -- --overwrite-data-of-peer primary all

10、第一次执行完成设置主节点命令后，如果需要设置哪个是主节点，可以使用以下命令设置

1	[iyunv@web1 ~]# /sbin/drbdadm primary r0 或 /sbin/drbdadm primary all

11、执行设置主节点命令后，查看两台机器的数据是否同步完成

1 2 3 4 5 6 7

[iyunv@web1 ~]# cat /proc/drbd version: 8.4.6 (api:1/proto:86-101) GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70 build by root@web1, 2016-03-19 04:35:38 0: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r-----     ns:2061312 nr:0 dw:0 dr:2061312 al:0 bm:0 lo:0 pe:4 ua:0 ap:0 ep:1 wo:f oos:39132     [==================>.] sync'ed: 98.3% (39132/2096348)K     finish: 0:00:01 speed: 38,348 (38,812) K/sec

1 2 3 4 5

[iyunv@web1 ~]# cat /proc/drbd version: 8.4.6 (api:1/proto:86-101) GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70 build by root@web1, 2016-03-19 04:35:38 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----     ns:2096348 nr:0 dw:0 dr:2096348 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

由这一行输出可以看到同步已经完成cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate


12、挂载DRBD磁盘

• 由于mount操作只能在主节点上进行，因此只有设置了主节点后才能格式化drbd磁盘分区,同时，两个节点中，同一时刻只能有一台处于primary状态，另一台处于secondary状态
  
  

13、查看drbd分区

1 2 3 4 5 6 7

[iyunv@web1 ~]# ll /dev/drbd* brw-rw---- 1 root disk 147, 0 Mar 19 19:06 /dev/drbd0 /dev/drbd: total 0 drwxr-xr-x 2 root root 60 Mar 19 19:06 by-disk drwxr-xr-x 2 root root 60 Mar 19 19:06 by-res

14、格式化drbd磁盘，并挂载在主节点上

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

[iyunv@web1 ~]# mkfs.ext4 /dev/drbd0 mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 131072 inodes, 524087 blocks 26204 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=536870912 16 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks:     32768, 98304, 163840, 229376, 294912 Writing inode tables: done                            Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 32 mounts or 180 days, whichever comes first.  Use tune2fs -c or -i to override.

15、挂载到/data/目录，创建测试文件

1 2 3 4 5

[iyunv@web1 ~]# mount /dev/drbd0 /data/ [iyunv@web1 ~]# ls /data/ lost+found [iyunv@web1 ~]# mkdir /data/www [iyunv@web1 ~]# dd if=/dev/zero of=/data/testdrbd.file bs=10M count=20

16、测试DRBD数据镜像，为了保证数据一致性，需要先停止备节点的drbd服务

1 2 3 4 5 6 7 8 9

[iyunv@web2 ~]# /etc/init.d/drbd stop Stopping all DRBD resources: . [iyunv@web2 ~]# mount /dev/sdb1 /mnt/ [iyunv@web2 ~]# ll !$ ll /mnt/ total 204820 drwx------ 2 root root     16384 Mar 19 19:34 lost+found -rw-r--r-- 1 root root 209715200 Mar 19 19:42 testdrbd.file drwxr-xr-x 2 root root      4096 Mar 19 19:37 www

• 由上面的输出可以看到，数据已经同步到slave节点上
  
• 备用节点切换为主节点后，需要重新挂载drbd磁盘到切换后的节点上