Heartbeat+DRBD+NFS高可用

trytete

一、环境
系统:     CentOS 6.4x64 最小化安装
node1:     192.168.1.13
node2:     192.168.1.14
vip:      192.168.1.15
nfs:      192.168.1.10
二、基础配置
node1和node2的操作一样

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

#关闭iptables和selinux [iyunv@node1 ~]# getenforce Disabled                        #确保这项是正确的 [iyunv@node1 ~]# service iptables stop #配置本地hosts解析 [iyunv@node1 ~]# echo "192.168.1.13  node1" >>/etc/hosts [iyunv@node1 ~]# echo "192.168.1.14  node2" >>/etc/hosts [iyunv@node1 ~]# cat /etc/hosts 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.13  node1 192.168.1.14  node2 #配置epel源 [iyunv@node1 ~]# rpm -ivh http://dl.fedoraproject.org/pub/ ... ease-6-8.noarch.rpm [iyunv@node1 ~]# sed -i 's@#b@b@g' /etc/yum.repos.d/epel.repo [iyunv@node1 ~]# sed  -i 's@mirrorlist@#mirrorlist@g' /etc/yum.repos.d/epel.repo #同步时间 [iyunv@node1 ~]# yum install ntp -y [iyunv@node1 ~]# echo "*/10 * * * * /usr/sbin/ntpdate asia.pool.ntp.org  &>/dev/null" >/var/spool/cron/root [iyunv@node1 ~]# ntpdate asia.pool.ntp.org 21 Jun 17:32:45 ntpdate[1561]: step time server 211.233.40.78 offset -158.552839 sec [iyunv@node1 ~]# hwclock -w #配置ssh互信 [iyunv@node1 ~]# ssh-keygen [iyunv@node1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@node2

三、安装配置heartbeat
(1).安装heartbeat

1 2	#在ha-node1和ha-node2都执行安装操作 [iyunv@node1 ~]# yum install heartbeat -y

(2).配置ha.cf

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

[iyunv@node1 ~]# cd /usr/share/doc/heartbeat-3.0.4/ [iyunv@node1 heartbeat-3.0.4]# cp authkeys ha.cf haresources /etc/ha.d/ [iyunv@node1 heartbeat-3.0.4]# cd /etc/ha.d/ [iyunv@node1 ha.d]# ls authkeys  ha.cf  harc  haresources  rc.d  README.config  resource.d  shellfuncs [iyunv@node1 ha.d]# egrep -v "^$|^#" /etc/ha.d/ha.cf logfile /var/log/ha-log logfacility local1 keepalive 2 deadtime 30 warntime 10 initdead 120 mcast eth0 225.0.10.1 694 1 0 auto_failback on node node1 node node2 crm no

(3).配置authkeys

1 2 3 4 5 6 7 8 9 10

[iyunv@node1 ha.d]# dd if=/dev/random bs=512 count=1 | openssl md5 0+1 records in 0+1 records out 21 bytes (21 B) copied, 3.1278e-05 s, 671 kB/s (stdin)= 4206bd8388c16292bc03710a0c747f59 [iyunv@node1 ha.d]# grep -v ^# /etc/ha.d/authkeys auth 1 1 md5 4206bd8388c16292bc03710a0c747f59 #将认证文件权限修改成600 [iyunv@node1 ~]# chmod 600 /etc/ha.d/authkeys

(4).配置haresource

1 2	[iyunv@node1 ha.d]# grep -v ^# /etc/ha.d/haresources node1 IPaddr::192.168.1.15/24/eth0

(5).启动heartbeat

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

[iyunv@node1 ha.d]# scp authkeys haresources  ha.cf node2:/etc/ha.d/ #node1启动服务 [iyunv@node1 ~]# /etc/init.d/heartbeat start Starting High-Availability services: INFO:  Resource is stopped Done. [iyunv@node1 ~]# chkconfig heartbeat off #说明：关闭开机自启动，当服务器重启时，需要人工去启动 #node2启动服务 [iyunv@node2 ~]# /etc/init.d/heartbeat start #查看结果 [iyunv@node1 ~]# ip a |grep eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 192.168.1.13/24 brd 192.168.1.255 scope global eth0     inet 192.168.1.15/24 brd 192.168.1.255 scope global secondary eth0     #vip在主节点上 [iyunv@node2 ~]# ip a |grep eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 192.168.1.14/24 brd 192.168.1.255 scope global eth0    #备节点上没有vip

(6).测试heartbeat
正常状态

1 2 3 4 5 6 7 8 9 10

#node1信息 [iyunv@node1 ~]# ip a |grep eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 192.168.1.13/24 brd 192.168.1.255 scope global eth0     inet 192.168.1.15/24 brd 192.168.1.255 scope global secondary eth0     #vip在主节点上 #node2信息 [iyunv@node2 ~]# ip a |grep eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 192.168.1.14/24 brd 192.168.1.255 scope global eth0    #备节点上没有vip

模拟主节点宕机后的状态信息

1 2 3 4 5 6 7 8 9 10 11 12

#在主节点node1停止heartbeat服务 [iyunv@node1 ~]# /etc/init.d/heartbeat stop Stopping High-Availability services: Done. [iyunv@node1 ~]# ip a |grep eth0               #主节点的heartbeat服务停止后，vip资源被抢走 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 192.168.1.13/24 brd 192.168.1.255 scope global eth0       #在备节点node2查看资源 [iyunv@node2 ~]# ip a |grep eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 192.168.1.14/24 brd 192.168.1.255 scope global eth0     inet 192.168.1.15/24 brd 192.168.1.255 scope global secondary eth0

恢复主节点的heartbeat服务

1 2 3 4 5 6 7 8 9 10 11 12 13 14

[iyunv@node1 ~]# /etc/init.d/heartbeat start Starting High-Availability services: INFO:  Resource is stopped Done. #主节点的heartbeat服务恢复后，将资源接管回来了 [iyunv@node1 ~]# ip a |grep eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 192.168.1.13/24 brd 192.168.1.255 scope global eth0     inet 192.168.1.15/24 brd 192.168.1.255 scope global secondary eth0       #查看备节点 [iyunv@node2 ~]# ip a |grep eth0         #vip资源已移除 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 192.168.1.14/24 brd 192.168.1.255 scope global eth0

四、安装部署DRBD

(1).对硬盘进行分区，node1和node2的操作一样

1 2 3 4 5 6 7 8 9

[iyunv@node1 ~]# fdisk /dev/sdb #说明：/dev/sdb分成2个分区/dev/sdb1和/dev/sdb2，/dev/sdb1=19G [iyunv@node1 ~]# partprobe /dev/sdb #对分区进行格式化 [iyunv@node1 ~]# mkfs.ext4 /dev/sdb1 说明：sdb2分区为meta data分区，不需要格式化操作 [iyunv@node1 ~]# tune2fs -c -1 /dev/sdb1 说明：设置最大挂载数为-1,关闭强制检查挂载次数限制

(2).安装DRBD

由于我们的系统是CentOS6.4的，所以我们还需要安装内核模块，版本需要和uname -r保持一致，安装包我们从系统安装软件中提取出来，过程略。node1和node2的安装过程一样，这里只给出node1的安装过程

1 2 3

#安装内核文件 [iyunv@node1 ~]# rpm -ivh kernel-devel-2.6.32-358.el6.x86_64.rpm kernel-headers-2.6.32-358.el6.x86_64.rpm [iyunv@node1 ~]# yum install drbd84 kmod-drbd84 -y

(3).配置DRBD

a.修改全局配置文件

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

[iyunv@node1 ~]# egrep -v "^$|^#|^[[:space:]]+#" /etc/drbd.d/global_common.conf global {     usage-count no; } common {     protocol C;     handlers {     }     startup {     }     options {     }     disk {                 on-io-error detach;         no-disk-flushes;         no-md-flushes;         rate 200M;     }     net {         sndbuf-size 512k;             max-buffers     8000;             unplug-watermark   1024;             max-epoch-size  8000;             cram-hmac-alg "sha1";             shared-secret "weyee2014";             after-sb-0pri disconnect;             after-sb-1pri disconnect;             after-sb-2pri disconnect;             rr-conflict disconnect;     } }

b.增加资源

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

[iyunv@node1 ~]# cat /etc/drbd.d/nfsdata.res resource nfsdata {     on node1 {         device /dev/drbd1;         disk    /dev/sdb1;         address 192.168.1.13:7789;         meta-disk /dev/sdb2 [0];     }     on node2 {         device /dev/drbd1;         disk  /dev/sdb1;         address 192.168.1.14:7789;         meta-disk /dev/sdb2 [0];        } }

c.将配置文件复制到node2上，重启系统加载drbd模块，初始化meta数据

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

[iyunv@node1 ~]# scp global_common.conf nfsdata.res node2:/etc/drbd.d/ [iyunv@node1 ~]# depmod [iyunv@node1 ~]# modprobe drbd [iyunv@node1 ~]# lsmod |grep drbd drbd                  365931  0 libcrc32c               1246  1 drbd #在node1初始化meta数据 [iyunv@node1 ~]# drbdadm create-md nfsdata initializing activity log NOT initializing bitmap Writing meta data... New drbd meta data block successfully created. #在node2上加载模块，初始化meta数据 [iyunv@node2 ~]# depmod [iyunv@node2 ~]# modprobe drbd [iyunv@node2 ~]# lsmod |grep drbd drbd                  365931  0 libcrc32c               1246  1 drbd [iyunv@node2 ~]# drbdadm create-md nfsdata initializing activity log NOT initializing bitmap Writing meta data... New drbd meta data block successfully created.

d.在node1和node2上启动drbd

1

#node1操作