红帽企业集群和存储管理之DRBD+Heartbeat+NFS实现详解
红帽企业集群和存储管理之DRBD+Heartbeat+NFS实现详解
案例应用背景
本实验部署DRBD + HEARDBEAT + NFS 环境，建立一个高可用(HA)的文件服务器集群。在方案中，通过DRBD保证了服务器数据的完整性和一致性。DRBD类似于一个网络RAID-1功能。当你将数据写入本地文件系统时，数据还将会被发送到网络中另一台主机上，以相同的形式记录在一个另文件系统中。主节点与备节点的数据可以保证实时相互同步。当本地主服务器出现故障时，备份服务器上还会保留有一份相同的数据，可以继续使用。在高可用(HA)中使用DRBD功能，可以代替使用一个共享盘阵。因为数据同时存在于本地主服务器和备份服务器上。切换时，远程主机只要使用它上面的那份备份数据，就可以继续提供主服务器上相同的服务，并且client用户对主服务器的故障无感知。
案例应用简化拓扑图
软件包下载地址：http://down.运维网.com/data/402474
http://blog.运维网.com/attachment/201205/210031635.png
案例应用具体实现步骤：
一.网络的基本配置
1.1node1基本配置&新建磁盘
1.1.1查看系统信息，同步时间
#uname -rv
2.6.18-164.el5 #1SMP Tue Aug 18 15:51:54 EDT 2009
# cat/etc/redhat-release
Red Hat EnterpriseLinux Server release 5.4 (Tikanga)
#hwclock -s
#date
Wed Feb8 13:55:50 CST 2012
1.1.2查看主机名称，修改查看ip地址
#hostname
node1.junjie.com
# cat/etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1.junjie.com
#setup
http://blog.运维网.com/attachment/201205/210056704.png
#service network restart
Shutting downinterface eth0:
Shutting downloopback interface:
Bringing up loopbackinterface:
Bringing upinterface eth0:
#ifconfig eth0
eth0 Link encap:EthernetHWaddr 00:0C:29:AE:83:D1
inet addr:192.168.101.211Bcast:192.168.101.255Mask:255.255.255.0
1.1.3配置/etc/hosts文件（就不用dns了）
# echo"192.168.101.211 node1.junjie.com node1">>/etc/hosts
# echo"192.168.101.212 node2.junjie.com node2">>/etc/hosts
1.1.4构建一个新的磁盘空间有利于实现DRBD技术
#fdisk -l
Disk /dev/sda: 21.4GB, 21474836480 bytes
255 heads, 63sectors/track, 2610 cylinders
Units = cylinders of16065 * 512 = 8225280 bytes
DeviceBoot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1318 10482412+83Linux
/dev/sda3 1319 1579 2096482+ 82Linux swap / Solaris
#fdisk /dev/sda
p/n/p//+1000M/p/w
#fdisk -l
Disk /dev/sda: 21.4GB, 21474836480 bytes
255 heads, 63sectors/track, 2610 cylinders
Units = cylinders of16065 * 512 = 8225280 bytes
DeviceBoot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83Linux
/dev/sda2 14 1318 10482412+83Linux
/dev/sda3 1319 1579 2096482+ 82Linux swap /Solaris
/dev/sda4 1580 1702 987997+ 83 Linux
#partprobe /dev/sda
# cat/proc/partitions
major minor#blocks name
8 0 20971520 sda
8 1 104391 sda1
8 2 10482412 sda2
8 3 2096482 sda3
8 4 987997 sda4
1.2 node2基本配置&新建磁盘
1.2.1查看系统信息，同步时间
#uname -rv
2.6.18-164.el5 #1SMP Tue Aug 18 15:51:54 EDT 2009
# cat/etc/redhat-release
Red Hat EnterpriseLinux Server release 5.4 (Tikanga)
#hwclock -s
#date
Wed Feb8 14:02:22 CST 2012
1.2.2查看主机名称，修改查看ip地址
#hostname
node2.junjie.com
# cat/etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node2.junjie.com
#setup
http://blog.运维网.com/attachment/201205/210119285.png
#service network restart
Shutting downinterface eth0:
Shutting downloopback interface:
Bringing up loopbackinterface:
Bringing upinterface eth0:
#ifconfig eth0
eth0 Link encap:EthernetHWaddr 00:0C:29:D1:D4:32
inet addr:192.168.101.212Bcast:192.168.101.255 Mask:255.255.255.0
1.2.3配置/etc/hosts文件（就不用dns了）
# echo"192.168.101.211 node1.junjie.com node1">>/etc/hosts
# echo"192.168.101.212 node2.junjie.com node2">>/etc/hosts
1.2.4构建一个新的磁盘空间有利于实现DRBD技术
#fdisk -l
Disk /dev/sda: 21.4GB, 21474836480 bytes
255 heads, 63sectors/track, 2610 cylinders
Units = cylinders of16065 * 512 = 8225280 bytes
DeviceBoot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1318 10482412+83Linux
/dev/sda3 1319 1579 2096482+ 82Linux swap /Solaris
#fdisk /dev/sda
p/n/p//+1000M/p/w
#fdisk -l
Disk /dev/sda: 21.4GB, 21474836480 bytes
255 heads, 63sectors/track, 2610 cylinders
Units = cylinders of16065 * 512 = 8225280 bytes
DeviceBoot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1318 10482412+83Linux
/dev/sda3 1319 1579 2096482+ 82Linux swap /Solaris
/dev/sda4 1580 1702 987997+ 83 Linux
#partprobe /dev/sda
# cat/proc/partitions
major minor#blocks name
8 0 20971520 sda
8 1 104391 sda1
8 2 10482412 sda2
8 3 2096482 sda3
8 4 987997 sda4
1.3 在node1和node2上配置ssh密钥信息，
有利于以后在一个节点对另一节点直接操作。
1.3.1 在node1上配置ssh密钥信息
#ssh-keygen -t rsa
#ssh-copy-id -i .ssh/id_rsa.pubroot@node2.junjie.com
1.3.2 在node2上配置ssh密钥信息
#ssh-keygen -t rsa
#ssh-copy-id -i .ssh/id_rsa.pubroot@node2.junjie.com
二、DRBD安装配置步骤
在node1和node2做以下操作：
我下载的软件包是：（我放在/root/下了）
drbd83-8.3.8-1.el5.centos.i386.rpm
kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
2.1、安装DRBD套件
# rpm-ivh drbd83-8.3.8-1.el5.centos.i386.rpm
# rpm-ivh kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
# rpm-ivh drbd83-8.3.8-1.el5.centos.i386.rpm
# rpm-ivh kmod-drbd83-8.3.8-1.el5.centos.i686.rpm
2.2、加载DRBD模块
#modprobe drbd
#lsmod | grep drbd
drbd 2285280
#
#modprobe drbd
#lsmod | grep drbd
drbd 2285280
#
2.3、修改配置文件
drbd.conf配置文件DRBD运行时,会读取一个配置文件/etc/drbd.conf.这个文件里描述了DRBD设备与硬盘分区的映射关系
2.3.1在node1上作以下配置
http://blog.运维网.com/attachment/201205/210817388.png
# cd/etc/drbd.d/
#ll
total4
-rwxr-xr-x 1 rootroot 1418 Jun42010global_common.conf
#cp global_common.conf global_common.conf.bak
http://blog.运维网.com/attachment/201205/210835314.png
http://blog.运维网.com/attachment/201205/210847245.png
2.3.2复制配置到node2上：
#scp /etc/drbd.conf node2:/etc/
drbd.conf 100%133 0.1KB/s 00:00
#scp /etc/drbd.d/* node2:/etc/drbd.d/
global_common.conf 100%427 0.4KB/s 00:00
global_common.conf.bak 100% 1418 1.4KB/s 00:00
nfs.res 100%330 0.3KB/s 00:00
#
2.4、 检测配置文件
// 检测配置文件
#drbdadm adjust nfs
0: Failure: (119) Novalid meta-data signature found.
==> Use 'drbdadm create-md res' to initializemeta-data area.0k out of rangek.
#
2.5、创建nfs 的资源
2.5.1 在node1上创建nfs的资源
#drbdadm create-md nfs
Writing metadata...
initializingactivity log
NOT initializedbitmap
New drbd meta datablock successfully created.
#ll /dev/drbd0
brw-r----- 1 rootdisk 147, 0 Feb8 14:27/dev/drbd0
2.5.2 在node2上创建nfs的资源
#ssh node2.junjie.com 'drbdadm create-md nfs'
NOT initializedbitmap
Writing metadata...
initializingactivity log
New drbd meta datablock successfully created.
#ssh node2.junjie.com 'ls -l /dev/drbd0'
brw-r----- 1 rootdisk 147, 0 Feb8 14:19/dev/drbd0
2.6 启动DRBD服务
#service drbd start
Starting DRBDresources: drbdsetup 0 show:5: delay-probe-volume 0k => 0k out of rangek.
#ssh node2.junjie.com 'service drbd start'
Starting DRBDresources: drbdsetup 0 show:5: delay-probe-volume 0k => 0k out of rangek.
#
2.7 启动DRBD服务，查看DRBD状态
#service drbdstatus
drbd driver loadedOK; device status:
version: 8.3.8(api:88/proto:86-94)
GIT-hash:d78846e52224fd00562f7c225bcc25b2d422321d build bymockbuild@builder10.centos.org, 2010-06-04 08:04:16
m:rescs ro ds pmounted fstype
0:nfsConnected Secondary/SecondaryInconsistent/InconsistentC
#ssh node2.junjie.com 'servicedrbd status'
drbd driver loadedOK; device status:
version: 8.3.8(api:88/proto:86-94)
GIT-hash:d78846e52224fd00562f7c225bcc25b2d422321d build bymockbuild@builder10.centos.org, 2010-06-04 08:04:16
m:rescs ro ds pmounted fstype
0:nfsConnected Secondary/SecondaryInconsistent/InconsistentC
#
#drbd-overview
0:nfsConnected Secondary/SecondaryInconsistent/Inconsistent C r----
#ssh node2.junjie.com 'drbd-overview'
0:nfsConnected Secondary/SecondaryInconsistent/Inconsistent C r----
#
#chkconfig drbd on
#chkconfig --list drbd
drbd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
#ssh node2.junjie.com 'chkconfig drbd on'
#ssh node2.junjie.com 'chkconfig --list drbd'
drbd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
2.8 在node1主节点上进行以下配置，并查看挂载信息。
#mkdir /mnt/nfs
#ssh node2.junjie.com 'mkdir /mnt/nfs'
#drbdsetup /dev/drbd0 primary -o
#mkfs.ext3 /dev/drbd0
#mount /dev/drbd0 /mnt/nfs/
http://blog.运维网.com/attachment/201205/210718814.png
至此DRBD配置成功！！！
三、NFS配置
两台服务器都修改nfs配置文件,都修改nfs 启动脚本,如下：
http://blog.运维网.com/attachment/201205/210453381.png
http://blog.运维网.com/attachment/201205/210504781.png
四、Heartbeat配置
在server1和server2做以下操作：
1、安装Heartbeat套件
# yumlocalinstall -y heartbeat-2.1.4-9.el5.i386.rpmheartbeat-pils-2.1.4-10.el5.i386.rpm heartbeat-stonith-2.1.4-10.el5.i386.rpmlibnet-1.1.4-3.el5.i386.rpm perl-MailTools-1.77-1.el5.noarch.rpm--nogpgcheck
#
# yumlocalinstall -y heartbeat-2.1.4-9.el5.i386.rpmheartbeat-pils-2.1.4-10.el5.i386.rpm heartbeat-stonith-2.1.4-10.el5.i386.rpmlibnet-1.1.4-3.el5.i386.rpm perl-MailTools-1.77-1.el5.noarch.rpm--nogpgcheck
#
2、拷贝配置文档
# cd/usr/share/doc/heartbeat-2.1.4/
# cp authkeys ha.cf haresources/etc/ha.d/
# cd /etc/ha.d/
3、修改配置文档
http://blog.运维网.com/attachment/201205/211334933.png
http://blog.运维网.com/attachment/201205/211348429.png
http://blog.运维网.com/attachment/201205/211359681.png
六、测试
1、在测试机上将192.168.101.210:/mnt/nfs挂到本地/data下
#mkdir /data
#mount 192.168.101.210:/mnt/nfs/ /data/
# cd/data/
#ll
total20
-rw-r--r-- 1 rootroot 4 Feb8 17:41 f1
drwx------ 2 rootroot 16384 Feb8 14:57lost+found
#touch f-client-1
#ll
total20
-rw-r--r-- 1 rootroot 0 Feb8 19:50f-client-1
-rw-r--r-- 1 rootroot 4 Feb8 17:41 f1
drwx------ 2 rootroot 16384 Feb8 14:57lost+found
#cd
#
2、在测试机上创建测试shell，每秒一次
http://blog.运维网.com/attachment/201205/211458932.png
3、将主节点node1 的heartbeat服务停止，则备节点node2接管服务
#service heartbeat stop
StoppingHigh-Availability services:
[ OK]
#drbd-overview
0:nfsConnected Secondary/PrimaryUpToDate/UpToDate C r----
#ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:AE:83:D1
UPBROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:67 Base address:0x2000
#
#drbd-overview
0:nfsConnected Primary/SecondaryUpToDate/UpToDate C r---- /mnt/nfs ext3 950M 18M 885M 2%
#ifconfig eth0:0
eth0:0 Link encap:EthernetHWaddr 00:0C:29:D1:D4:32
inet addr:192.168.101.210Bcast:192.168.101.254 Mask:255.255.255.0
UPBROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:67 Base address:0x2000
#
4、在客户端上运行nfs.sh测试文件，一直显示的信息如下：
#./nfs.sh
---> trying touchx : Wed Feb 8 20:00:58 CST 2012
trying touchx : Wed Feb 8 20:00:59 CST 2012
trying touchx : Wed Feb 8 20:01:00 CST 2012
trying touchx : Wed Feb 8 20:01:01 CST 2012
trying touchx : Wed Feb 8 20:01:02 CST 2012
trying touchx : Wed Feb 8 20:01:03 CST 2012
trying touchx : Wed Feb 8 20:01:04 CST 2012
trying touchx : Wed Feb 8 20:01:05 CST 2012
页:
[1]