GlusterFS分布式存储搭建双机复制卷结合Keepalived实现存储高可用
GlusterFS分布式存储高可用方案设计1、搭建GlusterFS复制卷(使用至少2个存储节点)
2、配置Keepalived管理GlusterFS主从存储节点
3、配置Keepalived浮动IP(VIP)对外提供存储服务
4、实现存储高可用(即使用两台GlusterFS服务器提供双节点复制卷，并提供快速故障转移，实现存储的持续高可用)
5、可以应用在关键性数据存储业务场景
一、环境准备
IP
Hostname
存储
系统
VIP
172.16.10.10
data-node-01
/dev/sdb1
CentOS7
172.16.10.220
说明：用于对外提供存储服务
172.16.10.11
data-node-02
/dev/sdb1
CentOS7
172.16.10.12
web-node-12
客户端
/etc/hosts配置
172.16.10.10 data-node-01
172.16.10.11 data-node-02 建立GlusterFS存储挂载点
mkdir -p /glusterfs/storage1
echo "/dev/sdb1 /glusterfs/storage1 xfs defaults 0 0" >> /etc/fstab
mount -a 二、安装GlusterFS服务端软件
1、安装gluster源，并安装glusterfs及相关软件包
yum install centos-release-gluster -y
yum install glusterfs glusterfs-server glusterfs-cli glusterfs-geo-replication glusterfs-rdma -y 2、客户端安装GlusterFS客户端软件
yum install glusterfs-fuse 3、启动Glusterd服务
systemctl start glusterd 4、在任意一个节点上添加信任节点
gluster peer probe data-node-02
gluster peer probe data-node-01
gluster peer status 5、在任意一个节点上建立复制卷
mkdir /glusterfs/storage1/rep_vol1
gluster volume create rep_vol1 replica 2 data-node-01:/glusterfs/storage1/rep_vol1 data-node-02:/glusterfs/storage1/rep_vol1 6、启动复制卷
gluster volume start rep_vol1 7、查看复制卷状态
gluster volume status
gluster volume info 8、客户端测试挂载复制卷
mount -t glusterfs data-node-01:rep_vol1 /data/ 9、客户端测试复制卷数据存储
for i in `seq -w 1 3`;do cp -rp /var/log/messages /data/test-$i;done
# ls /data/
1111.txt2.txtanaconda-ks.cfgtest-1test-2test-3 三、安装与配置Keepalived
1、安装Keepalived
yum -y install keepalived 2、启动keepalived服务
systemctl start keepalived 3、主节点keepalived配置
! Configuration File for keepalived
global_defs {
notification_email {
mail@huangming.org
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id GFS_HA_MASTER
vrrp_skip_check_adv_addr
}
vrrp_sync_group GFS_HA_GROUP {
group {
GFS_HA_1
}
}
vrrp_script monitor_glusterfs_status {
script "/etc/keepalived/scripts/monitor_glusterfs_status.sh"
interval 5
fall 3
rise 1
weight 20
}
vrrp_instance GFS_HA_1 {
state BACKUP
interface ens34
virtual_router_id 107
priority 100
advert_int 2
nopreempt
authentication {
auth_type PASS
auth_pass 11112222
}
virtual_ipaddress {
172.16.10.220/24 dev ens34
}
track_script {
monitor_glusterfs_status
}
track_interface {
ens34
}
notify_master "/etc/keepalived/scripts/keepalived_notify.sh master"
notify_backup "/etc/keepalived/scripts/keepalived_notify.sh backup"
notify_fault"/etc/keepalived/scripts/keepalived_notify.sh fault"
notify_stop "/etc/keepalived/scripts/keepalived_notify.sh stop"
} 4、备节点keepalived配置
! Configuration File for keepalived
global_defs {
notification_email {
mail@huangming.org
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id GFS_HA_MASTER
vrrp_skip_check_adv_addr
}
vrrp_sync_group GFS_HA_GROUP {
group {
GFS_HA_1
}
}
vrrp_script monitor_glusterfs_status {
script "/etc/keepalived/scripts/monitor_glusterfs_status.sh"
interval 5
fall 3
rise 1
weight 20
}
vrrp_instance GFS_HA_1 {
state BACKUP
interface ens34
virtual_router_id 107
priority 90
advert_int 2
authentication {
auth_type PASS
auth_pass 11112222
}
virtual_ipaddress {
172.16.10.220/24 dev ens34
}
track_script {
monitor_glusterfs_status
}
track_interface {
ens34
}
notify_master "/etc/keepalived/scripts/keepalived_notify.sh master"
notify_backup "/etc/keepalived/scripts/keepalived_notify.sh backup"
notify_fault"/etc/keepalived/scripts/keepalived_notify.sh fault"
notify_stop "/etc/keepalived/scripts/keepalived_notify.sh stop"
} 5、keepalived vrrp监控脚本
#!/bin/bash
#check glusterfsd and glusterd process
systemctl status glusterd &>/dev/null
if [ $? -eq 0 ];then
systemctl status glusterfsd &>/dev/null
if [ $? -eq 0 ];then
exit 0
else
exit 2
fi
else
systemctl start glusterd &>/dev/null
systemctl stop keepalived &>/dev/null && exit 1
fi 6、keepalived通知脚本(管理Glusterd服务)
#!/bin/bash
#keepalived script for glusterd
master() {
systemctl status glusterd
if [ $? -ne 0 ];then
systemctl start glusterd
else
systemctl restart glusterd
fi
}
backup() {
systemctl status glusterd
if [ $? -ne 0 ];then
systemctl start glusterd
fi
}
case $1 in
master)
master
;;
backup)
backup
;;
fault)
backup
;;
stop)
backup
systemctl restart keepalived
;;
*)
echo $"Usage: $0 {master|backup|fault|stop}"
esac 四、测试Keepalived自动接管GlusterFS服务及存储的可用性
1、重新启动keepalived服务
systemctl restart keepalived.service 2、查看VIP接管情况
## 节点1上
# ip a show dev ens34
3: ens34:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:b2:b5:2a brd ff:ff:ff:ff:ff:ff
inet 172.16.10.10/24 brd 172.16.10.255 scope global ens34
valid_lft forever preferred_lft forever
inet 172.16.10.220/24 scope global secondary ens34
valid_lft forever preferred_lft forever
inet6 fe80::ce9a:ee2e:7b6c:a6bb/64 scope link
valid_lft forever preferred_lft forever
## 节点2上
# ip a show dev ens34
3: ens34:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:ba:42:cf brd ff:ff:ff:ff:ff:ff
inet 172.16.10.11/24 brd 172.16.10.255 scope global ens34
valid_lft forever preferred_lft forever
inet6 fe80::e23:ce0:65c3:ffbf/64 scope link
valid_lft forever preferred_lft forever 3、客户端上使用VIP挂载GlusterFS提供的复制卷，并测试是否可用
mount -t glusterfs 172.16.10.220:rep_vol1 /data/
# ls /data/
1111.txt2.txtanaconda-ks.cfgtesttest-1test-2test-3
# mkdir /data/test
# echo 1111 >/data/test/1.txt
# ls /data/test
1.txt
# cat /data/test/1.txt
1111 查看GluserFS节点复制卷的使用情况
# ls /glusterfs/storage1/rep_vol1/
1111.txt2.txtanaconda-ks.cfgtesttest-1test-2test-3 3、测试GlusterFS服务故障转移
将主节点(节点1)关机或重启，查看GlusterFS服务与VIP是否转移至节点2
# reboot
# ip a show dev ens34
3: ens34:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:ba:42:cf brd ff:ff:ff:ff:ff:ff
inet 172.16.10.11/24 brd 172.16.10.255 scope global ens34
valid_lft forever preferred_lft forever
inet 172.16.10.220/24 scope global secondary ens34
valid_lft forever preferred_lft forever
inet6 fe80::e23:ce0:65c3:ffbf/64 scope link
valid_lft forever preferred_lft forever
# tail -f /var/log/messages
Aug 27 22:56:19 data-node-02 Keepalived_vrrp: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Aug 27 22:56:19 data-node-02 Keepalived_vrrp: Sync group GFS_HA_GROUP has only 1 virtual router(s) - removing
Aug 27 22:56:19 data-node-02 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) removing protocol VIPs.
Aug 27 22:56:19 data-node-02 Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Aug 27 22:56:19 data-node-02 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) Entering BACKUP STATE
Aug 27 22:56:19 data-node-02 Keepalived_vrrp: VRRP sockpool:
Aug 27 22:56:19 data-node-02 Keepalived_vrrp: VRRP_Script(monitor_glusterfs_status) succeeded
Aug 27 22:56:19 data-node-02 kernel: nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Aug 27 22:56:19 data-node-02 kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Aug 27 22:56:19 data-node-02 kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Aug 27 22:56:19 data-node-02 kernel: IPVS: Creating netns size=2040 id=0
Aug 27 22:56:19 data-node-02 kernel: IPVS: ipvs loaded.
Aug 27 22:56:19 data-node-02 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Aug 27 22:56:21 data-node-02 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) Changing effective priority from 90 to 110
Aug 27 23:01:01 data-node-02 systemd: Started Session 3 of user root.
Aug 27 23:01:01 data-node-02 systemd: Starting Session 3 of user root.
Aug 27 23:03:09 data-node-02 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) Transition to MASTER STATE
Aug 27 23:03:11 data-node-02 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) Entering MASTER STATE
Aug 27 23:03:11 data-node-02 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) setting protocol VIPs.
Aug 27 23:03:11 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220
Aug 27 23:03:11 data-node-02 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) Sending/queueing gratuitous ARPs on ens34 for 172.16.10.220
Aug 27 23:03:11 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220
Aug 27 23:03:11 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220
Aug 27 23:03:11 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220
Aug 27 23:03:11 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220
Aug 27 23:03:11 data-node-02 systemd: Stopping GlusterFS, a clustered file-system server...
Aug 27 23:03:11 data-node-02 systemd: Starting GlusterFS, a clustered file-system server...
Aug 27 23:03:12 data-node-02 systemd: Started GlusterFS, a clustered file-system server.
Aug 27 23:03:16 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220
Aug 27 23:03:16 data-node-02 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) Sending/queueing gratuitous ARPs on ens34 for 172.16.10.220
Aug 27 23:03:16 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220
Aug 27 23:03:16 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220
Aug 27 23:03:16 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220
Aug 27 23:03:16 data-node-02 Keepalived_vrrp: Sending gratuitous ARP on ens34 for 172.16.10.220 在客户端上测试存储是否仍然可用
# df -Th
Filesystem Type SizeUsed Avail Use% Mounted on
/dev/mapper/cl-root xfs 40G1.2G 39G 3% /
devtmpfs devtmpfs 1.9G 01.9G 0% /dev
tmpfs tmpfs 1.9G 01.9G 0% /dev/shm
tmpfs tmpfs 1.9G8.6M1.9G 1% /run
tmpfs tmpfs 1.9G 01.9G 0% /sys/fs/cgroup
/dev/sda1 xfs 1014M139M876M14% /boot
tmpfs tmpfs 378M 0378M 0% /run/user/0
172.16.10.220:rep_vol1 fuse.glusterfs 10G136M9.9G 2% /data
# ls /data/
1111.txt2.txtanaconda-ks.cfgtesttest-1test-2test-3
# touch /data/test.log
# ls -l /data/
total 964
drwxr-xr-x 3 root root 4096 Aug 27 21:58 111
-rw-r--r-- 1 root root 10 Aug 27 21:23 1.txt
-rw-r--r-- 1 root root 6 Aug 27 21:36 2.txt
-rw------- 1 root root 2135 Aug 27 21:44 anaconda-ks.cfg
drwxr-xr-x 2 root root 4096 Aug 27 22:59 test
-rw------- 1 root root 324951 Aug 27 21:23 test-1
-rw------- 1 root root 324951 Aug 27 21:23 test-2
-rw------- 1 root root 324951 Aug 27 21:23 test-3
-rw-r--r-- 1 root root 0 Aug 27 23:05 test.log 查看节点1状态
# ip a show dev ens34
3: ens34:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:b2:b5:2a brd ff:ff:ff:ff:ff:ff
inet 172.16.10.10/24 brd 172.16.10.255 scope global ens34
valid_lft forever preferred_lft forever
inet6 fe80::ce9a:ee2e:7b6c:a6bb/64 scope link
valid_lft forever preferred_lft forever 重新启动keepalived服务
# systemctl start keepalived.service 查看keepalived日志(主备状态)
Aug 27 23:07:42 data-node-01 systemd: Starting LVS and VRRP High Availability Monitor...
Aug 27 23:07:43 data-node-01 Keepalived: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Aug 27 23:07:43 data-node-01 Keepalived: Opening file '/etc/keepalived/keepalived.conf'.
Aug 27 23:07:43 data-node-01 Keepalived: Starting Healthcheck child process, pid=2916
Aug 27 23:07:43 data-node-01 systemd: Started LVS and VRRP High Availability Monitor.
Aug 27 23:07:43 data-node-01 Keepalived: Starting VRRP child process, pid=2917
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: Registering Kernel netlink reflector
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: Registering Kernel netlink command channel
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: Registering gratuitous ARP shared channel
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: Sync group GFS_HA_GROUP has only 1 virtual router(s) - removing
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) removing protocol VIPs.
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) Entering BACKUP STATE
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: VRRP sockpool:
Aug 27 23:07:43 data-node-01 Keepalived_vrrp: VRRP_Script(monitor_glusterfs_status) succeeded
Aug 27 23:07:43 data-node-01 kernel: nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Aug 27 23:07:43 data-node-01 kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Aug 27 23:07:43 data-node-01 kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Aug 27 23:07:43 data-node-01 kernel: IPVS: Creating netns size=2040 id=0
Aug 27 23:07:43 data-node-01 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Aug 27 23:07:43 data-node-01 kernel: IPVS: ipvs loaded.
Aug 27 23:07:45 data-node-01 Keepalived_vrrp: VRRP_Instance(GFS_HA_1) Changing effective priority from 100 to 120 由此可见当节点1故障恢复后，keepalived会进入到备用状态，同时继续监管GlusterFS服务，当节点2故障时会将服务、存储和VIP切换到节点1，继续对外提供存储服务，从而实现存储的高可用
页:
[1]