在Glusterfs上创建distributed volume,replicated volume,dispersed volume,combined volu
前面一篇写到了在CentOS上如何安装glusterfs,以及简单创建了一个volume并实现了native-mount,今天我们重点看一下在glusterfs上都可以创建哪种类型的volume.
1. 首先还是先介绍下实验环境，今天共用到了5台虚拟机，其中4个虚拟机做server端,分别是：
servera.lab.example.com
serverb.lab.example.com
serverc.lab.example.com
serverd.lab.example.com
1个虚拟机做client端：
workstation.lab.example.com
五台虚拟机之间都打开了防火墙，可以相互访问，并且可以相互解析hostname,4台server上已经安装好了glusterfs相关的package,并组成了一个trusted-storage-pool:
# gluster pool list
UUID Hostname State
d61aaee4-efe5-4f60-9216-c65fdb0b65f8 serverb.lab.example.com Connected
be6e1fe5-ae7d-40d1-901e-8b58fd0a4be3 serverc.lab.example.com Connected
25dda180-3285-47f6-8595-3cb9bdaab92f serverd.lab.example.com Connected
860cd46c-390b-430f-8a01-d3433fa2775c localhost Connected
2.在4台server上分别创建5个bricks,并挂载到/bricks/thinvol$n目录下，依次执行以下命令（如非特殊说明，以下命令均为在4台server上执行，文中以servera为示例）：
a. 实验环境中每个server上已经提前做好了一个lvm group vg_bricks
# vgs
VG #PV #LV #SN Attr VSizeVFree
vg_bricks 110 0 wz--n- 20.00g 9.97g
b. 创建瘦逻辑卷
# for i in {1..9}; do lvcreate -L 2G -T vg_bricks/pool$i; done
Logical volume "pool1" created.
Logical volume "pool2" created.
Logical volume "pool3" created.
Logical volume "pool4" created.
Logical volume "pool5" created.
Logical volume "pool6" created.
Logical volume "pool7" created.
Logical volume "pool8" created.
Logical volume "pool9" created.
# for i in {1..9}; do lvcreate -L 2G -T vg_bricks/pool$i; done
Logical volume pool1 already exists in Volume group vg_bricks.
Logical volume pool2 already exists in Volume group vg_bricks.
Logical volume pool3 already exists in Volume group vg_bricks.
Logical volume pool4 already exists in Volume group vg_bricks.
Logical volume pool5 already exists in Volume group vg_bricks.
Logical volume pool6 already exists in Volume group vg_bricks.
Logical volume pool7 already exists in Volume group vg_bricks.
Logical volume pool8 already exists in Volume group vg_bricks.
Logical volume pool9 already exists in Volume group vg_bricks. c. 格式化逻辑卷并挂载到指定目录上，生产环境中挂载时建议将挂载命令写入/etc/fstab中以实现开机自动挂载
# for i in {1..9}; do mkfs -t xfs -i size=512 /dev/mapper/vg_bricks-thinvol$i; done
# mkdir -p /bricks/thinvol{1..9}
# for i in {1..9}; do mount -t xfs /dev/mapper/vg_bricks-thinvol$i /bricks/thinvol$i; done d. 设置selinux安全上下文，这里默认selinux是打开的
# for i in {1..9}; do mkdir /bricks/thinvol$i/brick; done
# chcon -R -t glusterd_brick_t /bricks/thinvol{1..9} 至此，准备工作可以告一段落了
3. 创建分布式卷Distributed volume
# gluster volume create Test01 \
> servera.lab.example.com:/bricks/thinvol1/brick \
> serverb.lab.example.com:/bricks/thinvol1/brick \
> serverc.lab.example.com:/bricks/thinvol1/brick \
> serverd.lab.example.com:/bricks/thinvol1/brickvolume create: Test01: success: please start the volume to access data 查看volume的状态，可以看到刚创建完后，volume处在“not started”状态
# gluster volume status Test01
Volume Test01 is not started start volume
# gluster volume start Test01
volume start: Test01: success 此时查看volume的详细信息发现volume已started，volume 类型为distribute
# gluster volume info Test01
Volume Name: Test01
Type: Distribute
Volume ID: f40beb82-81ae-42d2-bd1a-a7b9a24abe63
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: servera.lab.example.com:/bricks/thinvol1/brick
Brick2: serverb.lab.example.com:/bricks/thinvol1/brick
Brick3: serverc.lab.example.com:/bricks/thinvol1/brick
Brick4: serverd.lab.example.com:/bricks/thinvol1/brick
Options Reconfigured:
performance.readdir-ahead: on 在workstation.lab.example.com上挂载volume Test01
# yum install -y glusterfs-fuse
# mkdir /mnt/Test01
# mount -t glusterfs servera.lab.example.com:Test01 /mnt/Test01
# df -h | grep Test01
servera.lab.example.com:Test018.0G131M7.9G 2% /mnt/Test01 这里可以看到一些distributed volume的特点，有点类似raid0,volume的大小是组成volume的4个brick之和（每个brick大小为2G，共计2G*4=8G）
在/mnt/Test01下创建100个测试文件，会发现文件会“随机”分配到4个server的brick下
# cd /mnt/Test01/
# touch {1..100}.file# ls /bricks/thinvol1/brick/
100.file13.file18.file29.file32.file37.file40.file47.file61.file6.file 76.file7.file 97.file
12.file 17.file28.file2.file 34.file39.file44.file54.file62.file75.file77.file94.file# ls /bricks/thinvol1/brick/
10.file1.file 25.file38.file42.file52.file56.file59.file66.file80.file86.file98.file
14.file24.file26.file41.file50.file53.file57.file5.file 74.file83.file87.file
...
4. 创建复制卷Replicated volume
# gluster volume create Test02 replica 4 \
> servera.lab.example.com:/bricks/thinvol2/brick \
> serverb.lab.example.com:/bricks/thinvol2/brick \
> serverc.lab.example.com:/bricks/thinvol2/brick \
> serverd.lab.example.com:/bricks/thinvol2/brick
volume create: Test02: success: please start the volume to access data start volume Test02并查看volume的详细信息，可以看到volume的类型为Replicate即复制卷
# gluster volume start Test02
volume start: Test02: success
# gluster volume info Test02
Volume Name: Test02
Type: Replicate
Volume ID: 305f2c2f-8550-40e0-9d75-a2bc8149a333
Status: Started
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: servera.lab.example.com:/bricks/thinvol2/brick
Brick2: serverb.lab.example.com:/bricks/thinvol2/brick
Brick3: serverc.lab.example.com:/bricks/thinvol2/brick
Brick4: serverd.lab.example.com:/bricks/thinvol2/brick
Options Reconfigured:
performance.readdir-ahead: on 将Test02挂载到client下的/mnt/Test02目录下
# mkdir /mnt/Test02
# mount -t glusterfs servera.lab.example.com:Test02 /mnt/Test02
# df -h | grep Test02servera.lab.example.com:Test022.0G 33M2.0G 2% /mnt/Test02 这里可以看到Test02挂载后的大小为2G，即和组成Test02的每个brick一样大小，其特点类似raid1
在/mnt/Test02 下创建100个测试文件，会发现每个server的brick下都创建了100个测试文件
# touch {1..100}.file# ls /bricks/thinvol2/brick/
100.file16.file22.file29.file35.file41.file48.file54.file60.file67.file73.file7.file 86.file92.file99.file
10.file 17.file23.file2.file 36.file42.file49.file55.file61.file68.file74.file80.file87.file93.file9.file
11.file 18.file24.file30.file37.file43.file4.file 56.file62.file69.file75.file81.file88.file94.file
12.file 19.file25.file31.file38.file44.file50.file57.file63.file6.file 76.file82.file89.file95.file
13.file 1.file 26.file32.file39.file45.file51.file58.file64.file70.file77.file83.file8.file 96.file
14.file 20.file27.file33.file3.file 46.file52.file59.file65.file71.file78.file84.file90.file97.file
15.file 21.file28.file34.file40.file47.file53.file5.file 66.file72.file79.file85.file91.file98.file# ls /bricks/thinvol2/brick/ | wc -w
100# ls /bricks/thinvol2/brick/ | wc -w
100# ls /bricks/thinvol2/brick/ | wc -w
100
5. 创建“分散卷”Dispersed volume
# for BRICKNUM in {3..5}; do
> for node in {a..d}; do
> echo server$node.lab.example.com:/bricks/thinvol$BRICKNUM/brick
> done
> done > /tmp/Test03.txt
# cat /tmp/Test03.txt
servera.lab.example.com:/bricks/thinvol3/brick
serverb.lab.example.com:/bricks/thinvol3/brick
serverc.lab.example.com:/bricks/thinvol3/brick
serverd.lab.example.com:/bricks/thinvol3/brick
servera.lab.example.com:/bricks/thinvol4/brick
serverb.lab.example.com:/bricks/thinvol4/brick
serverc.lab.example.com:/bricks/thinvol4/brick
serverd.lab.example.com:/bricks/thinvol4/brick
servera.lab.example.com:/bricks/thinvol5/brick
serverb.lab.example.com:/bricks/thinvol5/brick
serverc.lab.example.com:/bricks/thinvol5/brick
serverd.lab.example.com:/bricks/thinvol5/brick
# gluster volume create Test03 disperse-data 4 redundancy 2 $( serverb.lab.example.com:/bricks/thinvol6/brick \
> serverc.lab.example.com:/bricks/thinvol6/brick \
> servera.lab.example.com:/bricks/thinvol7/brick \
> serverb.lab.example.com:/bricks/thinvol7/brick \
> serverc.lab.example.com:/bricks/thinvol7/brick
volume create: Test04: success: please start the volume to access data
# gluster volume start Test04
volume start: Test04: success
# gluster volume info Test04
Volume Name: Test04
Type: Distributed-Replicate
Volume ID: 45df998d-2d14-4af2-83c7-e497bd5a8dd0
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: servera.lab.example.com:/bricks/thinvol6/brick
Brick2: serverb.lab.example.com:/bricks/thinvol6/brick
Brick3: serverc.lab.example.com:/bricks/thinvol6/brick
Brick4: servera.lab.example.com:/bricks/thinvol7/brick
Brick5: serverb.lab.example.com:/bricks/thinvol7/brick
Brick6: serverc.lab.example.com:/bricks/thinvol7/brick
Options Reconfigured:
performance.readdir-ahead: on 从上面可以看出volume Test04的类型为Distributed-Replicate
在client上挂载Test04，Test04的可用空间为4G
# mkdir /mnt/Test04
# mount -t glusterfs servera.lab.example.com:Test04 /mnt/Test04
# df -h | grep Test04
servera.lab.example.com:Test044.0G 66M4.0G 2% /mnt/Test04 在/mnt/Test04下创建100个测试文件，并查看下数据分布
# touch {1..100}.file# ls /bricks/thinvol4/brick/ | wc -w
48
# ls /bricks/thinvol5/brick/ | wc -w
52
## ls /bricks/thinvol4/brick/ | wc -w
48
# ls /bricks/thinvol5/brick/ | wc -w
52# ls /bricks/thinvol4/brick/ | wc -w
48
# ls /bricks/thinvol5/brick/ | wc -w
52 可以发现
Brick1: servera.lab.example.com:/bricks/thinvol4/brick
Brick2: serverb.lab.example.com:/bricks/thinvol4/brick
Brick3: serverc.lab.example.com:/bricks/thinvol4/brick 组成了一个3副本，每个brick上的数据一致
Brick4: servera.lab.example.com:/bricks/thinvol5/brick
Brick5: serverb.lab.example.com:/bricks/thinvol5/brick
Brick6: serverc.lab.example.com:/bricks/thinvol5/brick 组成了一个3副本，每个brick上的数据一致
当然了，这里的例子限于实验环境的局限性，每个server上选取了两个brick，生产环境中还是要尽量杜绝这种情况的。Ok，今天就到这里了。
页:
[1]