openstack运维实战系列(九)之cinder与glusterfs结合
1. 概述cinder作为openstack的快存储服务，为instance提供永久的volume服务，cinder作为一种可插拔式的服务，能够支持各种存储类型，包括专业的FC存储，如EMC，NetApp，HP，IBM，huawei等商场的专业存储服务器，存储厂商只要开发对应的驱动和cinder对接即可；此外，cinder还支持开源的分布式存储，如glusterfs，ceph，sheepdog，nfs等，通过开源的分布式存储方案，能够达到廉价的IP-SAN存储。本文以glusterfs构建分布式存储，以供cinder使用。
2. 构建glusterfs存储
glusterfs是一种开源的分布式存储解决方案，能够支持集中方式：1. replicate复制(类似于RAID1)，2.stripe分片(类似于RAID0),3. distribute-replicate分布式复制，4. distribute-replicate-stripe分布式复制和分片(类似于RAID10)，本文采用的方式。
[*]环境说明
本文有两台机器组件glusterfs集群，分别是:10.1.112.55和10.1.112.56,两台机器分别有11块盘，每块3T，磁盘名字从/dev/sdb至/dev/sdl,挂载至/data2-/data12,如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# df -h
Filesystem SizeUsed Avail Use% Mounted on
/dev/sda2 9.9G2.6G6.9G27% /
tmpfs 7.8G 07.8G 0% /dev/shm
/dev/sda1 1008M 82M876M 9% /boot
/dev/sda4 257G188M244G 1% /data1
/dev/sdb1 2.8T118M2.8T 1% /data2
/dev/sdc1 2.8T118M2.8T 1% /data3
/dev/sdd1 2.8T118M2.8T 1% /data4
/dev/sde1 2.8T118M2.8T 1% /data5
/dev/sdf1 2.8T118M2.8T 1% /data6
/dev/sdg1 2.8T118M2.8T 1% /data7
/dev/sdh1 2.8T118M2.8T 1% /data8
/dev/sdi1 2.8T117M2.8T 1% /data9
/dev/sdj1 2.8T118M2.8T 1% /data10
/dev/sdk1 2.8T118M2.8T 1% /data11
/dev/sdl1 2.8T118M2.8T 1% /data12
架构如下:
2. 激活glusterfs邻居peer
1
2
3
4
5
6
7
8
# gluster peer probe 10.1.112.56
peer probe: success.
#查看
# gluster peer status
Number of Peers: 1
Hostname: 10.1.112.56
Uuid: a720fd05-4fa7-4ff7-924e-2d8a40e48c18
State: Peer in Cluster (Connected)
3. 基于brick创建volume，切割成11份，分别存储在两台机器(类似于RAID10)
1
2
3
4
5
6
7
8
9
10
11
12
# gluster volume create openstack_cinder stripe 11 replica 2 transport tcp \
10.1.112.55:/data2/cinder 10.1.112.56:/data2/cinder \
10.1.112.55:/data3/cinder 10.1.112.56:/data3/cinder \
10.1.112.55:/data4/cinder 10.1.112.56:/data4/cinder \
10.1.112.55:/data5/cinder 10.1.112.56:/data5/cinder \
10.1.112.55:/data6/cinder 10.1.112.56:/data6/cinder \
10.1.112.55:/data7/cinder 10.1.112.56:/data7/cinder \
10.1.112.55:/data8/cinder 10.1.112.56:/data8/cinder \
10.1.112.55:/data9/cinder 10.1.112.56:/data9/cinder \
10.1.112.55:/data10/cinder 10.1.112.56:/data10/cinder \
10.1.112.55:/data11/cinder 10.1.112.56:/data11/cinder \
10.1.112.55:/data12/cinder 10.1.112.56:/data12/cinder
4. 查看glusterfs volume的结构
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# gluster volume info
Volume Name: openstack_cinder
Type: Striped-Replicate
Volume ID: c55ff01b-3be0-4514-b622-83677f95924a
Status: Started
Number of Bricks: 1 x 11 x 2 = 22
Transport-type: tcp
Bricks:
Brick1: 10.1.112.55:/data2/cinder
Brick2: 10.1.112.56:/data2/cinder
Brick3: 10.1.112.55:/data3/cinder
Brick4: 10.1.112.56:/data3/cinder
Brick5: 10.1.112.55:/data4/cinder
Brick6: 10.1.112.56:/data4/cinder
Brick7: 10.1.112.55:/data5/cinder
Brick8: 10.1.112.56:/data5/cinder
Brick9: 10.1.112.55:/data6/cinder
Brick10: 10.1.112.56:/data6/cinder
Brick11: 10.1.112.55:/data7/cinder
Brick12: 10.1.112.56:/data7/cinder
Brick13: 10.1.112.55:/data8/cinder
Brick14: 10.1.112.56:/data8/cinder
Brick15: 10.1.112.55:/data9/cinder
Brick16: 10.1.112.56:/data9/cinder
Brick17: 10.1.112.55:/data10/cinder
Brick18: 10.1.112.56:/data10/cinder
Brick19: 10.1.112.55:/data11/cinder
Brick20: 10.1.112.56:/data11/cinder
Brick21: 10.1.112.55:/data12/cinder
Brick22: 10.1.112.56:/data12/cinder
5. 启动glusterfs volume
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# gluster volume start openstack_cinder #开启glusterfs volume
#查看glusterfs volume的状态
# gluster volume status
Status of volume: openstack_cinder
Gluster process Port OnlinePid
------------------------------------------------------------------------------
Brick 10.1.112.55:/data2/cinder 59152 Y 4121
Brick 10.1.112.56:/data2/cinder 59152 Y 43596
Brick 10.1.112.55:/data3/cinder 59153 Y 4132
Brick 10.1.112.56:/data3/cinder 59153 Y 43607
Brick 10.1.112.55:/data4/cinder 59154 Y 4143
Brick 10.1.112.56:/data4/cinder 59154 Y 43618
Brick 10.1.112.55:/data5/cinder 59155 Y 4154
Brick 10.1.112.56:/data5/cinder 59155 Y 43629
Brick 10.1.112.55:/data6/cinder 59156 Y 4165
Brick 10.1.112.56:/data6/cinder 59156 Y 43640
Brick 10.1.112.55:/data7/cinder 59157 Y 4176
Brick 10.1.112.56:/data7/cinder 59157 Y 43651
Brick 10.1.112.55:/data8/cinder 59158 Y 4187
Brick 10.1.112.56:/data8/cinder 59158 Y 43662
Brick 10.1.112.55:/data9/cinder 59159 Y 4198
Brick 10.1.112.56:/data9/cinder 59159 Y 43673
Brick 10.1.112.55:/data10/cinder 59160 Y 4209
Brick 10.1.112.56:/data10/cinder 59160 Y 43684
Brick 10.1.112.55:/data11/cinder 59161 Y 4220
Brick 10.1.112.56:/data11/cinder 59161 Y 43695
Brick 10.1.112.55:/data12/cinder 59162 Y 4231
Brick 10.1.112.56:/data12/cinder 59162 Y 43706
NFS Server on localhost 2049 Y 4244
Self-heal Daemon on localhost N/A Y 4251
NFS Server on 10.1.112.56 2049 Y 43718
Self-heal Daemon on 10.1.112.56 N/A Y 43727
Task Status of Volume openstack_cinder
------------------------------------------------------------------------------
6. 挂载测试
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# mount.glusterfs 10.1.112.56:openstack_cinder /media/
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 10321208 2488348 730857226% /
tmpfs 8140364 0 8140364 0% /dev/shm
/dev/sda1 1032088 83596 896064 9% /boot
/dev/sda4 268751588191660 254908060 1% /data1
/dev/sdb1 2928834296 329722928801324 1% /data2
/dev/sdc1 2928834296 329722928801324 1% /data3
/dev/sdd1 2928834296 329722928801324 1% /data4
/dev/sde1 2928834296 329722928801324 1% /data5
/dev/sdf1 2928834296 329722928801324 1% /data6
/dev/sdg1 2928834296 329722928801324 1% /data7
/dev/sdh1 2928834296 329722928801324 1% /data8
/dev/sdi1 2928834296 329722928801324 1% /data9
/dev/sdj1 2928834296 329722928801324 1% /data10
/dev/sdk1 2928834296 329722928801324 1% /data11
/dev/sdl1 2928834296 329722928801324 1% /data12
10.1.112.56:openstack_cinder
32217177216362752 32216814464 1% /media #已经挂载
3. cinder和glusterfs结合
[*]cinder-volume端配置内容如下
1
2
3
4
5
6
7
enabled_backends = glusterfs
#最后添加
volume_driver = cinder.volume.drivers.glusterfs.GlusterfsDriver #驱动
glusterfs_shares_config = /etc/cinder/shares.conf #glusterfs存储
glusterfs_mount_point_base = /var/lib/cinder/volumes #挂载点
volume_backend_name = glusterfs #后端名字，用于在controller上和cinder的type结合@@
2. 配置glusterfs存储配置
1
2
# vim /etc/cinder/shares.conf
10.1.112.55:/openstack_cinder
3. 重启cinder-volume服务
1
2
3
4
# chkconfig openstack-cinder-volume on
# serviceopenstack-cinder-volume restart
Stopping openstack-cinder-volume:
Starting openstack-cinder-volume:
@@@两台机器，执行相同的操作，并检查日志信息，看是否有错误/var/log/cinder/volume.log@@@@
4. controller节点检查服务状态
1
2
3
4
5
6
7
8
# cinder service-list
+------------------+--------------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone |Status | State | Updated_at | Disabled Reason |
+------------------+--------------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | controller | nova | enabled | up| 2016-01-22T08:52:14.000000 | None |
|cinder-volume | YiZhuang_10_1_112_55@glusterfs | nova | enabled | up| 2016-01-22T08:52:17.000000 | None | #说明正常
|cinder-volume | YiZhuang_10_1_112_56@glusterfs | nova | enabled | up| 2016-01-22T08:52:04.000000 | None |
+------------------+--------------------------------+------+---------+-------+----------------------------+-----------------+
5. controller建立type
1
2
3
4
5
6
# cinder type-create glusterfs
+--------------------------------------+------------+
| ID | Name |
+--------------------------------------+------------+
| 6688e8f9-e744-4c21-b570-fd81b099d4c0 | glusterfs|
+--------------------------------------+------------+
6. controller配置cinder-type和volume_backend_name联动
1
2
3
4
5
6
7
8
9
# cinder type-key set6688e8f9-e744-4c21-b570-fd81b099d4c0 volume_backend_name=glusterfs
#查看type的设置情况
# cinder extra-specs-list
+--------------------------------------+-----------+----------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+-----------+----------------------------------------+
| 6688e8f9-e744-4c21-b570-fd81b099d4c0 | glusterfs | {u'volume_backend_name': u'glusterfs'} | #关联完毕
+--------------------------------------+-----------+----------------------------------------+
7. 重启controller的cinder服务
1
2
3
4
5
6
# /etc/init.d/openstack-cinder-apirestart
Stopping openstack-cinder-api:
Starting openstack-cinder-api:
# /etc/init.d/openstack-cinder-scheduler restart
Stopping openstack-cinder-scheduler:
Starting openstack-cinder-scheduler:
4. 功能测试
[*]创建cinder volume
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# cinder create --display-name "test1" --volume-type glusterfs 10 #执行cinder type的类型
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
|availability_zone| nova |
| bootable | false |
| created_at | 2016-01-22T09:01:48.978864 |
| display_description | None |
| display_name | test1 |
| encrypted | False |
| id | 3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001 |
| metadata | {} |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | glusterfs |
+---------------------+--------------------------------------+
# cinder show3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2016-01-22T09:01:48.000000 |
| display_description | None |
| display_name | test1 |
| encrypted | False |
| id | 3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001 |
| metadata | {} |
| os-vol-host-attr:host | YiZhuang_10_1_112_55@glusterfs | #落在10.1.112.55这台机器
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
|os-vol-tenant-attr:tenant_id| a49b16d5324a4d20bde2217b17200485 |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | available | #创建成功，状态为available
| volume_type | glusterfs |
+--------------------------------+--------------------------------------+
2. 校验glusterfs的切割情况
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# for num in {2..12}
> do
> ls -lh /data${num}/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
> done
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data2/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data3/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data4/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data5/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data6/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data7/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data8/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data9/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data10/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data11/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
-rw-rw-rw- 2 root root 931M Jan 22 17:01 /data12/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001 #10G的磁盘，切割为11份，分别落在11个磁盘上，达到负载均衡的效果，另外一台机器的数据是一模一样
3. 校验两台机器的数据的md5
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# for num in {2..12}; do md5sum /data${num}/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001; done
e0f8c6646f8ce81fe6be0b12f1511aa1/data2/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e0f8c6646f8ce81fe6be0b12f1511aa1/data3/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e0f8c6646f8ce81fe6be0b12f1511aa1/data4/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data5/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data6/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data7/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data8/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data9/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data10/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data11/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data12/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
另外一台机器
# for num in {2..12} ; do md5sum /data${num}/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001; done
e0f8c6646f8ce81fe6be0b12f1511aa1/data2/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e0f8c6646f8ce81fe6be0b12f1511aa1/data3/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e0f8c6646f8ce81fe6be0b12f1511aa1/data4/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data5/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data6/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data7/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data8/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data9/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data10/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data11/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
e18d850c6c53cdeb2e346fcd28c7a189/data12/cinder/volume-3f0577c0-2e64-4c8d-a8a8-2b8da6b8d001
对比发现，两者的md5一模一样，说明两者是相同的文件，互为镜像，至此，glusterfs和cinder联动配置完毕！！
5. 总结
cinder作为openstack中管理volume的一个服务，主要承担管理的角色，存储的功能，有专业的存储方案来完成，如本文的glusterfs开源分布式存储，此外，cinder还可以针对不同的后端设置不同的type，如后端可能是专业的存储服务器，或者是SSD构建的glusterfs，或者SATA构建的ceph存储，可以设置不同的type，分配给不同的虚拟机，已达到不同性能的需求，关于这些功能，参考openstack cinder的配置文档。
页:
[1]