Openstack 集成 Ceph 基础篇（一）

xyzjr

　　详细配置也可以参考Ceph官方文档：http://docs.ceph.org.cn/rbd/rbd-openstack/
　　环境介绍：
　　Openstack   controller+compute
　　Openstack-compute   compute
　　ceph001 OSD+Mon+MDS
　　ceph002 OSD+MDS
　　以上环境已提前准备完成，下面正式开始。

Ceph 中的配置
　　(1) 在 ceph 中创建三个 pool 分别给 Cinder，Glance 和 nova 使用



cephadmin@ceph001:~$ ceph osd pool create volumes 64
pool 'volumes' created
cephadmin@ceph001:~$ ceph osd pool create images 64
pool 'images' created
cephadmin@ceph001:~$ ceph osd pool create vms 64
pool 'vms' created

　　(2) 将Ceph配置文件Copy到Openstack各节点中
　　The nodes running glance-api, cinder-volume, nova-compute and cinder-backup act as Ceph clients. Each requires the ceph.conffile:



root@ceph001:~# ssh openstack tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf
root@openstack's password:
[global]
fsid = 4f1100a0-bc37-4472-b0b0-58b44eabac97
mon_initial_members = ceph001
mon_host = 192.168.20.178
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 2
root@ceph001:~# ssh openstack-compute tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf
The authenticity of host 'openstack-compute (192.168.20.182)' can't be established.
ECDSA key fingerprint is b7:bf:c5:81:0d:a0:2a:2d:94:2f:c1:16:78:f3:9f:b2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'openstack-compute,192.168.20.182' (ECDSA) to the list of known hosts.
root@openstack-compute's password:
[global]
fsid = 4f1100a0-bc37-4472-b0b0-58b44eabac97
mon_initial_members = ceph001
mon_host = 192.168.20.178
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_pool_default_size = 2

　　(3) 在各节点上安装ceph 客户端
　　http://docs.ceph.com/docs/master/rbd/rbd-openstack/
　　On the glance-api node, you’ll need the Python bindings for librbd:
　　#apt-get install python-ceph   //好像N版openstack已经自带了，并且官方介绍的是安装python-rbd 各种源都找不到这个包，手动下载也无法安装提示和python-ceph有冲突
　　On the nova-compute, cinder-backup and on the cinder-volume node, use both the Python bindings and the client command line tools:
　　#apt-get install ceph-common
　　(4) SETUP CEPH CLIENT AUTHENTICATION
　　If you have cephx authentication enabled, create a new user for Nova/Cinder and Glance. Execute the following:



#cinder 用户会被 cinder 和 nova 使用，需要访问三个pool: volumes,vms 为rwx权限，images为rx权限



#glance 用户只会被 Glance 使用，只需要访问 images 这个 pool



root@ceph001:~# ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'
[client.cinder]
key = AQBJg5hYuD0gHBAAproupKDYIuq0QemTnPMYdA==
root@ceph001:~# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
[client.glance]
key = AQC4g5hYMEVgGxAAoaVq50KgWV/y6YPzeHteGA==
root@ceph001:~#

　　(5) 将 client.cinder 和 client.glance 的 keystring 文件拷贝到各节点并设置访问权限



ceph auth get-or-create client.glance | ssh openstack sudo tee /etc/ceph/ceph.client.glance.keyring
ssh openstack  sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring    //我这里的openstack环境是devstack,所以应该将glance用户替换为stack
ceph auth get-or-create client.cinder | ssh openstack  sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh openstack sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring　　//我这里的openstack环境是devstack,所以应该将cinder用户替换为stack
ceph auth get-or-create client.cinder | ssh openstack-compute sudo tee /etc/ceph/ceph.client.cinder.keyring
　　Create a temporary copy of the secret key on the nodes running nova-compute:



ceph auth get-key client.cinder | ssh openstack tee client.cinder.key
ceph auth get-key client.cinder | ssh openstack-compute tee client.cinder.key

　　Then, on the compute nodes, add the secret key to libvirt and remove the temporary copy of the key:



root@openstack:~# uuidgen
2862d317-e8df-4a5c-af7a-387ab2bc7ef5
root@openstack:/etc/ceph# ls
ceph.client.cinder.keyring  ceph.client.glance.keyring  ceph.conf
root@openstack:/etc/ceph# cat > secret.xml <<EOF
> <secret ephemeral='no' private='no'>
>  <uuid>2862d317-e8df-4a5c-af7a-387ab2bc7ef5</uuid>
>  <usage type='ceph'>
>   <name>client.cinder secret</name>
>  </usage>
> </secret>
>
> EOF
root@openstack:~#  virsh secret-define --file secret.xml
Secret 2862d317-e8df-4a5c-af7a-387ab2bc7ef5 created
root@openstack:~# virsh secret-set-value --secret 2862d317-e8df-4a5c-af7a-387ab2bc7ef5 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml
Secret value set
root@openstack:~#
//在所有nova-compute节点都需要执行以上操作
//openstack-compute uuid c0f5d1a4-b086-4c0c-984e-2f4e84f0f9c5
//Important: You don’t necessarily need the UUID on all the compute nodes. However from a platform consistency perspective, it’s better to keep the same UUID.
CONFIGURE OPENSTACK TO USE CEPH
　　(1) CONFIGURING GLANCE
　　Edit /etc/glance/glance-api.conf and add under the [DEFAULT] section:



[glance_store]
stores = glance.store.rbd.Store
default_store = rbd
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8
If you want to enable copy-on-write cloning of images, also add under the [DEFAULT] section:
show_image_direct_url = True

　　(2) CONFIGURING CINDER
　　OpenStack requires a driver to interact with Ceph block devices. You must also specify the pool name for the block device. On your OpenStack node, edit /etc/cinder/cinder.conf by adding:



[DEFAULT]
...
enabled_backends = ceph
...
[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = stack
rbd_secret_uuid = 2862d317-e8df-4a5c-af7a-387ab2bc7ef5

　　(3) CONFIGURING NOVA



在每个计算节点上的 /etc/nova/nova.conf 文件中做如下修改：
[libvirt]
images_type = rbd #只有在 boot disk 放在 ceph 中才需要配置这个，否则，设置为 qcow2
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = e21a123a-31f8-425a-86db-7204c33a6161
disk_cachemodes="network=writeback"
hw_disk_discard = unmap
inject_password = false
inject_key = false
inject_partition = -2
live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"
　　重启服务

glance,cinder,nova


环境验证

　　未完待续...