openstack kilo 集成ceph
1，环境准备10.0.1.100 cephdeploy
10.0.1.110cephmon1
10.0.1.120cephmon2
10.0.1.130cephosd1
10.0.1.140cephosd2
10.0.1.150cephosd3
10.0.1.11 controller
10.0.1.31 compute1
10.0.1.41 block
ceph和 openstack的安装略
2.创建3个pool
ceph osd pool create volumes 128
ceph osd pool create images 128
ceph osd pool create vms 128 3，拷贝ceph.conf到openstack各节点
ssh {your-openstack-server} sudo tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf
4，安装ceph包 在openstack节点上
glance-api节点
apt-get installpython-rbd
其他节点nova 和volumes
apt-get install ceph-common
5，设置认证
ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'
ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images' 6，拷贝认证到各节点
ceph auth get-or-create client.glance | ssh 10.0.1.11 sudo tee /etc/ceph/ceph.client.glance.keyring
ssh 10.0.1.11 sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring
ceph auth get-or-create client.cinder | ssh 10.0.1.41 sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh 10.0.1.41 sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder | ssh 10.0.1.31 sudo tee /etc/ceph/ceph.client.cinder.keyring
ceph auth get-key client.cinder | ssh 10.0.1.31 tee client.cinder.key 5.compute节点配置
#uuidgen
17abcd6f-109d-4e95-8115-4c31a6caa084
#cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
<uuid>17abcd6f-109d-4e95-8115-4c31a6caa084</uuid>
<usage type='ceph'>
<name>client.cinder secret</name>
</usage>
</secret>
EOF
#sudo virsh secret-define --file secret.xml
#sudo virsh secret-set-value --secret 17abcd6f-109d-4e95-8115-4c31a6caa084 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml
编辑/etc/ceph/ceph.conf最后加上如下内容
rbd cache = true
rbd cache writethrough until flush = true
admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/qemu/qemu-guest-$pid.log
rbd concurrent management ops = 20 编辑/etc/nova/nova.conf 在字段配置加入如下，uuid参考上面的保持同步
libvirt_images_type = rbd
libvirt_images_rbd_pool = vms
libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf
libvirt_disk_cachemodes="network=writeback"
rbd_user = cinder
rbd_secret_uuid = 17abcd6f-109d-4e95-8115-4c31a6caa084
libvirt_inject_password = false
libvirt_inject_key = false
libvirt_inject_partition = -2
libvirt_live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED" 设置权限
mkdir -p /var/run/ceph/guests/ /var/log/qemu/
chown libvirt-qemu:libvirtd /var/run/ceph/guests /var/log/qemu/ 重启启动nova-compute服务
6，glance节点配置
编辑/etc/glance/glance-api.conf
show_image_direct_url = True
default_store = rbd
stores = rbd
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8
重启glance-api服务
7，cinder配置vim /etc/cinder/cinder.confuuid和之前一致
enabled_backends = ceph
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder
rbd_secret_uuid = 17abcd6f-109d-4e95-8115-4c31a6caa084
重启cinder-volumes服务
效果测试
我创建一个50g的硬盘 并连接c2
在这台主机上我们查看这个硬盘
在ceph-deploy上查看使用情况，发现volume已经开始在使用了
写入一个数据，我写入一个2G的文件
再次查看ceph-delpoy，发现volumes使用如下：
页:
[1]