OpenStack CEPH Liberty 统一存储 bug解决

82870034

　　1， 问题：
　　glance 镜像 raw 格式，启动云主机报错
2015-12-13 11:53:27.597 11654 INFO nova.virt.libvirt.driver [req-db7b8ac1-8218-4ab0-b769-9c4d4379843b e198d9f41e784308858a52b174fc822a a7fc63796b934498ba9b6d1a918cb2c7 - - -] [instance: 18efe519-e86b-43e9-842a-f974a751e29a] Creating image
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [req-db7b8ac1-8218-4ab0-b769-9c4d4379843b e198d9f41e784308858a52b174fc822a a7fc63796b934498ba9b6d1a918cb2c7 - - -] [instance: 18efe519-e86b-43e9-842a-f974a751e29a] Instance failed to spawn
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] Traceback (most recent call last):
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2155, in _build_resources
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     yield resources
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     block_device_info=block_device_info)
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2437, in spawn
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     admin_pass=admin_password)
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2861, in _create_image
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     instance, size, fallback_from_host)
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6282, in _try_fetch_image_cache
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     size=size)
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 240, in cache
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     *args, **kwargs)
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 777, in create_image
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     prepare_template(target=base, max_size=size, *args, **kwargs)
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in inner
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     return f(*args, **kwargs)
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 230, in fetch_func_sync
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     fetch_func(target=target, *args, **kwargs)
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2853, in clone_fallback_to_fetch
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     backend.clone(context, disk_images['image_id'])
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 808, in clone
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     if self.driver.is_cloneable(location, image_meta):
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 193, in is_cloneable
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a]     if image_meta.disk_format != 'raw':
2015-12-13 11:53:27.932 11654 ERROR nova.compute.manager [instance: 18efe519-e86b-43e9-842a-f974a751e29a] AttributeError: 'dict' object has no attribute 'disk_format'　　2，问题分析
1，查找问题定位：/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py
        import ipdb;ipdb.set_trace()
        if image_meta.disk_format != 'raw':
            reason = ("rbd image clone requires image format to be "
                      "'raw' but image {0} is '{1}'").format(
                          url, image_meta.disk_format)
            LOG.debug(reason)
            return False　　2, ipdb 启动服务追踪
[root@AthComputeLiberty10325 ~]# ipdb  /usr/bin/nova-compute --config-file=/etc/nova/nova.conf　　3，发现蛛丝马迹
AttributeError: "'dict' object has no attribute 'disk_format'"
> /usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py(194)is_cloneable()
    193
--> 194         if image_meta.disk_format != 'raw':
    195             reason = ("rbd image clone requires image format to be "
ipdb> args
self = <nova.virt.libvirt.storage.rbd_utils.RBDDriver object at 0x5f41850>
image_location = {'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}
image_meta = {'status': u'active', 'deleted_at': None, 'name': u'Ceph-linux6.6.v2.0', 'deleted': None, 'container_format': u'ovf', 'created_at': datetime.datetime(2015, 12, 12, 8, 47, 40, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'disk_format': u'raw', 'updated_at': datetime.datetime(2015, 12, 12, 9, 35, 22, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'locations': [{'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}], 'owner': u'a7fc63796b934498ba9b6d1a918cb2c7', 'id': u'05cfd97d-f0d8-43b4-9796-180f650e79ce', 'min_ram': 0, 'checksum': u'0909ad3eebe539efc2054196abffdb41', 'min_disk': 0, 'is_public': None, 'direct_url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'properties': {}, 'size': 128849018880}
ipdb> image_meta.disk_format
*** AttributeError: 'dict' object has no attribute 'disk_format'
ipdb> image_meta.disk_format()
*** AttributeError: 'dict' object has no attribute 'disk_format'
ipdb> image_meta.disk_format.values()
*** AttributeError: 'dict' object has no attribute 'disk_format'
ipdb> dir(image_meta.disk_format)
*** AttributeError: 'dict' object has no attribute 'disk_format'
ipdb> args
self = <nova.virt.libvirt.storage.rbd_utils.RBDDriver object at 0x5f41850>
image_location = {'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}
image_meta = {'status': u'active', 'deleted_at': None, 'name': u'Ceph-linux6.6.v2.0', 'deleted': None, 'container_format': u'ovf', 'created_at': datetime.datetime(2015, 12, 12, 8, 47, 40, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'disk_format': u'raw', 'updated_at': datetime.datetime(2015, 12, 12, 9, 35, 22, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'locations': [{'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}], 'owner': u'a7fc63796b934498ba9b6d1a918cb2c7', 'id': u'05cfd97d-f0d8-43b4-9796-180f650e79ce', 'min_ram': 0, 'checksum': u'0909ad3eebe539efc2054196abffdb41', 'min_disk': 0, 'is_public': None, 'direct_url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'properties': {}, 'size': 128849018880}
ipdb> image_meta
{'status': u'active', 'deleted_at': None, 'name': u'Ceph-linux6.6.v2.0', 'deleted': None, 'container_format': u'ovf', 'created_at': datetime.datetime(2015, 12, 12, 8, 47, 40, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'disk_format': u'raw', 'updated_at': datetime.datetime(2015, 12, 12, 9, 35, 22, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'locations': [{'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}], 'owner': u'a7fc63796b934498ba9b6d1a918cb2c7', 'id': u'05cfd97d-f0d8-43b4-9796-180f650e79ce', 'min_ram': 0, 'checksum': u'0909ad3eebe539efc2054196abffdb41', 'min_disk': 0, 'is_public': None, 'direct_url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'properties': {}, 'size': 128849018880}
ipdb> type(image_meta)
<type 'dict'>
ipdb> image_meta
{'status': u'active', 'deleted_at': None, 'name': u'Ceph-linux6.6.v2.0', 'deleted': None, 'container_format': u'ovf', 'created_at': datetime.datetime(2015, 12, 12, 8, 47, 40, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'disk_format': u'raw', 'updated_at': datetime.datetime(2015, 12, 12, 9, 35, 22, tzinfo=<iso8601.iso8601.Utc object at 0x604ae90>), 'locations': [{'url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'metadata': {}}], 'owner': u'a7fc63796b934498ba9b6d1a918cb2c7', 'id': u'05cfd97d-f0d8-43b4-9796-180f650e79ce', 'min_ram': 0, 'checksum': u'0909ad3eebe539efc2054196abffdb41', 'min_disk': 0, 'is_public': None, 'direct_url': u'rbd://03e180bd-7504-4883-8d0f-5506c63d4a4f/images/05cfd97d-f0d8-43b4-9796-180f650e79ce/snap', 'properties': {}, 'size': 128849018880}
ipdb> image_meta['disk_format']
u'raw'　　3，解决问题bug:
　　修改前：
　　if image_meta.disk_format != 'raw':
　　修改后：

　　if image_meta['disk_format'] != 'raw':
179
180     def is_cloneable(self, image_location, image_meta):
181         import ipdb;ipdb.set_trace()
182         url = image_location['url']
183         try:
184             fsid, pool, image, snapshot = self.parse_url(url)
185         except exception.ImageUnacceptable as e:
186             LOG.debug('not cloneable: %s', e)
187             return False
188
189         if self._get_fsid() != fsid:
190             reason = '%s is in a different ceph cluster' % url
191             LOG.debug(reason)
192             return False
193
194         # Add william , image_meta.disk_format != 'raw'
195         # so ...
196         if image_meta['disk_format'] != 'raw':
197             reason = ("rbd image clone requires image format to be "
198                       "'raw' but image {0} is '{1}'").format(
199                           url, image_meta.disk_format)
200             LOG.debug(reason)
201             return False　　

　　4, 完成操作

　　

         2015-12-13 13:18:36.659 19827 INFO nova.virt.libvirt.firewall [req-7d3b097b-701d-4352-a8cc-87bc6fb3f9f1 e198d9f41e784308858a52b174fc822a a7fc63796b934498ba9b6d1a918cb2c7 - - -] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] Called setup_basic_filtering in nwfilter
2015-12-13 13:18:36.660 19827 INFO nova.virt.libvirt.firewall [req-7d3b097b-701d-4352-a8cc-87bc6fb3f9f1 e198d9f41e784308858a52b174fc822a a7fc63796b934498ba9b6d1a918cb2c7 - - -] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] Ensuring static filters
2015-12-13 13:18:37.418 19827 INFO nova.compute.manager [-] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] VM Started (Lifecycle Event)
2015-12-13 13:18:37.441 19827 INFO nova.virt.libvirt.driver [-] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] Instance spawned successfully.
2015-12-13 13:18:37.525 19827 INFO nova.compute.manager [req-a92e6077-014c-421c-9fcc-9d21e6f7d420 - - - - -] [instance: 36025fe9-5abe-44de-bdb8-71983a3645f2] During sync_power_state the instance has a pending task (spawning). Skip.