设为首页 收藏本站
查看: 1157|回复: 0

[经验分享] OpenStack从数据库恢复Volume状态

[复制链接]
累计签到:1 天
连续签到:1 天
发表于 2016-1-9 09:11:51 | 显示全部楼层 |阅读模式
问题
  OpenStack中很容易导致数据库和真实状态不一致的情况。因为OpenStack中操作基本都是分步完成的,从api接受请求到调度再到具体的操作节点,每一步都有可能更新数据库状态,如果哪一个出错就会直接抛出异常导致整个操作链中断,然后数据库就处于上一个操作后的更新状态。比较典型的就是删除实例,如果在nova-compute出错那这个实例的状态就可能永远处于deleting状态了。
  现在我遇到这样一个问题,我有一个Volume挂载在一个实例上,但是不知道什么原因,这个Volume与这个实例的联系断了,在nova-volume通过tgtadm查看发现已经没有客户端连接到该Volume了。但是数据库中该记录还在,这导致以下结果:

1) 删除实例时无法删除,提示“Stderr: 'iscsiadm: No records found'”

2) 无法从实例卸载Volume 。于是只能直接操作数据库了。

与Volume相关的表
  数据库中与Volume直接相关的几个表如下所示
DSC0000.jpg

操作Volume时数据库的相关数据变化

新建Volume:

select * from volumes where id = 40\G
*************************** 1. row ***************************
created_at: 2012-10-29 07:00:23
updated_at: 2012-10-29 07:00:25
deleted_at: NULL
deleted: 0
id: 40
ec2_id: NULL
user_id: 397dd3be88b6492caa88521502b07617
project_id: c6159a4f3dd34a2b83527499a40dbd2b
host: store2.sigsit.org
size: 20
availability_zone: nova
instance_id: NULL
mountpoint: NULL
attach_time: NULL
status: available
attach_status: detached
scheduled_at: 2012-10-29 07:00:23
launched_at: 2012-10-29 07:00:25
terminated_at: NULL
display_name: test
display_description:
provider_location: 10.61.2.14:3260,5 iqn.2010-10.org.openstack:volume-00000028 1
provider_auth: NULL
snapshot_id: NULL
volume_type_id: NULL
select * from volume_metadata where volume_id = 40\G
select * from iscsi_targets where volume_id = 40\G
*************************** 1. row ***************************
created_at: 2012-09-24 09:00:36
updated_at: 2012-10-29 07:00:24
deleted_at: NULL
deleted: 0
id: 205
target_num: 5
host: store2.sigsit.org
volume_id: 40
select * from block_device_mapping where volume_id = 40\G
select * from sm_volume where id = 40\G

将该Volume挂载到一个实例后:

select * from volumes where id = 40\G
*************************** 1. row ***************************
created_at: 2012-10-29 07:00:23
updated_at: 2012-10-29 11:55:36
deleted_at: NULL
deleted: 0
id: 40
ec2_id: NULL
user_id: 397dd3be88b6492caa88521502b07617
project_id: c6159a4f3dd34a2b83527499a40dbd2b
host: store2.sigsit.org
size: 20
availability_zone: nova
instance_id: 70
mountpoint: /dev/vdc
attach_time: NULL
status: in-use
attach_status: attached
scheduled_at: 2012-10-29 07:00:23
launched_at: 2012-10-29 07:00:25
terminated_at: NULL
display_name: test
display_description:
provider_location: 10.61.2.14:3260,5 iqn.2010-10.org.openstack:volume-00000028 1
provider_auth: NULL
snapshot_id: NULL
volume_type_id: NULL
select * from volume_metadata where volume_id = 40\G
select * from iscsi_targets where volume_id = 40\G
*************************** 1. row ***************************
created_at: 2012-09-24 09:00:36
updated_at: 2012-10-29 07:00:24
deleted_at: NULL
deleted: 0
id: 205
target_num: 5
host: store2.sigsit.org
volume_id: 40
select * from block_device_mapping where volume_id = 40\G
*************************** 1. row ***************************
created_at: 2012-10-29 11:55:36
updated_at: NULL
deleted_at: NULL
deleted: 0
id: 49
instance_id: 70
device_name: /dev/vdc
delete_on_termination: 0
virtual_name: NULL
snapshot_id: NULL
volume_id: 40
volume_size: NULL
no_device: NULL
connection_info: {"driver_volume_type": "iscsi", "data": {"device_path": "/dev/disk/by-path/ip-10.61.2.14:3260-iscsi-iqn.2010-10.org.openstack:volume-00000028-lun-1", "target_discovered": false, "target_iqn": "iqn.2010-10.org.openstack:volume-00000028", "target_portal": "10.61.2.14:3260", "volume_id": 40, "target_lun": 1}}
select * from sm_volume where id = 40\G
select * from instances where id = 70\G
*************************** 1. row ***************************
created_at: 2012-09-10 02:32:36
updated_at: 2012-09-12 10:43:48
deleted_at: NULL
deleted: 0
id: 70
internal_id: NULL
user_id: 397dd3be88b6492caa88521502b07617
project_id: c6159a4f3dd34a2b83527499a40dbd2b
image_ref: 6c239063-9d2a-41ce-9612-bfe3564cc203
kernel_id:
ramdisk_id:
server_name: NULL
launch_index: 0
key_name: NULL
key_data: NULL
power_state: 1
vm_state: active
memory_mb: 1024
vcpus: 1
hostname: jiangyong-win7
host: stack6.sigsit.org
user_data:
reservation_id: r-h1yqckm4
scheduled_at: 2012-09-10 02:32:37
launched_at: 2012-09-10 02:32:48
terminated_at: NULL
display_name: jiangyong-win7
display_description: jiangyong-win7
availability_zone: NULL
locked: 0
os_type: NULL
launched_on: stack6.sigsit.org
instance_type_id: 19
vm_mode: NULL
uuid: 333f6afa-9009-40f7-a493-20b2382628b1
architecture: NULL
root_device_name: /dev/vda
access_ip_v4: NULL
access_ip_v6: NULL
config_drive:
task_state: NULL
default_ephemeral_device: NULL
default_swap_device: NULL
progress: 0
auto_disk_config: NULL
shutdown_terminate: 1
disable_terminate: 0
root_gb: 0
ephemeral_gb: 0
cell_name: NULL

将该Volume从实例卸载:

select * from volumes where id = 40\G
*************************** 1. row ***************************
created_at: 2012-10-29 07:00:23
updated_at: 2012-10-29 11:58:36
deleted_at: NULL
deleted: 0
id: 40
ec2_id: NULL
user_id: 397dd3be88b6492caa88521502b07617
project_id: c6159a4f3dd34a2b83527499a40dbd2b
host: store2.sigsit.org
size: 20
availability_zone: nova
instance_id: NULL
mountpoint: NULL
attach_time: NULL
status: available
attach_status: detached
scheduled_at: 2012-10-29 07:00:23
launched_at: 2012-10-29 07:00:25
terminated_at: NULL
display_name: test
display_description:
provider_location: 10.61.2.14:3260,5 iqn.2010-10.org.openstack:volume-00000028 1
provider_auth: NULL
snapshot_id: NULL
volume_type_id: NULL
select * from volume_metadata where volume_id = 40\G
select * from iscsi_targets where volume_id = 40\G
*************************** 1. row ***************************
created_at: 2012-09-24 09:00:36
updated_at: 2012-10-29 07:00:24
deleted_at: NULL
deleted: 0
id: 205
target_num: 5
host: store2.sigsit.org
volume_id: 40
select * from block_device_mapping where volume_id = 40\G
*************************** 1. row ***************************
created_at: 2012-10-29 11:55:36
updated_at: NULL
deleted_at: 2012-10-29 11:58:36
deleted: 1
id: 49
instance_id: 70
device_name: /dev/vdc
delete_on_termination: 0
virtual_name: NULL
snapshot_id: NULL
volume_id: 40
volume_size: NULL
no_device: NULL
connection_info: {"driver_volume_type": "iscsi", "data": {"device_path": "/dev/disk/by-path/ip-10.61.2.14:3260-iscsi-iqn.2010-10.org.openstack:volume-00000028-lun-1", "target_discovered": false, "target_iqn": "iqn.2010-10.org.openstack:volume-00000028", "target_portal": "10.61.2.14:3260", "volume_id": 40, "target_lun": 1}}
select * from sm_volume where id = 40\G
select * from instances where id = 70\G
*************************** 1. row ***************************
created_at: 2012-09-10 02:32:36
updated_at: 2012-09-12 10:43:48
deleted_at: NULL
deleted: 0
id: 70
internal_id: NULL
user_id: 397dd3be88b6492caa88521502b07617
project_id: c6159a4f3dd34a2b83527499a40dbd2b
image_ref: 6c239063-9d2a-41ce-9612-bfe3564cc203
kernel_id:
ramdisk_id:
server_name: NULL
launch_index: 0
key_name: NULL
key_data: NULL
power_state: 1
vm_state: active
memory_mb: 1024
vcpus: 1
hostname: jiangyong-win7
host: stack6.sigsit.org
user_data:
reservation_id: r-h1yqckm4
scheduled_at: 2012-09-10 02:32:37
launched_at: 2012-09-10 02:32:48
terminated_at: NULL
display_name: jiangyong-win7
display_description: jiangyong-win7
availability_zone: NULL
locked: 0
os_type: NULL
launched_on: stack6.sigsit.org
instance_type_id: 19
vm_mode: NULL
uuid: 333f6afa-9009-40f7-a493-20b2382628b1
architecture: NULL
root_device_name: /dev/vda
access_ip_v4: NULL
access_ip_v6: NULL
config_drive:
task_state: NULL
default_ephemeral_device: NULL
default_swap_device: NULL
progress: 0
auto_disk_config: NULL
shutdown_terminate: 1
disable_terminate: 0
root_gb: 0
ephemeral_gb: 0
cell_name: NULL

再次将该Volume挂载到该实例:

select * from volumes where id = 40\G
*************************** 1. row ***************************
created_at: 2012-10-29 07:00:23
updated_at: 2012-10-29 12:00:32
deleted_at: NULL
deleted: 0
id: 40
ec2_id: NULL
user_id: 397dd3be88b6492caa88521502b07617
project_id: c6159a4f3dd34a2b83527499a40dbd2b
host: store2.sigsit.org
size: 20
availability_zone: nova
instance_id: 70
mountpoint: /dev/vdc
attach_time: NULL
status: in-use
attach_status: attached
scheduled_at: 2012-10-29 07:00:23
launched_at: 2012-10-29 07:00:25
terminated_at: NULL
display_name: test
display_description:
provider_location: 10.61.2.14:3260,5 iqn.2010-10.org.openstack:volume-00000028 1
provider_auth: NULL
snapshot_id: NULL
volume_type_id: NULL
select * from volume_metadata where volume_id = 40\G
select * from iscsi_targets where volume_id = 40\G
*************************** 1. row ***************************
created_at: 2012-09-24 09:00:36
updated_at: 2012-10-29 07:00:24
deleted_at: NULL
deleted: 0
id: 205
target_num: 5
host: store2.sigsit.org
volume_id: 40
select * from block_device_mapping where volume_id = 40\G
*************************** 1. row ***************************
created_at: 2012-10-29 11:55:36
updated_at: NULL
deleted_at: 2012-10-29 11:58:36
deleted: 1
id: 49
instance_id: 70
device_name: /dev/vdc
delete_on_termination: 0
virtual_name: NULL
snapshot_id: NULL
volume_id: 40
volume_size: NULL
no_device: NULL
connection_info: {"driver_volume_type": "iscsi", "data": {"device_path": "/dev/disk/by-path/ip-10.61.2.14:3260-iscsi-iqn.2010-10.org.openstack:volume-00000028-lun-1", "target_discovered": false, "target_iqn": "iqn.2010-10.org.openstack:volume-00000028", "target_portal": "10.61.2.14:3260", "volume_id": 40, "target_lun": 1}}
*************************** 2. row ***************************
created_at: 2012-10-29 12:00:32
updated_at: NULL
deleted_at: NULL
deleted: 0
id: 50
instance_id: 70
device_name: /dev/vdc
delete_on_termination: 0
virtual_name: NULL
snapshot_id: NULL
volume_id: 40
volume_size: NULL
no_device: NULL
connection_info: {"driver_volume_type": "iscsi", "data": {"device_path": "/dev/disk/by-path/ip-10.61.2.14:3260-iscsi-iqn.2010-10.org.openstack:volume-00000028-lun-1", "target_discovered": false, "target_iqn": "iqn.2010-10.org.openstack:volume-00000028", "target_portal": "10.61.2.14:3260", "volume_id": 40, "target_lun": 1}}
select * from sm_volume where id = 40\G
select * from instances where id = 70\G
*************************** 1. row ***************************
created_at: 2012-09-10 02:32:36
updated_at: 2012-09-12 10:43:48
deleted_at: NULL
deleted: 0
id: 70
internal_id: NULL
user_id: 397dd3be88b6492caa88521502b07617
project_id: c6159a4f3dd34a2b83527499a40dbd2b
image_ref: 6c239063-9d2a-41ce-9612-bfe3564cc203
kernel_id:
ramdisk_id:
server_name: NULL
launch_index: 0
key_name: NULL
key_data: NULL
power_state: 1
vm_state: active
memory_mb: 1024
vcpus: 1
hostname: jiangyong-win7
host: stack6.sigsit.org
user_data:
reservation_id: r-h1yqckm4
scheduled_at: 2012-09-10 02:32:37
launched_at: 2012-09-10 02:32:48
terminated_at: NULL
display_name: jiangyong-win7
display_description: jiangyong-win7
availability_zone: NULL
locked: 0
os_type: NULL
launched_on: stack6.sigsit.org
instance_type_id: 19
vm_mode: NULL
uuid: 333f6afa-9009-40f7-a493-20b2382628b1
architecture: NULL
root_device_name: /dev/vda
access_ip_v4: NULL
access_ip_v6: NULL
config_drive:
task_state: NULL
default_ephemeral_device: NULL
default_swap_device: NULL
progress: 0
auto_disk_config: NULL
shutdown_terminate: 1
disable_terminate: 0
root_gb: 0
ephemeral_gb: 0
cell_name: NULL

再次卸载该Volume并删除:


select * from volumes where id = 40\G
*************************** 1. row ***************************
created_at: 2012-10-29 07:00:23
updated_at: 2012-10-29 13:41:56
deleted_at: 2012-10-29 13:44:36
deleted: 1
id: 40
ec2_id: NULL
user_id: 397dd3be88b6492caa88521502b07617
project_id: c6159a4f3dd34a2b83527499a40dbd2b
host: store2.sigsit.org
size: 20
availability_zone: nova
instance_id: NULL
mountpoint: NULL
attach_time: NULL
status: deleting
attach_status: detached
scheduled_at: 2012-10-29 07:00:23
launched_at: 2012-10-29 07:00:25
terminated_at: 2012-10-29 13:41:55
display_name: test
display_description:
provider_location: 10.61.2.14:3260,5 iqn.2010-10.org.openstack:volume-00000028 1
provider_auth: NULL
snapshot_id: NULL
volume_type_id: NULL
select * from volume_metadata where volume_id = 40\G
select * from iscsi_targets where volume_id = 40\G
select * from block_device_mapping where volume_id = 40\G
*************************** 1. row ***************************
created_at: 2012-10-29 11:55:36
updated_at: NULL
deleted_at: 2012-10-29 11:58:36
deleted: 1
id: 49
instance_id: 70
device_name: /dev/vdc
delete_on_termination: 0
virtual_name: NULL
snapshot_id: NULL
volume_id: 40
volume_size: NULL
no_device: NULL
connection_info: {"driver_volume_type": "iscsi", "data": {"device_path": "/dev/disk/by-path/ip-10.61.2.14:3260-iscsi-iqn.2010-10.org.openstack:volume-00000028-lun-1", "target_discovered": false, "target_iqn": "iqn.2010-10.org.openstack:volume-00000028", "target_portal": "10.61.2.14:3260", "volume_id": 40, "target_lun": 1}}
*************************** 2. row ***************************
created_at: 2012-10-29 12:00:32
updated_at: NULL
deleted_at: 2012-10-29 13:10:51
deleted: 1
id: 50
instance_id: 70
device_name: /dev/vdc
delete_on_termination: 0
virtual_name: NULL
snapshot_id: NULL
volume_id: 40
volume_size: NULL
no_device: NULL
connection_info: {"driver_volume_type": "iscsi", "data": {"device_path": "/dev/disk/by-path/ip-10.61.2.14:3260-iscsi-iqn.2010-10.org.openstack:volume-00000028-lun-1", "target_discovered": false, "target_iqn": "iqn.2010-10.org.openstack:volume-00000028", "target_portal": "10.61.2.14:3260", "volume_id": 40, "target_lun": 1}}
select * from sm_volume where id = 40\G

相关结论

创建Volume对数据库的修改
修改volumes表,添加一条Volume记录。
修改iscsi_targets表,寻找一个可用的target记录,将该记录的volume_id设置为新添volume的id。target记录有target_num和host信息,然后nova会用这两个值去相应的主机用这个target_num创建Volume。

挂载Volume对数据库的修改
修改volumes表,设置instance_id、mountpoint为实例id、设备名,修改Volume的状态status、attach_status为in-use、attached

修改block_device_mapping表,添加一条映射记录,包括包含实例和卷的信息,特别是卷的连接信息。
卸载Volume对数据库的修改
修改volumes表,设置instance_id、mountpoint为null,修改Volume的状态status、attach_status为available、detached

修改block_device_mapping表,修改相应的映射记录,设置deleted_at时间及deleted为1。
删除Volume对数据库的修改
修改volumes表,设置deleted_at时间及deleted为1。
修改iscsi_targets表,修改将volume_id对应的记录,设置volume_id为null。   在数据库中修改以上四个数值,Volume可以成功挂载、卸载到其它实例,原来的问题实例也可以正常删除了。当然具体情况还得具体分析,我已经查看过Volume的状态知道已经实质上被卸载了,没有客户端连接到该Volume上,所以只要在数据库中将该Volume恢复成未挂载状态即可。
  因此,当Volume处于不一致的状态时,首先应当登录Volume所在的存储节点,通过tgtadm --lld iscsi --mode target --op show命令查看Volume的状态:Volume是不是还存在?有没有连接的客户端?然后修改相关的数据库状态值。
  北方工业大学 |
云计算研究中心 | 姜永

运维网声明 1、欢迎大家加入本站运维交流群:群②:261659950 群⑤:202807635 群⑦870801961 群⑧679858003
2、本站所有主题由该帖子作者发表,该帖子作者与运维网享有帖子相关版权
3、所有作品的著作权均归原作者享有,请您和我们一样尊重他人的著作权等合法权益。如果您对作品感到满意,请购买正版
4、禁止制作、复制、发布和传播具有反动、淫秽、色情、暴力、凶杀等内容的信息,一经发现立即删除。若您因此触犯法律,一切后果自负,我们对此不承担任何责任
5、所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其内容的准确性、可靠性、正当性、安全性、合法性等负责,亦不承担任何法律责任
6、所有作品仅供您个人学习、研究或欣赏,不得用于商业或者其他用途,否则,一切后果均由您自己承担,我们对此不承担任何法律责任
7、如涉及侵犯版权等问题,请您及时通知我们,我们将立即采取措施予以解决
8、联系人Email:admin@iyunv.com 网址:www.yunweiku.com

所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其承担任何法律责任,如涉及侵犯版权等问题,请您及时通知我们,我们将立即处理,联系人Email:kefu@iyunv.com,QQ:1061981298 本贴地址:https://www.yunweiku.com/thread-162063-1-1.html 上篇帖子: 在CentOS上安装和配置OpenStack Nova 下篇帖子: 在OpenStack使用XenServer资源池浅析
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

扫码加入运维网微信交流群X

扫码加入运维网微信交流群

扫描二维码加入运维网微信交流群,最新一手资源尽在官方微信交流群!快快加入我们吧...

扫描微信二维码查看详情

客服E-mail:kefu@iyunv.com 客服QQ:1061981298


QQ群⑦:运维网交流群⑦ QQ群⑧:运维网交流群⑧ k8s群:运维网kubernetes交流群


提醒:禁止发布任何违反国家法律、法规的言论与图片等内容;本站内容均来自个人观点与网络等信息,非本站认同之观点.


本站大部分资源是网友从网上搜集分享而来,其版权均归原作者及其网站所有,我们尊重他人的合法权益,如有内容侵犯您的合法权益,请及时与我们联系进行核实删除!



合作伙伴: 青云cloud

快速回复 返回顶部 返回列表