opennebula4.2管理vmware esxi问题汇总

mahonglin123456

　　2013-11-04
　　问题1：failed: cp: omitting directory '/vmfs/volumes/1/34c24021539b1bc16507e09ad4553f9b'
　　Log：
　　Mon Nov 4 15:01:39 2013 [DiM][I]: New VM state is ACTIVE.
　　Mon Nov 4 15:01:39 2013 [LCM][I]: New VM state is PROLOG.
　　Mon Nov 4 15:01:39 2013 [TM][I]: Command execution fail: /srv/cloud/one/var/remotes/tm/shared/clone nebula:/srv/cloud/one/var/datastores/1/34c24021539b1bc16507e09ad4553f9b esxi01:/vmfs/volumes/0/1/disk.0 1 1
　　Mon Nov 4 15:01:39 2013 [TM][I]: clone: Cloning /vmfs/volumes/1/34c24021539b1bc16507e09ad4553f9b in esxi01:/vmfs/volumes/0/1/disk.0
　　Mon Nov 4 15:01:39 2013 [TM][E]: clone: Command "cd /vmfs/volumes/0/1; cp /vmfs/volumes/1/34c24021539b1bc16507e09ad4553f9b /vmfs/volumes/0/1/disk.0" failed: cp: omitting directory '/vmfs/volumes/1/34c24021539b1bc16507e09ad4553f9b'
　　Mon Nov 4 15:01:39 2013 [TM][E]: Error copying nebula:/srv/cloud/one/var/datastores/1/34c24021539b1bc16507e09ad4553f9b to esxi01:/vmfs/volumes/0/1/disk.0
　　Mon Nov 4 15:01:39 2013 [TM][I]: ExitCode: 1
　　Mon Nov 4 15:01:39 2013 [TM][E]: Error executing image transfer script: Error copying nebula:/srv/cloud/one/var/datastores/1/34c24021539b1bc16507e09ad4553f9b to esxi01:/vmfs/volumes/0/1/disk.0
　　Mon Nov 4 15:01:39 2013 [DiM][I]: New VM state is FAILED
　　

　　解决办法：
　　nebula脚本代码中CP文件夹命令错误，缺少-r参数：
　　[oneadmin@nebula ~]$ vi  /srv/cloud/one/var/remotes/tm/shared/clone
　　其中一行："cd $DST_DIR; cp $SRC_PATH $DST_PATH" \
　　修改为："cd $DST_DIR; cp -r $SRC_PATH $DST_PATH" \
　　*************************************************************************************************
　　

　　问题2：Operation not permitted (running as non-root?)
　　Log：

　　
　　Mon Nov 4 15:47:45 2013 [DiM][I]: New VM state is ACTIVE.
　　Mon Nov 4 15:47:45 2013 [LCM][I]: New VM state is PROLOG.
　　Mon Nov 4 15:48:02 2013 [LCM][I]: New VM state is BOOT
　　Mon Nov 4 15:48:02 2013 [VMM][I]: Generating deployment file: /srv/cloud/one/var/vms/4/deployment.0
　　Mon Nov 4 15:48:02 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/vnm/vmware/pre PFZNPjxJRD40PC9JRD48VUlEPjA8L1...................
　　Mon Nov 4 15:48:02 2013 [VMM][E]: pre: Error executing: ((esxcfg-vswitch vSwitch0 -l|grep one-pg-0) || /sbin/esxcfg-vswitch vSwitch0 --add-pg one-pg-0); /sbin/esxcfg-vswitch vSwitch0 -p one-pg-0 --vlan=10 on host: err: Error during version check: Failed to get vmkernel version: Operation not permitted (running as non-root?)
　　Mon Nov 4 15:48:02 2013 [VMM][I]: Error during version check: Failed to get vmkernel version: Operation not permitted (running as non-root?)
　　Mon Nov 4 15:48:02 2013 [VMM][I]: Error during version check: Failed to get vmkernel version: Operation not permitted (running as non-root?)
　　Mon Nov 4 15:48:02 2013 [VMM][I]: ExitCode: 255
　　Mon Nov 4 15:48:02 2013 [VMM][I]: out:
　　Mon Nov 4 15:48:02 2013 [VMM][I]: ExitCode: 0
　　Mon Nov 4 15:48:02 2013 [VMM][I]: Failed to execute network driver operation: pre.
　　Mon Nov 4 15:48:02 2013 [VMM][E]: Error deploying virtual machine
　　Mon Nov 4 15:48:02 2013 [DiM][I]: New VM state is FAILED
　　

　　解决办法：
　　esxi端使得普通用户拥有执行esxcfg-vswitch命令的权限：
　　~# chmod +s /sbin/esxcfg-vswitch
　　*************************************************************************************************
　　

　　问题3：deploy: Error executing: virsh -c 'esx://esxi02/?no_verify=1&auto_answer=1' start one-8
　　
　　

　　Log：
　　Mon Nov 4 16:23:54 2013 [DiM][I]: New VM state is ACTIVE.
　　Mon Nov 4 16:23:54 2013 [LCM][I]: New VM state is PROLOG.
　　Mon Nov 4 16:24:11 2013 [LCM][I]: New VM state is BOOT
　　Mon Nov 4 16:24:11 2013 [VMM][I]: Generating deployment file: /srv/cloud/one/var/vms/8/deployment.0
　　Mon Nov 4 16:24:11 2013 [VMM][I]: Successfully execute network driver operation: pre.
　　Mon Nov 4 16:24:25 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/vmm/vmware/deploy '/srv/cloud/one/var/vms/8/deployment.0' 'esxi02' 8 esxi02
　　Mon Nov 4 16:24:25 2013 [VMM][D]: deploy: Successfully defined domain one-8.
　　Mon Nov 4 16:24:25 2013 [VMM][E]: deploy: Error executing: virsh -c 'esx://esxi02/?no_verify=1&auto_answer=1' start one-8 err: ExitCode: 1
　　Mon Nov 4 16:24:25 2013 [VMM][I]: out:
　　Mon Nov 4 16:24:25 2013 [VMM][I]: error: Failed to start domain one-8
　　Mon Nov 4 16:24:25 2013 [VMM][I]: error: internal error Could not start domain: FileNotFound - File [0] 8/disk.0/disk.vmdk was not found
　　Mon Nov 4 16:24:25 2013 [VMM][I]:
　　Mon Nov 4 16:24:25 2013 [VMM][I]: ExitCode: 1
　　Mon Nov 4 16:24:25 2013 [VMM][I]: Failed to execute virtualization driver operation: deploy.
　　Mon Nov 4 16:24:25 2013 [VMM][E]: Error deploying virtual machine
　　Mon Nov 4 16:24:25 2013 [DiM][I]: New VM state is FAILED
　　

　　解决办法：
　　官方应用市场的ttylinux镜像命名存在一点问题，需要将ttylinux.vmdk名称改为disk.vmdk
　　[oneadmin@nebula ~]$ cd var/datastores/
　　[oneadmin@nebula datastores]$ ls
　　0  1  2
　　[oneadmin@nebula datastores]$ cd 1/
　　[oneadmin@nebula 1]$ ls
　　34c24021539b1bc16507e09ad4553f9b
　　[oneadmin@nebula 1]$ cd 34c24021539b1bc16507e09ad4553f9b/
　　[oneadmin@nebula 34c24021539b1bc16507e09ad4553f9b]$ ls
　　ttylinux-flat.vmdk  ttylinux.vmdk
　　[oneadmin@nebula 34c24021539b1bc16507e09ad4553f9b]$ mv ttylinux.vmdk disk.vmdk
　　[oneadmin@nebula 34c24021539b1bc16507e09ad4553f9b]$ ls
　　disk.vmdk  ttylinux-flat.vmdk
　　[oneadmin@nebula 34c24021539b1bc16507e09ad4553f9b]$
　　*************************************************************************************************

　　问题4：sh: /var/tmp/one/vmm/vmware/attach_disk: Permission denied
　　Log：
　　Wed Nov 6 14:34:51 2013 [VMM][I]: Successfully execute transfer manager driver operation: tm_attach.
　　Wed Nov 6 14:34:52 2013 [VMM][I]: Remote worker node files not found
　　Wed Nov 6 14:34:52 2013 [VMM][I]: Updating remotes
　　Wed Nov 6 14:35:10 2013 [VMM][I]: Command execution fail: /var/tmp/one/vmm/vmware/attach_disk 'one-18' '/vmfs/volumes/0/18/disk.2' 'VMWare' '4' 'drv_message' 18 esxi02
　　Wed Nov 6 14:35:10 2013 [VMM][I]: sh: /var/tmp/one/vmm/vmware/attach_disk: Permission denied
　　Wed Nov 6 14:35:10 2013 [VMM][I]: ExitCode: 126
　　Wed Nov 6 14:35:10 2013 [VMM][I]: Failed to execute virtualization driver operation: attach_disk.
　　Wed Nov 6 14:35:10 2013 [VMM][E]: Error attaching new VM Disk
解决办法：
在oned.conf中修改Nebula默认脚本路径：
　　SCRIPTS_REMOTE_DIR=/var/tmp/one
　　修改为目前的脚本路径，比如：
　　SCRIPTS_REMOTE_DIR=/srv/cloud/one/var/remotes
　　*************************************************************************************************
　　问题5：mkimage脚本文件
　　Log:
　　
　　Thu Nov 7 15:17:40 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/tm/shared/mkimage 256 vmdk_vmfs esxi02:/vmfs/volumes/0/36/disk.2 36 0
　　Thu Nov 7 15:17:40 2013 [VMM][I]: mkimage: Making filesystem of 256M and type vmdk_vmfs at esxi02:/vmfs/volumes/0/36/disk.2
　　Thu Nov 7 15:17:40 2013 [VMM][E]: mkimage: Command "export PATH=/usr/sbin:/sbin:$PATH
　　Thu Nov 7 15:17:40 2013 [VMM][I]: # export PATH=/usr/sbin:/sbin:/bin:/sbin:/usr/bin:/srv/cloud/one/bin:/usr/local/bin:/var/lib/gems/1.8/bin/:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/srv/cloud/one/bin
　　Thu Nov 7 15:17:40 2013 [VMM][I]: dd if=/dev/zero of=/vmfs/volumes/0/36/disk.2 bs=1 count=1 seek=256M
　　Thu Nov 7 15:17:40 2013 [VMM][I]: vmkfstools -U /vmfs/volumes/0/36/disk.2/disk.vmdk; rm -f /vmfs/volumes/0/36/disk.2/*; vmkfstools -c 256M -d thin /vmfs/volumes/0/36/disk.2/disk.vmdk" failed: 1+0 records in
　　Thu Nov 7 15:17:40 2013 [VMM][I]: 1+0 records out
　　Thu Nov 7 15:17:40 2013 [VMM][I]: Failed to delete virtual disk: Not a directory (1310729).
　　Thu Nov 7 15:17:40 2013 [VMM][I]: rm: cannot stat '/vmfs/volumes/0/36/disk.2/*': Not a directory
　　Thu Nov 7 15:17:40 2013 [VMM][I]: File_GetFreeSpace: Couldn't statfs /vmfs/volumes/0/36/disk.2/disk.vmdk
　　Thu Nov 7 15:17:40 2013 [VMM][I]: Failed to create virtual disk: One of the parameters supplied is invalid (1).
　　Thu Nov 7 15:17:40 2013 [VMM][E]: Could not create image /vmfs/volumes/0/36/disk.2
　　Thu Nov 7 15:17:40 2013 [VMM][I]: ExitCode: 255
　　Thu Nov 7 15:17:40 2013 [VMM][I]: Failed to execute transfer manager driver operation: tm_attach.
　　Thu Nov 7 15:17:40 2013 [VMM][E]: Error attaching new VM Disk: Could not create image /vmfs/volumes/0/36/disk.2
　　解决办法：

• 　　mkimage: Command "export PATH=/usr/sbin:/sbin:$PATH
  
  　　修改/srv/cloud/one/var/remotes/tm/shared/mkimage文件，将export PATH=/usr/sbin:/sbin:\$PATH修改为export PATH=/usr/sbin:/sbin:$PATH
  
• 　　Failed to delete virtual disk: Not a directory (1310729).
  
  

　　    修改/srv/cloud/one/var/remotes/tm/shared/mkimage文件，修改如下：
　　# CREATE_IMAGE="$DD if=/dev/zero of=$DST_PATH bs=1 count=1 seek=${SIZE}M"   //注销该行
　　  CREATE_IMAGE="mkdir -p $DST_PATH"              //插入改行，用于创建文件夹
　　保存退出；
　　*************************************************************************************************
　　
问题6：为esxi虚拟机挂载磁盘时提示没有权限创建路径/srv
Log:
　　省略
　　解决办法：
　　在esxi上以root登录执行：
　　
　　~ # mkdir -p /srv/cloud/one/var/remotes
　　~ # chown oneadmin:cloud /srv/cloud/one/var/remotes
　　其他(用于备忘)：
　　~ # chmod +s /sbin/vmkfstools
　　~ # chmod +s /bin/vim-cmd
　　*************************************************************************************************

　　问题6：ESXi5.0的SNMP配置
　　
　　vicfg-snmp --server esxi01 --username root --password passwd123 --show
　　vicfg-snmp --server esxi01 --username root --password passwd123 -c public
　　vicfg-snmp --server esxi01 --username root --password passwd123 -t opennebula@162/public
　　vicfg-snmp --server esxi01 --username root --password passwd123 --enable
　　vicfg-snmp --server esxi01 --username root --password passwd123 --test
　　
*************************************************************************************************

问题7：GenericVmConfigFault - Module DevicePowerOn power on failed
　　


　　Thu Nov 14 09:05:45 2013 [DiM][I]: New VM state is ACTIVE.
　　Thu Nov 14 09:05:45 2013 [LCM][I]: New VM state is PROLOG.
　　Thu Nov 14 09:10:44 2013 [LCM][I]: New VM state is BOOT
　　Thu Nov 14 09:10:44 2013 [VMM][I]: Generating deployment file: /srv/cloud/one/var/vms/74/deployment.0
　　Thu Nov 14 09:10:44 2013 [VMM][I]: Successfully execute network driver operation: pre.
　　Thu Nov 14 09:10:58 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/vmm/vmware/deploy '/srv/cloud/one/var/vms/74/deployment.0' 'esxi02' 74 esxi02
　　Thu Nov 14 09:10:58 2013 [VMM][D]: deploy: Successfully defined domain one-74.
　　Thu Nov 14 09:10:58 2013 [VMM][E]: deploy: Error executing: virsh -c 'esx://esxi02/?no_verify=1&auto_answer=1' start one-74 err: ExitCode: 1
　　Thu Nov 14 09:10:58 2013 [VMM][I]: out:
　　Thu Nov 14 09:10:58 2013 [VMM][I]: error: Failed to start domain one-74
　　Thu Nov 14 09:10:58 2013 [VMM][I]: error: internal error Could not start domain: GenericVmConfigFault - Module DevicePowerOn power on failed.
　　Thu Nov 14 09:10:58 2013 [VMM][I]:
　　Thu Nov 14 09:10:58 2013 [VMM][I]:
　　Thu Nov 14 09:10:58 2013 [VMM][I]: ExitCode: 1
　　Thu Nov 14 09:10:58 2013 [VMM][I]: Failed to execute virtualization driver operation: deploy.
　　Thu Nov 14 09:10:58 2013 [VMM][E]: Error deploying virtual machine
　　Thu Nov 14 09:10:58 2013 [DiM][I]: New VM state is FAILED
　　
解决办法：
当前磁盘使用的是sd模式，该模式改为hd，或者直接修改oned.conf也可。
*************************************************************************************************
问题8：disk.0/disk.vmdk was not found
　　Thu Nov 14 13:01:03 2013 [DiM][I]: New VM state is ACTIVE.
　　Thu Nov 14 13:01:03 2013 [LCM][I]: New VM state is PROLOG.
　　Thu Nov 14 13:04:00 2013 [LCM][I]: New VM state is BOOT
　　Thu Nov 14 13:04:00 2013 [VMM][I]: Generating deployment file: /srv/cloud/one/var/vms/85/deployment.0
　　Thu Nov 14 13:04:01 2013 [VMM][I]: Successfully execute network driver operation: pre.
　　Thu Nov 14 13:04:15 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/vmm/vmware/deploy '/srv/cloud/one/var/vms/85/deployment.0' 'esxi02' 85 esxi02
　　Thu Nov 14 13:04:15 2013 [VMM][D]: deploy: Successfully defined domain one-85.
　　Thu Nov 14 13:04:15 2013 [VMM][E]: deploy: Error executing: virsh -c 'esx://esxi02/?no_verify=1&auto_answer=1' start one-85 err: ExitCode: 1
　　Thu Nov 14 13:04:15 2013 [VMM][I]: out:
　　Thu Nov 14 13:04:15 2013 [VMM][I]: error: Failed to start domain one-85
　　Thu Nov 14 13:04:15 2013 [VMM][I]: error: internal error Could not start domain: FileNotFound - File [0] 85/disk.0/disk.vmdk was not found
　　Thu Nov 14 13:04:15 2013 [VMM][I]:
　　Thu Nov 14 13:04:15 2013 [VMM][I]: ExitCode: 1
　　Thu Nov 14 13:04:15 2013 [VMM][I]: Failed to execute virtualization driver operation: deploy.
　　Thu Nov 14 13:04:15 2013 [VMM][E]: Error deploying virtual machine
　　Thu Nov 14 13:04:15 2013 [DiM][I]: New VM state is FAILED
　　*************************************************************************************************

　　
解决办法：
出现这类问题可能有两种原因：

• 目录下不存在该文件，而存在vm-name-flat.vmdk和vm-name.vmdk，那么将vm-name.vmdk更改为disk.vmdk即可。
  
• 目录下存在该文件，那么请确定是否改过vm-name-flat.vmdk的名字，如果改过会出现上述问题。
  
  

　　如何导入esxi虚拟机：
　　将esxi虚拟机的vm-name.vmdk命名为disk.vmdk，并将该文件和vm-name-flat.vmdk文件压缩，比如：

　　[iyunv@opennebula CentOS6.4_mini]# ls
　　CentOS6.4_mini-00-flat.vmdk  CentOS6.4_mini-00.vmdk
　　[iyunv@opennebula CentOS6.4_mini]# mv CentOS6.4_mini-00.vmdk disk.vmdk
　　[iyunv@opennebula CentOS6.4_mini]# ls
　　CentOS6.4_mini-00-flat.vmdk  disk.vmdk
　　[iyunv@opennebula CentOS6.4_mini]# tar -jcvf ../CentOS6.4_mini-131114.tar.bz2 *
　　*************************************************************************************************
　　
问题9：为虚拟机添加Volatile Disk
现象：在nebula4.2环境中，为esxi虚拟机创建Volatile Disk的时候remotes/tm/shared/mkimage文件和remotes/vmm/vmware/attach_disk文件不匹配导致不能真确的创建和添加磁盘，同时由于磁盘格式不对，power off虚拟机后无法resume。
remotes/vmm/vmware/attach_disk添加配置文件的磁盘路径：
DISK_PATH="/vmfs/volumes/$DATASTORE/$VM_ID/$DISK_NAME/disk.vmdk"
　　remotes/tm/shared/mkimage创建磁盘的名称和路径：

　　CREATE_IMAGE="$DD if=/dev/zero of=$DST_PATH bs=1 count=1 seek=${SIZE}M"
　　该命令只是在/vmfs/volumes/$DATASTORE/$VM_ID下创建一个$DISK_NAME磁盘文件，而attach_disk对应的$DISK_NAME是一个目录，该目录中的disk.vmdk才是真正的磁盘文件。其二，使用qemu-img查看磁盘类型显示是raw格式，非vmware的vmdk格式，所有要修改mkimage脚本使其在正确路径下创建磁盘并且是vmdk格式的磁盘。
　　修改后的mkimage脚本：
　　
　　#!/bin/bash
　　SIZE=$1
　　FSTYPE=$2
　　DST=$3
　　

　　VMID=$4
　　DSID=$5
　　

　　if [ -z "${ONE_LOCATION}" ]; then
　　    TMCOMMON=/var/lib/one/remotes/tm/tm_common.sh
　　else
　　    TMCOMMON=$ONE_LOCATION/var/remotes/tm/tm_common.sh
　　fi
　　

　　. $TMCOMMON
　　

　　#-------------------------------------------------------------------------------
　　# Set dst path and dir
　　#-------------------------------------------------------------------------------
　　DST_PATH=`arg_path $DST`
　　DST_HOST=`arg_host $DST`
　　DST_DIR=`dirname $DST_PATH`
　　

　　#-------------------------------------------------------------------------------
　　# Make the new image (file-based)
　　#-------------------------------------------------------------------------------
　　# ssh_make_path $DST_HOST $DST_DIR
　　

　　MKDIR_CMD=`/bin/mkdir -p $DST_PATH`                       //创建目录
　　CREATE_IMAGE="/usr/bin/qemu-img create -f vmdk $DST_PATH/disk.vmdk ${SIZE}M"                 //使用qemu-img命令创建vmdk格式的disk.vmdk
　　

　　$MKDIR_CMD
　　$CREATE_IMAGE
　　

　　log "Making filesystem of ${SIZE}M and type $FSTYPE at $DST"
　　

　　exit 0
　　
替换原有mkimage脚本，并且创建/vmfs/目录指向存储目录：
　　[iyunv@nebula datastores]# ls
　　0  1  100  101  102  2
　　[iyunv@nebula datastores]# pwd
　　/srv/cloud/one/var/datastores
　　[iyunv@nebula datastores]# mkdir /vmfs
　　[iyunv@nebula datastores]# chown oneadmin:cloud /vmfs
　　[iyunv@nebula datastores]# ln -s /srv/cloud/one/var/datastores   /vmfs/volumes
　　*************************************************************************************************
　　问题10：
　　esxi新建用户需要执行一些命令，在esxi端要做的操作：
　　
　　chmod +s /sbin/vmkvsitools
　　chmod +s /sbin/esxcfg-vswitch
　　chmod +s /bin/vim-cmd
　　chmod +s /sbin/mkdir
　　*************************************************************************************************