Xen Live Migration with iSCSI

淡淡回忆

　　Xen Live Migration with iSCSI












Overview
　　This is a document about how to set up XEN and iSCSI on 3 Debian Stable (3.1 / Sarge) machines so that you can have a virtual host that can be migrated live between 2 of the machines. The third machine will be the common disk for the virtual machine that is seen via the 2 machines running XEN. I have assumed you have some knowledge of Linux specifically Debian, but not limited to. You may need to solve some dependencies that I have left out by yourself. I can't think of everything, and I didn't start with completely empty machines.


http://onexin.iyunv.com/source/plugin/onexin_bigdata/file:///C:/DOCUME%7E1/NIPENG%7E1.FLY/LOCALS%7E1/Temp/enhtmlclip/Image%284%29.png　　In this example we will use dangermouse character names for the machine names. Buggles and Dangermouse will be the XEN hosts, and Penfold will house the disk that is common for both. Virtual01 will be the name we give to the virtual machine, that can be live migrated between dangermouse and buggles, using the disk on penfold over iSCSI. That was a mouthful, but don't worry we will break it down into what is hopefully simple steps.











XEN





Installation
　　Assuming you have already got running debian stable machines using the 2.6.8 kernel that was shipped with Debian, we are ready to move on to installing XEN from backports.org. This process needs to be done on all three machines.
　　Prepare the apt sources by adding the backports repositories to the file /etc/apt/sources.list

dangermouse:~# echo "deb http://www.backports.org/debian/ sarge-backports main" >> /etc/apt/sources

　　Now tell apt that we don't normally want to install packages from backports.org unless we specifically ask.

dangermouse:~# cat  /etc/apt/preferencesPackage: *Pin: release a=sarge-backportsPin-Priority: 200!

　　Update the apt cache:

dangermouse:~# apt-get update

　　Now we want to install the xen kernel and hypervisor on every machine. This is so we get a consistent 2.6.16 kernel on every machine including the machine we will use for the disk so we can use the latest version of iSCSI Target software from IET.

dangermouse:~# aptitude -t sarge-backports install linux-image-2.6.16-1-xen-686 linux-modules-2.6.16-1-xen-686 grub xen-hypervisor-3.0-i386 xen-utils-3.0 udev

　　There is probably a few other packages that I have left out that will get automatically updated as well. They should all be fine.
　　We need to disable tls as it is not supported by XEN at this time. There may be a libc686-xen package available, and I would suggest installing that if possible instead of disabling /lib/tls.

dangermouse:~# mv /lib/tls /lib/tls.disabled

　　The Debian package of XEN kernel or hypervisor doesn't yet make an initrd image, so we need to make one manually.

dangermouse:~# mkinitrd -o /boot/initrd.img-2.6.16-1-xen-686 2.6.16-1-xen-686

　　Now prepare the boot loader. Firstly run update-grub to automatically populate /boot/grub/menu.1st, then fixup the created XEN entries, by adding two module lines and removing the initrd line.

dangermouse:~# update-grub

　　turn this:

kernel /boot/vmlinuz-2.6.16-1-xen-686 root=/dev/hda1 roinitrd /boot/initrd.img-2.6.16-1-xen-686

　　into this in the /boot/grub/menu.1st:

kernel /boot/xen-3.0-i386.gz console=vga dom0_mem=256000module /boot/vmlinuz-2.6.16-1-xen-686 root=/dev/hda1 ro xencons=offmodule /boot/initrd.img-2.6.16-1-xen-686

　　Replace /dev/hda1 with whatever is your root partition. dom0_mem is the amount of memory available to the host virtual machine or domain0. xencons turns off the use of the serial port. I need the serial ports for other purposes, so this one is up to you.
　　Now you need to reboot and confirm that the XEN hypervisor boots and domain0 starts as well. If this fails go over the steps and make sure you didn't leave a step out. Because you still have your Debian standard kerenl installed you can always recover to that with no ill affect





XEN Configuration
　　Now that you have XEN installed and you are successfully booting off the XEN Hypervisor we can configure the xen daemon. The configuration file is /etc/xen/xend-config.sxp.

(xend-relocation-server yes)(xend-relocation-hosts-allow '')(network-script 'network-bridge netdev=eth0')(vif-script vif-bridge)(dom0-min-mem 196)(dom0-cpus 0)

　　Now restart the xend process.

dangermouse:~# /etc/init.d/xend restart






iSCSI





uDev Setup
　　We setup udev to make the iSCSI devices a consistent name across all the machines, and you don't end up with /dev/sda on one machine and /dev/sdb on another.
　　On each of the clients do the following:

dangermouse:~# echo "BUS==\"scsi\", SYSFS{vendor}==\"IET\", SYSFS{model}==\"VIRTUAL-DISK\", \KERNEL==\"sd*\", NAME=\"%k\", PROGRAM=\"/lib/udev/getlun.sh $id\", SYMLINK+=\"iscsi_%c%n\"" \>>/etc/udev/rules.d/10_persistant_scsi.rulesdangermouse:~# cat  /lib/udev/getlun.sh#!/bin/bashecho $1 | awk -F":" '{print $NF}'!

　　What this will do is create a link in /dev called iscsi_01 for instance, where 0 is the LUN number, and 1 is the first partition, which might be a link to say /dev/sdb1.





iSCSI Target
　　Now we need to set up the target on the machine which won't run the virtual machine. This is penfold.
　　I used a block device for this on a SATA raid controller, but you could just use an ordinary file or a normal disk, or logical volume. The iSCSI Target documentation will give you more information on setting up something other than a block device.

penfold:~# apt-get install linux-headers-2.6.16-1-xen-686 openssl libssl-dev

　　Get the latest IET source code from your nearest sourceforge mirror, then uncompress and extract the files from the archive ready to build.

penfold:~# cd /usr/srcpenfold:~# wget http://sf.net...../iscsitarget-0.4.13.tar.gzpenfold:~# tar -zxf iscsitarget-0.4.13.tar.gzpenfold:~# cd iscsitarget-0.4.13penfold:~# makepenfold:~# make install

　　edit /etc/ietd.conf and make it look like the following.

IncomingUser joe secretOutgoingUser jack secret2    Target iqn.2006-05.com.performancemagic:penfold.disk1        IncomingUser jim othersecret        OutgoingUser james yetanothersecret        Lun 0 Path=/dev/sdb,Type=fileio        Alias Test        HeaderDigest None        DataDigest None        MaxConnections 1        InitialR2T Yes        ImmediateData No        MaxRecvDataSegmentLength 8192        MaxXmitDataSegmentLength 8192        MaxBurstLength 262144        FirstBurstLength 65536        DefaultTime2Wait 2        DefaultTime2Retain 20        MaxOutstandingR2T 8        DataPDUInOrder Yes        DataSequenceInOrder Yes        ErrorRecoveryLevel 0

　　Now start the iSCSI target.

penfold:~# /etc/init.d/iscsi-target startStarting iSCSI enterprise target service: succeeded.

　　(dmesg output abreviated)

penfold:~# dmesgiSCSI Enterprise Target Software - version 0.4.13iotype_init(90) register fileioiotype_init(90) register nulliosess_param(173) 1 1 1 8192 8192 262144 65536 2 20 1 1 1 0 1 1sess_param(173) 1 1 1 8192 8192 262144 65536 2 20 1 1 1 0 1 1





iSCSI Initiator
　　Now lets build and setup the iSCSI initiator.

dangermouse:~# cd /usr/srcdangermouse:~# svn co svn://svn.berlios.org/open-iscsi/

　　The revision I got was 581

dangermouse:~# cd open-iscsidangermouse:~# makedangermouse:~# make installdangermouse:~# echo "iqn.2006-05.com.performancemagic:dangermouse.01" > /etc/initiatorname.iscsi

　　Change dangermouse in the above example to buggles for the other XEN host.
　　Edit /etc/iscsid.conf and make it look like the following.

node.active_cnx = 1node.startup = manualnode.session.auth.username = jimnode.session.auth.password = othersecretnode.session.timeo.replacement_timeout = 120node.session.err_timeo.abort_timeout = 10node.session.err_timeo.reset_timeout = 30node.session.iscsi.InitialR2T = Nonode.session.iscsi.ImmediateData = Yesnode.session.iscsi.FirstBurstLength = 262144node.session.iscsi.MaxBurstLength = 16776192node.session.iscsi.DefaultTime2Wait = 0node.session.iscsi.DefaultTime2Retain = 0node.session.iscsi.MaxConnections = 1node.conn[0].iscsi.HeaderDigest = Nonenode.conn[0].iscsi.DataDigest = Nonenode.conn[0].iscsi.MaxRecvDataSegmentLength = 65536discovery.sendtargets.auth.username = joediscovery.sendtargets.auth.password = secret

　　Start the initiator.

dangermouse:~# /etc/init.d/open-iscsi startStarting iSCSI initiator service: succeeded.

　　(dmesg output abreviated)

dangermouse:~# dmesgSCSI subsystem initializediscsi: registered transport (tcp)





iSCSI Setup
　　Now we have the target setup and the initiator started, we need to find the targets on the initiator.

dangermouse:~# iscsiadm -m discovery -t sendtargets -p 127.0.0.1:3260[0f6cce] 127.0.0.1:3260,1 iqn.2006-05.com.performancemagic:penfold.disk1

　　now take the number in the [ ] brackets and login.

dangermouse:~# iscsiadm -m node -r 0f0cce -l

　　(dmesg output abreviated)

dangermouse:~# dmesgscsi0 : iSCSI Initiator over TCP/IP, v1.0-574sess_param(173) 1 0 1 8192 8192 262144 65536 2 0 1 1 1 0 1 1Vendor: IET Model: VIRTUAL-DISK Rev: 0Type: Direct-Access ANSI SCSI revision: 04SCSI device sda: 10000 512-byte hdwr sectors (5 MB)sda: Write Protect is offsda: Mode Sense: 77 00 00 08SCSI device sda: drive cache: write backSCSI device sda: 10000 512-byte hdwr sectors (5 MB)sda: Write Protect is offsda: Mode Sense: 77 00 00 08SCSI device sda: drive cache: write backsda: sda1sd 0:0:0:1: Attached scsi disk sda





XEN Virtual Machine





Create partition and File-system
　　Now that you can see the iSCSI device we can create partitions for the virtual machine to use. We just need to create the simplest of patitions, one for the root file-system of the virtual machine and one for swap. You can use fdisk or cfdisk for this, or any other partitioning programme. Your partitions in fdisk might look something like this:

dangermouse:~# fdisk /dev/iscsi_0Command (m for help): pDisk /dev/iscs_0: 60.0 GB, 60011642880 bytes255 heads, 63 sectors/track, 7296 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes   Device Boot      Start         End      Blocks   Id  System/dev/iscsi_01               1         249     2000061   82  Linux swap / Solaris/dev/iscsi_02             250        7296    56605027+  83  Linux

　　Now that we have created the partitions we need to create file-systems on them.

dangermouse:~# mkswap /dev/iscsi_01Setting up swapspace version 1, size = 5115 kBno label, UUID=055d2ab4-d2ab-4779-818c-71d4950c08dfdangermouse:~# mke2fs -j /dev/iscsi_02Filesystem label=OS type: LinuxBlock size=1024 (log=0)Fragment size=1024 (log=0)1256 inodes, 5000 blocks250 blocks (5.00%) reserved for the super userFirst data block=1Maximum filesystem blocks=52428801 block group8192 blocks per group, 8192 fragments per group1256 inodes per groupWriting inode tables: doneCreating journal (1024 blocks): doneWriting superblocks and filesystem accounting information: doneThis filesystem will be automatically checked every 28 mounts or180 days, whichever comes first.  Use tune2fs -c or -i to override.





Create Virtual Machine
　　I am going to create a Debian virtual machine using debootstrap.

dangermouse:~# mount /dev/iscsi_02 /mntdangermouse:~# debootstrap sarge /mntI: Retrieving ReleaseI: Retrieving Packages......

　　Once debootstrap is completed we need to setup the machine a little bit. We need to disable tls again, setup the /etc/fstab and the network interfaces and create tty and console devices. We also need to copy the kernel modules to the virtual machine.

dangermouse:~# mv /mnt/lib/tls /mnt/lib/tls.disableddangermouse:~# cat                proc            /proc           proc    defaults        0       0/dev/hda1       none            swap    sw              0       0/dev/hda2       /               ext3    defaults,errors=remount-ro 0       1!dangermouse:~# echo "virtaul01" > /mnt/etc/hostnamedangermouse:~# cat