How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.5

yanfangsheng123

How to Setup Xen4 and pvops 2.6.32.x on CentOS 5.52011-04-29 13:17Setting up Xen4 has been tricky, because of the move to the latest Linux kernels with paravirt_ops. Compiling it on CentOS, which uses 2.6.18kernel, and old toolset is a bit more challenging. The following are my notes for setting up Xen4 + pvops 2.6.32.x on CentOS 5.5 on BL460cblades.
Important note: I couldn’t get Xen 4.0.1 release work on CentOS 5.5 on our blades. You have to get the latest Xen and pvops kernel available from http://xenbits.xen.org/xen-4.0-testing.hgand http://www.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git/ (4.0.2-rc1-pre and 2.6.32.21 as of this writing). This may or may not be a problem for you.
Install prerequisite packages
yum -y groupinstall "Development Libraries"yum -y groupinstall "Development Tools"yum -y installtransfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-develSDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial glibc-develInstall rpmforge repository. Follow the instructions on CentOS Wiki.
wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpmrpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txtrpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpmrpm -i rpmforge-release-0.5.1-1.el5.rf.*.rpmyum check-updateInstall mercurial and git
yum -y install mercurial gitCompile Xen
hg clone http://xenbits.xen.org/xen-4.0-testing.hgcd xen-4.0-testing.hgmake -j16 xenmake -j16 toolsmake -j16 stubdomCompile pvops kernel
make kernelsmake -j16 distThis is the most critical step, and if you don’t configure the kernel correctly, booting will fail badly. make kernels builds the kernel with the default .config. Tomodify it, change to build-xxx directory and run make menuconfig.
Make sure to have the following in your config file
# These are important for CentOS/RHELCONFIG_SYSFS_DEPRECATED=yCONFIG_SYSFS_DEPRECATED_V2=y CONFIG_ACPI_PROCFS=yCONFIG_XEN=yCONFIG_XEN_MAX_DOMAIN_MEMORY=32CONFIG_XEN_SAVE_RESTORE=yCONFIG_XEN_PRIVILEGED_GUEST=yCONFIG_XEN_PCI=yCONFIG_PCI_XEN=yCONFIG_XEN_BLKDEV_FRONTEND=mCONFIG_NETXEN_NIC=mCONFIG_XEN_NETDEV_FRONTEND=mCONFIG_XEN_KBDDEV_FRONTEND=mCONFIG_HVC_XEN=yCONFIG_XEN_FBDEV_FRONTEND=mCONFIG_XEN_BALLOON=yCONFIG_XEN_SCRUB_PAGES=yCONFIG_XEN_DEV_EVTCHN=yCONFIG_XEN_BACKEND=yCONFIG_XEN_BLKDEV_BACKEND=yCONFIG_XEN_NETDEV_BACKEND=yCONFIG_XENFS=yCONFIG_XEN_COMPAT_XENFS=yCONFIG_XEN_XENBUS_FRONTEND=mCONFIG_XEN_PCIDEV_FRONTEND=yIf you want to compile xen-evtchn, xen-netback, xen-blkback and xenfs as modules, use the following in your config.
CONFIG_XEN_DEV_EVTCHN=mCONFIG_XEN_BACKEND=yCONFIG_XEN_BLKDEV_BACKEND=mCONFIG_XEN_NETDEV_BACKEND=mCONFIG_XENFS=mInstall Xen and kernel
make install-xenmake install-tools PYTHON_PREFIX_ARG=cd build-*make modules_installmake installFor make install-tools the PYTHON_PREFIX_ARG is REQUIRED. On Fedora systems, this is not required.
Create initrd (above make install usually takes care of creating an initrd matching your configuration, but just in case)
depmod 2.6.32.21mkinitrd -v -f /boot/initrd-2.6.32.21.img 2.6.32.21Update grub
title Xen 4 + 2.6.32.21    root   (hd0,0)    kernel  /xen-4.0.gz dom0_mem=4096M loglvl=all guest_loglvl=all    module  /vmlinuz-2.6.32.21 root=/dev/cciss/c0d0p1 ro nomodeset console=tty0 console=ttyS0,115200n8    module  /initrd-2.6.32.21.imgAdd the following to /etc/modprobe.conf to load all the Xen modules
install xen /sbin/modprobe xen-evtchn; /sbin/modprobe xen-netback; /sbin/modprobe xenfs; /sbin/modprobe xen-blkback; /bin/trueOR, add the following to /etc/rc.local (I had trouble making the modprobe work)
if( uname-r | grep"2.6.32.21"); then   formod inxen-evtchn xen-netback xen-blkback xenfs; do        /sbin/modprobe $mod   done   /etc/init.d/xend startfiAdd xenfs to /etc/fstab
none /proc/xen xenfs defaults 0 0Test it
echo "savedefault --default=X --once"| grub --batch (where X is the entry forXen)reboot/etc/init.d/xend startxm listTroubleshooting The biggest problem will be kernel not booting properly. There might be various reasons. Make sure to use the right set of configuration options for your hardware Make sure to have all Xen options correctly compiledout_of_memory kernel OOPS CONFIG_XEN_MAX_DOMAIN_MEMORY=32 # change this to a lesser number Add more memory to dom0 with dom0_mem=4096M Network card not appearing – make sure to have the right network module (bnx2x for BroadCom NetXtreme II card) inserted Missing/dev/xen entries. Means that you haven’t loaded the xen-evtchn and xen-gntdev. Run
modprobe xen-evtchn xen-gntdevSee step 9 above to update /etc/modprobe.conf
If you see the following in /var/log/messages
xenconsoled: Failed to contact xenstore (Connection refused)Do step 2 to insert all the appropriate modules, and verify that you can start xend
Installing VMs First, verify that the initrd includes all the necessary Xen frontend modules. This
is not necessary for hvm VMs, but for paravirtualization it’s crucial to have all
the right netfront and blkfront modules installed. A sample mkinitrd for domU is
below
mkinitrd -v-f /boot/initrd-2.6.32.21-domU.img  --with=blktap \        --with=xen-blkback --with=xenfs 2.6.32.21Install virt-manager
yum installvirt-managerConfigure /etc/xen/xend-config.sxp to enable access through libvirt and virt-manager. Add the following lines
(xend-http-server yes)(xend-unix-server yes)(xend-port        8000)(xend-relocation-port 8002)Restart xend
/etc/init.d/xend restartStart virt-manager as root Install a VM as usual Referenceshttp://wiki.xensource.com/xenwiki/2.6.18-to-2.6.31-and-higher http://wiki.xensource.com/xenwiki/XenParavirtOps. This is the most authoritative reference for compiling thepvops kernel. http://wiki.xensource.com/xenwiki/Xen4.0 http://bderzhavets.wordpress.com/2010/04/24/set-up-ubuntu-10-04-server-pv-domu-at-xen-4-0-dom0-pvops-2-6-32-10-kernel-dom0-on-top-of-ubuntu-10-04-server/ (this is Ubuntu specific, but has some useful details)