Setting up Red Hat 5.4 Xen for SR-IOV using the Intel 82576 GbE

yuanhaoliang

转载自：
https://communities.intel.com/community/wired/blog/2010/03/01/setting-up-red-hat-54-xen-for-sr-iov-using-the-intel-82576-gbe

Setting up Red Hat 5.4 Xenfor SR-IOV using the Intel 82576 GbE
发贴人 Patrick_Kutch 于2010-3-1 18:53:54在WiredEthernet　　So Intel has this new super-cool SR-IOV technology that allows you to directly assign multiple VMs to an Intel NIC based on the Intel® 82576 GbE Controller.  But how, you may be wondering do you get it to work?  I’ve writtena document that explains this, and am going to make a quick blog out of it.  I hope you find it useful, if you do, then my name is Patrick, if you don’t, then my name is Paul.:-)
　　
　　It should be noted that there is a bug in the version of the Xen kernel that shipped with Red Hat* 5.4 Xen that causes the SR-IOV performance to be poor.  The upcoming Red Hat 5.5 release will have the fix in it.

　　1  Introduction
　　This document is intended to help those wishing to setup and configure SR-IOV on Red Hat 5.4 Xen with the Intel® 82576 GbE controller.
　　Included within these instructions are step-by-step procedures for installing Red Hat 5.4 Xen along with configuring the installation for SR-IOV usage and how to assign a SR-IOV Virtual Function to a Virtual Machine.
2  Server Requirements
　　The server where Red Hat 5.4 Xen is to be installed must SR-IOV.  There are a number of platform requirements for SR-IOV that include BIOS support and I/O re-mapping capability such as that available as part of the Intel® Virtualization Technology for DirectedI/O (Intel® VT-d).
　　In addition a SR-IOV capable Ethernet controller is required.  The system used for the purposes of this paper has an Intel® 82576 GbE controller as the LOM, thus meeting the requirement for a SR-IOV capable Ethernet controller.
　　
2.1 Server Used
　　At the time of the writing of this document there are very few commercially available servers that support SR-IOV.  The system used for this paper is a Supermicro X8DTN+ with the UEFI BIOS which supports SR-IOV.
　　In addition to the on-board Intel® 82576 GbE Controller, an Intel® 82572EI GbE Controller has been added to the server to be used for non SR-IOV traffic.
2.2 BIOS Configuration
　　There will be some sort of BIOS configuration required to enable SR-IOV for the server.  This section details the configuration required for the Supermicro X8DTN+.
2.2.1 Enable Intel® VT-d
　　This configuration is found under the ChipsetàNorth BridgeàTylersburg IOH ConfigurationàIntel VT for Directed I/O Configuration menu.

　　Figure 1 BIOS SR-IOV Configuration
　　
2.2.2 Enable Intel® Virtualization Technology (VT-x)
　　In addition to enabling Intel® VT-d, the hypervisor itself also requires Intel VT-x to be enabled.  This is done through the BIOS option of:
　　Advanced->CPU Configuration->Intel Virtualization

　　Figure 2 Enable Intel VT-x
3  Installation of Red Hat 5.4 Xen
　　The first part of the installation is just like normal installation of a Red Hat distribution; simply put in the CD/DVD and boot up the computer.
3.1 Installation/SubscriptionNumber
　　When prompted for the Installation Number dialog box, enter the appropriate Installation Number:
　　
　　Figure 3 Installation Number
　　It is vital that a valid Installation/Subscription number be provided at this point.  If this is not done then the option to select Virtualization support (Xen) will not be available.
　　Continue with installation.
3.2 Selecting Virtualization Support
　　If a valid Installation/Subscription number was entered (see section 3.1) then a dialog should appear as follows:

　　Figure 4 Select Virtualization Support
　　If so, make sure Virtualization is selected and continue with installation as normal.
　　If, however the screen appears as:

　　Figure 5 Virtualization Option Not Available
　　This is an indication that a valid Installation/Subscription number was not entered as described in Section 3.1.
4  Configuration of Red Hat 5.4 Xen for SR-IOV Support
　　At this point Red Hat Xen 5.4 should be installed and running.  It will look just like ‘normal’ Red Hat Linux.  One difference will be the ability to launch the Virtual Machine Manager from the ApplicationàSystem Toolsmenu.
　　Additionally information about the Xen installation can be obtained from a command prompt by entering the command:
　　#xm info
4.1 Checking to see if any SR-IOV VF’s are Present
　　Now that Xen in installed and running, you may want to see if any SR-IOV Virtual Functions exist on the system (they won’t).  To verify this, enter the following command:
　　#lspci | grep Eth
　　The resulting output should look very similar to the following:
　　01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
　　01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
　　04:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
　　This shows three Intel® Ethernet controllers, including the dual port Intel® 82576 GbE controller; however no Virtual Functions appear yet.
4.2 Modifyingmodprobe.conf
　　The modprobe.conf file, located in the /etc/ directory needs to be updated with some options for the Intel igb driver.  The additions to the file are as follows:
　　options igb max_vfs=7
　　blacklist igbvf
　　options loop max_loop=128
　　Make these modifications and save the file.
　　The resulting file I ended up with is as follows:
　　alias eth0 e1000e
　　alias eth1 igb
　　alias eth2 igb
　　alias scsi_hostadapter ahci
　　options igb max_vfs=7
　　blacklist igbvf
　　options loop max_loop=128
　　
4.2.1 Explanation of Settings
　　·         options igb max_vfs=7
　　This option tells the igb driver to allocate sufficient SR-IOV resources for 7 Virtual Functions per port.
　　·         blacklist igbvf
　　The blacklist line makes it so modules ignore any igbvf devices that show up in the hypervisor PCIe space
　　·         options loop max_loop=128
　　By default Linux only has 7 loop devices.  SR-IOV requires many more, this option increases the number to 128
4.3 Modifyinggrub.conf
　　The grub.conf file presents options on how to load the kernel when booting.  This file is located in the /boot/grub/ directory.
　　This file needs to be updated to indicate that SR-IOV capability is desired.  This is accomplished by adding a single parameter to one of the lines in the grub.conf file.  And 2 additional parameters to another line.
　　Specifically “pci_pt_e820_access=on” needs to be added to the “module /vmlinuz-*” line.  While the kernel /xen.* line needs “iommu=1” and “msi=1” added.
　　My preference is to make 2 boot options, leaving the original one and making an alternate with SR-IOV enabled. To do this simply copy and paste the existing configuration to a new entry, rename it and then set the default tobe the SR-IOV entry.  The resulting grub.conf file is as follows:
　　# grub.conf generated by anaconda
　　#
　　# Note that you do not have to rerun grub after making changes to this file
　　# NOTICE:  You have a /boot partition.  This means that
　　#          all kernel and initrd paths are relative to /boot/, eg.
　　#          root (hd0,1)
　　#          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
　　#          initrd /initrd-version.img
　　#boot=/dev/sda
　　default=1
　　timeout=30
　　splashimage=(hd0,1)/grub/splash.xpm.gz
　　hiddenmenu
　　title Red Hat Enterprise Linux Server (2.6.18-164.el5xen)
　　       root (hd0,1)
　　       kernel /xen.gz-2.6.18-164.el5
　　       module /vmlinuz-2.6.18-164.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet
　　       module /initrd-2.6.18-164.el5xen.img
　　
　　title Red Hat Xen (2.6.18-164.el5xen)
　　       root (hd0,1)
　　       kernel /xen.gz-2.6.18-164.el5 iommu=1 msi=1
　　       module /vmlinuz-2.6.18-164.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet pci_pt_e820_access=on
　　       module /initrd-2.6.18-164.el5xen.img
　　After doing this, the “Red Hat Xen (2.6.18-164el15xen)” option is default.  When the system boots next time, if you press a key at grub startup you will see something like Figure 6.
4.3.1 Explanation of Settings
　　·         default=1
　　o    This indicates that of the two boot options now in the grub.conf file, the default will be the newly created Red Hat Xen (2.6.18-164.el5xen) option.
　　·         iommu=1
　　This option tells the kernel to enable SR-IOV
　　·         msi=1
　　This option tells the kernel to enable MSI-X interrupts
　　·         pci_pt_e820_access=on
　　This option enables direct assignment of a PCIe device to a Virtual Machine

　　Figure 6 Grub Menu, with SR-IOV Option
4.4 Checking Again for SR-IOV VF’s
　　After making the changes outlined in Sections 4.2 and 4.3 reboot the server.  Now try once again to see if there are SR-IOV Virtual Functions available for assignment to a Virtual Machine.
　　To verify this, enter the following command:
　　#lspci | grep Eth
　　The resulting output should look very similar to the following:
　　01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
　　01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
　　01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　01:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
　　04:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
　　
　　Note that as before, the two Intel® 82576 GbE controllers are still there, along with fourteen Virtual Functions.  Recall in Section 4.2 the options set 7 VF’s per port; with two ports that presents 14 Virtual Functionsfor assignment to a Virtual Machine.
　　If for some reason you do not see something similar, double check to make sure that the configuration steps in Sections 4.2 and 4.3 are correct – a simple typo will prevent things from working properly.
5  Assigning a Virtual Function to a Virtual Machine
　　This section will describe a mechanism for adding a SR-IOV VF to a Virtual Machine.  There are several ways to accomplish this - the explanation presented here will utilize the Virtual Machine Manager GUI provided withthe Red Hat 5.4 Xen.
　　The steps outlined here will begin with the creation of the Virtual Machine.  If you wish to add a Virtual Function to an existing Virtual Machine, skip to step 5.2.
5.1 Creating a Virtual Machine
　　Run the Virtual Machine Manager application via the Applications->System Tools menu.
5.1.1 Step 1 – Start Create Wizard
　　Select the localhost in the window, and then the ‘New’ button.

　　Figure 7 Virtual Machine Manager
　　Follow the steps within the wizard to create a new virtual machine until you come to the point where the wizard prompts for information about the network.
5.1.2 Step 2 – Network Configuration
　　When the Virtual Machine creation wizard prompts for information about the Network to be used, simply go with the default of ‘Virtual Network’

　　Figure 8 Network Setup
　　Finish with the wizard.  When the Virtual Machine begins to boot up for the 1st time, power it down using the Virtual Machine Manager.
5.2 Assigningthe SR-IOV Virtual Function
　　Select the powered down Virtual Machine from the Virtual Machine Manager.
5.2.1 Step 1 Hardware Configuration Tab
　　Then select the hardware tab.
　　If an emulated NIC already configured, it can be removed using this panel.

　　Figure 9 Removing Default NIC
　　  Step 2 – Add Hardware

Select the Add Hardware button at the bottom of the window.  The Adding Virtual Hardware window will appear:

　　Figure 10 Adding Virtual Hardware
　　Select “Physical Host Device” from the dropdown list, press the “Forward” Button”
5.2.2 Step 3 – Select 82576 Virtual Function
　　From the Device drop down list, scroll down until the 82576 Virtual Function devices appear and select one.

　　Figure 11 Select a Virtual Function
　　After selecting the Virtual Function, finish the wizard and boot up the Virtual Machine.  The new hardware will be discovered and depending upon the guest Operating System the driver will either installed automaticallyor you will be prompted for a driver CD.
　　Install the Intel® 82576 Virtual Function Driver and enjoy the SR-IOV functionality of the Intel® 82576 GbE Controller.
5.2.3 Virtual Function Number and Physical Port Relationship
　　Care should be taken when selecting the Virtual Function to be assigned to a Virtual Machine.  Each VF has a PCIe BUS:Device:Function ID’s.  As seen in Figure 11 the ID’s for the dual port Intel® 82576 GbE controllerare numbered 01:10:n and 01:11:n.
　　If the last number (n) is an even number, then the Virtual Function is on Physical Function (port) 0.  If the number (n)is odd, then the Virtual Function is on Physical Function (port) 1.
　　Note: Care should be taken not to assign the same Virtual Function number to more than one Virtual Machine.  The GUI will allow this to occur, however when the Virtual Machines are run, the first oneloaded will function properly, however any subsequently loaded Virtual Machines that are assigned the same Virtual Function will immediately fail with an error as shown in Figure 12.

　　Figure 12 Attempt to Use the Same VF Twice