GPU pass-through on KVM with Windows 7 Guest

sm702 · 发表于 2017-6-25 12:53:05

　　2015年1月13日发布
　　OS: CentOS 6.5
　　Graphic Cards: Embedded Graphic Card, Grid K1, Grid K2
　　Hypervisor: KVM
　　Benchmark tool: Specviewperf 11
　　
1. kernel recompile using linux-3.18.2
　　Download link:
　　https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.18.2.tar.gz
　　
Before compiling kernel, install needed packages:

yum install ncurses-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel qt-devel
yum groupinstall "Development Tools"

　　Extract to /usr/scr/ directory

tar -xvzf linux-3.18.2
cd linux-3.18.2
make mrproper
make clean
cp /boot/config-'uname -r' ./.config
make menuconfig

　　There are 3 big options to check and modify:

Bus Options (PCI etc.)
- <*> PCI Stub driver
Networking support (changing kernel from 2.x version to 3.x version)
- Networking options
  - Network packet filtering framework(Netfilter)
    - IP: Netfilter Configuration
      - <M> IPv4 NAT
    - IPv6: Netfilter Configuration
      - <M> IPv6 NAT
Device Drivers
- IOMMU Hardware Support
  - Support for IOMMU using DMA Remapping Devices
  - Enable DMA Remapping Devices by default
<*> VFIO Non-Privileged userspace driver framework
- <*> VFIO support for PCI devices
- VFIO PCI support for VGA devices

　　Continue to recompile kernel

make all // only when you first recompile kernel, after that, use make command
make modules_install
make install
vi /boot/grub/grub.conf

　　Here modify kernel booting priority: default =0
　　Add intel_iommu=on to kernel line

https://3.bp.blogspot.com/-Gx23kXzJyLY/VLS2gn75OvI/AAAAAAAAAuE/9bY2rd95dIc/s1600/1.PNG

　　
(If you are using an intel IGP on your host, you need the i915 vga arbiter patch)
　　i915 vga arbiter fixes file: https://lkml.org/lkml/2014/5/9/517
　　
Continue to recompile kernel

reboot

　　Check if your host is ready to PCI device pass-through using follow commands:

dmesg | grep -e DMAR -e IOMMU

　　ACPI: DMAR 0x00000000BD3346F4 000158 (v01 DELL PE_SC3 00000001 DELL 00000001)

　　Intel-IOMMU: enabled

　　dmar: IOMMU 0: reg_base_addr d5000000 ver 1:0 cap d2078c106f0466 ecap f020de

　　dmar: IOMMU 1: reg_base_addr df900000 ver 1:0 cap d2078c106f0466 ecap f020de

　　IOAPIC id 2 under DRHD base  0xd5000000 IOMMU 0

　　IOAPIC id 0 under DRHD base  0xdf900000 IOMMU 1

　　IOAPIC id 1 under DRHD base  0xdf900000 IOMMU 1

　　IOMMU 0 0xd5000000: using Queued invalidation

　　IOMMU 1 0xdf900000: using Queued invalidation

　　IOMMU: Setting RMRR:

　　IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf452000 - 0xbf452fff]

　　IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf450000 - 0xbf450fff]

　　IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf458000 - 0xbf46ffff]

　　IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf458000 - 0xbf46ffff]

　　IOMMU: Prepare 0-16MiB unity mapping for LPC

　　IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
　　
2. Install kvm

yum install kvm qemu-kvm python-virtinst libvirt libvirt-python virt-manager libguestfs-tools

　　(or)

yum groupinstall "Virtualisation Tools" "Virtualization Platform“
yum install python-virtins

　　Start kvm:

chkconfig libvirtd on
service libvirtd start

　　3. Create Windows 7 guest

ISO image file download link: Click Here
VirtIO drivers for windows guest download link: Click Here

　　Run virt-manager

virt-manager //when using vnc to access, use dbus-launch virt-manager command

　　Network bridge setup
　　Edit->Connection Detail->Network Interface
　　Ad br0 (bridge) connect to em1

https://1.bp.blogspot.com/-AyoE0Q0Qjt4/VLTD5aD42xI/AAAAAAAAAuU/p1lCcwoFSHU/s1600/2.PNG

Edit /etc/sysconfig/network-scripts/ifcfg-br0

Add GATEWAY and DNS1

Restart network service

service network restart

Create qcow2 Image Disk

cd /home/'username'/images/
qemu-img create -f qcow2-o preallocation=metadata win7.qcow2 60G

　　(image convert command is like this:

qemu-img convert -f raw -O qcow2 win7.dsk win7.qcow2
qemu-img convert -f vmdk -O raw win7.vmdk win7.img )

　　
Create new virtual machine (windows 7)

Add Windows 7 ISO file and virtIO driver file(Load VirtIO driver)

https://4.bp.blogspot.com/-zCtMAuOe9Qs/VLTKH60AG5I/AAAAAAAAAuk/DZvt0HbQ4Xs/s1600/3.PNG

Setup public ip to virtual machine

https://1.bp.blogspot.com/-G22DnojGPD0/VLTNrpgyS_I/AAAAAAAAAuw/Wg5hjFv5m7M/s1600/4.PNG

Setup remote access with port 2222(course there is no DVI output on GridK1,K2)

My Computer(Right Click) -> Option -> Remote Setup -> Access All

Control Panel -> Windows Firewall -> Advanced Setting -> Inbound Rule -> New Rule -> Port -> TCP and Special Port(?): 2222 -> Named "Remote Poart 2222"

https://2.bp.blogspot.com/-jSfSnjsDLo0/VLTRQnACTSI/AAAAAAAAAu8/mrdXylhFCMQ/s1600/5.PNG

https://1.bp.blogspot.com/-Ayj0Wx8yuyw/VLTRs72xdAI/AAAAAAAAAvE/K-NLpKlQ3_o/s1600/6.PNG

run->regedit

HKEY_LOCAL_MACHINE -> SYSTEM -> CurrentControlSet -> Control -> Terminal server -> Wds -> rdpwd -> Tds -> tcp ->PortNumber
HKEY_LOCAL_MACHINE -> SYSTEM -> CurrentControlSet -> Control -> Terminal server -> WinStations -> RDP-Tcp -> PortNumber

　　Edit 10 value as 2222

https://2.bp.blogspot.com/-qgpVrSyIiuU/VLTTOC_wwVI/AAAAAAAAAvQ/52tyYRE-9Wo/s1600/1.PNG

we can use xxx.xxx.xxx.x:2222 to remote access.

3. Install Needed Softwares

GPU-Z Click here to Download
Specviewperf 11 Click Here to Download

4. Bind PCI device to virtual machine using pci-stub driver

lspci -nn | grep VGA

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID K1] [10de:0ff2] (rev a1)
08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID K1] [10de:0ff2] (rev a1)
09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID K1] [10de:0ff2] (rev a1)
0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID K1] [10de:0ff2] (rev a1)

10:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. G200eR2 [102b:0534]

vi /usr/bin/pci-bind

　　-----------------------------------------------------------------------------------

　　#!/bin/bash

　　#modprobe pci-stub

　　for dev in "$@"; do

　　vendor=$(cat /sys/bus/pci/devices/$dev/vendor)

　　device=$(cat /sys/bus/pci/devices/$dev/device)

　　if [ -e /sys/bus/pci/devices/$dev/driver ]; then

　　echo $dev > /sys/bus/pci/devices/$dev/driver/unbind

　　fi

　　echo $vendor $device > /sys/bus/pci/drivers/pci-stub/new_id

　　done

　　------------------------------------------------------------------------------------

chmod 755 /usr/bin/pci-bind
pci-bind 0000:07:00.0
lspci -v -s 07:00.0

　　------------------------------------------------------------------------------------
　　07:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID K1] (rev a1) (prog-if 00 [VGA controller])
　　Subsystem: NVIDIA Corporation Device 1012
　　Flags: bus master, fast devsel, latency 0, IRQ 33
　　Memory at d4000000 (32-bit, non-prefetchable) [size=16M]
　　Memory at 3fff8000000 (64-bit, prefetchable) [size=128M]
　　Memory at 3fff6000000 (64-bit, prefetchable) [size=32M]
　　I/O ports at <ignored> [disabled]
　　Expansion ROM at d5800000 [disabled] [size=512K]
　　Capabilities: [60] Power Management version 3
　　Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
　　Capabilities: [78] Express Endpoint, MSI 00
　　Capabilities: [b4] Vendor Specific Information: Len=14 <?>
　　Capabilities: [100] Virtual Channel
　　Capabilities: [128] Power Budgeting <?>
　　Capabilities: [420] Advanced Error Reporting
　　Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
　　Capabilities: [900] #19
　　Kernel driver in use: pci-stub

　　------------------------------------------------------------------------------------

　　
　　5. Attach PCI device to virtual machine