Setting DPDK+OVS+QEMU on CentOS

blueice

　　Environment Build Step:
　　these packages are needed for building dpdk+ovs:



yum install -y make gcc glibc.i686 libgcc.i686 libstdc++.i686 glibc-devel.i686 glibc-devel.i686 libc6-dev-i386 glibc-devel.x86_64 libc6-dev clang autoconf automake libtool cmake python m4 openssl git libpcap-devel pciutils numactl-devel kernel-devel
　　First download latest dpdk & ovs :http://dpdk.org/download
　　git clone https://github.com/openvswitch/ovs
　　unzip these files.
　　then configure & compile dpdk. in config/common_linux add the following lines:



CONFIG_RTE_LIBRTE_PMD_PCAP=y
CONFIG_RTE_LIBRTE_PMD_RING=y
　　DPDK:
　　first need to know, if ovs-vsctl add-port, then the port is automatically binded to the NIC, by order.. so the name dpdk0 shouldn't be changed.. the dpdk[n]'s n is its order of the NICs..
　　also dpdkvhostuser[n], dpdkr[n].. and so on.

• before startup, computer must met: VT-d, (svm, vmx),  add following line to /etc/fstab (if wondering why 1GB rather than 2mb.. this page explains all)
  
  
  nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0
  
• add startup-config: default_hugepagesz=1GB hugepagesz=1GB hugepages=5 (5GB RAM), modify /boot/grub2/grub.cfg and reboot
  
• make sure these pre-requirements are met: make, gcc, gcc-multilib( glibc.i686, libgcc.i686, libstdc++.i686 and glibc-devel.i686 / glibc-devel.i686 / libc6-dev-i386; glibc-devel.x86_64 / libc6-dev )
  
• ( it's quite possible that this problem is not solved, it's v16.04 and still need this patch. the patch no. is 945 ) patch usage: patch -p1 < *.patch ( for more, see man patch )
  
• using its ./tools/setup.sh to compile at a suitable version of dpdk.  (not suggested, this step is too simple.. is divided as follows:
  
  
  • make install T=x86_64-native-linuxapp-gcc
    
  • cd x86_64-native-linuxapp-gcc
    
  • vi .config (note this step can modify the output of the libs to static libs. otherwise the .so files are needed  to set the paths. seems..
    
  • make (if need to reset .config, just make clean && make will done)
    
  • done.
    
  
  
• then to set environmental vars:
  
• 
  
  
  1 export RTE_SDK=/root/dpdk-16.04
  2 export RTE_TARGET=x86_64-....
  
• use ./tools/setup.sh again to insmod igb_uio kernel module. then create hugepages. then add the additional nic to igb_uio.
  
• *test with test programs
  
• ready to use
  

　　OVS installation:

• set the PATH for ovs:
  
  
  export DPDK_DIR=$HOME/dpdk-16.04
  export DPDK_TARGET=x86_64-native-linuxapp-gcc
  export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
  export OVS_DIR=$HOME/ovs
  export VM_NAME=Centos-vm
  export GUEST_MEM=1024M
  export QCOW2_IMAGE=/root/CentOS7_x86_64.qcow2
  export VHOST_SOCK_DIR=/usr/local/var/run/openvswitch
  export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
  
• yum install clang autoconf automake libtool
  
• ./boot.sh
  
• ./configure --with-dpdk=%DPDK_BUILD
  
• make install
  
• installation finished
  

　　OVS startup:

• *if started before..
  
  
  rm -f /usr/local/etc/openvswitch/conf.db /usr/local/var/run/openvswitch/db.sock
  ovs-appctl -t ovsdb-server exit
  ovs-appctl -t ovs-vswitchd exit
  
• source ./setup.sh content as follows: (start server, ovs and add port for qemu usage).  note sometimes there are 2 nodes of NUMA, so if dpdk ports are on the 2nd node, the socket mem should be set larger than 0, usually 1024.
  
  
  ovsdb-tool create /usr/local/etc/openvswitch/conf.db  \
  /usr/local/share/openvswitch/vswitch.ovsschema
  ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
  --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
  --pidfile --detach
  ovs-vsctl --no-wait init
  ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
  ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-hugepage-dir=/mnt/huge_1GB
  ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024,0"
  ovs-vswitchd unix:$DB_SOCK --pidfile --detach
  ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=6
  ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
  ovs-vsctl add-bond br0 dpdkbond dpdk0 dpdk1 -- set Interface dpdk0 type=dpdk -- set Interface dpdk1 type=dpdk
  ovs-vsctl add-port br0 dpdkvhostuser0 -- set Interface dpdkvhostuser0 type=dpdkvhostuser
  ovs-vsctl add-port br0 dpdkvhostuser1 -- set Interface dpdkvhostuser1 type=dpdkvhostuser
  
• done
  

　　QEMU:

• before starting Qemu, need to create a virtual tap/tun device, and bind it with NIC: reference the steps
  
  
  yum install brctl " a bridge creating pkg
  ip link set eno16777736 down " a nic, which is connected to outer web
  brctl addbr br1 " create a bridge where the outer web and the inner nic running on, letting the inner system get on web
  brctl addif eno16777736
  ip link set dev br1 promisc on " promisc mode on, in this mode, tap/nic both working..
  ip link set dev eno16777736 promisc on
  dhclient br1 " to give ip addr to every un-allocated [v]nics.. and if something goes wrong, make sure dhclient is not already running in the background..
  ip link set dev br1 up
  ip link set dev eno16777736 up
  ip tuntap add mode tap tap0
  ip link set dev tap0 promisc on
  
• start qemu vm by:
  
  
  qemu-system-x86_64 -name $VM_NAME -cpu host -enable-kvm -m $GUEST_MEM -object memory-backend-file,id=mem,size=$GUEST_MEM,mem-path=/mnt/huge_1GB,share=on -numa node,memdev=mem -mem-prealloc -smp sockets=1,cores=2 -drive file=$QCOW2_IMAGE -chardev socket,id=char0,path=$VHOST_SOCK_DIR/dpdkvhostuser0 -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off -chardev socket,id=char1,path=$VHOST_SOCK_DIR/dpdkvhostuser1 -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2,mrg_rxbuf=off -net nic,macaddr=00:00:00:00:00:21 -net tap,ifname=tap0,script=no,downscript=no -nographic -snapshot
  
• to enable connection between dpdkvhostuser ports, first need to set bridge br0 up by: (必要はない
  
  
  ip link set dev br0 up
  
• set the ip inside the VMs, by:
  
  
  ip addr add dev eth0 192.168.6.xx[n]
  ip link set eth0 up
  ip route add default via 192.168.6.1 " this gw should be found by traceroute www.baidu.com in the outer machine.. the first jump router is the gw..
  
• done, now host & guest is ping-free.. but still not able to ssh(securecrt) into the inside vm.. fixed by modifying the ssh configure settings and restart sshd service..
  

　　note: viewing logs created by any ovs program, using journalctl..



journalctl -t ovs-vswitchd
　　done