20. Intel Virtual Function Driver

20.1. SR-IOV Mode Utilization in a DPDK Environment

The DPDK uses the SR-IOV feature for hardware-based I/O sharing in IOV mode. Therefore, it is possible to partition SR-IOV capability on Ethernet controller NIC resources logically and expose them to a virtual machine as a separate PCI function called a “Virtual Function”. Refer to Fig. 20.1.

Therefore, a NIC is logically distributed among multiple virtual machines (as shown in Fig. 20.1), while still having global data in common to share with the Physical Function and other Virtual Functions. The DPDK fm10kvf, i40evf, igbvf or ixgbevf as a Poll Mode Driver (PMD) serves for the Intel® 82576 Gigabit Ethernet Controller, Intel® Ethernet Controller I350 family, Intel® 82599 10 Gigabit Ethernet Controller NIC, Intel® Fortville 10/40 Gigabit Ethernet Controller NIC’s virtual PCI function, or PCIe host-interface of the Intel Ethernet Switch FM10000 Series. Meanwhile the DPDK Poll Mode Driver (PMD) also supports “Physical Function” of such NIC’s on the host.

The DPDK PF/VF Poll Mode Driver (PMD) supports the Layer 2 switch on Intel® 82576 Gigabit Ethernet Controller, Intel® Ethernet Controller I350 family, Intel® 82599 10 Gigabit Ethernet Controller, and Intel® Fortville 10/40 Gigabit Ethernet Controller NICs so that guest can choose it for inter virtual machine traffic in SR-IOV mode.

For more detail on SR-IOV, please refer to the following documents:

• SR-IOV provides hardware based I/O sharing
• PCI-SIG-Single Root I/O Virtualization Support on IA
• Scalable I/O Virtualized Servers

Fig. 20.1 Virtualization for a Single Port NIC in SR-IOV Mode

20.2. Setting Up a KVM Virtual Machine Monitor

The following describes a target environment:

• Host Operating System: Fedora 14
• Hypervisor: KVM (Kernel Virtual Machine) with Qemu version 0.14.0
• Guest Operating System: Fedora 14
• Linux Kernel Version: Refer to the DPDK Getting Started Guide
• Target Applications: l2fwd, l3fwd-vf

The setup procedure is as follows:

1. Before booting the Host OS, open BIOS setup and enable Intel® VT features.

2. While booting the Host OS kernel, pass the intel_iommu=on kernel command line argument using GRUB. When using DPDK PF driver on host, pass the iommu=pt kernel command line argument in GRUB.

3. Download qemu-kvm-0.14.0 from http://sourceforge.net/projects/kvm/files/qemu-kvm/ and install it in the Host OS using the following steps:

   When using a recent kernel (2.6.25+) with kvm modules included:

   tar xzf qemu-kvm-release.tar.gz
   cd qemu-kvm-release
   ./configure --prefix=/usr/local/kvm
   make
   sudo make install
   sudo /sbin/modprobe kvm-intel
   

   When using an older kernel, or a kernel from a distribution without the kvm modules, you must download (from the same link), compile and install the modules yourself:

   tar xjf kvm-kmod-release.tar.bz2
   cd kvm-kmod-release
   ./configure
   make
   sudo make install
   sudo /sbin/modprobe kvm-intel
   

   qemu-kvm installs in the /usr/local/bin directory.

   For more details about KVM configuration and usage, please refer to:

   http://www.linux-kvm.org/page/HOWTO1.

4. Create a Virtual Machine and install Fedora 14 on the Virtual Machine. This is referred to as the Guest Operating System (Guest OS).

5. Download and install the latest ixgbe driver from:

   http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=14687

6. In the Host OS

   When using Linux kernel ixgbe driver, unload the Linux ixgbe driver and reload it with the max_vfs=2,2 argument:

   rmmod ixgbe
   modprobe ixgbe max_vfs=2,2
   

   When using DPDK PMD PF driver, insert DPDK kernel module igb_uio and set the number of VF by sysfs max_vfs:

   modprobe uio
   insmod igb_uio
   ./dpdk-devbind.py -b igb_uio 02:00.0 02:00.1 0e:00.0 0e:00.1
   echo 2 > /sys/bus/pci/devices/0000\:02\:00.0/max_vfs
   echo 2 > /sys/bus/pci/devices/0000\:02\:00.1/max_vfs
   echo 2 > /sys/bus/pci/devices/0000\:0e\:00.0/max_vfs
   echo 2 > /sys/bus/pci/devices/0000\:0e\:00.1/max_vfs
   

   Note

   You need to explicitly specify number of vfs for each port, for example, in the command above, it creates two vfs for the first two ixgbe ports.

   Let say we have a machine with four physical ixgbe ports:

   0000:02:00.0

   0000:02:00.1

   0000:0e:00.0

   0000:0e:00.1

   The command above creates two vfs for device 0000:02:00.0:

   ls -alrt /sys/bus/pci/devices/0000\:02\:00.0/virt*
   lrwxrwxrwx. 1 root root 0 Apr 13 05:40 /sys/bus/pci/devices/0000:02:00.0/virtfn1 -> ../0000:02:10.2
   lrwxrwxrwx. 1 root root 0 Apr 13 05:40 /sys/bus/pci/devices/0000:02:00.0/virtfn0 -> ../0000:02:10.0
   

   It also creates two vfs for device 0000:02:00.1:

   ls -alrt /sys/bus/pci/devices/0000\:02\:00.1/virt*
   lrwxrwxrwx. 1 root root 0 Apr 13 05:51 /sys/bus/pci/devices/0000:02:00.1/virtfn1 -> ../0000:02:10.3
   lrwxrwxrwx. 1 root root 0 Apr 13 05:51 /sys/bus/pci/devices/0000:02:00.1/virtfn0 -> ../0000:02:10.1
   

7. List the PCI devices connected and notice that the Host OS shows two Physical Functions (traditional ports) and four Virtual Functions (two for each port). This is the result of the previous step.

8. Insert the pci_stub module to hold the PCI devices that are freed from the default driver using the following command (see http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM Section 4 for more information):

   sudo /sbin/modprobe pci-stub
   

   Unbind the default driver from the PCI devices representing the Virtual Functions. A script to perform this action is as follows:

   echo "8086 10ed" > /sys/bus/pci/drivers/pci-stub/new_id
   echo 0000:08:10.0 > /sys/bus/pci/devices/0000:08:10.0/driver/unbind
   echo 0000:08:10.0 > /sys/bus/pci/drivers/pci-stub/bind
   

   where, 0000:08:10.0 belongs to the Virtual Function visible in the Host OS.

9. Now, start the Virtual Machine by running the following command:

   /usr/local/kvm/bin/qemu-system-x86_64 -m 4096 -smp 4 -boot c -hda lucid.qcow2 -device pci-assign,host=08:10.0
   

   where:

   — -m = memory to assign

   —-smp = number of smp cores

   — -boot = boot option

   —-hda = virtual disk image

   — -device = device to attach

   Note

   — The pci-assign,host=08:10.0 value indicates that you want to attach a PCI device to a Virtual Machine and the respective (Bus:Device.Function) numbers should be passed for the Virtual Function to be attached.

   — qemu-kvm-0.14.0 allows a maximum of four PCI devices assigned to a VM, but this is qemu-kvm version dependent since qemu-kvm-0.14.1 allows a maximum of five PCI devices.

   — qemu-system-x86_64 also has a -cpu command line option that is used to select the cpu_model to emulate in a Virtual Machine. Therefore, it can be used as:

   /usr/local/kvm/bin/qemu-system-x86_64 -cpu ?
   
   (to list all available cpu_models)
   
   /usr/local/kvm/bin/qemu-system-x86_64 -m 4096 -cpu host -smp 4 -boot c -hda lucid.qcow2 -device pci-assign,host=08:10.0
   
   (to use the same cpu_model equivalent to the host cpu)
   

   For more information, please refer to: http://wiki.qemu.org/Features/CPUModels.

10. If use vfio-pci to pass through device instead of pci-assign, steps 8 and 9 need to be updated to bind device to vfio-pci and replace pci-assign with vfio-pci when start virtual machine.

    sudo /sbin/modprobe vfio-pci
    
    echo "8086 10ed" > /sys/bus/pci/drivers/vfio-pci/new_id
    echo 0000:08:10.0 > /sys/bus/pci/devices/0000:08:10.0/driver/unbind
    echo 0000:08:10.0 > /sys/bus/pci/drivers/vfio-pci/bind
    
    /usr/local/kvm/bin/qemu-system-x86_64 -m 4096 -smp 4 -boot c -hda lucid.qcow2 -device vfio-pci,host=08:10.0
    

11. Install and run DPDK host app to take over the Physical Function. Eg.

    make install T=x86_64-native-linuxapp-gcc
    ./x86_64-native-linuxapp-gcc/app/testpmd -l 0-3 -n 4 -- -i
    

12. Finally, access the Guest OS using vncviewer with the localhost:5900 port and check the lspci command output in the Guest OS. The virtual functions will be listed as available for use.

13. Configure and install the DPDK with an x86_64-native-linuxapp-gcc configuration on the Guest OS as normal, that is, there is no change to the normal installation procedure.

    make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
    cd x86_64-native-linuxapp-gcc
    make
    

Fig. 20.2 Performance Benchmark Setup

20.3. DPDK SR-IOV PMD PF/VF Driver Usage Model

20.3.1. Fast Host-based Packet Processing

Software Defined Network (SDN) trends are demanding fast host-based packet handling. In a virtualization environment, the DPDK VF PMD driver performs the same throughput result as a non-VT native environment.

With such host instance fast packet processing, lots of services such as filtering, QoS, DPI can be offloaded on the host fast path.

Fig. 20.3 shows the scenario where some VMs directly communicate externally via a VFs, while others connect to a virtual switch and share the same uplink bandwidth.

Fig. 20.3 Fast Host-based Packet Processing

20.4. SR-IOV (PF/VF) Approach for Inter-VM Communication

Inter-VM data communication is one of the traffic bottle necks in virtualization platforms. SR-IOV device assignment helps a VM to attach the real device, taking advantage of the bridge in the NIC. So VF-to-VF traffic within the same physical port (VM0<->VM1) have hardware acceleration. However, when VF crosses physical ports (VM0<->VM2), there is no such hardware bridge. In this case, the DPDK PMD PF driver provides host forwarding between such VMs.

Fig. 20.4 shows an example. In this case an update of the MAC address lookup tables in both the NIC and host DPDK application is required.

In the NIC, writing the destination of a MAC address belongs to another cross device VM to the PF specific pool. So when a packet comes in, its destination MAC address will match and forward to the host DPDK PMD application.

In the host DPDK application, the behavior is similar to L2 forwarding, that is, the packet is forwarded to the correct PF pool. The SR-IOV NIC switch forwards the packet to a specific VM according to the MAC destination address which belongs to the destination VF on the VM.

Fig. 20.4 Inter-VM Communication