3.1. Setup

This section describes how to setup spp_vf and spp_mirror.

3.1.1. Environment

Here is a recommended versions of each of software.

  • Ubuntu 16.04
  • qemu-kvm 2.7 or later

Note

If you do not use --vhost-client as described in How to Use of spp_vf, you can use previous versions of qemu-kvm.

3.1.2. Edit Config

Uncomment user and group in /etc/libvirt/qemu.conf.

# /etc/libvirt/qemu.conf

user = "root"
group = "root"

To use hugepages with libvirt, change KVM_HUGEPAGES from 0 to 1 in /etc/default/qemu-kvm.

# /etc/default/qemu-kvm

KVM_HUGEPAGES=1

Change grub configuration for hugepages and isolcpus features. It is defined as GRUB_CMDLINE_LINUX_DEFAULT and takes parameters.

  • isolcpus: Core IDs separated with commna, such as 2,3,4,8
  • hugepagesz: The size of hugepage, such as 2M or 1G
  • hugepages: The number of hugepages, such as 8
  • default_hugepagesz: Default hugepage size, required only for 1G

In this setup guide, configure hugepagez and its default is 1G and the number of hugepages is 36. Core IDs of for isolcpus depends on your environment and usecase, but should be configured to have enough number of cores.

# /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=2,...,46 hugepagesz=1G hugepages=..."

For hugepages, isolcpus, refer to the dpdk documentation below.

You need to run update-grub and reboot to activate grub config.

$ sudo update-grub
$ sudo reboot

You can check hugepage settings as following.

$ cat /proc/meminfo | grep -i huge
AnonHugePages:      2048 kB
HugePages_Total:      36            #       /etc/default/grub
HugePages_Free:       36
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB         #       /etc/default/grub

$ mount | grep -i huge
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,...,nsroot=/)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,...,gid=117)
hugetlb on /run/lxcfs/controllers/hugetlb type cgroup (rw,...,nsroot=/)

Finally, you umount default hugepage.

$ sudo umount /dev/hugepages

3.1.3. Install DPDK

Install DPDK in any directory. This is a simple instruction and please refer Getting Started Guide for Linux for details.

$ cd /path/to/any_dir
$ git clone http://dpdk.org/git/dpdk
$ cd dpdk
$ git checkout [TAG_NAME(e.g. v17.05)]
$ export RTE_SDK=`pwd`
$ export RTE_TARGET=x86_64-native-linuxapp-gcc
$ make T=x86_64-native-linuxapp-gcc install

3.1.4. Install SPP

Clone SPP in any directory and compile it.

$ cd /path/to/any_dir
$ git clone http://dpdk.org/git/apps/spp
$ cd spp
$ make

3.1.5. Setup for DPDK

Load igb_uio module.

$ sudo modprobe uio
$ sudo insmod $RTE_SDK/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
$ lsmod | grep uio
igb_uio                16384  0  # igb_uio is loaded
uio                    20480  1 igb_uio

Then, bind your devices with PCI number by using dpdk-devbind.py. PCI number is inspected

# check your device for PCI_Number
$ $RTE_SDK/usertools/dpdk-devbind.py --status

$ sudo $RTE_SDK/usertools/dpdk-devbind.py --bind=igb_uio PCI_NUM

3.1.6. virsh setup

First of all, please check version of qemu-kvm.

$ qemu-system-x86_64 --version

If your system does not have qemu-kvm or the version of qemu is less than 2.7, then please install qemu following the instruction of https://wiki.qemu.org/index.php/Hosts/Linux to install qemu 2.7. You may need to install libvirt-bin, virtinst, bridge-utils packages via apt-get install to run virt-install.

virsh is a command line interface that can be used to create, destroy, stop start and edit VMs and configure. After create an image file, you can setup it with virt-install. --location is a URL of installer and it should be http://archive.ubuntu.com/ubuntu/dists/xenial/main/installer-amd64/ for amd64.

virt-install \
--name [VM_NAME] \
--ram 4096 \
--disk path=/var/lib/libvirt/images/[VM_NAME].img,size=30 \
--vcpus 4 \
--os-type linux \
--os-variant ubuntu16.04 \
--network network=default \
--graphics none \
--console pty,target_type=serial \
--location '[LOCATION]' \
--extra-args 'console=ttyS0,115200n8 serial'

You may need type the following commands through ssh to activate console.

$sudo systemctl enable serial-getty@ttyS0.service
$sudo systemctl start serial-getty@ttyS0.service

Edit VM configuration with virsh.

$ virsh edit [VM_NAME]

You need to add xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' into the domain tag because of adding <qemu:commandline> tag. In addition, you need to add the tag enclosed by <memoryBacking> and </memoryBacking>, <qemu:commandline> and </qemu:commandline> because SPP uses vhost-user as interface with VM. Note that number used in those tags should be the same value (e.g. chr0,sock0,vhost-net0) and these values should correspond to “add vhost N” (in this example 0). MAC address used in <qemu:arg value='virtio-net-pci,netdev=vhost-net0,mac=52:54:00:12:34:56'/> can be specified when registering MAC address to classifier using Secondary command.

The following is an example of modified xml file:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>spp-vm1</name>
  <uuid>d90f5420-861a-4479-8559-62d7a1545cb9</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/spp-vm1.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
      function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
      function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:99:aa:7f'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
      function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
      function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host'/>
    <qemu:arg value='-object'/>
    <qemu:arg
    value='memory-backend-file,id=mem,size=4096M,mem-path=/run/hugepages/kvm,share=on'/>
    <qemu:arg value='-numa'/>
    <qemu:arg value='node,memdev=mem'/>
    <qemu:arg value='-mem-prealloc'/>
    <qemu:arg value='-chardev'/>
    <qemu:arg value='socket,id=chr0,path=/tmp/sock0,server'/>
    <qemu:arg value='-device'/>
    <qemu:arg
    value='virtio-net-pci,netdev=vhost-net0,mac=52:54:00:12:34:56'/>
    <qemu:arg value='-netdev'/>
    <qemu:arg value='vhost-user,id=vhost-net0,chardev=chr0,vhostforce'/>
    <qemu:arg value='-chardev'/>
    <qemu:arg value='socket,id=chr1,path=/tmp/sock1,server'/>
    <qemu:arg value='-device'/>
    <qemu:arg
    value='virtio-net-pci,netdev=vhost-net1,mac=52:54:00:12:34:57'/>
    <qemu:arg value='-netdev'/>
    <qemu:arg value='vhost-user,id=vhost-net1,chardev=chr1,vhostforce'/>
  </qemu:commandline>
</domain>

3.1.7. Setup spp_mirror

Setup of spp_mirror is almost the same as SPP VF. Configuration of use of shallowcopy or deepcopy is different from spp_vf. It is defined in src/mirror/Makefile and which of copying is used is configured by editing CFLAG option. It is defined to use shallowcopy by default.

If you use deepcopy, comment out the line of -Dspp_mirror_SHALLOWCOPY to be disabled.

#CFLAGS += -Dspp_mirror_SHALLOWCOPY

Then, run make command to compile spp_mirror.

$ make

3.1.8. Trouble Shooting Guide

You might encounter a permission error for tmp/sockN because of appamor. In this case, you should try it.

#Edit /etc/libvirt/qemu.conf and set security_driver to none:
$sudo vi /etc/libvirt/qemu.conf
...
security_driver = "none"
...
#Restart libvirtd:
$sudo systemctl restart libvirtd.service

Or, you remove appamor.

$ sudo apt-get remove apparmor

If you use CentOS, not Ubuntu, confirm that SELinux doesn’t prevent for permission. SELinux should be disabled in this case.

# /etc/selinux/config
SELINUX=disabled

Check your SELinux configuration.

$ getenforce
Disabled