215. VM hotplug Tests

215.1. Description

Hotplug requires to let users plug out/in the NIC during runtime. The DPDK software should handle that well without any crash or similar. That means the interrupt event reporting needed to support that.

Note, this feature is about to fix the gap of passive SR-IOV live migration by failsafe PMD. So “plug out/in the NIC” typically does not the case that physically plug out/in a NIC from/to server, it should be case that remove/add a qemu device from/to a VM.

215.2. Hardware

Ixgbe, ice and i40e NICs

215.3. Note

Known issue for UIO in dpdk/doc/guides/rel_notes/known_issues.rst as below, This test plan only test VFIO scenario.

215.3.1. Kernel crash when hot-unplug igb_uio device while DPDK application is running

Description:

When device has been bound to igb_uio driver and application is running, hot-unplugging the device may cause kernel crash.

Reason:

When device is hot-unplugged, igb_uio driver will be removed which will destroy UIO resources. Later trying to access any uio resource will cause kernel crash.

Resolution/Workaround:

If using DPDK for PCI HW hot-unplug, prefer to bind device with VFIO instead of IGB_UIO.

Affected Environment/Platform:

ALL.

Driver/Module:

igb_uio module.

215.4. Test Case: one device

Bind host PF port 0 to vfio_pci:

modprobe vfio_pci
./usertools/dpdk-devbind.py -b vfio_pci 18:00.0

Passthrough PF and start qemu script as below, using “-monitor stdio” will send the monitor to the standard output:

taskset -c 0-7 qemu-system-x86_64 -enable-kvm \
-m 4096 -cpu host -smp 8 -name qemu-vm1 \
-monitor stdio \
-drive file=/home/vm_b/ubuntu-16.04_test_vfio.img \
-device vfio-pci,host=0000:18:00.0,id=dev1 \
-netdev tap,id=hostnet1,ifname=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,vhost=on \
-device rtl8139,netdev=hostnet1,id=net0,mac=00:00:00:14:c4:31,bus=pci.0,addr=0x1f \
-vnc :5

Log in VM, bind passthrough port 0 to vfio-pci:

modprobe -r vfio_iommu_type1
modprobe -r vfio
modprobe vfio enable_unsafe_noiommu_mode=1
modprobe vfio-pci
./usertools/dpdk-devbind.py -b vfio-pci 00:03.0

Start testpmd with “–hot-plug” enable, set rxonly forward mode and enable verbose output:

./dpdk-testpmd -c f -n 4 -- -i --hot-plug
testpmd> set fwd rxonly
testpmd> set verbose 1
testpmd> start

Send packets from tester, check RX could work successfully

Set txonly forward mode, send packet from testpmd, check TX could work successfully:

testpmd> set fwd txonly
testpmd> start

Remove device from qemu interface:

(qemu) device_del dev1

Check device is removed, no system hange and core dump:

./usertools/dpdk-devbind.py -s

Add device from qemu interface:

(qemu) device_add vfio-pci,host=18:00.0,id=dev1

Check driver adds the device, bind port to vfio-pci

Attach the VF from testpmd:

testpmd> port attach 00:03.0
testpmd> port start all

Check testpmd adds the device successfully, no hange and core dump

Check RX/TX could work successfully

Repeat above steps for 3 times

215.5. Test Case: one device + reset

Bind host PF port 0 to vfio_pci:

modprobe vfio_pci
./usertools/dpdk-devbind.py -b vfio_pci 18:00.0

Log in VM, passthrough PF and start qemu script same as above

Bind passthrough port 0 to vfio-pci:

modprobe -r vfio_iommu_type1
modprobe -r vfio
modprobe vfio enable_unsafe_noiommu_mode=1
modprobe vfio-pci
./usertools/dpdk-devbind.py -b vfio-pci 00:03.0

Start testpmd with “–hot-plug” enable, set rxonly forward mode and enable verbose output:

./dpdk-testpmd -c f -n 4 -- -i --hot-plug
testpmd> set fwd rxonly
testpmd> set verbose 1
testpmd> start

Send packets from tester, check RX could work successfully

Set txonly forward mode, send packet from testpmd, check TX could work successfully:

testpmd> set fwd txonly
testpmd> start

Remove device from qemu interface:

(qemu) device_del dev1

Quit testpmd

Check device is removed, no system hange and core dump:

./usertools/dpdk-devbind.py -s

Add device from qemu interface:

(qemu) device_add vfio-pci,host=18:00.0,id=dev1

Check driver adds the device, bind port to vfio-pci

Restart testpmd

Check testpmd adds the device successfully, no hange and core dump

Check RX/TX could work successfully

Repeat above steps for 3 times

215.6. Test Case: two/multi devices

Bind host PF port 0 and port 1 to vfio_pci:

modprobe vfio_pci
./usertools/dpdk-devbind.py -b vfio_pci 18:00.0 18:00.1

Passthrough PFs and start qemu script as below, using “-monitor stdio” will send the monitor to the standard output:

taskset -c 0-7 qemu-system-x86_64 -enable-kvm \
-m 4096 -cpu host -smp 8 -name qemu-vm1 \
-monitor stdio \
-drive file=/home/vm_b/ubuntu-16.04_test_vfio.img \
-device vfio-pci,host=0000:18:00.0,id=dev1 \
-device vfio-pci,host=0000:18:00.1,id=dev2 \
-netdev tap,id=hostnet1,ifname=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,vhost=on \
-device rtl8139,netdev=hostnet1,id=net0,mac=00:00:00:14:c4:31,bus=pci.0,addr=0x1f \
-vnc :5

Log in VM, bind passthrough port 0 and port 1 to vfio-pci:

modprobe -r vfio_iommu_type1
modprobe -r vfio
modprobe vfio enable_unsafe_noiommu_mode=1
modprobe vfio-pci
./usertools/dpdk-devbind.py -b vfio-pci 00:03.0 00:04.0

Start testpmd with “–hot-plug” enable, set rxonly forward mode and enable verbose output:

./dpdk-testpmd -c f -n 4 -- -i --hot-plug
testpmd> set fwd rxonly
testpmd> set verbose 1
testpmd> start

Send packets from tester, check RX could work successfully Set txonly forward mode, send packet from testpmd, check TX could work successfully:

testpmd> set fwd txonly
testpmd> start

Remove device 1 and device 2 from qemu interface:

(qemu) device_del dev1
(qemu) device_del dev2

Check devices are removed, no system hange and core dump:

./usertools/dpdk-devbind.py -s

Add devices from qemu interface:

(qemu) device_add vfio-pci,host=18:00.0,id=dev1
(qemu) device_add vfio-pci,host=18:00.1,id=dev2

Check driver adds the devices, bind port to vfio-pci

Attach the VFs from testpmd:

testpmd> port attach 00:03.0
testpmd> port attach 00:04.0
testpmd> port start all

Check testpmd adds the devices successfully, no hange and core dump

Check RX/TX could work successfully

Repeat above steps for 3 times

215.7. Test Case: two/multi devices + reset

Bind host PF port 0 and port 1 to vfio_pci:

modprobe vfio_pci
./usertools/dpdk-devbind.py -b vfio_pci 18:00.0 18:00.1

Passthrough PFs and start qemu script same as above

Log in VM, bind passthrough port 0 and port 1 to vfio-pci:

modprobe -r vfio_iommu_type1
modprobe -r vfio
modprobe vfio enable_unsafe_noiommu_mode=1
modprobe vfio-pci
./usertools/dpdk-devbind.py -b vfio-pci 00:03.0 00:04.0

Start testpmd with “–hot-plug” enable, set rxonly forward mode and enable verbose output:

./dpdk-testpmd -c f -n 4 -- -i --hot-plug
testpmd> set fwd rxonly
testpmd> set verbose 1
testpmd> start

Send packets from tester, check RX could work successfully

Set txonly forward mode, send packets from testpmd, check TX could work successfully:

testpmd> set fwd txonly
testpmd> start

Remove device 1 and device 2 from qemu interface:

(qemu) device_del dev1
(qemu) device_del dev2

Quit testpmd

Check devices are removed, no system hange and core dump:

./usertools/dpdik-devbind.py -s

Add devices from qemu interface:

(qemu) device_add vfio-pci,host=18:00.0,id=dev1
(qemu) device_add vfio-pci,host=18:00.1,id=dev2

Check driver adds the devices, bind ports to vfio-pci

Restart testpmd

Check testpmd adds the devices successfully, no hange and core dump

Check RX/TX could work successfully

Repeat above steps for 3 times