.. SPDX-License-Identifier: BSD-3-Clause
Copyright(c) 2023 Intel Corporation.
Using the AF_XDP driver in Kubernetes
=====================================
Introduction
------------
Two infrastructure components are needed in order to provision a pod
that is using the AF_XDP PMD in Kubernetes:
1. AF_XDP Device Plugin (DP).
2. AF_XDP Container Network Interface (CNI) binary.
Both of these components are available through
the `AF_XDP Device Plugin for Kubernetes`_ repository.
The AF_XDP DP provisions and advertises networking interfaces to Kubernetes,
while the CNI configures and plumbs network interfaces for the Pod.
This document explains how to use the `AF_XDP Device Plugin for Kubernetes`_
with a DPDK application using the :doc:`../nics/af_xdp`.
.. _AF_XDP Device Plugin for Kubernetes: https://github.com/redhat-et/afxdp-plugins-for-kubernetes
Background
----------
The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program
onto the kernel netdev to be used by the PMD.
This operation requires root or escalated Linux privileges
and thus prevents the PMD from working in an unprivileged container.
The AF_XDP Device Plugin handles this situation
by managing the eBPF program(s) on behalf of the Pod, outside of the pod context.
At a technical level the AF_XDP Device Plugin opens a Unix Domain Socket (UDS)
and listens for a client to make requests over that socket.
A DPDK application acting as a client connects and initiates a configuration "handshake".
After some validation on the Device Plugin side,
the client receives a file descriptor which points to the XSKMAP
associated with the loaded eBPF program.
The XSKMAP is an eBPF map of AF_XDP sockets (XSK).
The client can then proceed with creating an AF_XDP socket
and inserting that socket into the XSKMAP pointed to by the descriptor.
The EAL vdev argument ``use_cni`` is used to indicate that the user wishes
to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor
from the CNI.
When this flag is set,
the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag
should be used when creating the socket
to instruct libbpf not to load the default libbpf program on the netdev.
Instead the loading is handled by the AF_XDP Device Plugin.
The EAL vdev argument ``use_pinned_map`` is used indicate to the AF_XDP PMD
to retrieve the XSKMAP fd from a pinned eBPF map.
This map is expected to be pinned by an external entity like the AF_XDP Device Plugin.
This enabled unprivileged pods to create and use AF_XDP sockets.
When this flag is set, the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag
is used by the AF_XDP PMD when creating the AF_XDP socket.
The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` or ``use_pinned_map``
arguments to explicitly tell the AF_XDP PMD where to find either:
1. The UDS to interact with the AF_XDP Device Plugin. OR
2. The pinned xskmap to use when creating AF_XDP sockets.
If this argument is not passed alongside the ``use_cni`` or ``use_pinned_map`` arguments
then the AF_XDP PMD configures it internally to the `AF_XDP Device Plugin for Kubernetes`_.
.. note::
DPDK AF_XDP PMD <= v23.11 will only work with
the AF_XDP Device Plugin <= commit id `38317c2`_.
.. note::
DPDK AF_XDP PMD > v23.11 will work with latest version of the AF_XDP Device Plugin
through a combination of the ``dp_path`` and/or the ``use_cni`` parameter.
In these versions of the PMD if a user doesn't explicitly set the ``dp_path`` parameter
when using ``use_cni`` then that path is transparently configured in the AF_XDP PMD
to the default `AF_XDP Device Plugin for Kubernetes`_ mount point path.
The path can be overridden by explicitly setting the ``dp_path`` param.
.. note::
DPDK AF_XDP PMD > v23.11 is backwards compatible
with (older) versions of the AF_XDP DP <= commit id `38317c2`_
by explicitly setting ``dp_path`` to ``/tmp/afxdp.sock``.
.. _38317c2: https://github.com/redhat-et/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669
Prerequisites
-------------
Device Plugin and DPDK container prerequisites:
* Create a DPDK container image.
* Set up the device plugin and prepare the Pod Spec as described in
the instructions for `AF_XDP Device Plugin for Kubernetes`_.
* The Docker image should contain the libbpf and libxdp libraries,
which are dependencies for AF_XDP,
and should include support for the ``ethtool`` command.
* The Pod should have enabled the capabilities
``CAP_NET_RAW`` for AF_XDP socket creation,
``IPC_LOCK`` for umem creation and
``CAP_BPF`` (for Kernel < 5.19) along with support for hugepages.
.. note::
For Kernel versions < 5.19, all BPF sys calls required CAP_BPF,
to access maps shared between the eBFP program and the userspace program.
Kernels >= 5.19, only requires CAP_BPF for map creation (BPF_MAP_CREATE)
and loading programs (BPF_PROG_LOAD).
* Increase locked memory limit so containers have enough memory for packet buffers.
For example:
.. code-block:: console
cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf
[Service]
LimitMEMLOCK=infinity
EOF
* dpdk-testpmd application should have AF_XDP feature enabled.
For further information see the docs for the: :doc:`../../nics/af_xdp`.
Example
-------
Build a DPDK container image (using Docker)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Create a Dockerfile (should be placed in top level DPDK directory):
.. code-block:: console
FROM fedora:38
# Setup container to build DPDK applications
RUN dnf -y upgrade && dnf -y install \
libbsd-devel \
numactl-libs \
libbpf-devel \
libbpf \
meson \
ninja-build \
libxdp-devel \
libxdp \
numactl-devel \
python3-pyelftools \
python38 \
iproute
RUN dnf groupinstall -y 'Development Tools'
# Create DPDK dir and copy over sources
# Create DPDK dir and copy over sources
COPY ./ /dpdk
WORKDIR /dpdk
# Build DPDK
RUN meson setup build
RUN ninja -C build
2. Build a DPDK container image (using Docker)
.. code-block:: console
# docker build -t dpdk -f Dockerfile
Run dpdk-testpmd with the AF_XDP Device Plugin + CNI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Clone the AF_XDP Device plugin and CNI
.. code-block:: console
# git clone https://github.com/redhat-et/afxdp-plugins-for-kubernetes.git
.. note::
Ensure you have the AF_XDP Device Plugin + CNI prerequisites installed.
* Build the AF_XDP Device plugin and CNI
.. code-block:: console
# cd afxdp-plugins-for-kubernetes/
# make image
* Make sure to modify the image used by the `daemonset.yml`_ file
in the deployments directory with the following configuration:
.. _daemonset.yml : https://github.com/redhat-et/afxdp-plugins-for-kubernetes/blob/main/deployments/daemonset.yml
.. code-block:: yaml
image: afxdp-device-plugin:latest
.. note::
This will select the AF_XDP DP image that was built locally.
Detailed configuration options can be found in the AF_XDP Device Plugin `readme`_ .
.. _readme: https://github.com/redhat-et/afxdp-plugins-for-kubernetes#readme
* Deploy the AF_XDP Device Plugin and CNI
.. code-block:: console
# kubectl create -f deployments/daemonset.yml
* Create the Network Attachment definition
.. code-block:: console
# kubectl create -f nad.yaml
Sample nad.yml
.. code-block:: yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: afxdp-network
annotations:
k8s.v1.cni.cncf.io/resourceName: afxdp/myPool
spec:
config: '{
"cniVersion": "0.3.0",
"type": "afxdp",
"mode": "primary",
"logFile": "afxdp-cni.log",
"logLevel": "debug",
"ethtoolCmds" : ["-N -device- rx-flow-hash udp4 fn",
"-N -device- flow-type udp4 dst-port 2152 action 22"
],
"ipam": {
"type": "host-local",
"subnet": "192.168.1.0/24",
"rangeStart": "192.168.1.200",
"rangeEnd": "192.168.1.220",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "192.168.1.1"
}
}'
For further reference please use the example provided by the AF_XDP DP `nad.yaml`_
.. _nad.yaml: https://github.com/redhat-et/afxdp-plugins-for-kubernetes/blob/main/examples/network-attachment-definition.yaml
* Run the Pod
.. code-block:: console
# kubectl create -f pod.yaml
Sample pod.yaml:
.. code-block:: yaml
apiVersion: v1
kind: Pod
metadata:
name: dpdk
annotations:
k8s.v1.cni.cncf.io/networks: afxdp-network
spec:
containers:
- name: testpmd
image: dpdk:latest
command: ["tail", "-f", "/dev/null"]
securityContext:
capabilities:
add:
- NET_RAW
- IPC_LOCK
resources:
requests:
afxdp/myPool: '1'
limits:
hugepages-1Gi: 2Gi
cpu: 2
memory: 256Mi
afxdp/myPool: '1'
volumeMounts:
- name: hugepages
mountPath: /dev/hugepages
volumes:
- name: hugepages
emptyDir:
medium: HugePages
For further reference please see the `pod.yaml`_
.. _pod.yaml: https://github.com/redhat-et/afxdp-plugins-for-kubernetes/blob/main/examples/pod-spec.yaml
* Run DPDK with a command like the following:
.. code-block:: console
kubectl exec -i <Pod name> --container <containers name> -- \
/<Path>/dpdk-testpmd -l 0,1 --no-pci \
--vdev=net_af_xdp0,use_cni=1,iface=<interface name> \
--no-mlockall --in-memory \
-- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
Or
.. code-block:: console
kubectl exec -i <Pod name> --container <containers name> -- \
/<Path>/dpdk-testpmd -l 0,1 --no-pci \
--vdev=net_af_xdp0,use_cni=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/afxdp.sock" \
--no-mlockall --in-memory \
-- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
Or
.. code-block:: console
kubectl exec -i <Pod name> --container <containers name> -- \
/<Path>/dpdk-testpmd -l 0,1 --no-pci \
--vdev=net_af_xdp0,use_pinned_map=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/xsks_map" \
--no-mlockall --in-memory \
-- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
.. note::
If the ``dp_path`` parameter isn't explicitly set with ``use_cni`` or ``use_pinned_map``
the AF_XDP PMD will set the parameter values to the `AF_XDP Device Plugin for Kubernetes`_ defaults.