1. Using the AF_XDP driver in Kubernetes
1.1. Introduction
Two infrastructure components are needed in order to provision a pod that is using the AF_XDP PMD in Kubernetes:
AF_XDP Device Plugin (DP).
AF_XDP Container Network Interface (CNI) binary.
Both of these components are available through the AF_XDP Device Plugin for Kubernetes repository.
The AF_XDP DP provisions and advertises networking interfaces to Kubernetes, while the CNI configures and plumbs network interfaces for the Pod.
This document explains how to use the AF_XDP Device Plugin for Kubernetes with a DPDK application using the AF_XDP Poll Mode Driver.
1.2. Background
The standard AF_XDP Poll Mode Driver initialization process involves loading an eBPF program onto the kernel netdev to be used by the PMD. This operation requires root or escalated Linux privileges and thus prevents the PMD from working in an unprivileged container. The AF_XDP Device Plugin handles this situation by managing the eBPF program(s) on behalf of the Pod, outside of the pod context.
At a technical level the AF_XDP Device Plugin opens a Unix Domain Socket (UDS) and listens for a client to make requests over that socket. A DPDK application acting as a client connects and initiates a configuration “handshake”. After some validation on the Device Plugin side, the client receives a file descriptor which points to the XSKMAP associated with the loaded eBPF program. The XSKMAP is an eBPF map of AF_XDP sockets (XSK). The client can then proceed with creating an AF_XDP socket and inserting that socket into the XSKMAP pointed to by the descriptor.
The EAL vdev argument use_cni
is used to indicate that the user wishes
to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor
from the CNI.
When this flag is set,
the XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD
libbpf flag
should be used when creating the socket
to instruct libbpf not to load the default libbpf program on the netdev.
Instead the loading is handled by the AF_XDP Device Plugin.
The EAL vdev argument use_pinned_map
is used indicate to the AF_XDP PMD
to retrieve the XSKMAP fd from a pinned eBPF map.
This map is expected to be pinned by an external entity like the AF_XDP Device Plugin.
This enabled unprivileged pods to create and use AF_XDP sockets.
When this flag is set, the XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD
libbpf flag
is used by the AF_XDP PMD when creating the AF_XDP socket.
The EAL vdev argument dp_path
is used alongside the use_cni
or use_pinned_map
arguments to explicitly tell the AF_XDP PMD where to find either:
The UDS to interact with the AF_XDP Device Plugin. OR
The pinned xskmap to use when creating AF_XDP sockets.
If this argument is not passed alongside the use_cni
or use_pinned_map
arguments
then the AF_XDP PMD configures it internally to the AF_XDP Device Plugin for Kubernetes.
Note
DPDK AF_XDP PMD <= v23.11 will only work with the AF_XDP Device Plugin <= commit id 38317c2.
Note
DPDK AF_XDP PMD > v23.11 will work with latest version of the AF_XDP Device Plugin
through a combination of the dp_path
and/or the use_cni
parameter.
In these versions of the PMD if a user doesn’t explicitly set the dp_path
parameter
when using use_cni
then that path is transparently configured in the AF_XDP PMD
to the default AF_XDP Device Plugin for Kubernetes mount point path.
The path can be overridden by explicitly setting the dp_path
param.
Note
DPDK AF_XDP PMD > v23.11 is backwards compatible
with (older) versions of the AF_XDP DP <= commit id 38317c2
by explicitly setting dp_path
to /tmp/afxdp.sock
.
1.3. Prerequisites
Device Plugin and DPDK container prerequisites:
Create a DPDK container image.
Set up the device plugin and prepare the Pod Spec as described in the instructions for AF_XDP Device Plugin for Kubernetes.
The Docker image should contain the libbpf and libxdp libraries, which are dependencies for AF_XDP, and should include support for the
ethtool
command.The Pod should have enabled the capabilities
CAP_NET_RAW
for AF_XDP socket creation,IPC_LOCK
for umem creation andCAP_BPF
(for Kernel < 5.19) along with support for hugepages.Note
For Kernel versions < 5.19, all BPF sys calls required CAP_BPF, to access maps shared between the eBFP program and the userspace program. Kernels >= 5.19, only requires CAP_BPF for map creation (BPF_MAP_CREATE) and loading programs (BPF_PROG_LOAD).
Increase locked memory limit so containers have enough memory for packet buffers. For example:
cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf [Service] LimitMEMLOCK=infinity EOF
dpdk-testpmd application should have AF_XDP feature enabled.
For further information see the docs for the: AF_XDP Poll Mode Driver.
1.4. Example
1.4.1. Build a DPDK container image (using Docker)
Create a Dockerfile (should be placed in top level DPDK directory):
FROM fedora:38 # Setup container to build DPDK applications RUN dnf -y upgrade && dnf -y install \ libbsd-devel \ numactl-libs \ libbpf-devel \ libbpf \ meson \ ninja-build \ libxdp-devel \ libxdp \ numactl-devel \ python3-pyelftools \ python38 \ iproute RUN dnf groupinstall -y 'Development Tools' # Create DPDK dir and copy over sources # Create DPDK dir and copy over sources COPY ./ /dpdk WORKDIR /dpdk # Build DPDK RUN meson setup build RUN ninja -C build
Build a DPDK container image (using Docker)
# docker build -t dpdk -f Dockerfile
1.4.2. Run dpdk-testpmd with the AF_XDP Device Plugin + CNI
Clone the AF_XDP Device plugin and CNI
# git clone https://github.com/redhat-et/afxdp-plugins-for-kubernetes.git
Note
Ensure you have the AF_XDP Device Plugin + CNI prerequisites installed.
Build the AF_XDP Device plugin and CNI
# cd afxdp-plugins-for-kubernetes/ # make image
Make sure to modify the image used by the daemonset.yml file in the deployments directory with the following configuration:
image: afxdp-device-plugin:latest
Note
This will select the AF_XDP DP image that was built locally. Detailed configuration options can be found in the AF_XDP Device Plugin readme .
Deploy the AF_XDP Device Plugin and CNI
# kubectl create -f deployments/daemonset.yml
Create the Network Attachment definition
# kubectl create -f nad.yaml
Sample nad.yml
apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: afxdp-network annotations: k8s.v1.cni.cncf.io/resourceName: afxdp/myPool spec: config: '{ "cniVersion": "0.3.0", "type": "afxdp", "mode": "primary", "logFile": "afxdp-cni.log", "logLevel": "debug", "ethtoolCmds" : ["-N -device- rx-flow-hash udp4 fn", "-N -device- flow-type udp4 dst-port 2152 action 22" ], "ipam": { "type": "host-local", "subnet": "192.168.1.0/24", "rangeStart": "192.168.1.200", "rangeEnd": "192.168.1.220", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.1.1" } }'
For further reference please use the example provided by the AF_XDP DP nad.yaml
Run the Pod
# kubectl create -f pod.yaml
Sample pod.yaml:
apiVersion: v1 kind: Pod metadata: name: dpdk annotations: k8s.v1.cni.cncf.io/networks: afxdp-network spec: containers: - name: testpmd image: dpdk:latest command: ["tail", "-f", "/dev/null"] securityContext: capabilities: add: - NET_RAW - IPC_LOCK resources: requests: afxdp/myPool: '1' limits: hugepages-1Gi: 2Gi cpu: 2 memory: 256Mi afxdp/myPool: '1' volumeMounts: - name: hugepages mountPath: /dev/hugepages volumes: - name: hugepages emptyDir: medium: HugePages
For further reference please see the pod.yaml
Run DPDK with a command like the following:
kubectl exec -i <Pod name> --container <containers name> -- \ /<Path>/dpdk-testpmd -l 0,1 --no-pci \ --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \ --no-mlockall --in-memory \ -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
Or
kubectl exec -i <Pod name> --container <containers name> -- \ /<Path>/dpdk-testpmd -l 0,1 --no-pci \ --vdev=net_af_xdp0,use_cni=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/afxdp.sock" \ --no-mlockall --in-memory \ -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
Or
kubectl exec -i <Pod name> --container <containers name> -- \ /<Path>/dpdk-testpmd -l 0,1 --no-pci \ --vdev=net_af_xdp0,use_pinned_map=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/xsks_map" \ --no-mlockall --in-memory \ -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
Note
If the dp_path
parameter isn’t explicitly set with use_cni
or use_pinned_map
the AF_XDP PMD will set the parameter values to the AF_XDP Device Plugin for Kubernetes defaults.