4. Cross compiling DPDK for aarch64 and aarch32
This chapter describes how to cross compile DPDK for aarch64 on x86 build machine and compile 32-bit aarch32 DPDK on aarch64 build machine.
Note
Whilst it is recommended to natively build DPDK on aarch64 (just like with x86), it is also possible to cross compile DPDK for aarch64. An aarch64 cross compiler GNU toolchain or an LLVM/clang toolchain may be used for cross-compilation.
4.1. Prerequisites
4.1.1. NUMA library
NUMA is required by most modern machines, not needed for non-NUMA architectures.
Note
For compiling the NUMA lib, run libtool –version to ensure the libtool version >= 2.2, otherwise the compilation will fail with errors.
git clone https://github.com/numactl/numactl.git
cd numactl
git checkout v2.0.13 -b v2.0.13
./autogen.sh
autoconf -i
./configure --host=aarch64-linux-gnu CC=aarch64-none-linux-gnu-gcc --prefix=<numa install dir>
make install
Note
The compiler is aarch64-none-linux-gnu-gcc if you download GCC
using the below guide. If you’re using a different compiler,
make sure you’re using the proper executable name.
The numa header files and lib file is generated in the include and lib folder
respectively under <numa install dir>.
4.1.2. Meson prerequisites
Meson depends on pkgconfig to find the dependencies.
The package pkg-config-aarch64-linux-gnu is required for aarch64.
To install it in Ubuntu:
sudo apt install pkg-config-aarch64-linux-gnu
For aarch32, install pkg-config-arm-linux-gnueabihf:
sudo apt install pkg-config-arm-linux-gnueabihf
4.2. GNU toolchain
4.2.1. Get the cross toolchain
The latest GNU cross compiler toolchain can be downloaded from: https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads.
It is always recommended to check and get the latest compiler tool from the page and use it to generate better code. As of this writing 9.2-2019.12 is the newest, the following description is an example of this version.
For aarch64:
wget https://developer.arm.com/-/media/Files/downloads/gnu-a/9.2-2019.12/binrel/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu.tar.xz
tar -xvf gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu.tar.xz
export PATH=$PATH:<cross_install_dir>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu/bin
For aarch32:
wget https://developer.arm.com/-/media/Files/downloads/gnu-a/9.2-2019.12/binrel/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf.tar.xz
tar -xvf gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf.tar.xz
export PATH=$PATH:<cross_install_dir>/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf/bin
Note
For the host requirements and other info, refer to the release note section: https://releases.linaro.org/components/toolchain/binaries/
4.2.2. Augment the GNU toolchain with NUMA support
Copy the NUMA header files and lib to the cross compiler’s directories:
cp <numa_install_dir>/include/numa*.h <cross_install_dir>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu/aarch64-none-linux-gnu/libc/usr/include/
cp <numa_install_dir>/lib/libnuma.a <cross_install_dir>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu/lib/gcc/aarch64-none-linux-gnu/9.2.1/
cp <numa_install_dir>/lib/libnuma.so <cross_install_dir>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu/lib/gcc/aarch64-none-linux-gnu/9.2.1/
Note
Using LDFLAGS and CFLAGS is not a viable alternative to copying the files.
The Meson docs say it is not recommended, as there are many caveats
to their use with Meson, especially when rebuilding the project.
A viable alternative would be to use the c_args and c_link_args
options with Meson 0.51.0 and higher:
-Dc_args=-I<numa_install_dir>/include -Dc_link_args=-L<numa_install_dir>/lib
For Meson versions lower than 0.51.0, the c_args and c_link_args
options do not apply to cross compilation.
However, the compiler/linker flags may be added to cross files under [properties]:
c_args = ['-I<numa_install_dir>/include']
c_link_args = ['-L<numa_install_dir>/lib']
4.2.3. Cross Compiling DPDK with GNU toolchain using Meson
Note
The names of GCC binaries in cross files differ from the downloaded ones,
which have an extra -none- in their name.
Please modify the cross file binaries accordingly
when using the downloaded cross compilers.
An example cross file with modified names and added NUMA paths would look like this:
[binaries]
c = 'aarch64-none-linux-gnu-gcc'
cpp = 'aarch64-none-linux-gnu-cpp'
ar = 'aarch64-none-linux-gnu-gcc-ar'
strip = 'aarch64-none-linux-gnu-strip'
pkgconfig = 'aarch64-linux-gnu-pkg-config' # the downloaded binaries
# do not contain a pkgconfig binary, so it is not modified
pcap-config = ''
[host_machine]
system = 'linux'
cpu_family = 'aarch64'
cpu = 'armv8-a'
endian = 'little'
[properties]
# Generate binaries that are portable across all Armv8 machines
platform = 'generic'
c_args = ['-I<numa_install_dir>/include'] # replace <numa_install_dir>
c_link_args = ['-L<numa_install_dir>/lib'] # with your path
To cross-compile DPDK on a desired target machine we can use the following command:
meson setup cross-build --cross-file <target_machine_configuration>
ninja -C cross-build
For example if the target machine is aarch64 we can use the following command, provided the cross file has been modified accordingly:
meson setup aarch64-build-gcc --cross-file config/arm/arm64_armv8_linux_gcc
ninja -C aarch64-build-gcc
If the target machine is aarch32 we can use the following command, provided the cross file has been modified accordingly:
meson setup aarch32-build --cross-file config/arm/arm32_armv8_linux_gcc
ninja -C aarch32-build
4.3. LLVM/Clang toolchain
4.3.1. Obtain the cross tool chain
The latest LLVM/Clang cross compiler toolchain can be downloaded from: https://developer.arm.com/tools-and-software/open-source-software/developer-tools/llvm-toolchain.
# Ubuntu binaries
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/clang+llvm-10.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz
The LLVM/Clang toolchain does not implement the standard c library. The GNU toolchain ships an implementation we can use. Refer to obtain_GNU_toolchain to get the GNU toolchain.
4.3.2. Unzip and add into the PATH
tar -xvf clang+llvm-10.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz
export PATH=$PATH:<cross_install_dir>/clang+llvm-10.0.0-x86_64-linux-gnu-ubuntu-18.04/bin
4.3.3. Cross Compiling DPDK with LLVM/Clang toolchain using Meson
Note
To use the NUMA library follow the same steps as for augment_the_gnu_toolchain_with_numa_support.
The paths to GNU stdlib must be specified in a cross file.
Augmenting the default cross-file’s c_args and c_link_args
config/arm/arm64_armv8_linux_clang_ubuntu1804 would look like this:
...
c_args = ['-target', 'aarch64-linux-gnu', '--sysroot', '<cross_install_dir>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu/aarch64-none-linux-gnu/libc']
c_link_args = ['-target', 'aarch64-linux-gnu', '-fuse-ld=lld', '--sysroot', '<cross_install_dir>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu/aarch64-none-linux-gnu/libc', '--gcc-toolchain=<cross_install_dir>/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu']
Assuming the file with augmented c_args and c_link_args
is named arm64_armv8_linux_clang,
use the following command to cross-compile DPDK for the target machine:
meson setup aarch64-build-clang --cross-file config/arm/arm64_armv8_linux_clang
ninja -C aarch64-build-clang
4.3.4. Cross Compiling DPDK with LLVM/Clang toolchain using Meson on Ubuntu 18.04
On most popular Linux distribution it is not necessary to download the toolchains, but rather use the packages provided by said distributions. On Ubuntu 18.04, these packages are needed:
sudo apt-get install pkg-config-aarch64-linux-gnu clang llvm llvm-dev lld
libc6-dev-arm64-cross libatomic1-arm64-cross libgcc-8-dev-arm64-cross
Use the following command to cross-compile DPDK for the target machine:
meson setup aarch64-build-clang --cross-file config/arm/arm64_armv8_linux_clang_ubuntu1804
ninja -C aarch64-build-clang
4.4. Building for an aarch64 SoC on an aarch64 build machine
If you wish to build on an aarch64 build machine for a different aarch64 SoC, you don’t need a separate cross toolchain, just a different set of configuration options. To build for an aarch64 SoC, use the -Dplatform meson option:
meson setup soc_build -Dplatform=<target_soc>
Substitute <target_soc> with one of the supported SoCs
generic: Generic un-optimized build for armv8 aarch64 execution mode.
generic_aarch32: Generic un-optimized build for armv8 aarch32 execution mode.
altra: Ampere Altra/AltraMax
ampereone: Ampere AmpereOne
ampereoneac04: Ampere AmpereOneAC04
armada: Marvell ARMADA
bluefield: NVIDIA BlueField
bluefield3: NVIDIA BlueField-3
capri: AMD Pensando Capri
cdx: AMD CDX
centriq2400: Qualcomm Centriq 2400
cn9k: Marvell OCTEON 9
cn10k: Marvell OCTEON 10
dpaa: NXP DPAA
elba: AMD Pensando Elba
emag: Ampere eMAG
ft2000plus: Phytium FT-2000+
tys2500: Phytium TengYun S2500
grace: NVIDIA Grace
graviton2: AWS Graviton2
graviton3: AWS Graviton3
graviton4: AWS Graviton4
hip10: HiSilicon HIP10
kunpeng920: HiSilicon Kunpeng 920
kunpeng930: HiSilicon Kunpeng 930
n1sdp: Arm Neoverse N1SDP
n2: Arm Neoverse N2
odyssey: Marvell Odyssey
stingray: Broadcom Stingray
thunderx2: Marvell ThunderX2 T99
thunderxt88: Marvell ThunderX T88
thunderxt83: Marvell ThunderX T83
v2: Arm Neoverse V2
These SoCs are also used in cross files, e.g.:
[properties]
# Generate binaries that are portable across all Armv8 machines
platform = 'generic'
4.5. Supported SoC configuration
The SoC configuration is a combination of implementer and CPU part number configuration and SoC-specific configuration:
soc_<name> = {
'description': 'SoC Description', # mandatory
'implementer': <implementer_id>, # mandatory
'part_number': <part_number>, # mandatory
'numa': false, # optional, specify for non-NUMA SoCs
'enable_drivers': 'common/*,bus/*', # optional, comma-separated list of
# drivers to build, wildcards are accepted
'disable_drivers': 'crypto/*', # optional, comma-separated list of
# drivers to disable, wildcards are accepted
'flags': [
['RTE_MAX_LCORE', '16'],
['RTE_MAX_NUMA_NODES', '1']
] # optional, list of DPDK options that will be added
# or overwritten
}
Where <implementer_id> is a key defined in the implementers dictionary in config/arm/meson.build (e.g. 0x41) and part_number is a key defined in implementers[<implementer_id>][‘part_number_config’] dictionary (i.e. the part number must be defined for the implementer, e.g. for 0x41, a valid value is 0xd49, which is the neoverse-n2 SoC).