29. Development Kit Build System

29.1.1. Build Directory Concept

After installation, a build directory structure is created. Each build directory contains include files, libraries, and applications:

~/DPDK$ ls
app                               MAINTAINERS
config                            Makefile
COPYRIGHT                         mk
doc                               scripts
examples                          lib
tools                             x86_64-native-linuxapp-gcc
x86_64-native-linuxapp-icc        i686-native-linuxapp-gcc
i686-native-linuxapp-icc

...
~/DEV/DPDK$ ls i686-native-linuxapp-gcc

app build buildtools include kmod lib Makefile


~/DEV/DPDK$ ls i686-native-linuxapp-gcc/app/
cmdline_test   dump_cfg     test     testpmd
cmdline_test.map      dump_cfg.map   test.map
        testpmd.map


~/DEV/DPDK$ ls i686-native-linuxapp-gcc/lib/

libethdev.a  librte_hash.a  librte_mbuf.a librte_pmd_ixgbe.a

librte_cmdline.a librte_lpm.a librte_mempool.a librte_ring.a

librte_eal.a librte_pmd_e1000.a librte_timer.a


~/DEV/DPDK$ ls i686-native-linuxapp-gcc/include/
arch                       rte_cpuflags.h       rte_memcpy.h
cmdline_cirbuf.h           rte_cycles.h         rte_memory.h
cmdline.h                  rte_debug.h          rte_mempool.h
cmdline_parse_etheraddr.h  rte_eal.h            rte_memzone.h
cmdline_parse.h            rte_errno.h          rte_pci_dev_ids.h
cmdline_parse_ipaddr.h     rte_ethdev.h         rte_pci.h
cmdline_parse_num.h        rte_ether.h          rte_per_lcore.h
cmdline_parse_portlist.h   rte_fbk_hash.h       rte_prefetch.h
cmdline_parse_string.h     rte_hash_crc.h       rte_random.h
cmdline_rdline.h           rte_hash.h           rte_ring.h
cmdline_socket.h           rte_interrupts.h     rte_rwlock.h
cmdline_vt100.h            rte_ip.h             rte_sctp.h
exec-env                   rte_jhash.h          rte_spinlock.h
rte_alarm.h                rte_launch.h         rte_string_fns.h
rte_atomic.h               rte_lcore.h          rte_tailq.h
rte_branch_prediction.h    rte_log.h            rte_tcp.h
rte_byteorder.h            rte_lpm.h            rte_timer.h
rte_common.h               rte_malloc.h         rte_udp.h
rte_config.h               rte_mbuf.h

A build directory is specific to a configuration that includes architecture + execution environment + toolchain. It is possible to have several build directories sharing the same sources with different configurations.

For instance, to create a new build directory called my_sdk_build_dir using the default configuration template config/defconfig_x86_64-linuxapp, we use:

cd ${RTE_SDK}
make config T=x86_64-native-linuxapp-gcc O=my_sdk_build_dir

This creates a new my_sdk_build_dir directory. After that, we can compile by doing:

cd my_sdk_build_dir
make

which is equivalent to:

make O=my_sdk_build_dir

The content of the my_sdk_build_dir is then:

-- .config                         # used configuration

-- Makefile                        # wrapper that calls head Makefile
                                   # with $PWD as build directory


    -- build                              #All temporary files used during build
    +--app                                # process, including . o, .d, and .cmd files.
        |  +-- test                       # For libraries, we have the .a file.
        |  +-- test.o                     # For applications, we have the elf file.
        |  `-- ...
        +-- lib
            +-- librte_eal
            |   `-- ...
            +-- librte_mempool
            |  +--  mempool-file1.o
            |  +--  .mempool-file1.o.cmd
            |  +--  .mempool-file1.o.d
            |  +--   mempool-file2.o
            |  +--  .mempool-file2.o.cmd
            |  +--  .mempool-file2.o.d
            |  `--  mempool.a
            `-- ...

-- include                # All include files installed by libraries
    +-- librte_mempool.h  # and applications are located in this
    +-- rte_eal.h         # directory. The installed files can depend
    +-- rte_spinlock.h    # on configuration if needed (environment,
    +-- rte_atomic.h      # architecture, ..)
    `-- \*.h ...

-- lib                    # all compiled libraries are copied in this
    +-- librte_eal.a      # directory
    +-- librte_mempool.a
    `-- \*.a ...

-- app                    # All compiled applications are installed
+ --test                  # here. It includes the binary in elf format

Refer to Development Kit Root Makefile Help for details about make commands that can be used from the root of DPDK.

29.3.1. General Rules For DPDK Makefiles

In the DPDK, Makefiles always follow the same scheme:

1. Include $(RTE_SDK)/mk/rte.vars.mk at the beginning.

2. Define specific variables for RTE build system.

3. Include a specific $(RTE_SDK)/mk/rte.XYZ.mk, where XYZ can be app, lib, extapp, extlib, obj, gnuconfigure, and so on, depending on what kind of object you want to build. See Makefile Types below.

4. Include user-defined rules and variables.

   The following is a very simple example of an external application Makefile:

   include $(RTE_SDK)/mk/rte.vars.mk
   
   # binary name
   APP = helloworld
   
   # all source are stored in SRCS-y
   SRCS-y := main.c
   
   CFLAGS += -O3
   CFLAGS += $(WERROR_FLAGS)
   
   include $(RTE_SDK)/mk/rte.extapp.mk
   

29.3.2. Makefile Types

Depending on the .mk file which is included at the end of the user Makefile, the Makefile will have a different role. Note that it is not possible to build a library and an application in the same Makefile. For that, the user must create two separate Makefiles, possibly in two different directories.

In any case, the rte.vars.mk file must be included in the user Makefile as soon as possible.

29.3.3. Internally Generated Build Tools

app/dpdk-pmdinfogen

dpdk-pmdinfogen scans an object (.o) file for various well known symbol names. These well known symbol names are defined by various macros and used to export important information about hardware support and usage for pmd files. For instance the macro:

PMD_REGISTER_DRIVER(drv, name)

Creates the following symbol:

static char this_pmd_name0[] __attribute__((used)) = "<name>";

Which dpdk-pmdinfogen scans for. Using this information other relevant bits of data can be exported from the object file and used to produce a hardware support description, that dpdk-pmdinfogen then encodes into a json formatted string in the following format:

static char <name_pmd_string>="PMD_INFO_STRING=\"{'name' : '<name>', ...}\"";

These strings can then be searched for by external tools to determine the hardware support of a given library or application.

29.3.4. Useful Variables Provided by the Build System

• RTE_SDK: The absolute path to the DPDK sources. When compiling the development kit, this variable is automatically set by the framework. It has to be defined by the user as an environment variable if compiling an external application.
• RTE_SRCDIR: The path to the root of the sources. When compiling the development kit, RTE_SRCDIR = RTE_SDK. When compiling an external application, the variable points to the root of external application sources.
• RTE_OUTPUT: The path to which output files are written. Typically, it is $(RTE_SRCDIR)/build, but it can be overridden by the O= option in the make command line.
• RTE_TARGET: A string identifying the target for which we are building. The format is arch-machine-execenv-toolchain. When compiling the SDK, the target is deduced by the build system from the configuration (.config). When building an external application, it must be specified by the user in the Makefile or as an environment variable.
• RTE_SDK_BIN: References $(RTE_SDK)/$(RTE_TARGET).
• RTE_ARCH: Defines the architecture (i686, x86_64). It is the same value as CONFIG_RTE_ARCH but without the double-quotes around the string.
• RTE_MACHINE: Defines the machine. It is the same value as CONFIG_RTE_MACHINE but without the double-quotes around the string.
• RTE_TOOLCHAIN: Defines the toolchain (gcc , icc). It is the same value as CONFIG_RTE_TOOLCHAIN but without the double-quotes around the string.
• RTE_EXEC_ENV: Defines the executive environment (linuxapp). It is the same value as CONFIG_RTE_EXEC_ENV but without the double-quotes around the string.
• RTE_KERNELDIR: This variable contains the absolute path to the kernel sources that will be used to compile the kernel modules. The kernel headers must be the same as the ones that will be used on the target machine (the machine that will run the application). By default, the variable is set to /lib/modules/$(shell uname -r)/build, which is correct when the target machine is also the build machine.
• RTE_DEVEL_BUILD: Stricter options (stop on warning). It defaults to y in a git tree.

29.3.5. Variables that Can be Set/Overridden in a Makefile Only

• VPATH: The path list that the build system will search for sources. By default, RTE_SRCDIR will be included in VPATH.
• CFLAGS: Flags to use for C compilation. The user should use += to append data in this variable.
• LDFLAGS: Flags to use for linking. The user should use += to append data in this variable.
• ASFLAGS: Flags to use for assembly. The user should use += to append data in this variable.
• CPPFLAGS: Flags to use to give flags to C preprocessor (only useful when assembling .S files). The user should use += to append data in this variable.
• LDLIBS: In an application, the list of libraries to link with (for example, -L /path/to/libfoo -lfoo ). The user should use += to append data in this variable.
• SRC-y: A list of source files (.c, .S, or .o if the source is a binary) in case of application, library or object Makefiles. The sources must be available from VPATH.
• INSTALL-y-$(INSTPATH): A list of files to be installed in $(INSTPATH). The files must be available from VPATH and will be copied in $(RTE_OUTPUT)/$(INSTPATH). Can be used in almost any RTE Makefile.
• SYMLINK-y-$(INSTPATH): A list of files to be installed in $(INSTPATH). The files must be available from VPATH and will be linked (symbolically) in $(RTE_OUTPUT)/$(INSTPATH). This variable can be used in almost any DPDK Makefile.
• PREBUILD: A list of prerequisite actions to be taken before building. The user should use += to append data in this variable.
• POSTBUILD: A list of actions to be taken after the main build. The user should use += to append data in this variable.
• PREINSTALL: A list of prerequisite actions to be taken before installing. The user should use += to append data in this variable.
• POSTINSTALL: A list of actions to be taken after installing. The user should use += to append data in this variable.
• PRECLEAN: A list of prerequisite actions to be taken before cleaning. The user should use += to append data in this variable.
• POSTCLEAN: A list of actions to be taken after cleaning. The user should use += to append data in this variable.
• DEPDIR-y: Only used in the development kit framework to specify if the build of the current directory depends on build of another one. This is needed to support parallel builds correctly.

29.3.6. Variables that can be Set/Overridden by the User on the Command Line Only

Some variables can be used to configure the build system behavior. They are documented in Development Kit Root Makefile Help and External Application/Library Makefile Help

• WERROR_CFLAGS: By default, this is set to a specific value that depends on the compiler. Users are encouraged to use this variable as follows:

  CFLAGS += $(WERROR_CFLAGS)

This avoids the use of different cases depending on the compiler (icc or gcc). Also, this variable can be overridden from the command line, which allows bypassing of the flags for testing purposes.

29.3.7. Variables that Can be Set/Overridden by the User in a Makefile or Command Line

• CFLAGS_my_file.o: Specific flags to add for C compilation of my_file.c.
• LDFLAGS_my_app: Specific flags to add when linking my_app.
• EXTRA_CFLAGS: The content of this variable is appended after CFLAGS when compiling.
• EXTRA_LDFLAGS: The content of this variable is appended after LDFLAGS when linking.
• EXTRA_LDLIBS: The content of this variable is appended after LDLIBS when linking.
• EXTRA_ASFLAGS: The content of this variable is appended after ASFLAGS when assembling.
• EXTRA_CPPFLAGS: The content of this variable is appended after CPPFLAGS when using a C preprocessor on assembly files.