26. Multi-process Sample Application

This chapter describes example multi-processing applications that are included in the DPDK.

26.1. Example Applications

26.1.1. Building the Sample Applications

The multi-process example applications are built the same way as other sample applications, and as documented in the DPDK Getting Started Guide.

To compile the sample application see Compiling the Sample Applications.

The applications are located in the multi_process sub-directory.

Note

If only a specific multi-process application needs to be built, the final make command can be run just in that application’s directory.

26.1.2. Basic Multi-process Example

The examples/simple_mp folder contains a basic example application that demonstrates how two DPDK processes can work together to share information using queues and memory pools.

26.1.2.1. Running the Application

To run the application, start simple_mp binary in one terminal, passing at least two cores in the coremask/corelist:

./<build_dir>/examples/dpdk-simple_mp -l 0-1 -n 4 --proc-type=primary

For the first DPDK process run, the proc-type flag can be omitted or set to auto, since all DPDK processes will default to being a primary instance, meaning they have control over the hugepage shared memory regions. The process should start successfully and display a command prompt as follows:

$ ./<build_dir>/examples/dpdk-simple_mp -l 0-1 -n 4 --proc-type=primary
EAL: coremask set to 3
EAL: Detected lcore 0 on socket 0
EAL: Detected lcore 1 on socket 0
EAL: Detected lcore 2 on socket 0
EAL: Detected lcore 3 on socket 0
...

EAL: Requesting 2 pages of size 1073741824
EAL: Requesting 768 pages of size 2097152
EAL: Ask a virtual area of 0x40000000 bytes
EAL: Virtual area found at 0x7ff200000000 (size = 0x40000000)
...

EAL: check module finished
EAL: Main core 0 is ready (tid=54e41820)
EAL: Core 1 is ready (tid=53b32700)

Starting core 1

simple_mp >

To run the secondary process to communicate with the primary process, again run the same binary setting at least two cores in the coremask/corelist:

./<build_dir>/examples/dpdk-simple_mp -l 2-3 -n 4 --proc-type=secondary

When running a secondary process such as above, the proc-type parameter can be specified as auto. Omitting the parameter will cause the process to try and start as a primary rather than secondary process.

The process starts up, displaying similar status messages to the primary instance as it initializes then prints a command prompt.

Once both processes are running, messages can be sent between them using the send command. At any stage, either process can be terminated using the quit command.

EAL: Main core 10 is ready (tid=b5f89820)             EAL: Main core 8 is ready (tid=864a3820)
EAL: Core 11 is ready (tid=84ffe700)                  EAL: Core 9 is ready (tid=85995700)
Starting core 11                                      Starting core 9
simple_mp > send hello_secondary                      simple_mp > core 9: Received 'hello_secondary'
simple_mp > core 11: Received 'hello_primary'         simple_mp > send hello_primary
simple_mp > quit                                      simple_mp > quit

Note

If the primary instance is terminated, the secondary instance must also be shut-down and restarted after the primary. This is necessary because the primary instance will clear and reset the shared memory regions on startup, invalidating the secondary process’s pointers. The secondary process can be stopped and restarted without affecting the primary process.

26.1.2.2. How the Application Works

This application uses two queues and a single memory pool created in the primary process. The secondary process then uses lookup functions to attach to these objects.

if (rte_eal_process_type() == RTE_PROC_PRIMARY){
	send_ring = rte_ring_create(_PRI_2_SEC, ring_size, rte_socket_id(), flags);
	recv_ring = rte_ring_create(_SEC_2_PRI, ring_size, rte_socket_id(), flags);
	message_pool = rte_mempool_create(_MSG_POOL, pool_size,
			STR_TOKEN_SIZE, pool_cache, priv_data_sz,
			NULL, NULL, NULL, NULL,
			rte_socket_id(), flags);
} else {
	recv_ring = rte_ring_lookup(_PRI_2_SEC);
	send_ring = rte_ring_lookup(_SEC_2_PRI);
	message_pool = rte_mempool_lookup(_MSG_POOL);
}

Note, however, that the named ring structure used as send_ring in the primary process is the recv_ring in the secondary process.

The application has two threads:

sender: Reads from stdin, converts them to messages, and enqueues them to the ring.
receiver: Dequeues any messages on the ring, prints them, then frees the buffer.

26.1.3. Symmetric Multi-process Example

The symmetric multi process example demonstrates how a set of processes can run in parallel, with each process performing the same set of packet- processing operations. The following diagram shows the data-flow through the application, using two processes.

../_images/sym_multi_proc_app.png — Fig. 26.1 Example Data Flow in a Symmetric Multi-process Application

Each process reads packets from each of the network ports in use. RSS distributes incoming packets on each port to different hardware Rx queues. Each process reads a different RX queue on each port and so does not contend with any other process for that queue access. Each process writes outgoing packets to a different Tx queue on each port.

26.1.3.1. Running the Application

The first instance of the symmetric_mp process is the primary instance, with the EAL arguments:

-p <portmask>: The portmask is a hexadecimal bitmask of what ports on the system are to be used. For example: -p 3 to use ports 0 and 1 only.
--num-procs <N>: The N is the total number of symmetric_mp instances that will be run side-by-side to perform packet processing. This parameter is used to configure the appropriate number of receive queues on each network port.
--proc-id <n>: Where n is a numeric value in the range 0 <= n < N (number of processes, specified above). This identifies which symmetric_mp instance is being run, so that each process can read a unique receive queue on each network port.

The secondary instance must be started with similar EAL parameters. Example:

# ./<build_dir>/examples/dpdk-symmetric_mp -l 1 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=0
# ./<build_dir>/examples/dpdk-symmetric_mp -l 2 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=1
# ./<build_dir>/examples/dpdk-symmetric_mp -l 3 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=2
# ./<build_dir>/examples/dpdk-symmetric_mp -l 4 -n 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=3

Note

In the above example, auto is used so the first instance becomes the primary process.

26.1.3.2. How the Application Works

The primary instance creates the memory pool and initializes the network ports.

if (num_ports & 1)
	rte_exit(EXIT_FAILURE, "Application must use an even number of ports\n");
for(i = 0; i < num_ports; i++){
	if(proc_type == RTE_PROC_PRIMARY)
		if (smp_port_init(ports[i], mp, (uint16_t)num_procs) < 0)
			rte_exit(EXIT_FAILURE, "Error initialising ports\n");
}

The secondary instance gets the port information and exported by the primary process. The memory pool is accessed by doing a lookup for it by name:

if (proc_type == RTE_PROC_SECONDARY)
    mp = rte_mempool_lookup(_SMP_MBUF_POOL);
else
    mp = rte_mempool_create(_SMP_MBUF_POOL, NB_MBUFS, MBUF_SIZE, ... )

The main loop of each process, both primary and secondary, is the same. Each process reads from each port using the queue corresponding to its proc-id parameter, and writes to the corresponding transmit queue on the output port.

26.1.4. Client-Server Multi-process Example

The example multi-process application demonstrates a client-server type multi-process design. A single server process receives a set of packets from the ports and distributes these packets using round-robin ordering to the client processes, Each client processes packets and does level-2 forwarding by sending each packet out on a different network port.

The following diagram shows the data-flow through the application, using two client processes.

../_images/client_svr_sym_multi_proc_app.png — Fig. 26.2 Example Data Flow in a Client-Server Symmetric Multi-process Application

26.1.4.1. Running the Application

The server process must be run as the primary process to set up all memory structures. In addition to the EAL parameters, the application-specific parameters are:

-p <portmask >, where portmask is a hexadecimal bitmask of what ports on the system are to be used. For example: -p 3 to use ports 0 and 1 only.
-n <num-clients>, where the num-clients parameter is the number of client processes that will process the packets received by the server application.

Note

In the server process, has a single thread using the lowest numbered lcore in the coremask/corelist, performs all packet I/O. If coremask/corelist parameter specifies with more than a single lcore bit set, an additional lcore will be used for a thread to print packet count periodically.

The server application stores configuration data in shared memory, including the network ports used. The only application parameter needed by a client process is its client instance ID. To run a server application on lcore 1 (with lcore 2 printing statistics) along with two client processes running on lcores 3 and 4, the commands are:

# ./<build_dir>/examples/dpdk-mp_server -l 1-2 -n 4 -- -p 3 -n 2
# ./<build_dir>/examples/dpdk-mp_client -l 3 -n 4 --proc-type=auto -- -n 0
# ./<build_dir>/examples/dpdk-mp_client -l 4 -n 4 --proc-type=auto -- -n 1

Note

If the server application dies and needs to be restarted, all client applications also need to be restarted, as there is no support in the server application for it to run as a secondary process. Any client processes that need restarting can be restarted without affecting the server process.

26.1.4.2. How the Application Works

The server (primary) process performs the initialization of network port and data structure and stores its port configuration data in a memory zone in hugepage shared memory. The client process does not need the portmask parameter passed in via the command line. The server process is the primary process, and the client processes are secondary processes.

The server operates by reading packets from each network port and distributing those packets to the client queues. The client reads from the ring and routes the packet to a different network port. The routing used is very simple: all packets received on the first NIC port are transmitted back out on the second port and vice versa. Similarly, packets are routed between the 3rd and 4th network ports and so on.