21. DMA Device Library

The DMA library provides a DMA device framework for management and provisioning of hardware and software DMA poll mode drivers, defining generic API which support a number of different DMA operations.

21.1. Design Principles

The DMA framework provides a generic DMA device framework which supports both physical (hardware) and virtual (software) DMA devices, as well as a generic DMA API which allows DMA devices to be managed and configured, and supports DMA operations to be provisioned on DMA poll mode driver.

../_images/dmadev.svg

The above figure shows the model on which the DMA framework is built on:

  • The DMA controller could have multiple hardware DMA channels (aka. hardware DMA queues), each hardware DMA channel should be represented by a dmadev.

  • The dmadev could create multiple virtual DMA channels, each virtual DMA channel represents a different transfer context.

  • The DMA operation request must be submitted to the virtual DMA channel.

21.2. Device Management

21.2.1. Device Creation

Physical DMA controllers are discovered during the PCI probe/enumeration of the EAL function which is executed at DPDK initialization, this is based on their PCI BDF (bus/bridge, device, function). Specific physical DMA controllers, like other physical devices in DPDK can be listed using the EAL command line options.

The dmadevs are dynamically allocated by using the function rte_dma_pmd_allocate based on the number of hardware DMA channels.

21.2.2. Device Identification

Each DMA device, whether physical or virtual is uniquely designated by two identifiers:

  • A unique device index used to designate the DMA device in all functions exported by the DMA API.

  • A device name used to designate the DMA device in console messages, for administration or debugging purposes.

21.3. Device Features and Capabilities

DMA devices may support different feature sets. The rte_dma_info_get API can be used to get the device info and supported features.

Silent mode is a special device capability which does not require the application to invoke dequeue APIs.

21.3.1. Enqueue / Dequeue APIs

Enqueue APIs such as rte_dma_copy and rte_dma_fill can be used to enqueue operations to hardware. If an enqueue is successful, a ring_idx is returned. This ring_idx can be used by applications to track per operation metadata in an application-defined circular ring.

The rte_dma_submit API is used to issue doorbell to hardware. Alternatively the RTE_DMA_OP_FLAG_SUBMIT flag can be passed to the enqueue APIs to also issue the doorbell to hardware.

The following code demonstrates how to enqueue a burst of copies to the device and start the hardware processing of them:

struct rte_mbuf *srcs[DMA_BURST_SZ], *dsts[DMA_BURST_SZ];
unsigned int i;

for (i = 0; i < RTE_DIM(srcs); i++) {
   if (rte_dma_copy(dev_id, vchan, rte_pktmbuf_iova(srcs[i]),
         rte_pktmbuf_iova(dsts[i]), COPY_LEN, 0) < 0) {
      PRINT_ERR("Error with rte_dma_copy for buffer %u\n", i);
      return -1;
   }
}
rte_dma_submit(dev_id, vchan);

There are two dequeue APIs rte_dma_completed and rte_dma_completed_status, these are used to obtain the results of the enqueue requests. rte_dma_completed will return the number of successfully completed operations. rte_dma_completed_status will return the number of completed operations along with the status of each operation (filled into the status array passed by user). These two APIs can also return the last completed operation’s ring_idx which could help user track operations within their own application-defined rings.

21.3.2. Querying Device Statistics

The statistics from a dmadev device can be got via the statistics functions, i.e. rte_dma_stats_get(). The statistics returned for each device instance are:

  • submitted: The number of operations submitted to the device.

  • completed: The number of operations which have completed (successful and failed).

  • errors: The number of operations that completed with error.

The dmadev library has support for displaying DMA device information through the Telemetry interface. Telemetry commands that can be used are shown below.

  1. Get the list of available DMA devices by ID:

    --> /dmadev/list
{"/dmadev/list": [0, 1]}

  2. Get general information from a DMA device by passing the device id as a parameter:

    --> /dmadev/info,0
{"/dmadev/info": {"name": "0000:00:01.0", "nb_vchans": 1, "numa_node": 0, "max_vchans": 1, "max_desc": 4096,
"min_desc": 32, "max_sges": 0, "capabilities": {"mem2mem": 1, "mem2dev": 0, "dev2mem": 0, ...}}}

  3. Get the statistics for a particular DMA device and virtual DMA channel by passing the device id and vchan id as parameters (if a DMA device only has one virtual DMA channel you only need to pass the device id):

    --> /dmadev/stats,0,0
{"/dmadev/stats": {"submitted": 0, "completed": 0, "errors": 0}}

For more information on how to use the Telemetry interface, see the DPDK Telemetry User Guide.