35. Event Device Library

35.1.1. Event Metadata

The rte_event structure contains the following metadata fields, which the application fills in to have the event scheduled as required:

• flow_id - The targeted flow identifier for the enq/deq operation.
• event_type - The source of this event, eg RTE_EVENT_TYPE_ETHDEV or CPU.
• sub_event_type - Distinguishes events inside the application, that have the same event_type (see above)
• op - This field takes one of the RTE_EVENT_OP_* values, and tells the eventdev about the status of the event - valid values are NEW, FORWARD or RELEASE.
• sched_type - Represents the type of scheduling that should be performed on this event, valid values are the RTE_SCHED_TYPE_ORDERED, ATOMIC and PARALLEL.
• queue_id - The identifier for the event queue that the event is sent to.
• priority - The priority of this event, see RTE_EVENT_DEV_PRIORITY.

35.1.2. Event Payload

The rte_event struct contains a union for payload, allowing flexibility in what the actual event being scheduled is. The payload is a union of the following:

• uint64_t u64
• void *event_ptr
• struct rte_mbuf *mbuf

These three items in a union occupy the same 64 bits at the end of the rte_event structure. The application can utilize the 64 bits directly by accessing the u64 variable, while the event_ptr and mbuf are provided as convenience variables. For example the mbuf pointer in the union can used to schedule a DPDK packet.

35.1.3. Queues

An event queue is a queue containing events that are scheduled by the event device. An event queue contains events of different flows associated with scheduling types, such as atomic, ordered, or parallel.

35.1.4. Ports

Ports are the points of contact between worker cores and the eventdev. The general use-case will see one CPU core using one port to enqueue and dequeue events from an eventdev. Ports are linked to queues in order to retrieve events from those queues (more details in Linking Queues and Ports below).

35.2.1. Init and Config

The eventdev library uses vdev options to add devices to the DPDK application. The --vdev EAL option allows adding eventdev instances to your DPDK application, using the name of the eventdev PMD as an argument.

For example, to create an instance of the software eventdev scheduler, the following vdev arguments should be provided to the application EAL command line:

./dpdk_application --vdev="event_sw0"

In the following code, we configure eventdev instance with 3 queues and 6 ports as follows. The 3 queues consist of 2 Atomic and 1 Single-Link, while the 6 ports consist of 4 workers, 1 RX and 1 TX.

const struct rte_event_dev_config config = {
        .nb_event_queues = 3,
        .nb_event_ports = 6,
        .nb_events_limit  = 4096,
        .nb_event_queue_flows = 1024,
        .nb_event_port_dequeue_depth = 128,
        .nb_event_port_enqueue_depth = 128,
};
int err = rte_event_dev_configure(dev_id, &config);

The remainder of this walk-through assumes that dev_id is 0.

35.2.2. Setting up Queues

Once the eventdev itself is configured, the next step is to configure queues. This is done by setting the appropriate values in a queue_conf structure, and calling the setup function. Repeat this step for each queue, starting from 0 and ending at nb_event_queues - 1 from the event_dev config above.

struct rte_event_queue_conf atomic_conf = {
        .schedule_type = RTE_SCHED_TYPE_ATOMIC,
        .priority = RTE_EVENT_DEV_PRIORITY_NORMAL,
        .nb_atomic_flows = 1024,
        .nb_atomic_order_sequences = 1024,
};
struct rte_event_queue_conf single_link_conf = {
        .event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK,
};
int dev_id = 0;
int atomic_q_1 = 0;
int atomic_q_2 = 1;
int single_link_q = 2;
int err = rte_event_queue_setup(dev_id, atomic_q_1, &atomic_conf);
int err = rte_event_queue_setup(dev_id, atomic_q_2, &atomic_conf);
int err = rte_event_queue_setup(dev_id, single_link_q, &single_link_conf);

As shown above, queue IDs are as follows:

• id 0, atomic queue #1
• id 1, atomic queue #2
• id 2, single-link queue

These queues are used for the remainder of this walk-through.

35.2.3. Setting up Ports

Once queues are set up successfully, create the ports as required.

struct rte_event_port_conf rx_conf = {
        .dequeue_depth = 128,
        .enqueue_depth = 128,
        .new_event_threshold = 1024,
};
struct rte_event_port_conf worker_conf = {
        .dequeue_depth = 16,
        .enqueue_depth = 64,
        .new_event_threshold = 4096,
};
struct rte_event_port_conf tx_conf = {
        .dequeue_depth = 128,
        .enqueue_depth = 128,
        .new_event_threshold = 4096,
};
int dev_id = 0;
int rx_port_id = 0;
int err = rte_event_port_setup(dev_id, rx_port_id, &rx_conf);

for(int worker_port_id = 1; worker_port_id <= 4; worker_port_id++) {
        int err = rte_event_port_setup(dev_id, worker_port_id, &worker_conf);
}

int tx_port_id = 5;
int err = rte_event_port_setup(dev_id, tx_port_id, &tx_conf);

As shown above:

• port 0: RX core
• ports 1,2,3,4: Workers
• port 5: TX core

These ports are used for the remainder of this walk-through.

35.2.4. Linking Queues and Ports

The final step is to “wire up” the ports to the queues. After this, the eventdev is capable of scheduling events, and when cores request work to do, the correct events are provided to that core. Note that the RX core takes input from eg: a NIC so it is not linked to any eventdev queues.

Linking all workers to atomic queues, and the TX core to the single-link queue can be achieved like this:

uint8_t rx_port_id = 0;
uint8_t tx_port_id = 5;
uint8_t atomic_qs[] = {0, 1};
uint8_t single_link_q = 2;
uin8t_t priority = RTE_EVENT_DEV_PRIORITY_NORMAL;

for(int worker_port_id = 1; worker_port_id <= 4; worker_port_id++) {
        int links_made = rte_event_port_link(dev_id, worker_port_id, atomic_qs, NULL, 2);
}
int links_made = rte_event_port_link(dev_id, tx_port_id, &single_link_q, &priority, 1);

35.2.5. Starting the EventDev

A single function call tells the eventdev instance to start processing events. Note that all queues must be linked to for the instance to start, as if any queue is not linked to, enqueuing to that queue will cause the application to backpressure and eventually stall due to no space in the eventdev.

int err = rte_event_dev_start(dev_id);

35.2.6. Ingress of New Events

Now that the eventdev is set up, and ready to receive events, the RX core must enqueue some events into the system for it to schedule. The events to be scheduled are ordinary DPDK packets, received from an eth_rx_burst() as normal. The following code shows how those packets can be enqueued into the eventdev:

const uint16_t nb_rx = rte_eth_rx_burst(eth_port, 0, mbufs, BATCH_SIZE);

for (i = 0; i < nb_rx; i++) {
        ev[i].flow_id = mbufs[i]->hash.rss;
        ev[i].op = RTE_EVENT_OP_NEW;
        ev[i].sched_type = RTE_SCHED_TYPE_ATOMIC;
        ev[i].queue_id = atomic_q_1;
        ev[i].event_type = RTE_EVENT_TYPE_ETHDEV;
        ev[i].sub_event_type = 0;
        ev[i].priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
        ev[i].mbuf = mbufs[i];
}

const int nb_tx = rte_event_enqueue_burst(dev_id, rx_port_id, ev, nb_rx);
if (nb_tx != nb_rx) {
        for(i = nb_tx; i < nb_rx; i++)
                rte_pktmbuf_free(mbufs[i]);
}

35.2.7. Forwarding of Events

Now that the RX core has injected events, there is work to be done by the workers. Note that each worker will dequeue as many events as it can in a burst, process each one individually, and then burst the packets back into the eventdev.

The worker can lookup the events source from event.queue_id, which should indicate to the worker what workload needs to be performed on the event. Once done, the worker can update the event.queue_id to a new value, to send the event to the next stage in the pipeline.

int timeout = 0;
struct rte_event events[BATCH_SIZE];
uint16_t nb_rx = rte_event_dequeue_burst(dev_id, worker_port_id, events, BATCH_SIZE, timeout);

for (i = 0; i < nb_rx; i++) {
        /* process mbuf using events[i].queue_id as pipeline stage */
        struct rte_mbuf *mbuf = events[i].mbuf;
        /* Send event to next stage in pipeline */
        events[i].queue_id++;
}

uint16_t nb_tx = rte_event_enqueue_burst(dev_id, worker_port_id, events, nb_rx);

35.2.8. Egress of Events

Finally, when the packet is ready for egress or needs to be dropped, we need to inform the eventdev that the packet is no longer being handled by the application. This can be done by calling dequeue() or dequeue_burst(), which indicates that the previous burst of packets is no longer in use by the application.

An event driven worker thread has following typical workflow on fastpath:

while (1) {
        rte_event_dequeue_burst(...);
        (event processing)
        rte_event_enqueue_burst(...);
}