9. Generic flow API (rte_flow)

9.2.1. Description

A flow rule is the combination of attributes with a matching pattern and a list of actions. Flow rules form the basis of this API.

Flow rules can have several distinct actions (such as counting, encapsulating, decapsulating before redirecting packets to a particular queue, etc.), instead of relying on several rules to achieve this and having applications deal with hardware implementation details regarding their order.

Support for different priority levels on a rule basis is provided, for example in order to force a more specific rule to come before a more generic one for packets matched by both. However hardware support for more than a single priority level cannot be guaranteed. When supported, the number of available priority levels is usually low, which is why they can also be implemented in software by PMDs (e.g. missing priority levels may be emulated by reordering rules).

In order to remain as hardware-agnostic as possible, by default all rules are considered to have the same priority, which means that the order between overlapping rules (when a packet is matched by several filters) is undefined.

PMDs may refuse to create overlapping rules at a given priority level when they can be detected (e.g. if a pattern matches an existing filter).

Thus predictable results for a given priority level can only be achieved with non-overlapping rules, using perfect matching on all protocol layers.

Flow rules can also be grouped, the flow rule priority is specific to the group they belong to. All flow rules in a given group are thus processed either before or after another group.

Support for multiple actions per rule may be implemented internally on top of non-default hardware priorities, as a result both features may not be simultaneously available to applications.

Considering that allowed pattern/actions combinations cannot be known in advance and would result in an impractically large number of capabilities to expose, a method is provided to validate a given rule from the current device configuration state.

This enables applications to check if the rule types they need is supported at initialization time, before starting their data path. This method can be used anytime, its only requirement being that the resources needed by a rule should exist (e.g. a target RX queue should be configured first).

Each defined rule is associated with an opaque handle managed by the PMD, applications are responsible for keeping it. These can be used for queries and rules management, such as retrieving counters or other data and destroying them.

To avoid resource leaks on the PMD side, handles must be explicitly destroyed by the application before releasing associated resources such as queues and ports.

The following sections cover:

• Attributes (represented by struct rte_flow_attr): properties of a flow rule such as its direction (ingress or egress) and priority.
• Pattern item (represented by struct rte_flow_item): part of a matching pattern that either matches specific packet data or traffic properties. It can also describe properties of the pattern itself, such as inverted matching.
• Matching pattern: traffic properties to look for, a combination of any number of items.
• Actions (represented by struct rte_flow_action): operations to perform whenever a packet is matched by a pattern.

9.2.2. Attributes

9.2.3. Pattern item

Pattern items fall in two categories:

• Matching protocol headers and packet data (ANY, RAW, ETH, VLAN, IPV4, IPV6, ICMP, UDP, TCP, SCTP, VXLAN, MPLS, GRE, ESP and so on), usually associated with a specification structure.
• Matching meta-data or affecting pattern processing (END, VOID, INVERT, PF, VF, PORT and so on), often without a specification structure.

Item specification structures are used to match specific values among protocol fields (or item properties). Documentation describes for each item whether they are associated with one and their type name if so.

Up to three structures of the same type can be set for a given item:

• spec: values to match (e.g. a given IPv4 address).
• last: upper bound for an inclusive range with corresponding fields in spec.
• mask: bit-mask applied to both spec and last whose purpose is to distinguish the values to take into account and/or partially mask them out (e.g. in order to match an IPv4 address prefix).

Usage restrictions and expected behavior:

• Setting either mask or last without spec is an error.
• Field values in last which are either 0 or equal to the corresponding values in spec are ignored; they do not generate a range. Nonzero values lower than those in spec are not supported.
• Setting spec and optionally last without mask causes the PMD to use the default mask defined for that item (defined as rte_flow_item_{name}_mask constants).
• Not setting any of them (assuming item type allows it) is equivalent to providing an empty (zeroed) mask for broad (nonspecific) matching.
• mask is a simple bit-mask applied before interpreting the contents of spec and last, which may yield unexpected results if not used carefully. For example, if for an IPv4 address field, spec provides 10.1.2.3, last provides 10.3.4.5 and mask provides 255.255.0.0, the effective range becomes 10.1.0.0 to 10.3.255.255.

Example of an item specification matching an Ethernet header:

Non-masked bits stand for any value (shown as ? below), Ethernet headers with the following properties are thus matched:

• src: ??:01:02:03:??
• dst: ??:??:??:??:01
• type: 0x????

9.2.4. Matching pattern

A pattern is formed by stacking items starting from the lowest protocol layer to match. This stacking restriction does not apply to meta items which can be placed anywhere in the stack without affecting the meaning of the resulting pattern.

Patterns are terminated by END items.

Examples:

The above example shows how meta items do not affect packet data matching items, as long as those remain stacked properly. The resulting matching pattern is identical to “TCPv4 as L4”.

If supported by the PMD, omitting one or several protocol layers at the bottom of the stack as in the above example (missing an Ethernet specification) enables looking up anywhere in packets.

It is unspecified whether the payload of supported encapsulations (e.g. VXLAN payload) is matched by such a pattern, which may apply to inner, outer or both packets.

The above pattern is invalid due to a missing L3 specification between L2 (Ethernet) and L4 (UDP). Doing so is only allowed at the bottom and at the top of the stack.

9.2.5. Meta item types

They match meta-data or affect pattern processing instead of matching packet data directly, most of them do not need a specification structure. This particularity allows them to be specified anywhere in the stack without causing any side effect.

9.2.6. Data matching item types

Most of these are basically protocol header definitions with associated bit-masks. They must be specified (stacked) from lowest to highest protocol layer to form a matching pattern.

The following list is not exhaustive, new protocols will be added in the future.

0                     >= 10 B           == 20 B
|                  |<--------->|     |<--------->|
|                  |           |     |           |
|-----|------|-----|-----|-----|-----|-----------|-----|------|
| ETH | IPv4 | UDP | ... | baz | foo | ......... | bar | .... |
|-----|------|-----|-----|-----|-----|-----------|-----|------|
                         |                             |
                         |<--------------------------->|
                                     == 29 B

9.2.7. Actions

Each possible action is represented by a type. Some have associated configuration structures. Several actions combined in a list can be assigned to a flow rule. That list is not ordered.

They fall in three categories:

• Terminating actions (such as QUEUE, DROP, RSS, PF, VF) that prevent processing matched packets by subsequent flow rules, unless overridden with PASSTHRU.
• Non-terminating actions (PASSTHRU, DUP) that leave matched packets up for additional processing by subsequent flow rules.
• Other non-terminating meta actions that do not affect the fate of packets (END, VOID, MARK, FLAG, COUNT, SECURITY).

When several actions are combined in a flow rule, they should all have different types (e.g. dropping a packet twice is not possible).

Only the last action of a given type is taken into account. PMDs still perform error checking on the entire list.

Like matching patterns, action lists are terminated by END items.

Note that PASSTHRU is the only action able to override a terminating rule.

Example of action that redirects packets to queue index 10:

Action lists examples, their order is not significant, applications must consider all actions to be performed simultaneously:

In the above example, considering both actions are performed simultaneously, the end result is that only QUEUE has any effect.

As previously described, only the last action of a given type found in the list is taken into account. The above example also shows that VOID is ignored.

9.2.8. Action types

Common action types are described in this section. Like pattern item types, this list is not exhaustive as new actions will be added in the future.

9.2.9. Negative types

All specified pattern items (enum rte_flow_item_type) and actions (enum rte_flow_action_type) use positive identifiers.

The negative space is reserved for dynamic types generated by PMDs during run-time. PMDs may encounter them as a result but must not accept negative identifiers they are not aware of.

A method to generate them remains to be defined.

9.2.10. Planned types

Pattern item types will be added as new protocols are implemented.

Variable headers support through dedicated pattern items, for example in order to match specific IPv4 options and IPv6 extension headers would be stacked after IPv4/IPv6 items.

Other action types are planned but are not defined yet. These include the ability to alter packet data in several ways, such as performing encapsulation/decapsulation of tunnel headers.

9.3.1. Validation

Given that expressing a definite set of device capabilities is not practical, a dedicated function is provided to check if a flow rule is supported and can be created.

int
rte_flow_validate(uint16_t port_id,
                  const struct rte_flow_attr *attr,
                  const struct rte_flow_item pattern[],
                  const struct rte_flow_action actions[],
                  struct rte_flow_error *error);

The flow rule is validated for correctness and whether it could be accepted by the device given sufficient resources. The rule is checked against the current device mode and queue configuration. The flow rule may also optionally be validated against existing flow rules and device resources. This function has no effect on the target device.

The returned value is guaranteed to remain valid only as long as no successful calls to rte_flow_create() or rte_flow_destroy() are made in the meantime and no device parameter affecting flow rules in any way are modified, due to possible collisions or resource limitations (although in such cases EINVAL should not be returned).

Arguments:

• port_id: port identifier of Ethernet device.
• attr: flow rule attributes.
• pattern: pattern specification (list terminated by the END pattern item).
• actions: associated actions (list terminated by the END action).
• error: perform verbose error reporting if not NULL. PMDs initialize this structure in case of error only.

Return values:

• 0 if flow rule is valid and can be created. A negative errno value otherwise (rte_errno is also set), the following errors are defined.
• -ENOSYS: underlying device does not support this functionality.
• -EINVAL: unknown or invalid rule specification.
• -ENOTSUP: valid but unsupported rule specification (e.g. partial bit-masks are unsupported).
• EEXIST: collision with an existing rule. Only returned if device supports flow rule collision checking and there was a flow rule collision. Not receiving this return code is no guarantee that creating the rule will not fail due to a collision.
• ENOMEM: not enough memory to execute the function, or if the device supports resource validation, resource limitation on the device.
• -EBUSY: action cannot be performed due to busy device resources, may succeed if the affected queues or even the entire port are in a stopped state (see rte_eth_dev_rx_queue_stop() and rte_eth_dev_stop()).

9.3.2. Creation

Creating a flow rule is similar to validating one, except the rule is actually created and a handle returned.

struct rte_flow *
rte_flow_create(uint16_t port_id,
                const struct rte_flow_attr *attr,
                const struct rte_flow_item pattern[],
                const struct rte_flow_action *actions[],
                struct rte_flow_error *error);

Arguments:

• port_id: port identifier of Ethernet device.
• attr: flow rule attributes.
• pattern: pattern specification (list terminated by the END pattern item).
• actions: associated actions (list terminated by the END action).
• error: perform verbose error reporting if not NULL. PMDs initialize this structure in case of error only.

Return values:

A valid handle in case of success, NULL otherwise and rte_errno is set to the positive version of one of the error codes defined for rte_flow_validate().

9.3.3. Destruction

Flow rules destruction is not automatic, and a queue or a port should not be released if any are still attached to them. Applications must take care of performing this step before releasing resources.

int
rte_flow_destroy(uint16_t port_id,
                 struct rte_flow *flow,
                 struct rte_flow_error *error);

Failure to destroy a flow rule handle may occur when other flow rules depend on it, and destroying it would result in an inconsistent state.

This function is only guaranteed to succeed if handles are destroyed in reverse order of their creation.

Arguments:

• port_id: port identifier of Ethernet device.
• flow: flow rule handle to destroy.
• error: perform verbose error reporting if not NULL. PMDs initialize this structure in case of error only.

Return values:

• 0 on success, a negative errno value otherwise and rte_errno is set.

9.3.4. Flush

Convenience function to destroy all flow rule handles associated with a port. They are released as with successive calls to rte_flow_destroy().

int
rte_flow_flush(uint16_t port_id,
               struct rte_flow_error *error);

In the unlikely event of failure, handles are still considered destroyed and no longer valid but the port must be assumed to be in an inconsistent state.

Arguments:

• port_id: port identifier of Ethernet device.
• error: perform verbose error reporting if not NULL. PMDs initialize this structure in case of error only.

Return values:

• 0 on success, a negative errno value otherwise and rte_errno is set.

9.3.5. Query

Query an existing flow rule.

This function allows retrieving flow-specific data such as counters. Data is gathered by special actions which must be present in the flow rule definition.

int
rte_flow_query(uint16_t port_id,
               struct rte_flow *flow,
               enum rte_flow_action_type action,
               void *data,
               struct rte_flow_error *error);

Arguments:

• port_id: port identifier of Ethernet device.
• flow: flow rule handle to query.
• action: action type to query.
• data: pointer to storage for the associated query data type.
• error: perform verbose error reporting if not NULL. PMDs initialize this structure in case of error only.

Return values:

• 0 on success, a negative errno value otherwise and rte_errno is set.

9.6.1. Error initializer

static inline int
rte_flow_error_set(struct rte_flow_error *error,
                   int code,
                   enum rte_flow_error_type type,
                   const void *cause,
                   const char *message);

This function initializes error (if non-NULL) with the provided parameters and sets rte_errno to code. A negative error code is then returned.

9.9.1. Global bit-masks

Each flow rule comes with its own, per-layer bit-masks, while hardware may support only a single, device-wide bit-mask for a given layer type, so that two IPv4 rules cannot use different bit-masks.

The expected behavior in this case is that PMDs automatically configure global bit-masks according to the needs of the first flow rule created.

Subsequent rules are allowed only if their bit-masks match those, the EEXIST error code should be returned otherwise.

9.9.2. Unsupported layer types

Many protocols can be simulated by crafting patterns with the Item: RAW type.

PMDs can rely on this capability to simulate support for protocols with headers not directly recognized by hardware.

9.9.3. ANY pattern item

This pattern item stands for anything, which can be difficult to translate to something hardware would understand, particularly if followed by more specific types.

Consider the following pattern:

Knowing that TCP does not make sense with something other than IPv4 and IPv6 as L3, such a pattern may be translated to two flow rules instead:

Note that as soon as a ANY rule covers several layers, this approach may yield a large number of hidden flow rules. It is thus suggested to only support the most common scenarios (anything as L2 and/or L3).

9.9.4. Unsupported actions

• When combined with Action: QUEUE, packet counting (Action: COUNT) and tagging (Action: MARK or Action: FLAG) may be implemented in software as long as the target queue is used by a single rule.
• A rule specifying both Action: DUP + Action: QUEUE may be translated to two hidden rules combining Action: QUEUE and Action: PASSTHRU.
• When a single target queue is provided, Action: RSS can also be implemented through Action: QUEUE.

9.9.5. Flow rules priority

While it would naturally make sense, flow rules cannot be assumed to be processed by hardware in the same order as their creation for several reasons:

• They may be managed internally as a tree or a hash table instead of a list.
• Removing a flow rule before adding another one can either put the new rule at the end of the list or reuse a freed entry.
• Duplication may occur when packets are matched by several rules.

For overlapping rules (particularly in order to use Action: PASSTHRU) predictable behavior is only guaranteed by using different priority levels.

Priority levels are not necessarily implemented in hardware, or may be severely limited (e.g. a single priority bit).

For these reasons, priority levels may be implemented purely in software by PMDs.

• For devices expecting flow rules to be added in the correct order, PMDs may destroy and re-create existing rules after adding a new one with a higher priority.
• A configurable number of dummy or empty rules can be created at initialization time to save high priority slots for later.
• In order to save priority levels, PMDs may evaluate whether rules are likely to collide and adjust their priority accordingly.

9.11.1. MACVLAN to ETH → VF, PF

MACVLAN can be translated to a basic Item: ETH flow rule with a terminating Action: VF or Action: PF.

9.11.2. ETHERTYPE to ETH → QUEUE, DROP

ETHERTYPE is basically an Item: ETH flow rule with a terminating Action: QUEUE or Action: DROP.

9.11.3. FLEXIBLE to RAW → QUEUE

FLEXIBLE can be translated to one Item: RAW pattern with a terminating Action: QUEUE and a defined priority level.

9.11.4. SYN to TCP → QUEUE

SYN is a Item: TCP rule with only the syn bit enabled and masked, and a terminating Action: QUEUE.

Priority level can be set to simulate the high priority bit.

9.11.5. NTUPLE to IPV4, TCP, UDP → QUEUE

NTUPLE is similar to specifying an empty L2, Item: IPV4 as L3 with Item: TCP or Item: UDP as L4 and a terminating Action: QUEUE.

A priority level can be specified as well.

9.11.6. TUNNEL to ETH, IPV4, IPV6, VXLAN (or other) → QUEUE

TUNNEL matches common IPv4 and IPv6 L3/L4-based tunnel types.

In the following table, Item: ANY is used to cover the optional L4.

9.11.7. FDIR to most item types → QUEUE, DROP, PASSTHRU

FDIR is more complex than any other type, there are several methods to emulate its functionality. It is summarized for the most part in the table below.

A few features are intentionally not supported:

• The ability to configure the matching input set and masks for the entire device, PMDs should take care of it automatically according to the requested flow rules.

  For example if a device supports only one bit-mask per protocol type, source/address IPv4 bit-masks can be made immutable by the first created rule. Subsequent IPv4 or TCPv4 rules can only be created if they are compatible.

  Note that only protocol bit-masks affected by existing flow rules are immutable, others can be changed later. They become mutable again after the related flow rules are destroyed.

• Returning four or eight bytes of matched data when using flex bytes filtering. Although a specific action could implement it, it conflicts with the much more useful 32 bits tagging on devices that support it.

• Side effects on RSS processing of the entire device. Flow rules that conflict with the current device configuration should not be allowed. Similarly, device configuration should not be allowed when it affects existing flow rules.

• Device modes of operation. “none” is unsupported since filtering cannot be disabled as long as a flow rule is present.

• “MAC VLAN” or “tunnel” perfect matching modes should be automatically set according to the created flow rules.

• Signature mode of operation is not defined but could be handled through “FUZZY” item.

9.11.8. HASH

There is no counterpart to this filter type because it translates to a global device setting instead of a pattern item. Device settings are automatically set according to the created flow rules.

9.11.9. L2_TUNNEL to VOID → VXLAN (or others)

All packets are matched. This type alters incoming packets to encapsulate them in a chosen tunnel type, optionally redirect them to a VF as well.

The destination pool for tag based forwarding can be emulated with other flow rules using Action: DUP.