In addition to Poll Mode Drivers (PMDs) for physical and virtual hardware, DPDK also includes a pure-software library that allows physical PMD’s to be bonded together to create a single logical PMD.
Fig. 9.1 Bonded PMDs
The Link Bonding PMD library(librte_pmd_bond) supports bonding of groups of
rte_eth_dev ports of the same speed and duplex to provide
similar the capabilities to that found in Linux bonding driver to allow the
aggregation of multiple (slave) NICs into a single logical interface between a
server and a switch. The new bonded PMD will then process these interfaces
based on the mode of operation specified to provide support for features such
as redundant links, fault tolerance and/or load balancing.
The librte_pmd_bond library exports a C API which provides an API for the creation of bonded devices as well as the configuration and management of the bonded device and its slave devices.
Note
The Link Bonding PMD Library is enabled by default in the build
configuration files, the library can be disabled by setting
CONFIG_RTE_LIBRTE_PMD_BOND=n and recompiling the DPDK.
Currently the Link Bonding PMD library supports 4 modes of operation:
Fig. 9.2 Round-Robin (Mode 0)
This mode provides load balancing and fault tolerance by transmission of packets in sequential order from the first available slave device through the last. Packets are bulk dequeued from devices then serviced in a round-robin manner. This mode does not guarantee in order reception of packets and down stream should be able to handle out of order packets.
Fig. 9.3 Active Backup (Mode 1)
In this mode only one slave in the bond is active at any time, a different slave becomes active if, and only if, the primary active slave fails, thereby providing fault tolerance to slave failure. The single logical bonded interface’s MAC address is externally visible on only one NIC (port) to avoid confusing the network switch.
Fig. 9.4 Balance XOR (Mode 2)
This mode provides transmit load balancing (based on the selected transmission policy) and fault tolerance. The default policy (layer2) uses a simple calculation based on the packet flow source and destination MAC addresses as well as the number of active slaves available to the bonded device to classify the packet to a specific slave to transmit on. Alternate transmission policies supported are layer 2+3, this takes the IP source and destination addresses into the calculation of the transmit slave port and the final supported policy is layer 3+4, this uses IP source and destination addresses as well as the TCP/UDP source and destination port.
Note
The coloring differences of the packets are used to identify different flow classification calculated by the selected transmit policy
Fig. 9.5 Broadcast (Mode 3)
This mode provides fault tolerance by transmission of packets on all slave ports.
Fig. 9.6 Link Aggregation 802.3AD (Mode 4)
This mode provides dynamic link aggregation according to the 802.3ad specification. It negotiates and monitors aggregation groups that share the same speed and duplex settings using the selected balance transmit policy for balancing outgoing traffic.
DPDK implementation of this mode provide some additional requirements of the application.
- It needs to call
rte_eth_tx_burstandrte_eth_rx_burstwith intervals period of less than 100ms.- Calls to
rte_eth_tx_burstmust have a buffer size of at least 2xN, where N is the number of slaves. This is a space required for LACP frames. Additionally LACP packets are included in the statistics, but they are not returned to the application.
Fig. 9.7 Transmit Load Balancing (Mode 5)
This mode provides an adaptive transmit load balancing. It dynamically changes the transmitting slave, according to the computed load. Statistics are collected in 100ms intervals and scheduled every 10ms.
The librte_pmd_bond bonded device are compatible with the Ethernet device API exported by the Ethernet PMDs described in the DPDK API Reference.
The Link Bonding Library supports the creation of bonded devices at application
startup time during EAL initialization using the --vdev option as well as
programmatically via the C API rte_eth_bond_create function.
Bonded devices support the dynamical addition and removal of slave devices using
the rte_eth_bond_slave_add / rte_eth_bond_slave_remove APIs.
After a slave device is added to a bonded device slave is stopped using
rte_eth_dev_stop and then reconfigured using rte_eth_dev_configure
the RX and TX queues are also reconfigured using rte_eth_tx_queue_setup /
rte_eth_rx_queue_setup with the parameters use to configure the bonding
device.
Link bonding devices support the registration of a link status change callback,
using the rte_eth_dev_callback_register API, this will be called when the
status of the bonding device changes. For example in the case of a bonding
device which has 3 slaves, the link status will change to up when one slave
becomes active or change to down when all slaves become inactive. There is no
callback notification when a single slave changes state and the previous
conditions are not met. If a user wishes to monitor individual slaves then they
must register callbacks with that slave directly.
The link bonding library also supports devices which do not implement link
status change interrupts, this is achieve by polling the devices link status at
a defined period which is set using the rte_eth_bond_link_monitoring_set
API, the default polling interval is 10ms. When a device is added as a slave to
a bonding device it is determined using the RTE_PCI_DRV_INTR_LSC flag
whether the device supports interrupts or whether the link status should be
monitored by polling it.
The current implementation only supports devices that support the same speed and duplex to be added as a slaves to the same bonded device. The bonded device inherits these attributes from the first active slave added to the bonded device and then all further slaves added to the bonded device must support these parameters.
A bonding device must have a minimum of one slave before the bonding device itself can be started.
Like all other PMD, all functions exported by a PMD are lock-free functions that are assumed not to be invoked in parallel on different logical cores to work on the same target object.
It should also be noted that the PMD receive function should not be invoked directly on a slave devices after they have been to a bonded device since packets read directly from the slave device will no longer be available to the bonded device to read.
Link bonding devices are created using the rte_eth_bond_create API
which requires a unique device name, the bonding mode,
and the socket Id to allocate the bonding device’s resources on.
The other configurable parameters for a bonded device are its slave devices,
its primary slave, a user defined MAC address and transmission policy to use if
the device is in balance XOR mode.
Bonding devices support up to a maximum of RTE_MAX_ETHPORTS slave devices
of the same speed and duplex. Ethernet devices can be added as a slave to a
maximum of one bonded device. Slave devices are reconfigured with the
configuration of the bonded device on being added to a bonded device.
The bonded also guarantees to return the MAC address of the slave device to its original value of removal of a slave from it.
The primary slave is used to define the default port to use when a bonded device is in active backup mode. A different port will only be used if, and only if, the current primary port goes down. If the user does not specify a primary port it will default to being the first port added to the bonded device.
The bonded device can be configured with a user specified MAC address, this address will be inherited by the some/all slave devices depending on the operating mode. If the device is in active backup mode then only the primary device will have the user specified MAC, all other slaves will retain their original MAC address. In mode 0, 2, 3, 4 all slaves devices are configure with the bonded devices MAC address.
If a user defined MAC address is not defined then the bonded device will default to using the primary slaves MAC address.
There are 3 supported transmission policies for bonded device running in Balance XOR mode. Layer 2, Layer 2+3, Layer 3+4.
All these policies support 802.1Q VLAN Ethernet packets, as well as IPv4, IPv6 and UDP protocols for load balancing.
The librte_pmd_bond library support two modes of device creation, the libraries export full C API or using the EAL command line to statically configure link bonding devices at application startup. Using the EAL option it is possible to use link bonding functionality transparently without specific knowledge of the libraries API, this can be used, for example, to add bonding functionality, such as active backup, to an existing application which has no knowledge of the link bonding C API.
Using the librte_pmd_bond libraries API it is possible to dynamically create
and manage link bonding device from within any application. Link bonding
device are created using the rte_eth_bond_create API which requires a
unique device name, the link bonding mode to initial the device in and finally
the socket Id which to allocate the devices resources onto. After successful
creation of a bonding device it must be configured using the generic Ethernet
device configure API rte_eth_dev_configure and then the RX and TX queues
which will be used must be setup using rte_eth_tx_queue_setup /
rte_eth_rx_queue_setup.
Slave devices can be dynamically added and removed from a link bonding device
using the rte_eth_bond_slave_add / rte_eth_bond_slave_remove
APIs but at least one slave device must be added to the link bonding device
before it can be started using rte_eth_dev_start.
The link status of a bonded device is dictated by that of its slaves, if all slave device link status are down or if all slaves are removed from the link bonding device then the link status of the bonding device will go down.
It is also possible to configure / query the configuration of the control
parameters of a bonded device using the provided APIs
rte_eth_bond_mode_set/ get, rte_eth_bond_primary_set/get,
rte_eth_bond_mac_set/reset and rte_eth_bond_xmit_policy_set/get.
Link bonding devices can be created at application startup time using the
--vdev EAL command line option. The device name must start with the
eth_bond prefix followed by numbers or letters. The name must be unique for
each device. Each device can have multiple options arranged in a comma
separated list. Multiple devices definitions can be arranged by calling the
--vdev option multiple times.
Device names and bonding options must be separated by commas as shown below:
$RTE_TARGET/app/testpmd -c f -n 4 --vdev 'eth_bond0,bond_opt0=..,bond opt1=..'--vdev 'eth_bond1,bond _opt0=..,bond_opt1=..'
There are multiple ways of definitions that can be assessed and combined as long as the following two rules are respected:
The different options are:
mode=2
slave=0000:0a:00.0,slave=0000:0a:00.1
primary=0000:0a:00.0
socket_id=0
mac=00:1e:67:1d:fd:1d
xmit_policy=l23
lsc_poll_period_ms=100
up_delay=10
down_delay=50
Create a bonded device in round robin mode with two slaves specified by their PCI address:
$RTE_TARGET/app/testpmd -c '0xf' -n 4 --vdev 'eth_bond0,mode=0, slave=0000:00a:00.01,slave=0000:004:00.00' -- --port-topology=chained
Create a bonded device in round robin mode with two slaves specified by their PCI address and an overriding MAC address:
$RTE_TARGET/app/testpmd -c '0xf' -n 4 --vdev 'eth_bond0,mode=0, slave=0000:00a:00.01,slave=0000:004:00.00,mac=00:1e:67:1d:fd:1d' -- --port-topology=chained
Create a bonded device in active backup mode with two slaves specified, and a primary slave specified by their PCI addresses:
$RTE_TARGET/app/testpmd -c '0xf' -n 4 --vdev 'eth_bond0,mode=1, slave=0000:00a:00.01,slave=0000:004:00.00,primary=0000:00a:00.01' -- --port-topology=chained
Create a bonded device in balance mode with two slaves specified by their PCI addresses, and a transmission policy of layer 3 + 4 forwarding:
$RTE_TARGET/app/testpmd -c '0xf' -n 4 --vdev 'eth_bond0,mode=2, slave=0000:00a:00.01,slave=0000:004:00.00,xmit_policy=l34' -- --port-topology=chained