Programmer’s Guide

August 17, 2015

Contents

• 1. Introduction
  • 1.1. Documentation Roadmap
  • 1.2. Related Publications
• 2. Overview
  • 2.1. Development Environment
  • 2.2. Environment Abstraction Layer
  • 2.3. Core Components
    • 2.3.1. Ring Manager (librte_ring)
    • 2.3.2. Memory Pool Manager (librte_mempool)
    • 2.3.3. Network Packet Buffer Management (librte_mbuf)
    • 2.3.4. Timer Manager (librte_timer)
  • 2.4. Ethernet* Poll Mode Driver Architecture
  • 2.5. Packet Forwarding Algorithm Support
  • 2.6. librte_net
• 3. Environment Abstraction Layer
  • 3.1. EAL in a Linux-userland Execution Environment
    • 3.1.1. Initialization and Core Launching
    • 3.1.2. Multi-process Support
    • 3.1.3. Memory Mapping Discovery and Memory Reservation
    • 3.1.4. Xen Dom0 support without hugetbls
    • 3.1.5. PCI Access
    • 3.1.6. Per-lcore and Shared Variables
    • 3.1.7. Logs
    • 3.1.8. CPU Feature Identification
    • 3.1.9. User Space Interrupt Event
    • 3.1.10. Blacklisting
    • 3.1.11. Misc Functions
  • 3.2. Memory Segments and Memory Zones (memzone)
  • 3.3. Multiple pthread
    • 3.3.1. EAL pthread and lcore Affinity
    • 3.3.2. non-EAL pthread support
    • 3.3.3. Public Thread API
    • 3.3.4. Known Issues
    • 3.3.5. cgroup control
  • 3.4. Malloc
    • 3.4.1. Cookies
    • 3.4.2. Alignment and NUMA Constraints
    • 3.4.3. Use Cases
    • 3.4.4. Internal Implementation
• 4. Ring Library
  • 4.1. References for Ring Implementation in FreeBSD*
  • 4.2. Lockless Ring Buffer in Linux*
  • 4.3. Additional Features
    • 4.3.1. Name
    • 4.3.2. Water Marking
    • 4.3.3. Debug
  • 4.4. Use Cases
  • 4.5. Anatomy of a Ring Buffer
    • 4.5.1. Single Producer Enqueue
    • 4.5.2. Single Consumer Dequeue
    • 4.5.3. Multiple Producers Enqueue
    • 4.5.4. Modulo 32-bit Indexes
  • 4.6. References
• 5. Mempool Library
  • 5.1. Cookies
  • 5.2. Stats
  • 5.3. Memory Alignment Constraints
  • 5.4. Local Cache
  • 5.5. Use Cases
• 6. Mbuf Library
  • 6.1. Design of Packet Buffers
  • 6.2. Buffers Stored in Memory Pools
  • 6.3. Constructors
  • 6.4. Allocating and Freeing mbufs
  • 6.5. Manipulating mbufs
  • 6.6. Meta Information
  • 6.7. Direct and Indirect Buffers
  • 6.8. Debug
  • 6.9. Use Cases
• 7. Poll Mode Driver
  • 7.1. Requirements and Assumptions
  • 7.2. Design Principles
  • 7.3. Logical Cores, Memory and NIC Queues Relationships
  • 7.4. Device Identification and Configuration
    • 7.4.1. Device Identification
    • 7.4.2. Device Configuration
    • 7.4.3. On-the-Fly Configuration
    • 7.4.4. Configuration of Transmit and Receive Queues
    • 7.4.5. Hardware Offload
  • 7.5. Poll Mode Driver API
    • 7.5.1. Generalities
    • 7.5.2. Generic Packet Representation
    • 7.5.3. Ethernet Device API
• 8. IVSHMEM Library
  • 8.1. IVHSHMEM Library API Overview
  • 8.2. IVSHMEM Environment Configuration
  • 8.3. Best Practices for Writing IVSHMEM Applications
  • 8.4. Best Practices for Running IVSHMEM Applications
• 9. Link Bonding Poll Mode Driver Library
  • 9.1. Link Bonding Modes Overview
  • 9.2. Implementation Details
    • 9.2.1. Link Status Change Interrupts / Polling
    • 9.2.2. Requirements / Limitations
    • 9.2.3. Configuration
  • 9.3. Using Link Bonding Devices
    • 9.3.1. Using the Poll Mode Driver from an Application
    • 9.3.2. Using Link Bonding Devices from the EAL Command Line
• 10. Timer Library
  • 10.1. Implementation Details
  • 10.2. Use Cases
  • 10.3. References
• 11. Hash Library
  • 11.1. Hash API Overview
  • 11.2. Implementation Details
  • 11.3. Entry distribution in hash table
  • 11.4. Use Case: Flow Classification
  • 11.5. References
• 12. LPM Library
  • 12.1. LPM API Overview
  • 12.2. Implementation Details
    • 12.2.1. Addition
    • 12.2.2. Lookup
    • 12.2.3. Limitations in the Number of Rules
    • 12.2.4. Use Case: IPv4 Forwarding
    • 12.2.5. References
• 13. LPM6 Library
  • 13.1. LPM6 API Overview
    • 13.1.1. Implementation Details
    • 13.1.2. Addition
    • 13.1.3. Lookup
    • 13.1.4. Limitations in the Number of Rules
  • 13.2. Use Case: IPv6 Forwarding
• 14. Packet Distributor Library
  • 14.1. Distributor Core Operation
  • 14.2. Worker Operation
• 15. Reorder Library
  • 15.1. Operation
  • 15.2. Implementation Details
  • 15.3. Use Case: Packet Distributor
• 16. IP Fragmentation and Reassembly Library
  • 16.1. Packet fragmentation
  • 16.2. Packet reassembly
    • 16.2.1. IP Fragment Table
    • 16.2.2. Packet Reassembly
    • 16.2.3. Debug logging and Statistics Collection
• 17. Multi-process Support
  • 17.1. Memory Sharing
  • 17.2. Deployment Models
    • 17.2.1. Symmetric/Peer Processes
    • 17.2.2. Asymmetric/Non-Peer Processes
    • 17.2.3. Running Multiple Independent DPDK Applications
    • 17.2.4. Running Multiple Independent Groups of DPDK Applications
  • 17.3. Multi-process Limitations
• 18. Kernel NIC Interface
  • 18.1. The DPDK KNI Kernel Module
  • 18.2. KNI Creation and Deletion
  • 18.3. DPDK mbuf Flow
  • 18.4. Use Case: Ingress
  • 18.5. Use Case: Egress
  • 18.6. Ethtool
  • 18.7. Link state and MTU change
  • 18.8. KNI Working as a Kernel vHost Backend
    • 18.8.1. Overview
    • 18.8.2. Packet Flow
    • 18.8.3. Sample Usage
    • 18.8.4. Compatibility Configure Option
• 19. Thread Safety of DPDK Functions
  • 19.1. Fast-Path APIs
  • 19.2. Performance Insensitive API
  • 19.3. Library Initialization
  • 19.4. Interrupt Thread
• 20. Quality of Service (QoS) Framework
  • 20.1. Packet Pipeline with QoS Support
  • 20.2. Hierarchical Scheduler
    • 20.2.1. Overview
    • 20.2.2. Scheduling Hierarchy
    • 20.2.3. Application Programming Interface (API)
    • 20.2.4. Implementation
    • 20.2.5. Worst Case Scenarios for Performance
  • 20.3. Dropper
    • 20.3.1. Configuration
    • 20.3.2. Enqueue Operation
    • 20.3.3. Queue Empty Operation
    • 20.3.4. Source Files Location
    • 20.3.5. Integration with the DPDK QoS Scheduler
    • 20.3.6. Integration with the DPDK QoS Scheduler Sample Application
    • 20.3.7. Application Programming Interface (API)
  • 20.4. Traffic Metering
    • 20.4.1. Functional Overview
    • 20.4.2. Implementation Overview
• 21. Power Management
  • 21.1. CPU Frequency Scaling
  • 21.2. Core-load Throttling through C-States
  • 21.3. API Overview of the Power Library
  • 21.4. User Cases
  • 21.5. References
• 22. Packet Classification and Access Control
  • 22.1. Overview
    • 22.1.1. Rule definition
    • 22.1.2. RT memory size limit
    • 22.1.3. Classification methods
  • 22.2. Application Programming Interface (API) Usage
    • 22.2.1. Classify with Multiple Categories
• 23. Packet Framework
  • 23.1. Design Objectives
  • 23.2. Overview
  • 23.3. Port Library Design
    • 23.3.1. Port Types
    • 23.3.2. Port Interface
  • 23.4. Table Library Design
    • 23.4.1. Table Types
    • 23.4.2. Table Interface
    • 23.4.3. Hash Table Design
  • 23.5. Pipeline Library Design
    • 23.5.1. Connectivity of Ports and Tables
    • 23.5.2. Port Actions
    • 23.5.3. Table Actions
  • 23.6. Multicore Scaling
    • 23.6.1. Shared Data Structures
  • 23.7. Interfacing with Accelerators
• 24. Vhost Library
  • 24.1. Vhost API Overview
  • 24.2. Vhost Implementation
    • 24.2.1. Vhost cuse implementation
    • 24.2.2. Vhost user implementation
  • 24.3. Vhost supported vSwitch reference
• 25. Port Hotplug Framework
  • 25.1. Overview
  • 25.2. Port Hotplug API overview
  • 25.3. Reference
  • 25.4. Limitations
• 26. Source Organization
  • 26.1. Makefiles and Config
  • 26.2. Libraries
  • 26.3. Drivers
  • 26.4. Applications
• 27. Development Kit Build System
  • 27.1. Building the Development Kit Binary
    • 27.1.1. Build Directory Concept
  • 27.2. Building External Applications
  • 27.3. Makefile Description
    • 27.3.1. General Rules For DPDK Makefiles
    • 27.3.2. Makefile Types
    • 27.3.3. Useful Variables Provided by the Build System
    • 27.3.4. Variables that Can be Set/Overridden in a Makefile Only
    • 27.3.5. Variables that can be Set/Overridden by the User on the Command Line Only
    • 27.3.6. Variables that Can be Set/Overridden by the User in a Makefile or Command Line
• 28. Development Kit Root Makefile Help
  • 28.1. Configuration Targets
  • 28.2. Build Targets
  • 28.3. Install Targets
  • 28.4. Test Targets
  • 28.5. Documentation Targets
  • 28.6. Deps Targets
  • 28.7. Misc Targets
  • 28.8. Other Useful Command-line Variables
  • 28.9. Make in a Build Directory
  • 28.10. Compiling for Debug
• 29. Extending the DPDK
  • 29.1. Example: Adding a New Library libfoo
    • 29.1.1. Example: Using libfoo in the Test Application
• 30. Building Your Own Application
  • 30.1. Compiling a Sample Application in the Development Kit Directory
  • 30.2. Build Your Own Application Outside the Development Kit
  • 30.3. Customizing Makefiles
    • 30.3.1. Application Makefile
    • 30.3.2. Library Makefile
    • 30.3.3. Customize Makefile Actions
• 31. External Application/Library Makefile help
  • 31.1. Prerequisites
  • 31.2. Build Targets
  • 31.3. Help Targets
  • 31.4. Other Useful Command-line Variables
  • 31.5. Make from Another Directory
• 32. Performance Optimization Guidelines
  • 32.1. Introduction
• 33. Writing Efficient Code
  • 33.1. Memory
    • 33.1.1. Memory Copy: Do not Use libc in the Data Plane
    • 33.1.2. Memory Allocation
    • 33.1.3. Concurrent Access to the Same Memory Area
    • 33.1.4. NUMA
    • 33.1.5. Distribution Across Memory Channels
  • 33.2. Communication Between lcores
  • 33.3. PMD Driver
    • 33.3.1. Lower Packet Latency
  • 33.4. Locks and Atomic Operations
  • 33.5. Coding Considerations
    • 33.5.1. Inline Functions
    • 33.5.2. Branch Prediction
  • 33.6. Setting the Target CPU Type
• 34. Profile Your Application
• 35. Glossary

Figures

Fig. 2.1 Core Components Architecture

Fig. 3.1 EAL Initialization in a Linux Application Environment

Fig. 3.2 Example of a malloc heap and malloc elements within the malloc library

Fig. 4.1 Ring Structure

Fig. 4.2 Enqueue first step

Fig. 4.3 Enqueue second step

Fig. 4.4 Enqueue last step

Fig. 4.5 Dequeue last step

Fig. 4.6 Dequeue second step

Fig. 4.7 Dequeue last step

Fig. 4.8 Multiple consumer enqueue first step

Fig. 4.9 Multiple consumer enqueue second step

Fig. 4.10 Multiple consumer enqueue third step

Fig. 4.11 Multiple consumer enqueue fourth step

Fig. 4.12 Multiple consumer enqueue last step

Fig. 4.13 Modulo 32-bit indexes - Example 1

Fig. 4.14 Modulo 32-bit indexes - Example 2

Fig. 5.1 Two Channels and Quad-ranked DIMM Example

Fig. 5.2 Three Channels and Two Dual-ranked DIMM Example

Fig. 5.3 A mempool in Memory with its Associated Ring

Fig. 6.1 An mbuf with One Segment

Fig. 6.2 An mbuf with Three Segments

Fig. 17.1 Memory Sharing in the DPDK Multi-process Sample Application

Fig. 18.1 Components of a DPDK KNI Application

Fig. 18.2 Packet Flow via mbufs in the DPDK KNI

Fig. 18.3 vHost-net Architecture Overview

Fig. 18.4 KNI Traffic Flow

Fig. 20.1 Complex Packet Processing Pipeline with QoS Support

Fig. 20.2 Hierarchical Scheduler Block Internal Diagram

Fig. 20.3 Scheduling Hierarchy per Port

Fig. 20.4 Internal Data Structures per Port

Fig. 20.5 Prefetch Pipeline for the Hierarchical Scheduler Enqueue Operation

Fig. 20.6 Pipe Prefetch State Machine for the Hierarchical Scheduler Dequeue Operation

Fig. 20.7 High-level Block Diagram of the DPDK Dropper

Fig. 20.8 Flow Through the Dropper

Fig. 20.9 Example Data Flow Through Dropper

Fig. 20.10 Packet Drop Probability for a Given RED Configuration

Fig. 20.11 Initial Drop Probability (pb), Actual Drop probability (pa) Computed Using a Factor 1 (Blue Curve) and a Factor 2 (Red Curve)

Fig. 23.1 Example of Packet Processing Pipeline where Input Ports 0 and 1 are Connected with Output Ports 0, 1 and 2 through Tables 0 and 1

Fig. 23.2 Sequence of Steps for Hash Table Operations in a Packet Processing Context

Fig. 23.3 Data Structures for Configurable Key Size Hash Tables

Fig. 23.4 Bucket Search Pipeline for Key Lookup Operation (Configurable Key Size Hash Tables)

Fig. 23.5 Data Structures for 8-byte Key Hash Tables

Fig. 23.6 Data Structures for 16-byte Key Hash Tables

Fig. 23.7 Bucket Search Pipeline for Key Lookup Operation (Single Key Size Hash Tables)

Tables

Table 20.1 Packet Processing Pipeline Implementing QoS

Table 20.2 Infrastructure Blocks Used by the Packet Processing Pipeline

Table 20.3 Port Scheduling Hierarchy

Table 20.4 Scheduler Internal Data Structures per Port

Table 20.5 Ethernet Frame Overhead Fields

Table 20.6 Token Bucket Generic Operations

Table 20.7 Token Bucket Generic Parameters

Table 20.8 Token Bucket Persistent Data Structure

Table 20.9 Token Bucket Operations

Table 20.10 Subport/Pipe Traffic Class Upper Limit Enforcement Persistent Data Structure

Table 20.11 Subport/Pipe Traffic Class Upper Limit Enforcement Operations

Table 20.12 Weighted Round Robin (WRR)

Table 20.13 Subport Traffic Class Oversubscription

Table 20.14 Watermark Propagation from Subport Level to Member Pipes at the Beginning of Each Traffic Class Upper Limit Enforcement Period

Table 20.15 Watermark Calculation

Table 20.16 RED Configuration Parameters

Table 20.17 Relative Performance of Alternative Approaches

Table 20.18 RED Configuration Corresponding to RED Configuration File

Table 23.1 Port Types

Table 23.2 20 Port Abstract Interface

Table 23.3 Table Types

Table 23.5 Configuration Parameters Common for All Hash Table Types

Table 23.6 Configuration Parameters Specific to Extendible Bucket Hash Table

Table 23.7 Configuration Parameters Specific to Pre-computed Key Signature Hash Table

Table 23.8 Main Large Data Structures (Arrays) used for Configurable Key Size Hash Tables

Table 23.9 Field Description for Bucket Array Entry (Configurable Key Size Hash Tables)

Table 23.10 Description of the Bucket Search Pipeline Stages (Configurable Key Size Hash Tables)

Table 23.11 Lookup Tables for Match, Match_Many and Match_Pos

Table 23.12 Collapsed Lookup Tables for Match, Match_Many and Match_Pos

Table 23.13 Main Large Data Structures (Arrays) used for 8-byte and 16-byte Key Size Hash Tables

Table 23.14 Field Description for Bucket Array Entry (8-byte and 16-byte Key Hash Tables)

Table 23.15 Description of the Bucket Search Pipeline Stages (8-byte and 16-byte Key Hash Tables)

Table 23.16 Next Hop Actions (Reserved)

Table 23.17 User Action Examples

Table 11.1 Entry distribution measured with an example table with 1024 random entries using jhash algorithm

Table 11.2 Entry distribution measured with an example table with 1 million random entries using jhash algorithm