test_suite - Common Test Suite Features

Features common to all test suites.

The module defines the TestSuite class which doesn’t contain any test cases, and as such must be extended by subclasses which add test cases. The TestSuite contains the basics needed by subclasses:

  • Testbed (SUT, TG) configuration,

  • Packet sending and verification,

  • Test case verification.

class TestSuite

Bases: TestProtocol

The base class with building blocks needed by most test cases.

  • Test suite setup/cleanup methods to override,

  • Test case setup/cleanup methods to override,

  • Test case verification,

  • Testbed configuration,

  • Traffic sending and verification.

Test cases are implemented by subclasses. Test cases are all methods starting with test_, further divided into performance test cases (starting with test_perf_) and functional test cases (all other test cases).

By default, all test cases will be executed. A list of testcase names may be specified in the YAML test run configuration file and in the --test-suite command line argument or in the DTS_TESTCASES environment variable to filter which test cases to run. The union of both lists will be used. Any unknown test cases from the latter lists will be silently ignored.

The methods named [set_up|tear_down]_[suite|test_case] should be overridden in subclasses if the appropriate test suite/test case fixtures are needed.

The test suite is aware of the testbed (the SUT and TG) it’s running on. From this, it can properly choose the IP addresses and other configuration that must be tailored to the testbed.

sut_node

The SUT node where the test suite is running.

Type:

framework.testbed_model.sut_node.SutNode

tg_node

The TG node where the test suite is running.

Type:

framework.testbed_model.tg_node.TGNode

is_blocking: ClassVar[bool] = False

Whether the test suite is blocking. A failure of a blocking test suite will block the execution of all subsequent test suites in the current test run.

__init__(sut_node: SutNode, tg_node: TGNode, topology: Topology)

Initialize the test suite testbed information and basic configuration.

Find links between ports and set up default IP addresses to be used when configuring them.

Parameters:

  • sut_node (SutNode) – The SUT node where the test suite will run.

  • tg_node (TGNode) – The TG node where the test suite will run.

  • topology (Topology) – The topology where the test suite will run.

classmethod get_test_cases() → list[type[framework.test_suite.TestCase]]

A list of all the available test cases.

classmethod filter_test_cases(test_case_sublist: collections.abc.Sequence[str] | None = None) → tuple[set[type[framework.test_suite.TestCase]], set[type[framework.test_suite.TestCase]]]

Filter test_case_sublist from this class.

Test cases are regular (or bound) methods decorated with func_test() or perf_test().

Parameters:

test_case_sublist (collections.abc.Sequence[str] | None) – Test case names to filter from this class. If empty or None, return all test cases.

Returns:

The filtered test case functions. This method returns functions as opposed to methods, as methods are bound to instances and this method only has access to the class.

Raises:

ConfigurationError – If a test case from test_case_sublist is not found.

Return type:

tuple[set[type[framework.test_suite.TestCase]], set[type[framework.test_suite.TestCase]]]

set_up_suite() → None

Set up test fixtures common to all test cases.

This is done before any test case has been run.

tear_down_suite() → None

Tear down the previously created test fixtures common to all test cases.

This is done after all test have been run.

set_up_test_case() → None

Set up test fixtures before each test case.

This is done before each test case.

tear_down_test_case() → None

Tear down the previously created test fixtures after each test case.

This is done after each test case.

send_packet_and_capture(packet: scapy.packet.Packet, filter_config: PacketFilteringConfig = PacketFilteringConfig(no_lldp=True, no_arp=True), duration: float = 1) → list[scapy.packet.Packet]

Send and receive packet using the associated TG.

Send packet through the appropriate interface and receive on the appropriate interface. Modify the packet with l3/l2 addresses corresponding to the testbed and desired traffic.

Parameters:

  • packet (scapy.packet.Packet) – The packet to send.

  • filter_config (PacketFilteringConfig) – The filter to use when capturing packets.

  • duration (float) – Capture traffic for this amount of time after sending packet.

Returns:

A list of received packets.

Return type:

list[scapy.packet.Packet]

send_packets_and_capture(packets: list[scapy.packet.Packet], filter_config: PacketFilteringConfig = PacketFilteringConfig(no_lldp=True, no_arp=True), duration: float = 1) → list[scapy.packet.Packet]

Send and receive packets using the associated TG.

Send packets through the appropriate interface and receive on the appropriate interface. Modify the packets with l3/l2 addresses corresponding to the testbed and desired traffic.

Parameters:

  • packets (list[scapy.packet.Packet]) – The packets to send.

  • filter_config (PacketFilteringConfig) – The filter to use when capturing packets.

  • duration (float) – Capture traffic for this amount of time after sending packet.

Returns:

A list of received packets.

Return type:

list[scapy.packet.Packet]

send_packets(packets: list[scapy.packet.Packet]) → None

Send packets using the traffic generator and do not capture received traffic.

Parameters:

packets (list[scapy.packet.Packet]) – Packets to send.

get_expected_packets(packets: list[scapy.packet.Packet]) → list[scapy.packet.Packet]

Inject the proper L2/L3 addresses into packets.

Inject the L2/L3 addresses expected at the receiving end of the traffic generator.

Parameters:

packets (list[scapy.packet.Packet]) – The packets to modify.

Returns:

packets with injected L2/L3 addresses.

Return type:

list[scapy.packet.Packet]

get_expected_packet(packet: scapy.packet.Packet) → scapy.packet.Packet

Inject the proper L2/L3 addresses into packet.

Inject the L2/L3 addresses expected at the receiving end of the traffic generator.

Parameters:

packet (scapy.packet.Packet) – The packet to modify.

Returns:

packet with injected L2/L3 addresses.

Return type:

scapy.packet.Packet

verify(condition: bool, failure_description: str) → None

Verify condition and handle failures.

When condition is False, raise an exception and log the last 10 commands executed on both the SUT and TG.

Parameters:

  • condition (bool) – The condition to check.

  • failure_description (str) – A short description of the failure that will be stored in the raised exception.

Raises:

TestCaseVerifyError – condition is False.

verify_packets(expected_packet: scapy.packet.Packet, received_packets: list[scapy.packet.Packet]) → None

Verify that expected_packet has been received.

Go through received_packets and check that expected_packet is among them. If not, raise an exception and log the last 10 commands executed on both the SUT and TG.

Parameters:

  • expected_packet (scapy.packet.Packet) – The packet we’re expecting to receive.

  • received_packets (list[scapy.packet.Packet]) – The packets where we’re looking for expected_packet.

Raises:

TestCaseVerifyError – expected_packet is not among received_packets.

match_all_packets(expected_packets: list[scapy.packet.Packet], received_packets: list[scapy.packet.Packet]) → None

Matches all the expected packets against the received ones.

Matching is performed by counting down the occurrences in a dictionary which keys are the raw packet bytes. No deep packet comparison is performed. All the unexpected packets (noise) are automatically ignored.

Parameters:

  • expected_packets (list[scapy.packet.Packet]) – The packets we are expecting to receive.

  • received_packets (list[scapy.packet.Packet]) – All the packets that were received.

Raises:

TestCaseVerifyError – if and not all the expected_packets were found in received_packets.

TestSuiteMethodType

The generic type for a method of an instance of TestSuite

alias of TypeVar(‘TestSuiteMethodType’, bound=Callable[TestSuite, None])

class TestCaseType

Bases: Enum

The types of test cases.

FUNCTIONAL = 1
PERFORMANCE = 2
class TestCase

Bases: TestProtocol, Protocol[TestSuiteMethodType]

Definition of the test case type for static type checking purposes.

The type is applied to test case functions through a decorator, which casts the decorated test case function to TestCase and sets common variables.

name: ClassVar[str]
test_type: ClassVar[TestCaseType]
classmethod make_decorator(test_case_type: TestCaseType) → Callable[[TestSuiteMethodType], type[framework.test_suite.TestCase]]

Create a decorator for test suites.

The decorator casts the decorated function as TestCase, sets it as test_case_type and initializes common variables defined in RequiresCapabilities.

Parameters:

test_case_type (TestCaseType) – Either a functional or performance test case.

Returns:

The decorator of a functional or performance test case.

Return type:

Callable[[TestSuiteMethodType], type[framework.test_suite.TestCase]]

func_test(func: TestSuiteMethodType) → type[framework.test_suite.TestCase]

The decorator for functional test cases.

perf_test(func: TestSuiteMethodType) → type[framework.test_suite.TestCase]

The decorator for performance test cases.

class TestSuiteSpec

Bases: object

A class defining the specification of a test suite.

Apart from defining all the specs of a test suite, a helper function discover_all() is provided to automatically discover all the available test suites.

module_name

The name of the test suite’s module.

Type:

str

TEST_SUITES_PACKAGE_NAME = 'tests'
TEST_SUITE_MODULE_PREFIX = 'TestSuite_'
TEST_SUITE_CLASS_PREFIX = 'Test'
TEST_CASE_METHOD_PREFIX = 'test_'
__init__(module_name: str) → None
FUNC_TEST_CASE_REGEX = 'test_(?!perf_)'
PERF_TEST_CASE_REGEX = 'test_perf_'
property name: str

The name of the test suite’s module.

property module: module

A reference to the test suite’s module.

property class_name: str

The name of the test suite’s class.

property class_obj: type[framework.test_suite.TestSuite]

A reference to the test suite’s class.

classmethod discover_all(package_name: str | None = None, module_prefix: str | None = None) → list[typing_extensions.Self]

Discover all the test suites.

The test suites are discovered in the provided package_name. The full module name, expected under that package, is prefixed with module_prefix. The module name is a standard filename with words separated with underscores. For each module found, search for a TestSuite class which starts with TEST_SUITE_CLASS_PREFIX, continuing with the module name in PascalCase.

The PascalCase convention applies to abbreviations, acronyms, initialisms and so on:

OS -> Os
TCP -> Tcp
Parameters:

  • package_name (str | None) – The name of the package where to find the test suites. If None, the TEST_SUITES_PACKAGE_NAME is used.

  • module_prefix (str | None) – The name prefix defining the test suite module. If None, the TEST_SUITE_MODULE_PREFIX constant is used.

Returns:

A list containing all the discovered test suites.

Return type:

list[typing_extensions.Self]

AVAILABLE_TEST_SUITES: list[framework.test_suite.TestSuiteSpec] = [TestSuiteSpec(module_name='TestSuite_blocklist'), TestSuiteSpec(module_name='TestSuite_checksum_offload'), TestSuiteSpec(module_name='TestSuite_dynamic_queue_conf'), TestSuiteSpec(module_name='TestSuite_hello_world'), TestSuiteSpec(module_name='TestSuite_l2fwd'), TestSuiteSpec(module_name='TestSuite_mac_filter'), TestSuiteSpec(module_name='TestSuite_pmd_buffer_scatter'), TestSuiteSpec(module_name='TestSuite_smoke_tests'), TestSuiteSpec(module_name='TestSuite_vlan')]

Constant to store all the available, discovered and imported test suites.

The test suites should be gathered from this list to avoid importing more than once.

find_by_name(name: str) → framework.test_suite.TestSuiteSpec | None

Find a requested test suite by name from the available ones.