test_suite - Common Test Suite Features

Features common to all test suites.

The module defines the TestSuite class which doesn’t contain any test cases, and as such must be extended by subclasses which add test cases. The TestSuite contains the basics needed by subclasses:

  • Testbed (SUT, TG) configuration,

  • Packet sending and verification,

  • Test case verification.

class BaseConfig

Bases: FrozenModel

Base for a custom test suite configuration.

class TestSuite

Bases: TestProtocol

The base class with building blocks needed by most test cases.

  • Test suite setup/cleanup methods to override,

  • Test case setup/cleanup methods to override,

  • Test case verification,

  • Testbed configuration,

  • Traffic sending and verification.

Test cases are implemented by subclasses. Test cases are all methods starting with test_, further divided into performance test cases (starting with test_perf_) and functional test cases (all other test cases).

By default, all test cases will be executed. A list of testcase names may be specified in the YAML test run configuration file and in the --test-suite command line argument or in the DTS_TEST_SUITES environment variable to filter which test cases to run. The union of both lists will be used. Any unknown test cases from the latter lists will be silently ignored.

The methods named [set_up|tear_down]_[suite|test_case] should be overridden in subclasses if the appropriate test suite/test case fixtures are needed.

The test suite is aware of the testbed (the SUT and TG) it’s running on. From this, it can properly choose the IP addresses and other configuration that must be tailored to the testbed.

config

The test suite configuration.

Type:

framework.test_suite.BaseConfig

is_blocking: ClassVar[bool] = False

Whether the test suite is blocking. A failure of a blocking test suite will block the execution of all subsequent test suites in the current test run.

__init__(config: BaseConfig) → None

Initialize the test suite testbed information and basic configuration.

Parameters:

config (BaseConfig) – The test suite configuration.

property name: str

The name of the test suite.

property topology: Topology

The current topology in use.

classmethod get_test_cases() → list[type[framework.test_suite.TestCase]]

A list of all the available test cases.

classmethod filter_test_cases(test_case_sublist: collections.abc.Sequence[str] | None = None) → tuple[set[type[framework.test_suite.TestCase]], set[type[framework.test_suite.TestCase]]]

Filter test_case_sublist from this class.

Test cases are regular (or bound) methods decorated with func_test() or perf_test().

Parameters:

test_case_sublist (collections.abc.Sequence[str] | None) – Test case names to filter from this class. If empty or None, return all test cases.

Returns:

The filtered test case functions. This method returns functions as opposed to methods, as methods are bound to instances and this method only has access to the class.

Raises:

ConfigurationError – If a test case from test_case_sublist is not found.

Return type:

tuple[set[type[framework.test_suite.TestCase]], set[type[framework.test_suite.TestCase]]]

set_up_suite() → None

Set up test fixtures common to all test cases.

This is done before any test case has been run.

tear_down_suite() → None

Tear down the previously created test fixtures common to all test cases.

This is done after all test have been run.

set_up_test_case() → None

Set up test fixtures before each test case.

This is done before each test case.

tear_down_test_case() → None

Tear down the previously created test fixtures after each test case.

This is done after each test case.

TestSuiteMethodType

The generic type for a method of an instance of TestSuite

alias of TypeVar(‘TestSuiteMethodType’, bound=Callable[TestSuite, None])

class TestCaseType

Bases: Enum

The types of test cases.

FUNCTIONAL = 1
PERFORMANCE = 2
class TestCase

Bases: TestProtocol, Protocol[TestSuiteMethodType]

Definition of the test case type for static type checking purposes.

The type is applied to test case functions through a decorator, which casts the decorated test case function to TestCase and sets common variables.

name: ClassVar[str]
test_type: ClassVar[TestCaseType]
classmethod make_decorator(test_case_type: TestCaseType) → Callable[[TestSuiteMethodType], type[framework.test_suite.TestCase]]

Create a decorator for test suites.

The decorator casts the decorated function as TestCase, sets it as test_case_type and initializes common variables defined in RequiresCapabilities.

Parameters:

test_case_type (TestCaseType) – Either a functional or performance test case.

Returns:

The decorator of a functional or performance test case.

Return type:

Callable[[TestSuiteMethodType], type[framework.test_suite.TestCase]]

func_test(func: TestSuiteMethodType) → type[framework.test_suite.TestCase]

The decorator for functional test cases.

perf_test(func: TestSuiteMethodType) → type[framework.test_suite.TestCase]

The decorator for performance test cases.

class TestSuiteSpec

Bases: object

A class defining the specification of a test suite.

Apart from defining all the specs of a test suite, a helper function discover_all() is provided to automatically discover all the available test suites.

module_name

The name of the test suite’s module.

Type:

str

TEST_SUITES_PACKAGE_NAME = 'tests'
TEST_SUITE_MODULE_PREFIX = 'TestSuite_'
TEST_SUITE_CLASS_PREFIX = 'Test'
TEST_CASE_METHOD_PREFIX = 'test_'
FUNC_TEST_CASE_REGEX = 'test_(?!perf_)'
PERF_TEST_CASE_REGEX = 'test_perf_'
property name: str

The name of the test suite’s module.

__init__(module_name: str) → None
property module: module

A reference to the test suite’s module.

property class_name: str

The name of the test suite’s class.

property class_obj: type[framework.test_suite.TestSuite]

A reference to the test suite’s class.

Raises:

InternalError – If the test suite class is missing from the module.

property config_obj: type[framework.test_suite.BaseConfig]

A reference to the test suite’s configuration class.

classmethod discover_all(package_name: str | None = None, module_prefix: str | None = None) → list[typing_extensions.Self]

Discover all the test suites.

The test suites are discovered in the provided package_name. The full module name, expected under that package, is prefixed with module_prefix. The module name is a standard filename with words separated with underscores. For each module found, search for a TestSuite class which starts with TEST_SUITE_CLASS_PREFIX, continuing with the module name in PascalCase.

The PascalCase convention applies to abbreviations, acronyms, initialisms and so on:

OS -> Os
TCP -> Tcp
Parameters:

  • package_name (str | None) – The name of the package where to find the test suites. If None, the TEST_SUITES_PACKAGE_NAME is used.

  • module_prefix (str | None) – The name prefix defining the test suite module. If None, the TEST_SUITE_MODULE_PREFIX constant is used.

Returns:

A list containing all the discovered test suites.

Return type:

list[typing_extensions.Self]

AVAILABLE_TEST_SUITES: list[framework.test_suite.TestSuiteSpec] = [TestSuiteSpec(module_name='TestSuite_blocklist'), TestSuiteSpec(module_name='TestSuite_checksum_offload'), TestSuiteSpec(module_name='TestSuite_dual_vlan'), TestSuiteSpec(module_name='TestSuite_dynamic_config'), TestSuiteSpec(module_name='TestSuite_dynamic_queue_conf'), TestSuiteSpec(module_name='TestSuite_hello_world'), TestSuiteSpec(module_name='TestSuite_l2fwd'), TestSuiteSpec(module_name='TestSuite_mac_filter'), TestSuiteSpec(module_name='TestSuite_mtu'), TestSuiteSpec(module_name='TestSuite_packet_capture'), TestSuiteSpec(module_name='TestSuite_pmd_buffer_scatter'), TestSuiteSpec(module_name='TestSuite_port_control'), TestSuiteSpec(module_name='TestSuite_port_restart_config_persistency'), TestSuiteSpec(module_name='TestSuite_port_stats'), TestSuiteSpec(module_name='TestSuite_promisc_support'), TestSuiteSpec(module_name='TestSuite_queue_start_stop'), TestSuiteSpec(module_name='TestSuite_rte_flow'), TestSuiteSpec(module_name='TestSuite_smoke_tests'), TestSuiteSpec(module_name='TestSuite_softnic'), TestSuiteSpec(module_name='TestSuite_uni_pkt'), TestSuiteSpec(module_name='TestSuite_vlan')]

Constant to store all the available, discovered and imported test suites.

The test suites should be gathered from this list to avoid importing more than once.

find_by_name(name: str) → framework.test_suite.TestSuiteSpec | None

Find a requested test suite by name from the available ones.