runner - Testbed Setup and Test Suite Runner

Test suite runner module.

The module is responsible for running DTS in a series of stages:

Test run stage,

DPDK build stage,

Test suite stage,

Test case stage.

The test run stage sets up the environment before running test suites. The test suite stage sets up steps common to all test cases and the test case stage runs test cases individually.

class DTSRunner

Bases: object

Test suite runner class.

The class is responsible for running tests on testbeds defined in the test run configuration. Each setup or teardown of each stage is recorded in a DTSResult or one of its subclasses. The test case results are also recorded.

If an error occurs, the current stage is aborted, the error is recorded, everything in the inner stages is marked as blocked and the run continues in the next iteration of the same stage. The return code is the highest severity of all DTSErrors.

Example

An error occurs in a test suite setup. The current test suite is aborted, all its test cases are marked as blocked and the run continues with the next test suite. If the errored test suite was the last one in the given test run, the next test run begins.

__init__(): Initialize the instance with configuration, logger, result and string constants.

run() → None

Run all test runs from the test run configuration.

Before running test suites, test runs are first set up. The test runs defined in the test run configuration are iterated over. The test runs define which tests to run and where to run them.

The test suites are set up for each test run and each discovered test case within the test suite is set up, executed and torn down. After all test cases have been executed, the test suite is torn down and the next test suite will be run. Once all test suites have been run, the next test run will be tested.

In order to properly mark test suites and test cases as blocked in case of a failure, we need to have discovered which test suites and test cases to run before any failures happen. The discovery happens at the earliest point at the start of each test run.

All the nested steps look like this:

Test run setup

Test suite setup

Test case setup

Test case logic

Test case teardown

Test suite teardown

Test run teardown

The test cases are filtered according to the specification in the test run configuration and the --test-suite command line argument or the DTS_TESTCASES environment variable.