test_run - Test Run Execution

Test run module.

The test run is implemented as a finite state machine which maintains a globally accessible Context and each state implements State.

To spin up the test run state machine call spin().

The following graph represents all the states and steps of the state machine. Each node represents a state labelled with the initials, e.g. TRS is represented by TestRunSetup. States represented by a double green circle are looping states. These states are only exited through:

next which progresses to the next test suite/case.

end which indicates that no more test suites/cases are available and the loop is terminated.

Red dashed links represent the path taken when an exception is raised in the origin state. If a state does not have one, then the execution progresses as usual. When InternalError is raised in any state, the state machine execution is immediately terminated. Orange dashed links represent exceptional conditions. Test suites and cases can be blocked or skipped in the following conditions:

If a blocking test suite fails, the blocked flag is raised.

If the user sends a SIGINT signal, the blocked flag is raised.

If a test suite and/or test case requires a capability unsupported by the test run, then this is skipped and the state restarts from the beginning.

Finally, test cases retry when they fail and DTS is configured to re-run.

+-----------------+---------------------------+-----------------------------------+
| Current State   | Next Possible State       | Notes                             |
+-----------------+---------------------------+-----------------------------------+
| TRS             | TRE, TRT                  | Normal transition                 |
| TRE             | TRT, TSS, TRE (self-loop) | "end" → TRT, "next" → TSS,        |
|                 |                           | "next (skipped)" → TRE            |
| TRT             | exit                      | Normal transition                 |
| TSS             | TSE, TST                  | "next" → TSE, normal flow to TST  |
| TSE             | TST, TCS, TSE (self-loop) | "end" → TST, "next" → TCS,        |
|                 |                           | "next (blocked, skipped)" → TSE   |
| TST             | TRE                       | Normal transition                 |
| TCS             | TCE                       | Normal transition                 |
| TCE             | TCT, TCE (self-loop)      | "next" → TCT, "retry" → TCE       |
| TCT             | TSE                       | Normal transition                 |
| InternalError   | exit                      | Error handling                    |
| exit            | -                         | Final state                       |
+-----------------+---------------------------+-----------------------------------+

class TestRun

Bases: object

A class representing a test run.

The class is responsible for running tests on testbeds defined in the test run configuration. Each setup or teardown of each stage is recorded in a DTSResult or one of its subclasses. The test case results are also recorded.

If an error occurs, the current stage is aborted, the error is recorded, everything in the inner stages is marked as blocked and the run continues in the next iteration of the same stage. The return code is the highest severity of all DTSErrors.

Example

An error occurs in a test suite setup. The current test suite is aborted, all its test cases are marked as blocked and the run continues with the next test suite. If the errored test suite was the last one in the given test run, the next test run begins.

config

The test run configuration.

Type:: framework.config.test_run.TestRunConfiguration

logger

A reference to the current logger.

Type:: framework.logger.DTSLogger

state

The current state of the state machine.

Type:: framework.test_run.State

ctx

The test run’s runtime context.

Type:: framework.context.Context

result

The test run’s execution result.

Type:: framework.test_result.TestRunResult

selected_tests

The test suites and cases selected in this test run.

Type:: list[tuple[type[framework.test_suite.TestSuite], collections.deque[type[framework.test_suite.TestCase]]]]

blocked

True if the test run execution has been blocked.

Type:: bool

remaining_tests

The remaining tests in the execution of the test run.

Type:: collections.deque[tuple[type[framework.test_suite.TestSuite], collections.deque[type[framework.test_suite.TestCase]]]]

remaining_test_cases

The remaining test cases in the execution of a test suite within the test run’s state machine.

Type:: collections.deque[type[framework.test_suite.TestCase]]

supported_capabilities

All the capabilities supported by this test run.

Type:: set[framework.testbed_model.capability.Capability]

__init__(config: TestRunConfiguration, nodes: Iterable[Node], result: TestRunResult)

Test run constructor.

Parameters:

config (TestRunConfiguration) – The test run’s own configuration.
nodes (Iterable[Node]) – A reference to all the available nodes.
result (TestRunResult) – A reference to the test run result object.

property required_capabilities: set[framework.testbed_model.capability.Capability]: The capabilities required to run this test run in its totality.

spin(): Spin the internal state machine that executes the test run.

init_random_seed() → None

Initialize the random seed to use for the test run.

class State

Bases: Protocol

Protocol indicating the state of the test run.

before(): Hook before the state is processed.

after(): Hook after the state is processed.

property description: str: State description.

property logger: DTSLogger: A reference to the root logger.

get_log_file_name() → str | None

Name of the log file for this state.

property log_file_path: pathlib.Path | None: Path to the log file for this state.

next() → Optional[State]

Next state.

on_error(ex: Exception) → Optional[State]

Next state on error.

handle_exception(ex: Exception) → Optional[State]

Handles an exception raised by next.

__init__(*args, **kwargs)

class TestRunSetup

Bases: State

Test run setup.

property description: str: State description.

next() → framework.test_run.State | None

Process state and return the next one.

on_error(ex: Exception) → framework.test_run.State | None

Next state on error.

__init__(test_run: TestRun, result: TestRunResult) → None

class TestRunExecution

Bases: State

Test run execution.

property description: str: State description.

next() → framework.test_run.State | None

Next state.

on_error(ex: Exception) → framework.test_run.State | None

Next state on error.

__init__(test_run: TestRun, result: TestRunResult) → None

class TestRunTeardown

Bases: State

Test run teardown.

property description: str: State description.

next() → framework.test_run.State | None

Next state.

on_error(ex: Exception) → framework.test_run.State | None

Next state on error.

__init__(test_run: TestRun, result: TestRunResult) → None

class TestSuiteState

Bases: State

A test suite state template.

get_log_file_name() → str | None

Get the log file name.

__init__(test_run: TestRun, test_suite: TestSuite, result: TestSuiteResult) → None

class TestSuiteSetup

Bases: TestSuiteState

Test suite setup.

property description: str: State description.

next() → framework.test_run.State | None

Next state.

on_error(ex: Exception) → framework.test_run.State | None

Next state on error.

__init__(test_run: TestRun, test_suite: TestSuite, result: TestSuiteResult) → None

class TestSuiteExecution

Bases: TestSuiteState

Test suite execution.

property description: str: State description.

next() → framework.test_run.State | None

Next state.

on_error(ex: Exception) → framework.test_run.State | None

Next state on error.

__init__(test_run: TestRun, test_suite: TestSuite, result: TestSuiteResult) → None

class TestSuiteTeardown

Bases: TestSuiteState

Test suite teardown.

property description: str: State description.

next() → framework.test_run.State | None

Next state.

on_error(ex: Exception) → framework.test_run.State | None

Next state on error.

after(): Hook after state is processed.

__init__(test_run: TestRun, test_suite: TestSuite, result: TestSuiteResult) → None

class TestCaseState

Bases: State

A test case state template.

get_log_file_name() → str | None

Get the log file name.

__init__(test_run: TestRun, test_suite: TestSuite, test_suite_result: TestSuiteResult, test_case: type[framework.test_suite.TestCase], result: TestCaseResult) → None

class TestCaseSetup

Bases: TestCaseState

Test case setup.

property description: str: State description.

next() → framework.test_run.State | None

Next state.

on_error(ex: Exception) → framework.test_run.State | None

Next state on error.

__init__(test_run: TestRun, test_suite: TestSuite, test_suite_result: TestSuiteResult, test_case: type[framework.test_suite.TestCase], result: TestCaseResult) → None

class TestCaseExecution

Bases: TestCaseState

Test case execution.

__init__(test_run: TestRun, test_suite: TestSuite, test_suite_result: TestSuiteResult, test_case: type[framework.test_suite.TestCase], result: TestCaseResult, reattempts_left: int) → None

property description: str: State description.

next() → framework.test_run.State | None

Next state.

on_error(ex: Exception) → framework.test_run.State | None

Next state on error.

class TestCaseTeardown

Bases: TestCaseState

Test case teardown.

__init__(test_run: TestRun, test_suite: TestSuite, test_suite_result: TestSuiteResult, test_case: type[framework.test_suite.TestCase], result: TestCaseResult) → None

property description: str: State description.

next() → framework.test_run.State | None

Next state.

on_error(ex: Exception) → framework.test_run.State | None

Next state on error.