test_result - Test Results Records

Record and process DTS results.

The results are recorded in a hierarchical manner:

DTSResult contains

TestRunResult contains

TestSuiteResult contains

TestCaseResult

Each result may contain multiple lower level results, e.g. there are multiple TestSuiteResults in a TestRunResult. The results have common parts, such as setup and teardown results, captured in BaseResult, which also defines some common behaviors in its methods.

Each result class has its own idiosyncrasies which they implement in overridden methods.

The --output command line argument and the DTS_OUTPUT_DIR environment variable modify the directory where the files with results will be stored.

class TestSuiteWithCases

Bases: object

A test suite class with test case methods.

An auxiliary class holding a test case class with test case methods. The intended use of this class is to hold a subset of test cases (which could be all test cases) because we don’t have all the data to instantiate the class at the point of inspection. The knowledge of this subset is needed in case an error occurs before the class is instantiated and we need to record which test cases were blocked by the error.

test_suite_class

The test suite class.

Type:: type[framework.test_suite.TestSuite]

test_cases

The test case methods.

Type:: list[type[framework.test_suite.TestCase]]

required_capabilities

The combined required capabilities of both the test suite and the subset of test cases.

Type:: set[framework.testbed_model.capability.Capability]

create_config() → TestSuiteConfig

Generate a TestSuiteConfig from the stored test suite with test cases.

Returns:: The TestSuiteConfig representation.
Return type:: TestSuiteConfig

mark_skip_unsupported(supported_capabilities: set[framework.testbed_model.capability.Capability]) → None

Mark the test suite and test cases to be skipped.

The mark is applied if object to be skipped requires any capabilities and at least one of them is not among supported_capabilities.

Parameters:: supported_capabilities (set[framework.testbed_model.capability.Capability]) – The supported capabilities.

property skip: bool

Skip the test suite if all test cases or the suite itself are to be skipped.

Returns:: True if the test suite should be skipped, False otherwise.

__init__(test_suite_class: type[framework.test_suite.TestSuite], test_cases: list[type[framework.test_suite.TestCase]]) → None

class Result

Bases: Enum

The possible states that a setup, a teardown or a test case may end up in.

PASS = 1

FAIL = 2

ERROR = 3

BLOCK = 4

SKIP = 5

class TestCaseResultDict

Bases: TypedDict

Represents the TestCaseResult results.

test_case_name

The name of the test case.

Type:: str

result

The result name of the test case.

Type:: str

class TestSuiteResultDict

Bases: TypedDict

Represents the TestSuiteResult results.

test_suite_name

The name of the test suite.

Type:: str

test_cases

A list of test case results contained in this test suite.

Type:: list[framework.test_result.TestCaseResultDict]

class TestRunResultDict

Bases: TypedDict

Represents the TestRunResult results.

compiler_version

The version of the compiler used for the DPDK build.

Type:: str | None

dpdk_version

The version of DPDK being tested.

Type:: str | None

ports

A list of ports associated with the test run.

Type:: list[dict[str, Any]]

test_suites

A list of test suite results included in this test run.

Type:: list[framework.test_result.TestSuiteResultDict]

summary

A dictionary containing overall results, such as pass/fail counts.

Type:: dict[str, int | float]

class DtsRunResultDict

Bases: TypedDict

Represents the DtsRunResult results.

test_runs

A list of test run results.

Type:: list[framework.test_result.TestRunResultDict]

summary

A summary dictionary containing overall statistics for the test runs.

Type:: dict[str, int | float]

class FixtureResult

Bases: object

A record that stores the result of a setup or a teardown.

FAIL is a sensible default since it prevents false positives (which could happen if the default was PASS).

Preventing false positives or other false results is preferable since a failure is mostly likely to be investigated (the other false results may not be investigated at all).

result

The associated result.

Type:: framework.test_result.Result

error

The error in case of a failure.

Type:: Exception | None

__init__(result: Result = Result.FAIL, error: Exception | None = None)

Initialize the constructor with the fixture result and store a possible error.

Parameters:

result (Result) – The result to store.
error (Exception | None) – The error which happened when a failure occurred.

class BaseResult

Bases: object

Common data and behavior of DTS results.

Stores the results of the setup and teardown portions of the corresponding stage. The hierarchical nature of DTS results is captured recursively in an internal list. A stage is each level in this particular hierarchy (pre-run or the top-most level, test run, test suite and test case).

setup_result

The result of the setup of the particular stage.

Type:: framework.test_result.FixtureResult

teardown_result

The results of the teardown of the particular stage.

Type:: framework.test_result.FixtureResult

child_results

The results of the descendants in the results hierarchy.

Type:: collections.abc.MutableSequence[framework.test_result.BaseResult]

__init__(): Initialize the constructor.

update_setup(result: Result, error: Exception | None = None) → None

Store the setup result.

If the result is BLOCK, ERROR or FAIL, then the corresponding child results in result hierarchy are also marked with BLOCK.

Parameters:

result (Result) – The result of the setup.
error (Exception | None) – The error that occurred in case of a failure.

update_teardown(result: Result, error: Exception | None = None) → None

Store the teardown result.

Parameters:

result (Result) – The result of the teardown.
error (Exception | None) – The error that occurred in case of a failure.

get_errors() → list[Exception]

Compile errors from the whole result hierarchy.

Returns:: The errors from setup, teardown and all errors found in the whole result hierarchy.
Return type:: list[Exception]

to_dict(): Convert the results hierarchy into a dictionary representation.

add_result(results: dict[str, int])

Collate the test case result to the given result hierarchy.

Parameters:: results (dict[str, int]) – The dictionary in which results will be collated.

generate_pass_rate_dict(test_run_summary) → dict[str, float]

Generate a dictionary with the PASS/FAIL ratio of all test cases.

Parameters:: test_run_summary – The summary dictionary containing test result counts.
Returns:: A dictionary with the PASS/FAIL ratio of all test cases.
Return type:: dict[str, float]

class DTSResult

Bases: BaseResult

Stores environment information and test results from a DTS run.

Test run level information, such as testbed, the test suite list and DPDK build configuration (compiler, target OS and cpu),

Test suite and test case results,

All errors that are caught and recorded during DTS execution.

The information is stored hierarchically. This is the first level of the hierarchy and as such is where the data form the whole hierarchy is collated or processed.

The internal list stores the results of all test runs.

__init__(output_dir: str, logger: DTSLogger)

Extend the constructor with top-level specifics.

Parameters:

output_dir (str) – The directory where DTS logs and results are saved.
logger (DTSLogger) – The logger instance the whole result will use.

add_test_run(test_run_config: TestRunConfiguration) → TestRunResult

Add and return the child result (test run).

Parameters:: test_run_config (TestRunConfiguration) – A test run configuration.
Returns:: The test run’s result.
Return type:: TestRunResult

add_error(error: Exception) → None

Record an error that occurred outside any test run.

Parameters:: error (Exception) – The exception to record.

process() → None

Process the data after a whole DTS run.

The data is added to child objects during runtime and this object is not updated at that time. This requires us to process the child data after it’s all been gathered.

The processing gathers all errors and the statistics of test case results.

get_return_code() → int

Go through all stored Exceptions and return the final DTS error code.

Returns:: The highest error code found.
Return type:: int

to_dict() → DtsRunResultDict

Convert DTS result into a dictionary format.

The dictionary contains test runs and summary of test runs.

Returns:: A dictionary representation of the DTS result
Return type:: DtsRunResultDict

class TestRunResult

Bases: BaseResult

The test run specific result.

The internal list stores the results of all test suites in a given test run.

compiler_version: The DPDK build compiler version.

dpdk_version: The built DPDK version.

sut_os_name: The operating system of the SUT node.

sut_os_version: The operating system version of the SUT node.

sut_kernel_version: The operating system kernel version of the SUT node.

__init__(test_run_config: TestRunConfiguration)

Extend the constructor with the test run’s config.

Parameters:: test_run_config (TestRunConfiguration) – A test run configuration.

add_test_suite(test_suite_with_cases: TestSuiteWithCases) → TestSuiteResult

Add and return the child result (test suite).

Parameters:: test_suite_with_cases (TestSuiteWithCases) – The test suite with test cases.
Returns:: The test suite’s result.
Return type:: TestSuiteResult

property test_suites_with_cases: list[framework.test_result.TestSuiteWithCases]

The test suites with test cases to be executed in this test run.

The test suites can only be assigned once.

Returns:: The list of test suites with test cases. If an error occurs between the initialization of TestRunResult and assigning test cases to the instance, return an empty list, representing that we don’t know what to execute.

property ports: list[framework.testbed_model.port.Port]: Get the list of ports associated with this test run.

property sut_info: framework.testbed_model.os_session.OSSessionInfo | None: Get the SUT OS session information associated with this test run.

property dpdk_build_info: framework.testbed_model.sut_node.DPDKBuildInfo | None: Get the DPDK build information associated with this test run.

to_dict() → TestRunResultDict

Convert the test run result into a dictionary.

The dictionary contains test suites in this test run, and a summary of the test run and information about the DPDK version, compiler version and associated ports.

Returns:: A dictionary representation of the test run result.
Return type:: TestRunResultDict

class TestSuiteResult

Bases: BaseResult

The test suite specific result.

The internal list stores the results of all test cases in a given test suite.

test_suite_name

The test suite name.

Type:: str

__init__(test_suite_with_cases: TestSuiteWithCases)

Extend the constructor with test suite’s config.

Parameters:: test_suite_with_cases (TestSuiteWithCases) – The test suite with test cases.

add_test_case(test_case_name: str) → TestCaseResult

Add and return the child result (test case).

Parameters:: test_case_name (str) – The name of the test case.
Returns:: The test case’s result.
Return type:: TestCaseResult

to_dict() → TestSuiteResultDict

Convert the test suite result into a dictionary.

The dictionary contains a test suite name and test cases given in this test suite.

class TestCaseResult

Bases: BaseResult, FixtureResult

The test case specific result.

Stores the result of the actual test case. This is done by adding an extra superclass in FixtureResult. The setup and teardown results are FixtureResults and the class is itself a record of the test case.

test_case_name

The test case name.

Type:: str

__init__(test_case_name: str)

Extend the constructor with test case’s name.

Parameters:: test_case_name (str) – The test case’s name.

update(result: Result, error: Exception | None = None) → None

Update the test case result.

This updates the result of the test case itself and doesn’t affect the results of the setup and teardown steps in any way.

Parameters:

result (Result) – The result of the test case.
error (Exception | None) – The error that occurred in case of a failure.

to_dict() → TestCaseResultDict

Convert the test case result into a dictionary.

The dictionary contains a test case name and the result name.

add_result(results: dict[str, int])

Add the test case result to the results.

The base method goes through the hierarchy recursively and this method is here to stop the recursion, as the TestCaseResult are the leaves of the hierarchy tree.

Parameters:: results (dict[str, int]) – The dictionary to which results will be collated.

class TextSummary

Bases: object

Generates and saves textual summaries of DTS run results.

The summary includes: * Results of test cases, * Compiler version of the DPDK build, * DPDK version of the DPDK source tree, * Overall summary of results when multiple test runs are present.

__init__(dts_run_result: DTSResult)

Initializes with a DTSResult object and converts it to a dictionary format.

Parameters:: dts_run_result (DTSResult) – The DTS result.

save(output_path: Path)

Generate and save text statistics to a file.

Parameters:: output_path (Path) – The path where the text file will be saved.

class JsonResults

Bases: object

Save DTS run result in JSON format.

__init__(dts_run_result: DTSResult)

Initializes with a DTSResult object and converts it to a dictionary format.

Parameters:: dts_run_result (DTSResult) – The DTS result.

save(output_path: Path)

Save the result to a file as JSON.

Parameters:: output_path (Path) – The path where the JSON file will be saved.