2.7 test-runner

SYNOPSIS

test-runner [options] [suite-pattern ...]

DESCRIPTION

List or run Simics tests.

Simics tests are placed in test suites, which are directories containing a SUITEINFO file. By default, the test-runner command searches for test suites in the directories test, targets and modules of the Simics project, and in the test directory of installed Simics packages.

If the program is run with the flag --list, the names of all found test suites are listed:

$ ./bin/test-runner --list
modules/bar/test
modules/foo/test
test/t15_checkpoints: [simics]/test/t15_checkpoints

The name of a suite is the path of the directory, relative to the project or Simics installation path, using / as directory separator. If the test suite is located in the Simics installation, then the absolute path is printed as well.

The flag --test can be used to list the individual tests of each test suite:

$ ./bin/test-runner --tests --suite=modules/bar/test
Suite: modules/foo/test
    s-foo
    s-info-status

If neither --list nor --tests is passed, all found test suites will be run. For example, the command ./bin/test-runner --suite=modules/bar/test will run all tests of the test suite modules/bar/test. If test-runner is run with no arguments, then all tests in all test suites will be run.

Other than the argument --suite, a suite pattern can also be used to filter test suites. If one or more suite patterns are provided, only the suites matching at least one of the patterns will be run. A suite matches a suite pattern if the pattern is a substring of the suite's name. For example, if there is a suite modules/foo/test, then the command ./bin/test-runner foo will run that suite, and possibly other suites as well.

When an environment variable TEST_PATTERN is set, the value will be used as a pattern to filter the tests to be run within the test suite. The pattern is interpreted according to Python's regular expression syntax. For example, if the variable is set as export TEST_PATTERN="s-.+DMA-", then test-runner will run tests named s-xyz-DMA-read and s-abc-DMA-write, and skip tests named Xs-xyz-DMA-read or s-abc-DMAwrite.

The variable can also be set to multiple patterns with the semicolon as a separator, e.g. export TEST_PATTERN="pattern0;pattern1". This will run tests that match any one of the given patterns.

By default, all tests run sequentially. When the -j option is passed, automatically added s-*.py scripts, may be executed in parallel. The tests will be executed in an unspecified order, so -j should only be used if you know that there are no dependencies between the tests of the suite.

OPTIONS

-h, --help

show this help message and exit

--details

print detail help information

--logdir

the directory tests save logfiles in, defaults to [project]/logs/tests

--basedir

the base directory where the modules containing the tests exist, defaults to 'modules'

--moduledirs

directories with modules containing tests,separated by ';'

--project-only

disable the tests from Simics packages

-l, --list

list the suites matching the patterns

-t, --tests

list tests matching the given patterns

--tag TAG

Run all suites where a tag in SUITEINFO matches TAG

--suite SUITE

specify a test suite to be executed

-n NAME, --name NAME

specify a test name to be executed

-j N, --jobs N

run up to N Simics script tests in parallel in each test suite; N will be automatically set to the number of processors on the machine if not specified

-v, --verbose

print more verbose information when running

--py3

Deprecated, no effect.

--py2

Deprecated, no effect.

RETURN VALUE

The exit status is 0 if selected test suites succeed, 1 if there is an internal error, and 2 if any test fails.

SEE ALSO

The test framework in the API Reference Manual.