This chapter describes how to write functional tests for device models using the test framework in Simics Model Builder. The test framework supports device model testing as well as testing entire target systems, but this chapter focuses on the former.
Tests are written in Python and can be run as part of building your devices, or as a separate step. The basics of writing and running tests are described in section 4.2.3. This chapter goes into more detail, but assumes that you know the basics described in that section.
The Simics Reference Manual contains more detailed information about the testing framework in Simics, including the API of the libraries used in this chapter, as well as the command line program test-runner that allows the user to run tests in a project in a flexible way.
Functional tests of a device model should only test the model under test. This means that they should depend on as little as possible of the surrounding system. The test framework includes the Python libraries dev_util and pyobj which help you achieve this goal. They provide ways to interact with the device model and to fake the parts of the system the device model interacts with.
Testing should be an integrated part of normal device model development, preferably written before the implementation. Beyond the basic regression testing that automated tests give you, writing tests first helps you catch errors quickly, and helps you focus the implementation effort and design.
This chapter starts with an overview of the testing process. Then it shows you where to find some example models which include tests. Finally it goes into more detail about how to write tests.
The basics of a functional test of a device model is to write Python code which interacts with the device in a way that other parts of a real system would and to check that the device behaves as expected.
The most basic form of interaction with a device is reading and writing its registers. You can also interact with a device by calling methods in interfaces it implements. To check that the device behaves as expected, you check that the register accesses and interface calls have the expected results. A device may also perform its own interface calls as a side effect of register accesses or interface calls, for example to access memory, raise interrupt signals, or send network packets. To verify this, add fake objects implementing the interfaces and check that the device performs the expected calls. The pyobj library helps with this interfacing with the model. The stest library is used to check that the results are as expected.
Some Simics objects that a device model collaborates with cannot be easily faked. Models depending on timing, for example models which implement timers, or models timing when performing DMA, need a clock to keep track of the time and handle events. This is easier to do with an instance of the clock class instead of trying to fake it.
If a model uses an image object to store large data structures, it is easiest to consider this an internal implementation detail and provide a real image image in the test.
Simics Model Builder includes a couple of sample devices which include tests: DS12887 and DEC21140A-dml. The tests do not cover all the functionality, but are provided as a demonstration of basic test techniques. To get access to the tests, run the following:
project$ bin/project-setup --copy-module=DS12887
and
project$ bin/project-setup --copy-module=DEC21140A-dml
Now you can try running the tests:
project$ make test
After copying, the tests can be found in the [project]/modules/DS12887/test and [project]/modules/DS12887/test directories.
The DS12887 suite consists of three tests, sharing some definitions from a common Python file (common.py). Each test sets up a simple configuration consisting of an instance of the device and a small number of other objects. No actual processor is used; a clock object is used to allow time to pass.
The DEC21140A-dml suite contains similar tests for the DEC21140A Ethernet controller. These tests are far from exhaustive, but they demonstrate more in depth the techniques of testing a device interacting with other objects; these objects are faked in the test. The suite also shows how to check that the interface calls performed by the model happen in the expected order.
The key to useful functional tests of a device model is to test the entire model, but only the model. That is, the set-up for each test should not be bigger than needed.
When you use project-setup to create a new device skeleton, you also get a test template in the test directory of the device. You should extend this template with your functional tests for the model.
A test suite for a Simics module is placed in the test directory of the module's source. It has the following contents:
SUITEINFO
The existence of this file is what tells the test system that the directory contains a test suite. It needs to exist, but can be empty. It can optionally contain configuration parameters for the test suite, but it is usually empty.README or README.txt
An optional file, which is ignored by the test system. It should contain a description of the suite, in human-readable format.tests.py
An optional file which generates the set of tests in the suite. If it does not exist, the test system will automatically generate a test for every file named s-*.py, where * can be any string. This is usually all that is needed for functional tests of modules.s-*.py
Each file whose name matches this pattern is by default considered a test by the test system, and will be run in its own Simics process.This is just a high level description of the files in a test suite. See the Simics Reference Manual for the details.
Ideally, only one instance of the model under test should be needed for each test. In this case, just create the instance using SIM_add_configuration.
my_dev = pre_conf_object('dev', 'my_dev_class_name')
my_dev.attr1 = 'foo'
my_dev.attr2 = 4711
SIM_add_configuration([my_dev], None)
# Replace our pre_conf_object reference with
# a reference to the Simics obj
my_dev = conf.my_dev
In practice the device model may require connections to other models. These models can frequently be faked. This means that instead of creating the objects that would normally be used, an extremely simple class that only implements the necessary interfaces is used. This can usually be done in Python.
For example, many devices need to be able to signal interrupts. Normally an actual interrupt controller object, implementing the signal interface, is used. When testing, it is advantageous to create a small class implementing the same interface using pyobj:
import stest
import pyobj
# Create a simple device class implementing the signal interface
class FakePic(pyobj.ConfObject):
class raised(pyobj.SimpleAttribute(0, 'i')):
'''An attribute to store the signal state'''
# The signal interface
class signal(pyobj.Interface):
def signal_raise(self):
self._up.raised.val += 1
def signal_lower (self):
self._up.raised.val -= 1
Create such fake objects before the model under test. Then you can configure the model under test to connect to the fake object.
# Create a device instance of the fake PIC
fake_pic = pre_conf_object('fake_pic', 'FakePic')
# Create device and connect it to fake PIC
my_dev = pre_conf_object('dev', 'my_dev_class_name')
my_dev.pic = fake_pic
SIM_add_configuration([my_dev, fake_pic], None)
my_dev = conf.my_dev
fake_pic = conf.fake_pic
# Run test code
...
# Verify that interrupts were raised
stest.expect_equal(fake_pic.raised, 1, 'signal not raised')
The advantages of using fake objects compared to actual models are:
Objects that cannot be faked are those that cannot be implemented in Python because they use data types that cannot be translated from C.
If you cannot wrap an interface in Python, you can write a simple device in DML which translates between pseudo attributes and interface calls instead, or translates between an unwrappable interface and a wrappable one. Potentially, one might have to create a wrappable interface first. See chapter 11 in this guide. Then you can use pyobj to create devices that implement the wrappable interface.
When a configuration has been created, it is time to perform the actual tests. This requires the device to be configured, usually through register writes.
Thus we need a way of writing to the device. Attributes are not recommended for this, and will in most cases not work anyway, as attribute setters have no side-effects. The io_memory or transaction interfaces, which are used for register accesses, could be used, but require the construction of (generic) transaction objects. If the test focus is the registers, their values and their side effects, using transactions complicates the test code, requires to keep endianness in mind and has hence a higher risk of having errors in the test code itself. Here, being able to conveniently perform register accesses that trigger side effects just like accesses via the memory path will keep tests focused and simple.
The dev_util library handles this. It allows you to define Python bank proxies that wrap a register bank of the device and allow you to directly retrieve and modify register values. This also supports the use of fields.
import dev_util
from stest import expect_equal
my_device = pre_conf_object('dev', 'my_device_class')
SIM_add_configuration([my_device], None)
# Create a bank proxy for bank 'regs'of the device instance named 'dev';
# Some info to understand the following code:
# the bank has two registers r1 and r2
# r2 has the fields ctrl, flags, counter and status
regs = dev_util.bank_regs(conf.dev.bank.regs)
# Writing and reading the entire register r1
regs.r1.write(0xdeadbeef)
expect_equal(regs.r1.read(), 0xdeadbeef)
# There is no such thing as "writing only one field", so
# writing bit fields is overlaid on a full register value;
# the 'READ' value means that the current value of the
# register will be read first, giving you read-modify-write behavior.
# Note that not all fields need to be specified
regs.r2.write(dev_util.READ, ctrl = 0xA, counter = 2)
# Field reads are full register reads that extract field
# values for convenience
expect_equal(regs.r2.field.status.read(), 1)
expect_equal(regs.r2.field.flags.read(), 0x42)
# Of course, we can read/write the entire register without field values as well
regs.r2.write(0x47)
expect_equal(regs.r2.read(), 0x47)
As can be seen, registers are accessed by name through the bank proxy object. endianness and fields of registers are taken from the information retrieved from the bank. This is the recommended approach when the test focus is on proper functionality of registers.
If endianness, register offsets and the correct bit-to-field associations are part of the test focus, then using the bank register would not be able to catch such errors in the device, as all of that is extracted from the bank itself. In such a case, one can use Register_BE and Register_LE which allow the tester to define endianness, expected register offsets and bit-to-field associations. If there are disagreements between test and model regarding any of this the test would catch it.
The same example as above would then be:
import dev_util
from stest import expect_equal
my_device = pre_conf_object('dev', 'my_device_class')
SIM_add_configuration([my_device], None)
# Create a register proxies for each register
r1 = dev_util.Register_LE(conf.dev.bank.regs, # bank
0x0, # offset in bank
size=4)
r2 = dev_util.Register_LE(
dev.bank.regs, 0x4, size=4,
bitfield = dev_util.Bitfield_LE(
ctrl=(31,24), # Bits 31-24
flags=(23,5), # Bits 23-5
counter=(4,1), # Bits 4-1
status=0 # Bit 0
)
)
# Writing and reading the entire register r1
r1.write(0xdeadbeef)
expect_equal(r1.read(), 0xdeadbeef)
# There is no such thing as "just writing a field", so
# a read-modify-write behavior is implied, reading the full
# register value, then set the field values and then write it back
# Note that not all fields need to be specified
r2.write(ctrl = 0xA, counter = 2)
# Field reads are full register reads that extract field
# values for convenience
expect_equal(r2.status, 1)
expect_equal(r2.flags, 0x42)
# Quick access to a single field
# (these are implicit read-modify-write accesses again)
r2.flags = 0x66
r2.status = 0
# Of course, we can read/write the entire register without field values as well
r2.write(0x47)
expect_equal(r2.read(), 0x47)
As can be seen, registers can either have little- or big-endian byte order (Register_LE/Register_BE). Similarly, bitfields can either have little- or big-endian bit-order (Bitfield_LE/Bitfield_BE). LE/BE registers can be freely mixed with LE/BE bit fields. This works analogously to registers and fields in DML. You can read more about byte order and bit order in the application note Byte Order and Byte Swapping in Simics. Register_LE/Register_BE should only be used when testing if a model adheres to specified endiannesses, register offsets and bit-to-field associations.
It is quite common that a device performs DMA transfers. These transfers are often configured with descriptors, i.e., in-memory structures that tell the device how to perform the transfer (e.g., size and location of the transfer). A network controller would be a typical example of such a device. Network controllers are usually configured with receive and transmit descriptors.
The dev_util library provides two classes that makes it easier to test DMA transfers: the Memory class and the Layout class.
The Memory class replaces a memory-space and ram configuration. The advantage of using the Memory class over regular Simics RAM is twofold: it is possible to track which addresses have been written to; and you will get an exception if test code, or a device, tries to read from uninitialized addresses.
import dev_util
from stest import expect_equal, expect_different
mem = dev_util.Memory()
dma_dev = pre_conf_object('dev', 'my_dev_class')
dma_dev.phys_mem = mem.obj
SIM_add_configuration([dma_dev], None)
# Create a layout at address 0x1234.
# The descriptor looks like this:
#
# --------------------
# 0: | reg1 | reg2 |
# --------------------
# 4: | reg3 |
# --------------------
# 8: | f1 | f2 |////////|
# --------------------
#
# Total descriptor size is 10 bytes.
desc = dev_util.Layout_LE(
mem, 0x1234,
{'reg1' : (0, 2), # offset = 0, size = 2
'reg2' : (2, 2), # offset = 2, size = 2
'reg3' : (4, 4),
'reg4' : (8, 2,
dev_util.Bitfield_BE({'f1' : (15, 8),
'f2' : (7, 0)})
)})
# Initialize the descriptor
desc.reg1 = 0xffff
desc.reg2 = 0xabab
desc.reg3 = 0xdeadbeef
desc.reg4.write(0, f1=5, f2=27)
# Fill memory with test data (this is the data the device
# will read, in addition to the descriptor above).
mem.write(0xabab, tuple(i for i in xrange(256)))
# Run test
...
# If the device updates the descriptor with status information,
# we should check that now
expect_different(desc.reg4.f2, 29)
# Check that data was copied from 0xabab to 0xffff
expect_equal(mem.read(0xffff, 256), range(256))
One thing that should be noted is that it is not possible to access (read from or write to) a field in an uninitialized register. This is because the entire register must be loaded before altering a single field, and the Memory class will raise an exception when reading uninitialized memory. To handle this, either do a write to the entire register before setting the field, or set all the fields in a single write operation with a default value of, for example, zero:
layout.reg.write(0)
layout.reg.field = 1
or
layout.reg.write(0, field=1)
Testing devices also requires invoking the interfaces they implement. The interfaces related to bank accesses have been covered above, but a device can also implement other interfaces. If an interface can be Python wrapped, you can call it directly on the device or port via <dev>.iface.<interfaceName>.<functionName>(<functionArguments>) or <dev>.port.<portName>.iface.<interfaceName>.<functionName>(<functionArguments>).
For example, if your device implements the signal interface at the device level as well as in a port called reset then you can trigger it as follows:
dev = pre_conf_object('dev', 'my_dev_class')
SIM_add_configuration([dev], None)
dev = conf.dev
# raise signal on device
dev.iface.signal.signal_raise()
# spike signal on a port
dev.port.reset.iface.signal.signal_raise()
dev.port.reset.iface.signal.signal_lower()
# lower signal on device
dev.iface.signal.signal_lower()
Sending a transaction means to just call an interface on a device with a transaction object. The basics have been covered in the previous sections, but transaction object handling may need some more explanations. In many cases you will not need to manually create transactions, because you can either use the bank_regs or Register_LE/BE from dev_util or map you device into a memory space and use the provided read/write functions of the memory space.
However, if your device makes use of custom transaction atoms, you will have to manually create transactions and send them into your device. Assume you have defined a custom atom called extended_id in a module name extended-id-atom, then the steps to create and send a transaction is as follows.
import simics
# Create the device under test
dev = pre_conf_object('dev', 'my_dev_class')
SIM_add_configuration([dev], None)
dev = conf.dev
# Load the module that defines the custom atom
SIM_load_module('extended-id-atom')
# Create a 4 byte write transaction with extended_id = 3000
txn = simics.transaction_t(size=4, write=True,
value_le=0xdeadbeef,
extended_id=3000)
# Send transaction to offset 0x420 in bank 'regs'
exc = simics.SIM_issue_transaction(dev.bank.regs, txn, 0x420)
If the device under test sends transactions with custom atoms whose values have to be checked, you can create a pyobj based device as described in the section 16.3.2 that implements the transaction interface. In there you can inspect the state of transaction atoms and possibly then forward them to a test memory as described in the section 16.3.4. Below is an example of this.
import simics
import stest
import pyobj
import dev_util
# load the module that defines the custom atom
SIM_load_module('extended-id-atom')
# Create a simple device class that can inspect transactions and forward them
class txn_checker(pyobj.ConfObject):
class last_extended_id(pyobj.SimpleAttribute(0, 'i')):
'''An attribute to store the last seen extended id'''
# The transaction interface
class transaction(pyobj.Interface):
def issue(self, txn, addr):
self._up.last_extended_id.val = txn.extended_id
return self._up.to_mem.val.iface.transaction.issue(txn, addr)
class to_mem(pyobj.SimpleAttribute(None, 'o', simics.Sim_Attr_Optional)):
'''Connect to the memory space'''
mem = dev_util.Memory()
dev = pre_conf_object('dev','my_dev_class')
chk = pre_conf_object('chk','txn_checker')
dev.to_mem = chk
chk.to_mem = mem.obj
SIM_add_configuration([dev, chk],None)
dev = conf.dev
chk = conf.chk
regs = dev_util.bank_regs(dev.bank.regs)
# trigger a txn to addr 0x4242 by writing to r1
regs.r1.write(0x4242)
# we expect 0xdeadbeef to be written to the address we just wrote to r1
stest.expect_equal(mem.read(0x4242, 4),
list(0xdeadbeef.to_bytes(4, 'little')),
"Incorrect value written"
)
# we expect the last extended id to be 0x4711
stest.expect_equal(chk.last_extended_id, 0x4711, "Incorrect extended ID seen")
Tests should obviously be as complete as possible, covering all the implemented functionality of a device model. Anything that is implemented but not tested risks to stop working at any moment due to later changes to the code.
On the other hand, it is important to make tests complete quickly. Ideally, a programmer should be able to re-run all tests for a device model even after small changes, without having to wait a long time for the tests to finish. If the tests take too long, they will not be run as often and will then be less effective in helping development.
As a rule of thumb, the time taken by a test should be dominated by the Simics start up time, and all tests of a single device model no more than half a minute, preferably even shorter. Most device tests should only take a few seconds on a modern machine. Exceptions should only be allowed if there are very good reasons for them.
This means that not everything can be tested, but clever testing can cover a lot of functionality in a short time.
For tests to be useful and maintainable, you should document what is being tested, both for the entire test suite (in a README file) and each test (as comments in the respective files). Any omissions should be noted; this is important to assess coverage.
The test code itself is subject to the same commenting as any other program code; i.e., describe anything unusual and the reasoning behind the design, but do not over-comment the code itself.
When a test fails, it must be reasonably easy to find out exactly what has failed and why. This can be achieved by a combination of error messages when the failure occurs, logging done during the test run, and comments in the test code. Raising a Python exception at the point of error also helps locating it, since a complete traceback will be printed automatically. The checking functions in stest already raise exceptions if the check fails.
When dividing tests into subtests, the following should be taken into account:
As with all programming, it is always a good idea to factor out common test code into separate Python files. This can include configuration, definition of fake objects, functions for register access and other helper functions. See the sample tests for how it can be done.