Writing a software tracker consists of creating two objects, a tracker and a mapper, plus adding a composition class.
The idea is that the tracker keeps track of the machine state, passes that data on to the tracker state in the OS Awareness framework, which then passes this on to the mapper that maps the tracker data into a node tree. The framework provides interfaces for users to read and get updates for the node tree. More about using OS Awareness can be found in Analyzer User’s Guide. For a tracker writer the framework provides interfaces for reading and getting updates to the system, the machine state.
The directions of communication between different parts of OS Awareness is shown in figure 49.
The tracker state in the framework is used as a step in the communication between the tracker and the mapper. The concept of having both a tracker and a mapper that communicates via the framework gives some advantages:
The framework will combine different updates from the tracker into as few updates as possible for the mapper:
The framework will take care of storing and checkpointing the tracker state data.
The output of one tracker could be represented in different ways using different mappers.
The framework will help to provide error checks and logging.
See section 50.2.1.1 for information about the interface the tracker uses to add tracker state to the framework and section 50.3.2 for interfaces the mapper can use to retrieve tracker state.
A tracker could create a node tree directly, without using the tracker state or having a mapper. But this documentation will show the recommended way, which is having both a tracker and a mapper.
The node tree is how the software tracker represents the tracked target software (or hardware) state. The mapper will start by creating one root node to which it can then add nodes to. The tree can have any width or depth of nodes. It is up to the mapper to represent the tracked data in a good way.
See section 50.3.3.1 for the interface used to create a node tree.
The node-tree command of the software object can be used to print the node
tree.
See Analyzer User’s Guide for more information about how a node tree looks and how the user can script against it.
The easiest way to get started with writing a software tracker in C is to use
the project-setup command with the –osa-tracker option in a project. This
will add skeleton code for a tracker, a mapper and a composition class which can
be used as a base to implement the new tracker from.
The source code includes an example of how to implement a tracker and mapper,
sample-linux-tracker. The goal is to introduce the reader to important OS
Awareness concepts and interfaces required to create a new tracker and mapper.
The example is a Linux tracker with limited functionality. The sample Linux
tracker will work for single threaded programs running a single core version of
qsp-x86 using the qsp-linux-common.simics target script.
To get this sample into a project run the following command:
project-setup --copy-module sample-linux-tracker
The sample Linux tracker will introduce the following concepts:
The Linux tracker example is not meant to explain how the Linux kernel stores it task structures and other data. For information about that, turn to the Linux kernel source code or some documentation about the kernel.
The tracker is the object that tracks software or hardware by monitoring parts of the machine state (memory, registers, etc) and collecting data it is interested in. The tracker will provide the data it has collected to the framework for it to be mapped into a node tree by a mapper object.
The tracker will pass on its state to the framework using the
osa_tracker_state interface. See 50.1.1 for information
about the tracker state in the framework.
The tracker and mapper will communicate with entities, which are a unique number that the tracker decides, this can for example be the address of a task struct (as for the Linux tracker) or something else that can be represented as a unique number. Associated with the entity there will be a dictionary of properties for that entity. The keys of the dictionary must be of string type. The mapper will then interpret each entity and represent their properties as nodes in the node tree.
This section will specify how to use this interface while writing a tracker, for details about each function and its arguments see the API reference manual.
Begins a transaction. This must be done before modifying any entity. The
transaction will be ended with the end function call. When all transactions
have ended the updates will be sent to any mapper subscribing to updates from
this tracker.
Several stacked calls to begin is allowed as long as the last started
transaction is ended before any previously started transaction is ended. For
stacked transactions the initiator of the first transaction will be passed as
the initiator to the mapper.
Add a new entity with some specified properties. This entity id must not exist in the framework since before.
This should be called when the tracker finds a new entity, for example a new task if that is what the tracker is tracking. The properties argument will contain the data, in form of a string-indexed dictionary, for that entity.
Remove an existing entity. The entity id must exist in the framework.
Will, for example, be called when a task is found to be removed, if that is what the tracker is tracking.
Remove all entities for the tracker that started the transaction.
Updates the properties of an entity that has been added earlier. This can update one or more properties for the entity. If the key of the property does not exist since before this property will be added. To remove a property set the value of that property to nil.
Similar to update, but this can just update one property for the entity. When
using this function there is no need to build up a dictionary for the updated
property, instead the key and value are passed to the function.
Events are used for passing data that is not persistent, so this will not be stored in the framework or checkpointed. The event will just be passed on to the mapper when the transaction is ended. As an example, this is how the Linux tracker handles system calls.
In order for a tracker to be able to read out the current state of the system and get notifications for changes of the target system state the OS Awareness framework provides two different interfaces.
This interface provides functions to get the current state for system. This includes reading the registers of a processor, reading memory and getting processor mode.
See the API documentation for details about what exact functions this interface provides and how to use them.
This interface provides functions for getting callbacks when the state of the system changes. This can provide notification for when memory is accessed, a register is written, processor mode changes or an exception occurs.
Notifications are provided as callbacks, meaning that the tracker should provide a function that the parent will call once the state changes.
Every function in this interface will return an id which can be used to cancel
the notification by providing this id to the cancel function. Remember to
cancel all notifications once a processor is removed or when the tracker is
disabled.
See the API documentation for details about what functions this interface provides and how the callback functions should be implemented.
This interface is called from the trackers parent to do such things as enabling or disabling the tracker and adding or removing processors to the tracker. The tracker needs to implement this interface and all its functions in order to work. How to use the functions is described below, more information about the functions and their arguments can be found in the API reference manual.
This function is called when the parent wants to enable the tracker. The tracker can either have a previous state (after a checkpoint, see section 50.5) or an empty state at this time.
If the tracker is enabled without any previous state the enable function will
be called before add_processor is called so no processors will be available.
At this point the tracker cannot do all that much except maybe adding a root
entity to pass some information on to the mapper about its current
configuration.
If on the other hand there was a previous state there will be no calls to
add_processor after enabling the tracker. This means that the tracker might
have to install any callbacks to the osa_machine_notification interface at
this point. If the tracker adds any entities directly at enable this should not
be done if the tracker already has a state as this will have been checkpointed
in the framework.
This functions should return true if enabling the tracker went well.
This function is called by the framework or a parent tracker to tell the tracker that the provided processor is available for the tracker.
If the tracker is on the top level with the framework as parent, this will be
called after enable for each processor in the system.
For a guest tracker, the parent tracker will decide when to add or remove
processors. This makes it possible to reflect the software state. For a
hypervisor system, the hypervisor tracker will typically do this when the guest
OS is scheduled or unscheduled. However, the hypervisor tracker must first have
enabled the guest tracker by calling enable first.
When this is called the tracker should start tracking machine state for the added processor. The current state can be read and notifications should be installed.
The function should return true if adding the processor went well.
This is called from a parent tracker when a processor no longer is available to the system that the tracker is tracking. The tracker should then remove any machine state notifications it has installed on this processor and do any necessary tracker state updates related to this processor.
This function should return true if removing the processor went well.
> [!NOTE]
> This function is currently called before `disable` for all processors, but
> that might change so make sure that `disable` handles removing all
> remaining processors.
When receiving this callback the tracker should clear all its internal state, including removing all processors.
The tracker state in the framework will be cleared once the call to
clear_state is done. So the tracker does not have to remove its entities, but
it is allowed.
This will be called when a checkpoint with OS Awareness state (framework was enabled when taking the checkpoint, see 50.5) has been loaded, but the framework was not enabled before advancing the simulation.
After this has been called the tracker should be able to be enabled just as if it hadn’t had any previous state.
This will only be called while the tracker is disabled. At this point no notifications should be installed.
This should do the same things as clear_state when it comes to clearing
internal state. But as this is called when the tracker is enabled it should also
cancel any notifications that it has installed for the
osa_machine_notification interface.
A mapper object is used to map tracker data into a node tree. The node tree should represent the state of the tracked system. See 50.1.2 for more information about the node tree.
One tracker could potentially have different mappers that build up different node trees from the data provided.
This section describes the interfaces that need to be implemented by a mapper. For detailed information about the functions and their arguments see the API documentation for that specific interface.
The mapper must implement the osa_mapper_control interface which provides
functions that are called from the OS Awareness framework when enabling,
disabling or when mapper state should be cleared. Below are descriptions of what
each of these functions are meant to do when called.
This function is called when the framework (or a parent mapper) wants to enable the mapper.
Upon enable, the mapper can have a state, if it was enabled after loading a checkpoint taken with an active OS Awareness configuration. In this case, the mapper does not usually need to do anything as the state is already stored in the framework. See 50.5 for information about how the mapper handles checkpointing.
When the mapper is enabled without having a previous state, the following should
be done:
1. Create the root node by calling create in the osa_node_tree_admin
interface (50.3.3.1) of its parent.
1. Add any additional base nodes by calling add in the
osa_node_tree_admin interface with the newly created root node as
parent.
1. Subscribe to the tracker associated with the mapper, using the
subscribe_tracker function in the osa_tracker_state_notification
interface (50.3.2.2) of its parent.
1. Get any existing entities for the tracker that the mapper is associated
with using the get_entities function in the osa_tracker_state_query
interface (50.3.2.1) of its parent and add nodes
for these entities. Under normal circumstances there should be no
existing entities when the mapper is enabled as the mapper is enabled
before the tracker by the framework, but if the mapper is a guest under
a parent mapper this might not be the case. Doing this is a good idea,
and it should usually be done in the same way as when adding an entity
upon a tracker_update call.
The function should return true if enable went well.
This function is called when the framework (or parent mapper) needs the mapper
to clear its state so that the mapper can later be enabled again from a clean
state. This is called when the state has been restored from a checkpoint, but
the simulation advances before the OS Awareness framework is enabled, see
50.5 for more information. In this case the mapper needs to clear
all its internal state so that when enable is called it will perform all the
steps for the case when the mapper is enabled without a state.
The mapper does not need to remove nodes in the node tree when clearing state as the framework will clear the node tree state afterwards, but is allowed to remove nodes if it wants to.
This function will only be called when the mapper is disabled.
This function is called when the framework (or a parent mapper) wants to disable
the mapper. This should do the same things as the clear_state function in this
interface.
The difference from this compared to clear_state is that this function is
called when the mapper is enabled. In general disable and clear_state will
call a common function for clearing state, but if anything additional needs to
be done for an enabled mapper that should be done just for the disable
function.
This interface only provides one function; tracker_updated. This function is
called from the OS Awareness framework when the tracker state has been updated
for a tracker that the mapper is subscribing to.
The function will provide a changeset, providing data for what has been updated in the tracker state, and an initiator telling what processor initiated this change.
The mapper is supposed to map added, removed or modified entities to nodes in
the node tree and update the node tree using the osa_node_tree_admin interface
(50.3.3.1).
The mapper can also filter out unwanted changes, such as entities or properties that it does not care about. The mapper itself can also modify properties or add additional properties to suit what data it wants to provide the node tree with.
The changeset also provides events that the tracker has sent out. Events are
not stored in the tracker state; they contain non-persistent data which the
mapper can use to call event in the osa_node_tree_admin interface
(50.3.3.1) to provide the node tree with an event.
This can, for example, be a system call in a Linux tracker. The mapper could
also choose to filter out an event or modify it before passing it on to the node
tree.
For details on the tracker_updated function see the API reference manual.
Both functions in this interface are optional, if not implemented the function should be set to nil in the interface.
Needs to be implemented in order for the list command to work. See details on
format in the API documentation.
Only needed for stacked mappers that have other mappers as guests. If used, then this should return the mapper object for which a certain node belongs to. If this function is not implemented the framework assumes that the node belongs to the mapper object that created the node tree.
The mapper is supposed to build up node trees from the tracker state. In order to do so the mapper must be able to get the current tracker state and be able to get updates once the tracker state is updated. The OS Awareness framework provides two interfaces for reading the current tracker state and for getting notifications when the tracker state is updated.
This interface provides functions for getting tracker state from the OS
Awareness framework. The functions it provides are the get_entity function,
which allows getting the properties of a certain entity, and the get_entities
which returns a dictionary with all the entities for a certain tracker. Neither
of these functions are necessary for writing a working mapper, but the
get_entities function is recommended to use during the enable phase to make
sure that the mapper works as a guest to some other mapper.
This interface provides functions for subscribing to updates from a certain
tracker. The subscribe_tracker function should be called during the enabling
phase to tell the framework that this mapper wants to receive tracker state
updates (through calls to the tracker_updated function in the
osa_mapper_admin interface) for a certain tracker.
The tracker object to subscribe to is usually specified by having an attribute in the mapper.
The unsubscribe_tracker function is normally not used, this will tell the
framework that the mapper no longer wants updated for the specified tracker.
This is only useful if a mapper subscribes to several trackers and no longer
wants updated for some of them, for example a stacked mapper were one of the
guests is removed.
When the framework is disabled it will automatically remove any subscriptions so
the next time it is enabled again the mapper will have to call
subscribe_tracker again.
The purpose of a mapper is to map the tracker state into a node tree. The OS Awareness framework provides an interface for creating and modifying a node tree which the mapper should use.
The osa_node_tree_admin interface is part of the OS Awareness framework and is
the part used to create node trees. The mapper calls the functions of this
interface to create and maintain a node tree.
All modifications to the node tree has to be inside a transaction, which is
started with the begin function and ended by the end function of this
interface. Transactions can be stacked as long as the order is so that a
transaction started later is ended before any earlier started transaction.
Notifications for the node tree changes will be sent out when all transactions
have ended.
A mapper should start out by creating a root node using create once the mapper
is enabled. This will give the mapper the root node id of its node tree. After
that the mapper can add child nodes by using the add function. Nodes can be
updated using the update or set_property functions, removed using the
remove function and set as activated or deactivated using the activate and
deactivate functions.
Removing a node will also remove all of its child nodes. So removing the root node will end up removing all nodes in the node tree.
The activate function marks a node as active for a certain processor for a
node in the node tree. The deactivate function deactivates an active node.
Only one node can be set as active per processor at once, so when setting a new
active node there is no need to call deactivate on the previously active node
as that will be deactivated when a new node is activated. For an active node all
its ancestors will be seen as active on that processor in the node tree too.
A mapper can also do some formatting of outputted data for certain properties
using the register_formatter function. This formatter will apply when
get_formatted_properties in the osa_node_tree_query interface is called. One
case were this can be used is when a number should be displayed in hex format.
There are some node properties that have special meaning. These are not necessary, but might be needed for certain built-in functions and services to work.
In order for node path patterns and most OS Awareness provided commands to work a node must contain a name property. The value of this property must be a string.
It is recommended that the mapper sets the *name* property on all its nodes.
A list containing names of properties that uniquely identify nodes on the same level. If not given, the name is assumed to be unique.
A boolean that is used by some services to check if a node can be active on more than one processor at a time.
Used by some services to identify the top most node that a node shares the memory map for. For example, a thread node will have the parent process node as the memory space in the Linux tracker.
Some services also requires a leaf node to only be active on one processor at a time.
In order for the tracker and mapper to work with the OS Awareness framework a tracker composition object should be added. This allows adding and removing a tracker to a system and implements so that the tracker and mapper will receive control signals from the framework.
The class will be written in python and it should inherit from the
framework.tracker_composition class of the os-awareness module. So the
composition source should import the following:
from simmod.os_awareness import framework
Then the composition class should be created as follows:
class empty_software_tracker_comp(framework.tracker_composition):
One current limitation in the system is that the composition class must always
be named as the tracker class with a _comp suffix. In the example above this
composition class would be for the tracker with class empty_software_tracker.
Should return the tracker object. This can return Nil in case there is no tracker.
Should return the mapper object. This can return Nil in case there is no mapper.
This should create and configure the tracker and mapper objects. Commonly this sets the parent object of the tracker and mapper and for the mapper it sets which tracker it should be associated with.
The framework will set three attributes which can be used when configuring the objects:
- <span class="term">osa\_admin</span>
The OS awareness object that implements the OS awareness interfaces.
- <span class="term">tracker\_domain</span>
The parent tracker. This is what a parent attribute of the tracker would usually be set to. - mapper_domain
The parent mapper. This is what a parent attribute of the mapper would usually be set to.
It is also recommended that the composition class implements the info and status commands by adding the
_infoand_statusmethods.
In order to be able to support parameters the composition class need to
implement the osa_parameters interface. This will be done by adding the
following line, followed by the implementation of the three interface functions:
class osa_parameters(pyobj.Interface):
Should return a list on the following formats upon success:
[True, [tracker kind, parameters dictionary]]
The first element in the list is a boolean returning true if reading parameters went well.
The tracker kind should be the class of tracker associated with the composition.
This must be the name of the composition class without the _comp suffix,
otherwise the framework will not be able to automatically create the correct
tracker composition object when loading parameters.
If getting the parameters fails then the first value in the returned list should
be false while the second element of the list should be an error message of what
went wrong:
[False, error string]
The include_children argument is only used for stacked trackers. See API reference manual for information.
Should set parameters for the tracker. This function is provided with parameters on the format:
```
[tracker kind, parameters dictionary]
```
The tracker kind and parameters dictionary are the values saved when doing
get_parameters. It is then up to this function to make sure the tracker and/or
mapper objects are configured with these parameters.
Will tell if a certain kind of parameters are supported by the tracker. This
will in general only do the check that the kind corresponds to the tracker kind
set by get_parameters.
The details for the osa_parameters interface can be found in the API
documentation.
The OSA framework also provides a helper function, save_parameters_file, which
can be used to save a parameters file. This function is only available from
Python and located in simmod.os_awareness.framework. It takes four options,
filename, tracker_cls, parameters, and overwrite. In case of failure, a
FrameworkException exception will be raised.
It can be called like this:
framework.save_parameters_file(
"linux.params", "sample_linux_tracker", {...}, False)
Checkpointing for OS Awareness works a bit special. The enable state of the OS
Awareness framework is not checkpointed, so the framework will always be
disabled after a checkpoint is loaded. This means that if the framework is not
enabled before starting the simulation, after loading a checkpoint, the state of
the tracker, mapper and framework might be inconsistent with the target
software. So if the simulation starts before enabling the framework will send
out a clear_state call to all trackers and mappers. When receiving such a call
the tracker or mapper should clear all its internal state so that it can be
enabled from scratch again, without any state in the node tree or framework
tracker state.
If on the other hand the framework is enabled directly after loading a checkpoint the tracker and mapper should just continue as before the checkpoint.
The following should be thought about in order for trackers and mappers to work with checkpointing.
The object needs to keep track of if it has any previous state when enable is
called. This can, for example, be done by having an attribute that tells whether
or not the root node has been created (for the mapper) or a root entity has been
added (for the tracker).
If nodes already exist in the node tree no new nodes should be added when the
mapper enable function is called as these nodes will already exist in the node
tree as part of the checkpoint.
The same goes for the tracker that should not add any entities to the framework tracker state as that state will have been checkpointed. But for the tracker no machine state notifications will have been checkpointed so the tracker needs to install new notification callbacks.
The mapper needs to keep all its internal state that needs to survive a checkpoint in attributes. Specifically the mappings between entities and nodes have to be checkpointed as the mapper will receive updates on the same entities once the framework is enabled again.
When the framework is enabled after a checkpoint the tracker will not receive
any add_processor function calls as the tracker is expected to have saved the
processors in its internal state.
In order for a tracker or mapper to support snapshots there are some things to think about:
osa_machine_query cannot be called from an attribute
setter when the simulation is restoring state. This will give an error.osa_machine_notification interface are not
checkpointed, so the tracker will have to cancel notifications that are no
longer valid and install new notifications for the restored state.In order to help models support snapshots, the OS Awareness framework provides
an interface, osa_micro_checkpoint. This interface has two functions:
Called after Simics has restored the state for all objects. At this point it is
okay to call the functions in the osa_machine_query interface.
At this point the tracker should update its notifications in the
osa_machine_notification interface to match the newly restored state.