Hybrid setups where part of the platform runs outside Simics is a common use case. One challenge with hybrid platforms is to properly integrate it into Simics. There needs to be a layer working in both directions to translate transactions, events, API calls from one system to the other. This chapter focuses on integrating Emulators, external Simulators or Hardware into a Simics PCIe Hierarchy. The word Shim is defined in this chapter as the integration layer between Simics PCIe and an external process.
The Shim acts upstream as a Simics PCIe device and for downstream it will follow the IPC protocol shared with the external PCIe system. The Shim captures all upstream Simics PCIe traffic and events and converts them to the IPC protocol. The Shim supports asynchronous responses and concurrent requests made by the external device. For each transaction coming from upstream the Shim blocks Simics from progressing in time until the external device has responded to the transaction. This is done to ensure that Simics virtual simulation time is not affected by the response time of the external process.
For PCIe writes and PCIe messages the Shim will wait for the external device to have handled the event before completing the transaction to the initiator which could be a CPU or device.
For PCIe reads the Shim will wait until the external device has responded with the read payload.
While the Shim waits for a response to a PCIe transaction it sent to the external PCIe system, it can receive upstream facing PCIe transactions from the external PCIe system. The Shim will forward those requests upstream and send responses back to the external PCIe system. All of this is done in a thread safe manner.
The Shim supports both regular PCIe EPs and RCiEPs.
The Shim Frontend is a DML device appearing as a regular Simics PCIe device.
It has two purposes:
Shim C++ device on a single
interface.The Shim C++ device is responsible for:
The external-connection module is a separate generic module in Simics responsible for IPC.
It currently has classes to do communication over Unix sockets, TCP or Windows named pipes. The Shim
will interact with it over the external_connection_ctl and external_connection_events interfaces.
The Shim is not aware whether the communication is over TCP, Unix Socket or named pipes.
All of that is hidden inside the external-connection module.
The external-connection classes are unix-socket-server, tcp-server and named-pipe-server.
The setup to integrate an external PCIe hierarchy to Simics is very similar
to an Endpoint. The Shim Frontend, written in DML, is the only difference
from the Endpoint setup on the Simics side.
The Shim Frontend when connecting an external PCIe Switch represents a shell
of the USP of the switch. It snoops on write accesses to the BAR registers of the USP
similar to the Shim Frontend of the Endpoint.
But it also snoops on the (Prefetchable) Memory Base, Secondary Bus Number and
Subordinate Bus Number registers of the USP of the Switch in order to map
in the downstream resources of the USP device into Simics.
The source code for the Shim is distributed in module sample-pcie-shim.
It can be used as a starting point when integrating external PCIe subsystems.
There are two variants of the Shim Frontend. One for PCIe ports and one
for endpoints. The file pcie-shim-frontend-common.dml contains the base
templates for the port and endpoints and can be reused as is.
Files pcie-endpoint-shim-frontend.dml and pcie-port-shim-frontend.dml
defines the sample device classes for the PCIe port and endpoint frontends.
They contain BAR registers that mirror the real hardware. These two files
would need to be redefined for each project in order to match the
BARs for the external PCIe device.
The frontend does not need to mirror the capability set
of the external device since it delegates all transactions to it.
The Shim base class, ShimPcie, is defined in the files pcie-shim.h and pcie-shim.cc.
The class handles the interface calls to and from the Shim PCIe DML Frontend.
The exchanges with the external process must be implemented by the user
inheriting the ShimPcie class. A set of virtual methods are declared,
these are the entry points for the external exchange implementation.
The ShimPcie class can be reused as is.
The Simics side of the Shim consists of two parts:
First, the implementation of the Simics interfaces to receive upstream transactions
and Hot Reset. Method ShimPcie::issue receives all transactions and translates
the PCIe atoms into arguments forwarded to the different function handlers
(Config, Memory, Message, IO) named ShimPcie::forward....
The ShimPcie::forward_... functions have to be implemented by the user integrating
the external simulator/emulator/hardware.
Function ShimPcie::hot_reset must be implemented by the user.
Secondly, an implementation of functions ShimPcie::upstream_message,
ShimPcie::upstream_mem_read and ShimPcie::upstream_mem_write are part of
the ShimPcie. These functions create a corresponding Simics PCIe transaction
and forward it upstream.
The sample-pcie-shim module also contains sample code to showcase an
IPC implementation between Simics and an external PCIe subsystem. It gives
the user a good starting point for integrating the external PCIe subsystem with Simics.
File pcie-external-connection.cc defines the class PcieExternalConnection.
It inherits the base class ShimPcie and implements the IPC functionality.
The PcieExternalConnection class implements interface external_connection_event
in order to interact with the Simics external-connection module classes.
Supported IPC mechanisms in the external-connection module are
Unix Sockets, TCP or Windows Named Pipes.
File ExternalPcieFormat.h defines the packet format used for the
sample PCIe IPC traffic.
Methods:
PcieExternalConnection::write_async sends the packet to the
external process.PcieExternalConnection::wait_for_response After a request packet is sent to the external
process Simics will wait in this method for a response. This method blocks Simics from
progressing in virtual simulation time. Concurrent packet requests coming from the
external process is handled by this method as well. It called in Simics cell context.PcieExternalConnection::on_input is called whenever there is packet data to be read.
This method runs in Simics Threaded context. Only a small subset of the Simics API functions
are allowed to be executed in this context. The method reads the available data and
pushes it into a ring buffer. It then either posts a Simics event to schedule processing
of the packet in Simics Cell Context or it wakes up the blocking thread that executes
the PcieExternalConnection::wait_for_response method.PcieExternalConnection::read_input reads N number of bytes of packet data and pushes
it into the ring buffer.The C++ code is based on the Simics C++ Device API v2 library. Please check out the C++ Device API v2 manual.
The sample-pcie-shim module provides a sample component, sample_pcie_shim_comp.py
that adds the Shim Frontend and the Sample Shim in a component and registers
PciBusUpConnector to easily create and connect the objects into
the Simics PCIe hierarchy. It registers two components: sample-pcie-switch-shim-comp
to connect external switches and sample-pcie-endpoint-shim-comp to connect external
endpoints.
It can be connected to the Simics PCIe hierarchy with the below commands:
simics> load-module sample-pcie-shim
simics> new-sample-pcie-switch-shim-comp name = shim socket_type = tcp
simics> shim.connect-to dst = root_port
Connecting shim.upstream_target to root_port.slot[0]