This section of the documentation focuses on providing a guide for how to
migrate PCIe devices written with the legacy PCI library (devices that import
pci/common.dml for endpoint implementations or pci/pcie-to-pcie.dml for port
implementations). Additionally a short overview of some high-level differences
between the two libraries is provided. For a proper understanding of this
library, it is important to read the other sections of the PCIE Modelling
Library Documentation.
The new PCIe library utilizes modern Simics features, such as transaction_t,
atom_t, map_target_t and transaction_translator in its implementation. The
transaction_t data type has a number of advantages compared to the
generic_memory_operation data type that the prior library was based on. It
provides better traceability, easier debugging, asynchronous operations and the
ability to easily add inspectable extensions. Improved performance for an device
doing DMA is also a benefit. In addition, atom_t maps well to different TLP
headers or fields.
Furthermore, the new library is implemented from the ground up using modern DML 1.4. While there is a 1.4 version of the old library, it was initially implemented using 1.2 and the 1.4 is a compatibility port. With modern DML, more PCIe specific functionality can be handled by the library without the user of the library having to do it. An example of this is having to reference all the functions banks for multi-function endpoint in a param. Similarly with BAR registers of a function having to be referenced in param. It also simplifies for the user to override aspects of the library without changes necessarily needed in the library itself or having to fork the library.
The library also provides support for partial migration of parts of a PCIe
topology implemented using the legacy PCI library.
pcie-downstream-port-legacy, which can be utilized in switch and root ports
implemented with the new PCIe library, allows for connecting devices that are
implemented using the legacy PCI library. Similarly, the
legacy-upstream-pcie-adapter allows for connecting devices implemented with
the new PCIe library, to a port implemented using the legacy PCI library.
pci-bus and pcie-downstream-portThe legacy PCI library, which implements a PCI(e) hierarchy using a pci-bus or
pcie-bus. These are devices that implement the pci_bus Simics interface and
represent a PCI(e) bus. An endpoint or root/switch port would then reside on
such a bus by being set in the bus' pci_devices attribute. Say you have a
switch, it has an internal bus stemming from the upstream port where only the
upstream downstream ports may reside. Then each downstream port creates a new
bus underneath it where endpoints or upstream ports would reside. Each bus in
this case would be a pci-bus or pcie-bus object when using the legacy PCI
library. You could say that the legacy PCI(e) library was bus oriented i.e. the
main abstraction was the pci-bus or pcie-bus and that stems from that it was
originally developed for PCI which is bus oriented and PCIe support was added on
top of it. The new library drops the bus centric legacy and is modelled around
ports. This maps more easily to the current PCIe specification which to a large
extent is focused on point to point links with ports at both ends (with the
obvious exception of PCIE switches). Hence, with the new library the main
abstractions are instead pcie-downstream-port or
pcie-downstream-port-legacy. Each root/switch port implemented with the new
library (a device or subdevice that instantiates one of the template
pcie_<root/upstream/downstream>_port) will also create a subdevice object of
the class pcie-downstream-port (can be overridden to the legacy compatible
variant or a custom variant). Devices would then reside on the devices
attribute of the pcie-downstream-port object or (pci_devices attribute in
the case of connecting a legacy device to pcie-downstream-port-legacy).
pcie-downstream-port:s, unlike pci-bus, does not require creating
memory-space objects for each of cfg, mem and io space, as it is handles
this internally. This simplifies component code from
bus = self.add_pre_obj('pci_bus', ‘pcie-bus’)
bus.memory_space = self.add_pre_obj('pci_mem', 'memory-space’)
bus.conf_space = self.add_pre_obj('pci_conf', 'memory-space')
bus.io_space = self.add_pre_obj('pci_io', 'memory-space')
bus.pci_devices = […]
to
dp = self.add_pre_obj(‘pcie_dp', ‘pcie-downstream-port’)
dp.devices = […]
Additionally, when using pci-bus, even after placing a PCIe port device in the
pci_devices attribute, one also has to set the secondary_bus of the port
device to the bus as well. This is not needed with pcie-downstream-port as the
port implemented with the new library only has to be put in the devices or
pci_device attribute of the pcie-downstream-port(-legacy).
Refer to the High level design section of the documentation to get a more elaborate high-level overview of the new PCIe library.
The new PCIe library supports many more features from more recent versions of the PCIe specification. No new features are being implemented for the old library as it is considered legacy.
This section describes common steps for migrating different PCIe devices implemented with the legacy PCI library. Keep in mind that the device that is being migrated might be doing some custom utilization of the legacy library, and migrating such things is most likely not covered by this section. However, reading these steps, along with the rest of the documentation for this library, might provide clues on how to go about migrating that custom logic.
Migration typically involves writing and rewriting a large number of params.
To guard against misspellings, consider enabling the provisional language feature explicit_param_decls.
import pci/common.dml ->
import pcie/common.dmlpcie_device, instead instantiate
pcie_endpoint. Also remove the pci_hotplug param if defined.pcie_version param to a suitable value.pci_config bank to pcie_config.base_address_registers param in the pcie_config bankmemory_base_address_64 or memory_base_address_32. Remove the
map_func params from these registers and add a new param map_obj.
This param should be an object that should be mapped by this bar
register. The object will start at offset 0 in the mapped area by the BAR
register. This object could for instance be a bank (example where the
device has a bank defined in dml with the name "my_bank"): param map_obj = my_bank.obj.no_base_address_64 or no_base_address_32 templates, delete them.function param and the function_mapped_bank template
instantiations in banks that were mapped by BAR registers.pci_raise_interrupt(...)/pci_lower_interrupt(...) to either
pcie_config.raise_legacy_interrupt()/pcie_config.lower_legacy_interrupt()
Note that if the device instead uses
pci_raise_interrupt_pin()/pci_lower_interrupt_pin() and uses INTx
emulation in a way that is not allowed by the PCIe spec (for example
having a single function being able to assert multiple virtual pins). One
has to utilize the pcie_config.set_legacy_interrupt() method along with
checking and setting the applicable INTx registers in the config bank:
dml 1.4;
device legacy_irq_dev;
import "pcie/common.dml";
is pcie_endpoint;
template custom_legacy_irq {
method assert_intx(pci_interrupt_pin_t pin) {
this.status.ins.set(1);
if (this.command.id.val == 0)
this.set_legacy_interrupt(pin, 1);
}
attribute raise_legacy_irq is (pseudo_attr, uint64_attr) {
method set(attr_value_t val) throws {
local pci_interrupt_pin_t pin = SIM_attr_integer(val);
if ((pin >= PCI_INTERRUPT_INTA) && (pin <= PCI_INTERRUPT_INTD))
pcie_config.assert_intx(pin);
}
}
}
bank pcie_config is custom_legacy_irq {
// ...
}
Where pin would be the +1 value of that would have been passed to
pci_raise_interrupt_pin().
Calls to pci_data_from_memory(...)/pci_data_to_memory(...) should be
replaced with calls to
pcie_config.memory.read_bytes()/pcie_config.memory.write_bytes().
Calls to pci_value_from_memory(...)/pci_value_to_memory(...) should be replaced with calls to pcie_config.memory.read()/pcie_config.memory.write().
Alternatively rewrite them to use bytes_t objects and then use read_bytes()/write_bytes().
pcie_config.msi.raise()/pcie_config.msi.lower().msix_data_bank param in the MSI-X capability must be set. This should be a
bank that has instantiated the msix_table template. This bank in turn
should set a param msix_bank to a bank that is associated with this MSIX
table (in a single function endpoint that would be pcie_config). Then
instead of calling
pci_config.pci_raise_msix_interrupt()/pci_config.pci_lower_msix_interrupt(),
one would call pcie_config.msix.raise()/pcie_config.msix.lower().Similar process to the single function endpoint migration process. Steps that differ will be mentioned below.
pci_mf_device, instead import
pcie_multifunction_endpoint. Also remove the param pcie_device=true from
the device (and pci_hotplug if defined).mf_banks param.mf_fun param with function
(with the same value).pci_config_type_0_mf instantiation with type_0_bank template.providing_legacy_interrupts and
providing_dma_methods. Legacy interrupts and DMA can be achieved similarly
to the single endpoints explained previously (call the interrupt or memory
methods defined in the config bank).defining_pcie_ari_capability_v3) and instead use
defining_ari_capability.capability register
in the ari group of the function bank.Similar process to the single/multi-function endpoint migration. Additional steps described below.
pci_config or banks that instantiate
pci_config_type_0_mf) that are associated with virtual functions.sriov_vf_bars param and the sriov_total_vfs register.sriov (created by the
instantiation of defining_sriov_capability). Have these registers
still instantiating the vf_bar_64 or the vf_bar_32 template (there's
a new implementation for these templates in the new library).sriov group.
It's value should be set with param init_val.get_offset() method in the sriov group.get_stride() method in the sriov group.sriov_virtual_functions in each physical function that
instantiates the defining_sriov_capability template. This param should point
to an array of banks that instantiate the vf_type_0_bank template. In each
SR-IOV BAR register, set the param map_obj to an array of banks that is equal to
the number of virtual functions.PCIe port devices are used to construct Root Complexes (root ports) and Switches
(upstream and downstream ports). With the legacy PCI library, all the
connections for a PCIe port are handled through pcie-bus. The port would
reside on a bus by being placed in the pci_devices attribute of a pcie-bus,
and the bus underneath the port, the secondary bus, would be a (different)
pcie_bus that is set in the secondary_bus attribute of the port. These
connection schemes are typically handled in a component that creates multiple
instances of PCIe port devices and pcie-bus. With the new library, there is
more flexibility in regard to how to construct a Switch or Root Complex. For
instance, it is possible to construct the them as one Simics device, where all
the ports in the device would be subdevices. One could then place the downstream
port subdevice objects in the devices attribute of the upstream port subdevice
during post_init. However, it is still possible to construct them as a
component similarly to how it is done with the legacy library. The difference
being that one only needs to create object instances of the applicable ports and
connect them to each other using the devices attribute of the
pcie-downstream-port subdevice in each port, with no need for pcie-bus like
objects.
While PCIe ports are represented as PCI-PCI bridges in Switches and Root Complexes and are PCIe function in their own right with a dedicated device/port type in the PCI Express Capability Structure, Host Bridges in a Root Complex are more loosely defined. A Host Bridge is defined is "Part of a Root Complex that connects a host CPU or CPUs to a Hierarchy". How it is implemented is not defined, meaning that different devices have likely chosen different ways of implementing it. Sometimes the Host Bridge is represented as a function in the Root Complex, other times that is not the case. This also means that the implementations of Root Complex device models will not look the same in regard to how the Host Bridge is handled.
An example would be the Host Bridge has been implemented as a function by
instantiating the pcie_device template from the legacy library and
implementing the legacy pci_upstream interface to handle upstream memory
transactions. Then it routes the transaction to some applicable device or memory
space. Migrating such a Host Bridge to the new library would entail first
treating it as a single function endpoint. Then remove the pci_upstream
interface implementation and instead implement the translator interface or
transaction_translator interface. Here, one has to keep in mind that the
legacy pci_upstream receives an old transaction type generic_transaction_t,
while the transaction_translator receives transaction of type transaction_t.
How these transactions differ from the older transaction type is out of the
scope of this documentation. An overview of transactions can be found in the
Transactions section of the
Model Builder User Guide.
Similar process to the single function endpoint migration process. Steps that differ will be mentioned below.
pci\pcie-to-pcie or pci/bridge.dml, remove these
imports (import pcie/common.dml instead).forward_io, forward_io32, forward_mem,
forward_prefetchable, forward_prefetchable64 and pci_hotplug.pcie_root_port, pcie_downstream_port or
pcie_upstream_port.param has_hotplug_capable_slot = true; in the exp group of the PCI Express Capability structure that
should be present in the pcie_config bank.Devices (both endpoints and switches) implemented with the legacy PCI library
can be connected to root/downstream ports in a PCIe hierarchy implemented with
the new library. For this to work, the root/downstream port must ensure that the
instantiated downstream port object in the root/downstream port must be of the
class pcie-downstream-port-legacy. This can be achieved by setting the
classname param accordingly in the downstream_port connect of the device
that instantiated pcie_<root/upstream/downstream>_port.