Single Root I/O Virtualization and Sharing (SR-IOV) DML template reference
PCIe Modeling Library  / 

Link Training

Link training in the PCIe library is essentially a simplification of the T1 and T2 ordered sets that are actually communicated in a real PCIe link. The template pcie_phy adds a port to a device with the name phy. This is intended to be used as a target for transactions which are related to the physical layer in PCIe. This port is only added in PCIe Downstream Components (not to be confused with Downstream Port, the Downstream component on a Link is the component farther from the Root Complex) which includes Endpoints and Upstream Switch Ports.

Transactions in this layer are expected to have a BDF in address[31:16]. The bus number is unused as the transactions only traverse over one link. The function is currently also unused as the transaction will end up in the top-level of a device.

Note that there is no implementation of a Link Training and Status State Machine (LTSSM).

The Upstream component is the initiator of link training. This is done by sending a transaction to the phy port of the Downstream component. For this to happen when devices are added to the pcie-downstream-port associated with the Upstream component, the pcie_link_training template must be instantiated on the Upstream Component. Otherwise it will only happen when the Retrain Link field in the Link Control register is written to.

The Upstream component advertises its target link speed adding the pcie_link_negotiation atom to the transaction. It also advertises that it supports Flit Mode by including the pcie_link_flit_mode atom to the transaction. Upstream components are also capable of handling Device Readiness Status (DRS) messages. Link training is triggered in the do_link_training() method of the exp_link capability (which is instantiated in the exp_capability template if the has_links param is set to true). The do_link_training() method issues a transaction to the Upstream component which includes atoms for negotiation of the link properties.

The link training method inspects the following register (all in the PCI Express Capability Structure) fields during handling of link training:

The link training method sets the following register fields during handling of link training:

The current default transaction handler of the phy port handles certain aspects of link training. This includes link speed and width negotiation, as well as Flit Mode negotiation. The handler also sends a DRS PCIe message to the link training initiator (Upstream Component). The handler thus inspects the following atoms:

The handler inspects the following register (all in the PCI Express Capability Structure) fields during handling of link training:

The handler sets the following register fields during handling of link training:

The Downstream component finally sends a DRS message to the Upstream component if both link speed/width and Flit Mode negotiation were successful. This PCIe message includes the pcie_message_type atom set to PCIE_Vendor_Defined_Type_1, pcie_msg_vdm_vendor_id_t atom set to 0x0001 and pcie_msg_vdm_subtype_t atom set to PCIE_VDM_Device_Readiness_Status.

The link training handling can be overridden, both on the initiator side of the link (Upstream component) and the Downstream component. One reason for wanting to override the link training implementation provided by this library could be that the negotiated aspects of the link provided might not be enough. An override could thus add additional custom atoms and act upon those in both the Upstream and Downstream component.

Overriding Upstream Components

To override the link training of the link training initiator, it makes the most sense to override the default implementation of the do_link_training_custom(uint16 device_id, atom_t *extra_atoms) and the link_training_result_custom(const transaction_t *t, exception_type_t exc) -> (bool) methods in the exp_link capability.

The most sensible override for do_link_training_custom() would be to create a new atom array where any potential atoms passed to the method to begin are added, as well as any custom atoms that required the override. Then call the default() implementation. See the example below for more details.

dml 1.4;
device rp_link_training_override;

import "pcie/common.dml";

is pcie_root_port;
is pcie_link_training;

bank pcie_config {
    is defining_exp_capability;
    param exp_offset = 0x40;
    param exp_next_ptr = 0;
    param exp_dp_type = PCIE_DP_Type_RP;
    group exp {
        param has_links = true;
        group link {
            method do_link_training_custom(uint16 device_id, atom_t *extra_atoms) ->
                (bool) {
                local int num_extra_atoms;
                if (extra_atoms != NULL) {
                    local const atom_t *a;
                    for (a = extra_atoms; a->id != Sim_Atom_Id_list_end; a++)
                        num_extra_atoms++;
                }

                /*
                  Let's assume we want to add one atom in this override
                  We need two extra slots in the array to also fit ATOM_list_end(0)
                */
                local atom_t atoms[num_extra_atoms + 2];
                local int j;

                atoms[j++] = ATOM_pcie_ecs(PCIE_ECS_Extended);
                for (local int i = 0; i < num_extra_atoms; i++)
                    atoms[j++] = extra_atoms[i];
                atoms[j] = ATOM_list_end(0);

                return default(device_id, atoms);
            }
        }
    }
}

link_training_result_custom is called after the transaction has been processed by the Downstream component. This call is done by the default implementation of the do_link_training() method. An override of the link_training_result_custom() could for example inspect values of custom atoms if they were to be pointers expected to be set by the Downstream component. It is also possible to override the methods link_training_width_speed_result() and link_training_flit_mode_result() if it is desired to modify the default behavior of the library regarding link speed, width and Flit Mode negotiation in an Upstream component.

Overriding Downstream Components

To override the link training in an Downstream component, it makes the most sense to override the shared method handle_link_custom(const transaction_t *t) -> (exception_type_t) method of the phy port in Downstream components. This handler should be treating any custom atoms that may potentially have been added by the Upstream component. If it is desired to modify the default behavior of library for link speed, width and Flit mode negotiation (handle_link_flit_mode() and handle_link_flit_mode()).

Take the standard PCIe switch distributed as part of Simics Base. It indicates support of link speeds and widths using extended capability structures. Additionally, the supported values have been set in the link registers of the PCI Express Capability Structure. It also supports Hot-Plug along with having an attention button and a power indicator. The latter two are useful for Hot-Plug removal and software reporting status of Hot-Plug operations. Support for these features are enabled using params found in the exp_capability and exp_slot templates. This will result in the device emitting interrupts for Slot and Link related events if software has enabled it. In the case where interrupts might be generated by firmware in the device rather by hardware in the device, shared methods found in the exp_slot template can be overridden to fit a certain use case.

Figure 20. PCIe Switch supporting Hot-Plug and Link Training
dml 1.4;

device standard_pcie_switch;
param classname = "standard-pcie-switch";

param desc = "standard PCIe switch";

param documentation = "A standard PCIe switch with 4 downstream slots that"
                    + " contains the mandatory capabilities for a PCIe"
                    + " function in all ports.";

import "pcie/common.dml";

param pcie_version = 6.0;


template switch_port is pcie_port {
    bank pcie_config {
        register device_id { param init_val = 0x0370; }
        register vendor_id { param init_val = 0x8086; }
        register capabilities_ptr { param init_val = 0x40; }

        register bar0 @ 0x10 is (memory_base_address_64) {
            param size_bits = 14;
        }

        is defining_pm_capability;
        param pm_offset = capabilities_ptr.init_val;
        param pm_next_ptr = pm_offset + 0x08;

        is defining_msix_capability;
        param msix_offset = pm_next_ptr;
        param msix_next_ptr = msix_offset + 0x0C;
        param msix_num_vectors = 32;
        param msix_table_offset_bir = 0;
        param msix_pba_offset_bir = (0x10 * msix_num_vectors) << 3;
        param msix_data_bank = msix_data;

        is defining_exp_capability;
        param exp_offset = msix_next_ptr;
        param exp_next_ptr = 0x0;
        group exp {
            param has_links = true;
            group link {
                param max_link_speed = PCIE_Link_Speed_32;
                param max_link_width = PCIE_Link_Width_x8;
            }
        }

        is defining_dlf_capability;
        param dlf_offset = 0x100;
        param dlf_next_ptr = dlf_offset + 0x0C;
        is defining_pl16g_capability;
        param pl16g_offset = dlf_next_ptr;
        param pl16g_next_ptr = pl16g_offset + 0x28;
        param pl16g_max_link_width = PCIE_Link_Width_x8;
        param pl16g_max_lanes = PCIE_Link_Width_x8;
        is defining_pl32g_capability;
        param pl32g_offset = pl16g_next_ptr;
        param pl32g_next_ptr = 0;
        param pl32g_max_lanes = PCIE_Link_Width_x8;
    }

    bank msix_data is msix_table {
        param msix_bank = pcie_config;
    }
}

subdevice usp is (pcie_upstream_port, switch_port) {
    bank pcie_config {
        param exp_dp_type = PCIE_DP_Type_UP;
    }
}

subdevice dsp[i < 4] is (pcie_downstream_port, pcie_link_training,
                         switch_port, post_init) {
    bank pcie_config {
        param exp_dp_type = PCIE_DP_Type_DP;

        group exp {
            param has_hotplug_capable_slot = true;
            param has_attention_button_slot = true;
            group slot {
                param has_power_indicator = true;
            }
        }
    }


    method post_init() {
        pcie_device.connected(usp.downstream_port.obj, i << 3);
    }
}

Note that the downstream ports also have to instantiate the template pcie_link_training for link training support. This will ensure that when a device is connected, link training will be initiated to the device on the other side of the link. For link training to be successful, the device on the other side of the link also has to have a function(s) that contain link attributes in their PCIe Express Capability Structure (for example by setting the params max_link_speed and max_link_width in the link group) as done for the switch in the example above.

Single Root I/O Virtualization and Sharing (SR-IOV) DML template reference