Single Root I/O Virtualization and Sharing (SR-IOV)

The SR-IOV capability is modelled with template sriov_capability. See Single Root I/O Virtualization (SR-IOV) Extended Capability registers for the template and its methods definitions.

Template vf_type_0_bank shall be used to implement the virtual functions, and templates vf_bar_64/vf_bar_32 should be used to implement the BARs for them.

Since an SR-IOV device contains more than one function (at least one physical and one virtual), the device that implements the SR-IOV capability shall instantiate the pcie_multifunction_endpoint template.

Physical Function (PF)

The physical function always instantiates type_0_bank and contains the SR-IOV capability, and instantiating this capability requires the implementation of the methods get_offset() and get_stride() as well as setting the virtual_functions param appropriately. The methods are self-explanatory, while setting the virtual_functions param may not be as obvious. The idea here is to set the it to a sequence of type vf_type_0_bank using the "Each-in" expressions. An example of this would be param virtual_functions = (each vf_type_0_bank in (dev));

BAR registers for the virtual functions reside within the SR-IOV capability structure. Note that a BAR register in the SR-IOV capability is used for all virtual functions that are associated with the physical function. It maps the same resource type for all virtual functions that are enabled. Implementing SR-IOV BAR registers is done by the aforementioned vf_bar_64/vf_bar_32 templates. These should reside on an offset relative to the capability base that is in line with what is specified in the PCI Express® Base Specification. Instantiation of these templates require the implementation of the get_vf_bar_map() method, which can for example return a bank object.

A complete SR-IOV capability instantiation could look like this:

dml 1.4;
device setting_virtual_functions;
param desc = "SR-IOV Example 1";
param documentation = "An example of how to instantiate the SR-IOV capability";

import "pcie/common.dml";

is pcie_multifunction_endpoint;

param number_of_vfs = 10;

subdevice PF {
    bank pcie_config is type_0_bank {
        // Other capabilities here...

        is defining_sriov_capability;
        param sriov_offset = 0x100;
        param sriov_next_ptr = 0;
        param sriov_virtual_functions = (each vf_type_0_bank in (dev));
        group sriov {
            register vf_bar_01 @ base + 0x24 is vf_bar_64 "VF BAR0-1" {
                method get_vf_bar_map(uint64 vf_number) -> (conf_object_t *) throws {
                    if ((vf_number == 0) || (vf_number > VF.len))
                        throw;
                    return VF[vf_number - 1].bar01.obj;
                }
            }
            method get_offset() -> (uint16) { return 0x100; }
            method get_stride() -> (uint16) { return 0x10; }
        }
    }
    // Other banks here...
}

subdevice VF[i < number_of_vfs] {
    bank pcie_config is vf_type_0_bank { param vf_number = i + 1; }
    bank bar01 { }
}

Virtual functions (VFs)

Virtual functions are implemented by configuration banks that instantiate the vf_type_0_bank. Note that given a number of virtual functions the SR-IOV capability supports, that number of virtual function configuration banks have to be statically allocated during compile time.

The PCIe capabilities in virtual functions should not instantiate the same capability templates as the physical functions. There are SR-IOV equivalent capability templates that are prepended with sriov_ before the capability type (for example defining_sriov_msix_capability instead of defining_msix_capability). Note that there are currently only a few SR-IOV variants of the capability templates available in the library, but more will be added in the future.

As the PF might share logical implementation details with the VFs, the subdevice structure in the example above lends it self well to not having to accommodate the logical implementation for the PF and the VFs respectively, as it should work the same for both function types. In the example below, more implementation details are added to the example above to clarify this statement.

dml 1.4;
device setting_virtual_functions;
param desc = "SR-IOV Example 2";
param documentation = "An example of how to structure an SR-IOV capable device";

import "pcie/common.dml";

is pcie_multifunction_endpoint;

param number_of_vfs = 10;

template common_function_logic {
    bank pcie_config {
        register class_code { param init_val = 0x123456; }
        register capabilities_ptr {
            param init_val = 0x40;
        }

        param msix_offset           default 0x40;
        param msix_next_ptr         default 0;
        param msix_num_vectors      default 1024;
        param msix_table_offset_bir default 0x1003;
        param msix_pba_offset_bir   default 0x5003;
        param msix_data_bank        default msix_data;
    }

    bank msix_data is msix_table {
        param msix_bank = pcie_config;
    }

    bank bar01 {
        register hello size 2 @ 0x0 is write {
            method write(uint64 value) {
                default(value);
                log info, 1: "Hello from %s", this.qname;
            }
        }
    }
}

subdevice PF is common_function_logic {
    bank pcie_config is type_0_bank {
        register vendor_id {param init_val = 0x1234; }
        register device_id { param init_val = 0xabcd; }
        register bar01 @ 0x10 is (memory_base_address_64) {
            param map_obj = PF.bar01.obj;
        }

        is defining_msix_capability;

        is defining_sriov_capability;
        param sriov_offset = 0x100;
        param sriov_next_ptr = 0;
        param sriov_virtual_functions = (each vf_type_0_bank in (dev));
        group sriov {
            register vf_device_id { param init_val = 0xabcd; }
            register vf_bar_01 @ base + 0x24 is vf_bar_64 "VF BAR0-1" {
                method get_vf_bar_map(uint64 vf_number) -> (conf_object_t *) throws {
                    if ((vf_number == 0) || (vf_number > VF.len))
                        throw;
                    return VF[vf_number - 1].bar01.obj;
                }
            }
            method get_offset() -> (uint16) { return 0x100; }
            method get_stride() -> (uint16) { return 0x10; }
        }
    }
}

subdevice VF[i < number_of_vfs] is common_function_logic {
    bank pcie_config is vf_type_0_bank {
        param vf_number = i + 1;
        is defining_sriov_msix_capability;
    }
}

Sample SR-IOV Endpoint

Refer to the sample SR-IOV device in src/devices/sample-pcie-sriov-device included with the Simics base package for a more elaborate example including two physical functions with a number of virtual functions each.