Peer-to-peer (P2P) routing allows PCIe transactions to be routed directly between functions or ports within the same device or switch, without traversing the upstream link to the root complex. The PCIe library provides two templates for P2P routing support:
device_local_p2p — for multi-function (MF) and SR-IOV endpoint
devicesswitch_p2p — for PCIe switch downstream portsBoth templates hook into the upstream egress path. When a function or downstream port sends a transaction upstream, the P2P routing logic in the upstream egress path intercepts it and checks whether the destination can be reached locally. If so, the transaction is redirected to the local ingress path of the matching target.
ACS can be used to control peer-to-peer routing, like preventing peer-to-peer routing between certain switch ports and forcing everything upstream. Currently peer-to-peer routing is NOT yet connected to the ACS feature in the PCIe library, meaning ACS cannot yet control this peer-to-peer routing automatically. This will be added in the future.
P2P routing supports two types of transactions:
PCIE_Type_Mem): routed based on BAR
address ranges.PCIE_Type_Msg with
PCIE_Msg_Route_ID): routed based on device ID (BDF) matching for
endpoints, or bus number range matching for switches.Other transaction types (e.g., broadcast messages, IO, or configuration) are not P2P-routable and are always forwarded upstream.
The device_local_p2p template enables P2P routing between functions
within the same PCIe device. This covers:
To use device_local_p2p, instantiate it on the device.
By default can_route_p2p returns (PCIE_P2P_ROUTE_UPSTREAM, t), where t is the
transaction that was passed into the call. Hence, P2P routing must be
explicitly enabled by overriding this method. This reflects the fact
that device-local P2P routing is an optional feature in real hardware.
Override can_route_p2p in the
device_local_p2p group to control which function pairs are allowed
to communicate:
is device_local_p2p;
group device_local_p2p {
// we have a single PF bank at top level and 512 VFs
// in group "vfs". The test will only use two VFs
// hence our callback only deals with these two
method can_route_p2p(uint16 rid, uint16 tid, transaction_t *t, uint64 addr)
-> (pcie_p2p_routing_decision_t, transaction_t*) default {
// we allow the PF to talk to all VFs
if (rid == pf.get_device_id())
return (PCIE_P2P_ROUTE_P2P, t);
// we do not allow direct access to the PF from the VFs
if (tid == pf.get_device_id())
return (PCIE_P2P_ROUTE_UPSTREAM, t);
// we allow VF1 to talk to VF0 but not the other way around
if (rid == vfs.vf[1].get_device_id()
&& tid == vfs.vf[0].get_device_id())
return (PCIE_P2P_ROUTE_P2P, t);
return default(rid, tid, t, addr);
}
}
For memory transactions, the routing logic performs a lookup
if the targeted address is covered by any BAR range in any config
bank of the sending device.
For ID-routed messages, it checks if the targeted ID is owned by
the sending device. If this is the case, it then calls can_route_p2p
and if that returns true, sends the transaction directly to the
device-local destination and does not send it upstream.
See device_local_p2p for the full template reference.
The switch_p2p template enables P2P routing between downstream
ports of a PCIe switch. When one downstream port sends a transaction
upstream, the switch checks whether the destination matches the
address range or bus number range of another downstream port and, if
so, redirects the transaction to that port’s ingress.
To use switch_p2p, instantiate it on each downstream port
subdevice and set parameter downstream_ports in the switch_p2p
group to the sequence of all downstream ports in the switch.
The sequence does not have to be sorted. The switch_p2p template
will sort it according to the BDFs of the downstream ports.
subdevice dsp[i < 4] is (pcie_downstream_port, switch_p2p) {
in group switch_p2p {
param downstream_ports = (each pcie_downstream_port in (dev));
// [...]
By default can_route_p2p returns (PCIE_P2P_ROUTE_P2P, t) for switches, where t is the
transaction that was passed into the call, because P2P
routing is the standard behavior of a switch. Override can_route_p2p to restrict
which port pairs are allowed. The example below also shows how you
could use an attribute to select default switch behavior
or filtered behavior.
// Still in scope of dsp[i < 4] is (pcie_downstream_port, switch_p2p)
// and group switch_p2p
method can_route_p2p(uint64 target_port_index, transaction_t *t, uint64 addr)
-> (pcie_p2p_routing_decision_t, transaction_t *) default {
log info, 3: "Switch can_route_p2p: target_port=%04x addr=%016x",
target_port_index, addr;
if (!filter_peer_to_peer.val) // an attribute to control selective P2P
return default(target_port_index, t, addr);
// note: source port (index i) is counted from 0 while target_port_index is
// counted from 1, so we compare to i+1
switch (i + 1) {
// port 1 can peer to peer route to itself and target port 2
case 1:
if (target_port_index == 1 || target_port_index == 2)
return (PCIE_P2P_ROUTE_P2P, t);
break;
// port 2 can peer to peer route only to itself
case 2:
if (target_port_index == 2)
return (PCIE_P2P_ROUTE_P2P, t);
break;
// port 3 can only peer to peer route to port 4
case 3:
if (target_port_index == 4)
return (PCIE_P2P_ROUTE_P2P, t);
break;
// port 4 can peer to peer route to everyone
case 4:
return (PCIE_P2P_ROUTE_P2P, t);
default:
break;
}
return (PCIE_P2P_ROUTE_UPSTREAM, t);
}
} // end group switch_p2p
} // end subdevice dsp[i < 4]
Note that target_port_index is 1-based in order to keep it aligned
to port numbering in ACS where port index 0 is always the upstream
port of a switch. Hence, the downstream port indices start at index 1.
Index 1 is always the downstream port with the lowest BDF, index 2
is the next higher BDF and so on and so forth.
For memory transactions, the switch matches against the Type 1 BAR memory ranges of each downstream port. For ID-routed messages, it checks whether the target bus number falls within the secondary/subordinate bus number range of a downstream port.
See switch_p2p for the full template reference.