• Logan Gunthorpe's avatar
    PCI/P2PDMA: Support peer-to-peer memory · 52916982
    Logan Gunthorpe authored
    Some PCI devices may have memory mapped in a BAR space that's intended for
    use in peer-to-peer transactions.  To enable such transactions the memory
    must be registered with ZONE_DEVICE pages so it can be used by DMA
    interfaces in existing drivers.
    
    Add an interface for other subsystems to find and allocate chunks of P2P
    memory as necessary to facilitate transfers between two PCI peers:
    
      struct pci_dev *pci_p2pmem_find[_many]();
      int pci_p2pdma_distance[_many]();
      void *pci_alloc_p2pmem();
    
    The new interface requires a driver to collect a list of client devices
    involved in the transaction then call pci_p2pmem_find() to obtain any
    suitable P2P memory.  Alternatively, if the caller knows a device which
    provides P2P memory, they can use pci_p2pdma_distance() to determine if it
    is usable.  With a suitable p2pmem device, memory can then be allocated
    with pci_alloc_p2pmem() for use in DMA transactions.
    
    Depending on hardware, using peer-to-peer memory may reduce the bandwidth
    of the transfer but can significantly reduce pressure on system memory.
    This may be desirable in many cases: for example a system could be designed
    with a small CPU connected to a PCIe switch by a small number of lanes
    which would maximize the number of lanes available to connect to NVMe
    devices.
    
    The code is designed to only utilize the p2pmem device if all the devices
    involved in a transfer are behind the same PCI bridge.  This is because we
    have no way of knowing whether peer-to-peer routing between PCIe Root Ports
    is supported (PCIe r4.0, sec 1.3.1).  Additionally, the benefits of P2P
    transfers that go through the RC is limited to only reducing DRAM usage
    and, in some cases, coding convenience.  The PCI-SIG may be exploring
    adding a new capability bit to advertise whether this is possible for
    future hardware.
    
    This commit includes significant rework and feedback from Christoph
    Hellwig.
    Signed-off-by: default avatarChristoph Hellwig <hch@lst.de>
    Signed-off-by: default avatarLogan Gunthorpe <logang@deltatee.com>
    [bhelgaas: fold in fix from Keith Busch <keith.busch@intel.com>:
    https://lore.kernel.org/linux-pci/20181012155920.15418-1-keith.busch@intel.com,
    to address comment from Dan Carpenter <dan.carpenter@oracle.com>, fold in
    https://lore.kernel.org/linux-pci/20181017160510.17926-1-logang@deltatee.com]
    Signed-off-by: default avatarBjorn Helgaas <bhelgaas@google.com>
    52916982
Kconfig 4.65 KB