• Alexey Kardashevskiy's avatar
    powerpc/powernv/npu: Add compound IOMMU groups · 0bd97167
    Alexey Kardashevskiy authored
    At the moment the powernv platform registers an IOMMU group for each
    PE. There is an exception though: an NVLink bridge which is attached
    to the corresponding GPU's IOMMU group making it a master.
    
    Now we have POWER9 systems with GPUs connected to each other directly
    bypassing PCI. At the moment we do not control state of these links so
    we have to put such interconnected GPUs to one IOMMU group which means
    that the old scheme with one GPU as a master won't work - there will
    be up to 3 GPUs in such group.
    
    This introduces a npu_comp struct which represents a compound IOMMU
    group made of multiple PEs - PCI PEs (for GPUs) and NPU PEs (for
    NVLink bridges). This converts the existing NVLink1 code to use the
    new scheme. >From now on, each PE must have a valid
    iommu_table_group_ops which will either be called directly (for a
    single PE group) or indirectly from a compound group handlers.
    
    This moves IOMMU group registration for NVLink-connected GPUs to
    npu-dma.c. For POWER8, this stores a new compound group pointer in the
    PE (so a GPU is still a master); for POWER9 the new group pointer is
    stored in an NPU (which is allocated per a PCI host controller).
    Signed-off-by: default avatarAlexey Kardashevskiy <aik@ozlabs.ru>
    [mpe: Initialise npdev to NULL in pnv_try_setup_npu_table_group()]
    Signed-off-by: default avatarMichael Ellerman <mpe@ellerman.id.au>
    0bd97167
pci.h 7.63 KB