Commit 2a8d78d5 authored by Alex Deucher's avatar Alex Deucher Committed by Jonathan Corbet

Documentation: x86: rework IOMMU documentation

Add preliminary documentation for AMD IOMMU and combine
with the existing Intel IOMMU documentation and clean
up and modernize some of the existing documentation to
align with the current state of the kernel.
Signed-off-by: default avatarAlex Deucher <alexander.deucher@amd.com>
Acked-by: default avatarJoerg Roedel <jroedel@suse.de>
Link: https://lore.kernel.org/r/20220422192656.624920-1-alexander.deucher@amd.comSigned-off-by: default avatarJonathan Corbet <corbet@lwn.net>
parent f5461124
...@@ -22,7 +22,7 @@ x86-specific Documentation ...@@ -22,7 +22,7 @@ x86-specific Documentation
mtrr mtrr
pat pat
intel-hfi intel-hfi
intel-iommu iommu
intel_txt intel_txt
amd-memory-encryption amd-memory-encryption
amd_hsmp amd_hsmp
......
=================== =================
Linux IOMMU Support x86 IOMMU Support
=================== =================
The architecture spec can be obtained from the below location. The architecture specs can be obtained from the below locations.
http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf - Intel: http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
- AMD: https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf
This guide gives a quick cheat sheet for some basic understanding. This guide gives a quick cheat sheet for some basic understanding.
Some Keywords
- DMAR - DMA remapping
- DRHD - DMA Remapping Hardware Unit Definition
- RMRR - Reserved memory Region Reporting Structure
- ZLR - Zero length reads from PCI devices
- IOVA - IO Virtual address.
Basic stuff Basic stuff
----------- -----------
ACPI enumerates and lists the different DMA engines in the platform, and ACPI enumerates and lists the different IOMMUs on the platform, and
device scope relationships between PCI devices and which DMA engine controls device scope relationships between devices and which IOMMU controls
them. them.
What is RMRR? Some ACPI Keywords:
-------------
- DMAR - Intel DMA Remapping table
- DRHD - Intel DMA Remapping Hardware Unit Definition
- RMRR - Intel Reserved Memory Region Reporting Structure
- IVRS - AMD I/O Virtualization Reporting Structure
- IVDB - AMD I/O Virtualization Definition Block
- IVHD - AMD I/O Virtualization Hardware Definition
What is Intel RMRR?
^^^^^^^^^^^^^^^^^^^
There are some devices the BIOS controls, for e.g USB devices to perform There are some devices the BIOS controls, for e.g USB devices to perform
PS2 emulation. The regions of memory used for these devices are marked PS2 emulation. The regions of memory used for these devices are marked
...@@ -33,57 +35,79 @@ regions will fail. Hence BIOS uses RMRR to specify these regions along with ...@@ -33,57 +35,79 @@ regions will fail. Hence BIOS uses RMRR to specify these regions along with
devices that need to access these regions. OS is expected to setup devices that need to access these regions. OS is expected to setup
unity mappings for these regions for these devices to access these regions. unity mappings for these regions for these devices to access these regions.
How is IOVA generated? What is AMD IVRS?
---------------------- ^^^^^^^^^^^^^^^^^
Well behaved drivers call pci_map_*() calls before sending command to device The architecture defines an ACPI-compatible data structure called an I/O
that needs to perform DMA. Once DMA is completed and mapping is no longer Virtualization Reporting Structure (IVRS) that is used to convey information
required, device performs a pci_unmap_*() calls to unmap the region. related to I/O virtualization to system software. The IVRS describes the
configuration and capabilities of the IOMMUs contained in the platform as
well as information about the devices that each IOMMU virtualizes.
The IVRS provides information about the following:
The Intel IOMMU driver allocates a virtual address per domain. Each PCIE - IOMMUs present in the platform including their capabilities and proper configuration
device has its own domain (hence protection). Devices under p2p bridges - System I/O topology relevant to each IOMMU
share the virtual address with all devices under the p2p bridge due to - Peripheral devices that cannot be otherwise enumerated
transaction id aliasing for p2p bridges. - Memory regions used by SMI/SMM, platform firmware, and platform hardware. These are generally exclusion ranges to be configured by system software.
IOVA generation is pretty generic. We used the same technique as vmalloc() How is an I/O Virtual Address (IOVA) generated?
but these are not global address spaces, but separate for each domain. -----------------------------------------------
Different DMA engines may support different number of domains.
We also allocate guard pages with each mapping, so we can attempt to catch Well behaved drivers call dma_map_*() calls before sending command to device
any overflow that might happen. that needs to perform DMA. Once DMA is completed and mapping is no longer
required, driver performs dma_unmap_*() calls to unmap the region.
Intel Specific Notes
--------------------
Graphics Problems? Graphics Problems?
------------------ ^^^^^^^^^^^^^^^^^^
If you encounter issues with graphics devices, you can try adding If you encounter issues with graphics devices, you can try adding
option intel_iommu=igfx_off to turn off the integrated graphics engine. option intel_iommu=igfx_off to turn off the integrated graphics engine.
If this fixes anything, please ensure you file a bug reporting the problem. If this fixes anything, please ensure you file a bug reporting the problem.
Some exceptions to IOVA Some exceptions to IOVA
----------------------- ^^^^^^^^^^^^^^^^^^^^^^^
Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff). Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
The same is true for peer to peer transactions. Hence we reserve the The same is true for peer to peer transactions. Hence we reserve the
address from PCI MMIO ranges so they are not allocated for IOVA addresses. address from PCI MMIO ranges so they are not allocated for IOVA addresses.
AMD Specific Notes
------------------
Graphics Problems?
^^^^^^^^^^^^^^^^^^
If you encounter issues with integrated graphics devices, you can try adding
option iommu=pt to the kernel command line use a 1:1 mapping for the IOMMU. If
this fixes anything, please ensure you file a bug reporting the problem.
Fault reporting Fault reporting
--------------- ---------------
When errors are reported, the DMA engine signals via an interrupt. The fault When errors are reported, the IOMMU signals via an interrupt. The fault
reason and device that caused it with fault reason is printed on console. reason and device that caused it is printed on the console.
See below for sample.
Kernel Log Samples
------------------
Boot Message Sample Intel Boot Messages
------------------- ^^^^^^^^^^^^^^^^^^^
Something like this gets printed indicating presence of DMAR tables Something like this gets printed indicating presence of DMAR tables
in ACPI. in ACPI:
ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0 ::
ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0
When DMAR is being processed and initialized by ACPI, prints DMAR locations When DMAR is being processed and initialized by ACPI, prints DMAR locations
and any RMRR's processed:: and any RMRR's processed:
::
ACPI DMAR:Host address width 36 ACPI DMAR:Host address width 36
ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000 ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
...@@ -92,13 +116,14 @@ and any RMRR's processed:: ...@@ -92,13 +116,14 @@ and any RMRR's processed::
ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
When DMAR is enabled for use, you will notice.. When DMAR is enabled for use, you will notice:
PCI-DMA: Using DMAR IOMMU ::
-------------------------
Fault reporting PCI-DMA: Using DMAR IOMMU
^^^^^^^^^^^^^^^
Intel Fault reporting
^^^^^^^^^^^^^^^^^^^^^
:: ::
...@@ -107,9 +132,20 @@ Fault reporting ...@@ -107,9 +132,20 @@ Fault reporting
DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000 DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
DMAR:[fault reason 05] PTE Write access is not set DMAR:[fault reason 05] PTE Write access is not set
TBD AMD Boot Messages
---- ^^^^^^^^^^^^^^^^^
Something like this gets printed indicating presence of the IOMMU:
::
iommu: Default domain type: Translated
iommu: DMA domain TLB invalidation policy: lazy mode
AMD Fault reporting
^^^^^^^^^^^^^^^^^^^
::
- For compatibility testing, could use unity map domain for all devices, just AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xffffc02000 flags=0x0000]
provide a 1-1 for all useful memory under a single domain for all devices. AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xffffc02000 flags=0x0000]
- API for paravirt ops for abstracting functionality for VMM folks.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment