- 21 Jul, 2017 6 commits
-
-
Kirill A. Shutemov authored
On x86, 5-level paging enables 56-bit userspace virtual address space. Not all user space is ready to handle wide addresses. It's known that at least some JIT compilers use higher bits in pointers to encode their information. It collides with valid pointers with 5-level paging and leads to crashes. To mitigate this, we are not going to allocate virtual address space above 47-bit by default. But userspace can ask for allocation from full address space by specifying hint address (with or without MAP_FIXED) above 47-bits. If hint address set above 47-bit, but MAP_FIXED is not specified, we try to look for unmapped area by specified address. If it's already occupied, we look for unmapped area in *full* address space, rather than from 47-bit window. A high hint address would only affect the allocation in question, but not any future mmap()s. Specifying high hint address on older kernel or on machine without 5-level paging support is safe. The hint will be ignored and kernel will fall back to allocation from 47-bit address space. This approach helps to easily make application's memory allocator aware about large address space without manually tracking allocated virtual address space. The patch puts all machinery in place, but not yet allows userspace to have mappings above 47-bit -- TASK_SIZE_MAX has to be raised to get the effect. Signed-off-by:
Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Dave Hansen <dave.hansen@intel.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/20170716225954.74185-7-kirill.shutemov@linux.intel.comSigned-off-by:
Ingo Molnar <mingo@kernel.org>
-
Kirill A. Shutemov authored
MPX (without MAWA extension) cannot handle addresses above 47 bits, so we need to make sure that MPX cannot be enabled if we already have a VMA above the boundary and forbid creating such VMAs once MPX is enabled. The patch implements mpx_unmapped_area_check() which is called from all variants of get_unmapped_area() to check if the requested address fits mpx. On enabling MPX, we check if we already have any vma above 47-bit boundary and forbit the enabling if we do. As long as DEFAULT_MAP_WINDOW is equal to TASK_SIZE_MAX, the change is nop. It will change when we allow userspace to have mappings above 47-bits. Signed-off-by:
-
Kirill A. Shutemov authored
Rename these helpers to be consistent with spelling of TASK_SIZE and related constants. Signed-off-by:
-
Kirill A. Shutemov authored
XEN_ELFNOTE_INIT_P2M has to be 512GB for both 4- and 5-level paging. (PUD_SIZE * PTRS_PER_PUD) would do this. Unfortunately, we cannot use P4D_SIZE, which would fit here. With current headers structure it cannot be used in assembly, if p4d level is folded. Signed-off-by:
Juergen Gross <jgross@suse.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Dave Hansen <dave.hansen@intel.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/20170716225954.74185-4-kirill.shutemov@linux.intel.comSigned-off-by:
-
Kirill A. Shutemov authored
Modify printk_prot() and callers to print out additional page table level correctly. Signed-off-by:
-
Kirill A. Shutemov authored
Modify normalize_addr to handle different sizes of virtual address space. It's preparation for enabling 5-level paging. Signed-off-by:
-
- 20 Jul, 2017 1 commit
-
-
Ingo Molnar authored
The following commit: 5997efb9 ("x86/boot: Use memremap() to map the MPF and MPC data") causes new build failures on certain randconfigs: drivers/sfi/sfi_core.c: In function ‘sfi_map_memory’: drivers/sfi/sfi_core.c:104:10: error: implicit declaration of function ‘memremap’ [-Werror=implicit-function-declaration] drivers/sfi/sfi_core.c:104:31: error: ‘MEMREMAP_WB’ undeclared (first use in this function) This is a case of parasitic header dependencies: the sfi_core.c file indirectly includes <linux/io.h> for typical - but not all configs. Including <linux/io.h> explicitly solves the build failure. Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: arnd@arndb.de Cc: aryabinin@virtuozzo.com Cc: bp@alien8.de Cc: bp@suse.de Cc: brijesh.singh@amd.com Cc: corbet@lwn.net Cc: dvyukov@google.com Cc: dyoung@redhat.com Cc: glider@google.com Cc: konrad.wilk@oracle.com Cc: linux-tip-commits@vger.kernel.org Cc: luto@kernel.org Cc: lwoodman@redhat.com Cc: matt@codeblueprint.co.uk Cc: mst@redhat.com Cc: pbonzini@redhat.com Cc: riel@redhat.com Cc: rkrcmar@redhat.com Cc: thomas.lendacky@amd.com Cc: toshi.kani@hpe.com Link: http://lkml.kernel.org/r/20170720092307.6xslahuaclmsiffe@gmail.comSigned-off-by:
-
- 18 Jul, 2017 33 commits
-
-
Tom Lendacky authored
Add support to check if SME has been enabled and if memory encryption should be activated (checking of command line option based on the configuration of the default state). If memory encryption is to be activated, then the encryption mask is set and the kernel is encrypted "in place." Signed-off-by:
Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by:
Thomas Gleixner <tglx@linutronix.de> Cc: Alexander Potapenko <glider@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Borislav Petkov <bp@alien8.de> Cc: Brijesh Singh <brijesh.singh@amd.com> Cc: Dave Young <dyoung@redhat.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Cc: Larry Woodman <lwoodman@redhat.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Matt Fleming <matt@codeblueprint.co.uk> Cc: Michael S. Tsirkin <mst@redhat.com> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Radim Krčmář <rkrcmar@redhat.com> Cc: Rik van Riel <riel@redhat.com> Cc: Toshimitsu Kani <toshi.kani@hpe.com> Cc: kasan-dev@googlegroups.com Cc: kvm@vger.kernel.org Cc: linux-arch@vger.kernel.org Cc: linux-doc@vger.kernel.org Cc: linux-efi@vger.kernel.org Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/5f0da2fd4cce63f556117549e2c89c170072209f.1500319216.git.thomas.lendacky@amd.comSigned-off-by:
-
Tom Lendacky authored
Create a new function attribute, __nostackprotector, that can used to turn off stack protection on a per function basis. Signed-off-by:
-
Tom Lendacky authored
Add a cmdline_find_option() function to look for cmdline options that take arguments. The argument is returned in a supplied buffer and the argument length (regardless of whether it fits in the supplied buffer) is returned, with -1 indicating not found. Signed-off-by:
-
Tom Lendacky authored
Add the support to encrypt the kernel in-place. This is done by creating new page mappings for the kernel - a decrypted write-protected mapping and an encrypted mapping. The kernel is encrypted by copying it through a temporary buffer. Signed-off-by:
-
Tom Lendacky authored
Currently, native_make_p4d() is only defined when CONFIG_PGTABLE_LEVELS is greater than 4. Create a macro that will allow for defining and using native_make_p4d() when CONFIG_PGTABLES_LEVELS is not greater than 4. Signed-off-by:
-
Tom Lendacky authored
When accessing memory using /dev/mem (or /dev/kmem) use the proper encryption attributes when mapping the memory. To insure the proper attributes are applied when reading or writing /dev/mem, update the xlate_dev_mem_ptr() function to use memremap() which will essentially perform the same steps of applying __va for RAM or using ioremap() if not RAM. To insure the proper attributes are applied when mmapping /dev/mem, update the phys_mem_access_prot() to call phys_mem_access_encrypted(), a new function which will check if the memory should be mapped encrypted or not. If it is not to be mapped encrypted then the VMA protection value is updated to remove the encryption bit. Signed-off-by:
Borislav Petkov <bp@suse.de> Cc: Alexander Potapenko <glider@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Borislav Petkov <bp@alien8.de> Cc: Brijesh Singh <brijesh.singh@amd.com> Cc: Dave Young <dyoung@redhat.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Cc: Larry Woodman <lwoodman@redhat.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Matt Fleming <matt@codeblueprint.co.uk> Cc: Michael S. Tsirkin <mst@redhat.com> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Radim Krčmář <rkrcmar@redhat.com> Cc: Rik van Riel <riel@redhat.com> Cc: Toshimitsu Kani <toshi.kani@hpe.com> Cc: kasan-dev@googlegroups.com Cc: kvm@vger.kernel.org Cc: linux-arch@vger.kernel.org Cc: linux-doc@vger.kernel.org Cc: linux-efi@vger.kernel.org Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/c917f403ab9f61cbfd455ad6425ed8429a5e7b54.1500319216.git.thomas.lendacky@amd.comSigned-off-by:
-
Tom Lendacky authored
Xen does not currently support SME for PV guests. Clear the SME CPU capability in order to avoid any ambiguity. Signed-off-by:
-
Tom Lendacky authored
Provide support so that kexec can be used to boot a kernel when SME is enabled. Support is needed to allocate pages for kexec without encryption. This is needed in order to be able to reboot in the kernel in the same manner as originally booted. Additionally, when shutting down all of the CPUs we need to be sure to flush the caches and then halt. This is needed when booting from a state where SME was not active into a state where SME is active (or vice-versa). Without these steps, it is possible for cache lines to exist for the same physical location but tagged both with and without the encryption bit. This can cause random memory corruption when caches are flushed depending on which cacheline is written last. Signed-off-by:
-
Tom Lendacky authored
Update the KVM support to work with SME. The VMCB has a number of fields where physical addresses are used and these addresses must contain the memory encryption mask in order to properly access the encrypted memory. Also, use the memory encryption mask when creating and using the nested page tables. Signed-off-by:
-
Tom Lendacky authored
Since video memory needs to be accessed decrypted, be sure that the memory encryption mask is not set for the video ranges. Signed-off-by:
-
Tom Lendacky authored
Add support to check if memory encryption is active in the kernel and that it has been enabled on the AP. If memory encryption is active in the kernel but has not been enabled on the AP, then set the memory encryption bit (bit 23) of MSR_K8_SYSCFG to enable memory encryption on that AP and allow the AP to continue start up. Signed-off-by:
-
Tom Lendacky authored
The IOMMU is programmed with physical addresses for the various tables and buffers that are used to communicate between the device and the driver. When the driver allocates this memory it is encrypted. In order for the IOMMU to access the memory as encrypted the encryption mask needs to be included in these physical addresses during configuration. The PTE entries created by the IOMMU should also include the encryption mask so that when the device behind the IOMMU performs a DMA, the DMA will be performed to encrypted memory. Signed-off-by:
Joerg Roedel <jroedel@suse.de> Cc: <iommu@lists.linux-foundation.org> Cc: Alexander Potapenko <glider@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Borislav Petkov <bp@alien8.de> Cc: Brijesh Singh <brijesh.singh@amd.com> Cc: Dave Young <dyoung@redhat.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Cc: Larry Woodman <lwoodman@redhat.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Matt Fleming <matt@codeblueprint.co.uk> Cc: Michael S. Tsirkin <mst@redhat.com> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Radim Krčmář <rkrcmar@redhat.com> Cc: Rik van Riel <riel@redhat.com> Cc: Toshimitsu Kani <toshi.kani@hpe.com> Cc: kasan-dev@googlegroups.com Cc: kvm@vger.kernel.org Cc: linux-arch@vger.kernel.org Cc: linux-doc@vger.kernel.org Cc: linux-efi@vger.kernel.org Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/3053631ea25ba8b1601c351cb7c541c496f6d9bc.1500319216.git.thomas.lendacky@amd.comSigned-off-by:
-
Tom Lendacky authored
Move the setting of the cpuinfo_x86.microcode field from amd_init() to early_amd_init() so that it is available earlier in the boot process. This avoids having to read MSR_AMD64_PATCH_LEVEL directly during early boot. Signed-off-by:
-
Tom Lendacky authored
Add warnings to let the user know when bounce buffers are being used for DMA when SME is active. Since the bounce buffers are not in encrypted memory, these notifications are to allow the user to determine some appropriate action - if necessary. Actions can range from utilizing an IOMMU, replacing the device with another device that can support 64-bit DMA, ignoring the message if the device isn't used much, etc. Signed-off-by:
-
Tom Lendacky authored
Since DMA addresses will effectively look like 48-bit addresses when the memory encryption mask is set, SWIOTLB is needed if the DMA mask of the device performing the DMA does not support 48-bits. SWIOTLB will be initialized to create decrypted bounce buffers for use by these devices. Signed-off-by:
-
Tom Lendacky authored
When Secure Memory Encryption is enabled, the trampoline area must not be encrypted. A CPU running in real mode will not be able to decrypt memory that has been encrypted because it will not be able to use addresses with the memory encryption mask. Signed-off-by:
-
Tom Lendacky authored
Add support for changing the memory encryption attribute for one or more memory pages. This will be useful when we have to change the AP trampoline area to not be encrypted. Or when we need to change the SWIOTLB area to not be encrypted in support of devices that can't support the encryption mask range. Signed-off-by:
-
Tom Lendacky authored
Persistent memory is expected to persist across reboots. The encryption key used by SME will change across reboots which will result in corrupted persistent memory. Persistent memory is handed out by block devices through memory remapping functions, so be sure not to map this memory as encrypted. Signed-off-by:
-
Tom Lendacky authored
The SMP MP-table is built by UEFI and placed in memory in a decrypted state. These tables are accessed using a mix of early_memremap(), early_memunmap(), phys_to_virt() and virt_to_phys(). Change all accesses to use early_memremap()/early_memunmap(). This allows for proper setting of the encryption mask so that the data can be successfully accessed when SME is active. Signed-off-by:
-
Tom Lendacky authored
Boot data (such as EFI related data) is not encrypted when the system is booted because UEFI/BIOS does not run with SME active. In order to access this data properly it needs to be mapped decrypted. Update early_memremap() to provide an arch specific routine to modify the pagetable protection attributes before they are applied to the new mapping. This is used to remove the encryption mask for boot related data. Update memremap() to provide an arch specific routine to determine if RAM remapping is allowed. RAM remapping will cause an encrypted mapping to be generated. By preventing RAM remapping, ioremap_cache() will be used instead, which will provide a decrypted mapping of the boot related data. Signed-off-by:
Matt Fleming <matt@codeblueprint.co.uk> Reviewed-by:
-
Tom Lendacky authored
When SME is active, pagetable entries created for EFI need to have the encryption mask set as necessary. When the new pagetable pages are allocated they are mapped encrypted. So, update the efi_pgt value that will be used in CR3 to include the encryption mask so that the PGD table can be read successfully. The pagetable mapping as well as the kernel are also added to the pagetable mapping as encrypted. All other EFI mappings are mapped decrypted (tables, etc.). Signed-off-by:
-
Tom Lendacky authored
The efi_mem_type() function currently returns a 0, which maps to EFI_RESERVED_TYPE, if the function is unable to find a memmap entry for the supplied physical address. Returning EFI_RESERVED_TYPE implies that a memmap entry exists, when it doesn't. Instead of returning 0, change the function to return a negative error value when no memmap entry is found. Signed-off-by:
-
Tom Lendacky authored
Add a function that will determine if a supplied physical address matches the address of an EFI table. Signed-off-by:
-
Tom Lendacky authored
Add a function that will return the E820 type associated with an address range. Signed-off-by:
-
Tom Lendacky authored
The boot data and command line data are present in memory in a decrypted state and are copied early in the boot process. The early page fault support will map these areas as encrypted, so before attempting to copy them, add decrypted mappings so the data is accessed properly when copied. For the initrd, encrypt this data in place. Since the future mapping of the initrd area will be mapped as encrypted the data will be accessed properly. Signed-off-by:
-
Tom Lendacky authored
Add support to be able to either encrypt or decrypt data in place during the early stages of booting the kernel. This does not change the memory encryption attribute - it is used for ensuring that data present in either an encrypted or decrypted memory area is in the proper state (for example the initrd will have been loaded by the boot loader and will not be encrypted, but the memory that it resides in is marked as encrypted). Signed-off-by:
-
Tom Lendacky authored
Add early_memremap() support to be able to specify encrypted and decrypted mappings with and without write-protection. The use of write-protection is necessary when encrypting data "in place". The write-protect attribute is considered cacheable for loads, but not stores. This implies that the hardware will never give the core a dirty line with this memtype. Signed-off-by:
-
Tom Lendacky authored
The CR3 register entry can contain the SME encryption mask that indicates the PGD is encrypted. The encryption mask should not be used when creating a virtual address from the CR3 register, so remove the SME encryption mask in the read_cr3_pa() function. During early boot SME will need to use a native version of read_cr3_pa(), so create native_read_cr3_pa(). Signed-off-by:
-
Tom Lendacky authored
Changes to the existing page table macros will allow the SME support to be enabled in a simple fashion with minimal changes to files that use these macros. Since the memory encryption mask will now be part of the regular pagetable macros, we introduce two new macros (_PAGE_TABLE_NOENC and _KERNPG_TABLE_NOENC) to allow for early pagetable creation/initialization without the encryption mask before SME becomes active. Two new pgprot() macros are defined to allow setting or clearing the page encryption mask. The FIXMAP_PAGE_NOCACHE define is introduced for use with MMIO. SME does not support encryption for MMIO areas so this define removes the encryption mask from the page attribute. Two new macros are introduced (__sme_pa() / __sme_pa_nodebug()) to allow creating a physical address with the encryption mask. These are used when working with the cr3 register so that the PGD can be encrypted. The current __va() macro is updated so that the virtual address is generated based off of the physical address without the encryption mask thus allowing the same virtual address to be generated regardless of whether encryption is enabled for that physical location or not. Also, an early initialization function is added for SME. If SME is active, this function: - Updates the early_pmd_flags so that early page faults create mappings with the encryption mask. - Updates the __supported_pte_mask to include the encryption mask. - Updates the protection_map entries to include the encryption mask so that user-space allocations will automatically have the encryption mask applied. Signed-off-by:
-
Tom Lendacky authored
Create a pgd_pfn() macro similar to the p[4um]d_pfn() macros and then use the p[g4um]d_pfn() macros in the p[g4um]d_page() macros instead of duplicating the code. Signed-off-by:
-
Tom Lendacky authored
Add support to the early boot code to use Secure Memory Encryption (SME). Since the kernel has been loaded into memory in a decrypted state, encrypt the kernel in place and update the early pagetables with the memory encryption mask so that new pagetable entries will use memory encryption. The routines to set the encryption mask and perform the encryption are stub routines for now with functionality to be added in a later patch. Signed-off-by:
-
Tom Lendacky authored
Currently there is a check if the address being mapped is in the ISA range (is_ISA_range()), and if it is, then phys_to_virt() is used to perform the mapping. When SME is active, the default is to add pagetable mappings with the encryption bit set unless specifically overridden. The resulting pagetable mapping from phys_to_virt() will result in a mapping that has the encryption bit set. With SME, the use of ioremap() is intended to generate pagetable mappings that do not have the encryption bit set through the use of the PAGE_KERNEL_IO protection value. Rather than special case the SME scenario, remove the ISA range check and usage of phys_to_virt() and have ISA range mappings continue through the remaining ioremap() path. Signed-off-by:
-
Tom Lendacky authored
Add support for Secure Memory Encryption (SME). This initial support provides a Kconfig entry to build the SME support into the kernel and defines the memory encryption mask that will be used in subsequent patches to mark pages as encrypted. Signed-off-by:
-
