• Dave Hansen's avatar
    x86/pti: Never implicitly clear _PAGE_GLOBAL for kernel image · 39114b7a
    Dave Hansen authored
    Summary:
    
    In current kernels, with PTI enabled, no pages are marked Global. This
    potentially increases TLB misses.  But, the mechanism by which the Global
    bit is set and cleared is rather haphazard.  This patch makes the process
    more explicit.  In the end, it leaves us with Global entries in the page
    tables for the areas truly shared by userspace and kernel and increases
    TLB hit rates.
    
    The place this patch really shines in on systems without PCIDs.  In this
    case, we are using an lseek microbenchmark[1] to see how a reasonably
    non-trivial syscall behaves.  Higher is better:
    
      No Global pages (baseline): 6077741 lseeks/sec
      88 Global Pages (this set): 7528609 lseeks/sec (+23.9%)
    
    On a modern Skylake desktop with PCIDs, the benefits are tangible, but not
    huge for a kernel compile (lower is better):
    
      No Global pages (baseline): 186.951 seconds time elapsed  ( +-  0.35% )
      28 Global pages (this set): 185.756 seconds time elapsed  ( +-  0.09% )
                                   -1.195 seconds (-0.64%)
    
    I also re-checked everything using the lseek1 test[1]:
    
      No Global pages (baseline): 15783951 lseeks/sec
      28 Global pages (this set): 16054688 lseeks/sec
    			     +270737 lseeks/sec (+1.71%)
    
    The effect is more visible, but still modest.
    
    Details:
    
    The kernel page tables are inherited from head_64.S which rudely marks
    them as _PAGE_GLOBAL.  For PTI, we have been relying on the grace of
    $DEITY and some insane behavior in pageattr.c to clear _PAGE_GLOBAL.
    This patch tries to do better.
    
    First, stop filtering out "unsupported" bits from being cleared in the
    pageattr code.  It's fine to filter out *setting* these bits but it
    is insane to keep us from clearing them.
    
    Then, *explicitly* go clear _PAGE_GLOBAL from the kernel identity map.
    Do not rely on pageattr to do it magically.
    
    After this patch, we can see that "GLB" shows up in each copy of the
    page tables, that we have the same number of global entries in each
    and that they are the *same* entries.
    
      /sys/kernel/debug/page_tables/current_kernel:11
      /sys/kernel/debug/page_tables/current_user:11
      /sys/kernel/debug/page_tables/kernel:11
    
      9caae8ad6a1fb53aca2407ec037f612d  current_kernel.GLB
      9caae8ad6a1fb53aca2407ec037f612d  current_user.GLB
      9caae8ad6a1fb53aca2407ec037f612d  kernel.GLB
    
    A quick visual audit also shows that all the entries make sense.
    0xfffffe0000000000 is the cpu_entry_area and 0xffffffff81c00000
    is the entry/exit text:
    
      0xfffffe0000000000-0xfffffe0000002000           8K     ro                 GLB NX pte
      0xfffffe0000002000-0xfffffe0000003000           4K     RW                 GLB NX pte
      0xfffffe0000003000-0xfffffe0000006000          12K     ro                 GLB NX pte
      0xfffffe0000006000-0xfffffe0000007000           4K     ro                 GLB x  pte
      0xfffffe0000007000-0xfffffe000000d000          24K     RW                 GLB NX pte
      0xfffffe000002d000-0xfffffe000002e000           4K     ro                 GLB NX pte
      0xfffffe000002e000-0xfffffe000002f000           4K     RW                 GLB NX pte
      0xfffffe000002f000-0xfffffe0000032000          12K     ro                 GLB NX pte
      0xfffffe0000032000-0xfffffe0000033000           4K     ro                 GLB x  pte
      0xfffffe0000033000-0xfffffe0000039000          24K     RW                 GLB NX pte
      0xffffffff81c00000-0xffffffff81e00000           2M     ro         PSE     GLB x  pmd
    
    [1.] https://github.com/antonblanchard/will-it-scale/blob/master/tests/lseek1.cSigned-off-by: default avatarDave Hansen <dave.hansen@linux.intel.com>
    Cc: Andrea Arcangeli <aarcange@redhat.com>
    Cc: Andy Lutomirski <luto@kernel.org>
    Cc: Arjan van de Ven <arjan@linux.intel.com>
    Cc: Borislav Petkov <bp@alien8.de>
    Cc: Dan Williams <dan.j.williams@intel.com>
    Cc: David Woodhouse <dwmw2@infradead.org>
    Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
    Cc: Hugh Dickins <hughd@google.com>
    Cc: Josh Poimboeuf <jpoimboe@redhat.com>
    Cc: Juergen Gross <jgross@suse.com>
    Cc: Kees Cook <keescook@google.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Nadav Amit <namit@vmware.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: linux-mm@kvack.org
    Link: http://lkml.kernel.org/r/20180406205517.C80FBE05@viggo.jf.intel.comSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
    39114b7a
init.c 25.3 KB