• Nick Piggin's avatar
    x86: optimise x86's do_page_fault (C entry point for the page fault path) · 92181f19
    Nick Piggin authored
    Impact: cleanup, restructure code to improve assembly
    
    gcc isn't _all_ that smart about spilling registers to stack or reusing
    stack slots, even with branch annotations. do_page_fault contained a lot
    of functionality, so split unlikely paths into their own functions, and
    mark them as noinline just to be sure. I consider this actually to be
    somewhat of a cleanup too: the main function now contains about half
    the number of lines so the normal path is easier to read, while the error
    cases are also nicely split away.
    
    Also, ensure the order of arguments to functions is always the same: regs,
    addr, error_code. This can reduce code size a tiny bit, and just looks neater
    too.
    
    And add a couple of branch annotations.
    
    Before:
      do_page_fault:
              subq    $360, %rsp      #,
    
    After:
      do_page_fault:
              subq    $56, %rsp       #,
    
    bloat-o-meter:
      add/remove: 8/0 grow/shrink: 0/1 up/down: 2222/-1680 (542)
      function                                     old     new   delta
      __bad_area_nosemaphore                         -     506    +506
      no_context                                     -     474    +474
      vmalloc_fault                                  -     424    +424
      spurious_fault                                 -     358    +358
      mm_fault_error                                 -     272    +272
      bad_area_access_error                          -      89     +89
      bad_area                                       -      89     +89
      bad_area_nosemaphore                           -      10     +10
      do_page_fault                               2464     784   -1680
    
    Yes, the total size increases by 542 bytes, due to the extra function calls.
    But these will very rarely be called (except for vmalloc_fault) in a normal
    workload. Importantly, do_page_fault is less than 1/3rd it's original size,
    and touches far less stack.
    
    Existing gotos and branch hints did move a lot of the infrequently used text
    out of the fastpath, but that's even further improved after this patch.
    Signed-off-by: default avatarNick Piggin <npiggin@suse.de>
    Acked-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
    92181f19
fault.c 25 KB