• Uros Bizjak's avatar
    x86/percpu: Introduce const-qualified const_pcpu_hot to micro-optimize code generation · ed2f752e
    Uros Bizjak authored
    Some variables in pcpu_hot, currently current_task and top_of_stack
    are actually per-thread variables implemented as per-CPU variables
    and thus stable for the duration of the respective task.  There is
    already an attempt to eliminate redundant reads from these variables
    using this_cpu_read_stable() asm macro, which hides the dependency
    on the read memory address. However, the compiler has limited ability
    to eliminate asm common subexpressions, so this approach results in a
    limited success.
    
    The solution is to allow more aggressive elimination by aliasing
    pcpu_hot into a const-qualified const_pcpu_hot, and to read stable
    per-CPU variables from this constant copy.
    
    The current per-CPU infrastructure does not support reads from
    const-qualified variables. However, when the compiler supports segment
    qualifiers, it is possible to declare the const-aliased variable in
    the relevant named address space. The compiler considers access to the
    variable, declared in this way, as a read from a constant location,
    and will optimize reads from the variable accordingly.
    
    By implementing constant-qualified const_pcpu_hot, the compiler can
    eliminate redundant reads from the constant variables, reducing the
    number of loads from current_task from 3766 to 3217 on a test build,
    a -14.6% reduction.
    
    The reduction of loads translates to the following code savings:
    
            text           data     bss      dec            hex filename
      25,477,353        4389456  808452 30675261        1d4113d vmlinux-old.o
      25,476,074        4389440  808452 30673966        1d40c2e vmlinux-new.o
    
    representing a code size reduction of -1279 bytes.
    
    [ mingo: Updated the changelog, EXPORT(const_pcpu_hot). ]
    Co-developed-by: default avatarNadav Amit <namit@vmware.com>
    Signed-off-by: default avatarNadav Amit <namit@vmware.com>
    Signed-off-by: default avatarUros Bizjak <ubizjak@gmail.com>
    Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
    Link: https://lore.kernel.org/r/20231020162004.135244-1-ubizjak@gmail.com
    ed2f752e
processor.h 19 KB