• Nicholas Piggin's avatar
    powerpc/64: allow compiler to cache 'current' · e354d7dc
    Nicholas Piggin authored
    current may be cached by the compiler, so remove the volatile asm
    restriction. This results in better generated code, as well as being
    smaller and fewer dependent loads, it can avoid store-hit-load flushes
    like this one that shows up in irq_exit():
    
        preempt_count_sub(HARDIRQ_OFFSET);
        if (!in_interrupt() && ...)
    
    Which ends up as:
    
        ((struct thread_info *)current)->preempt_count -= HARDIRQ_OFFSET;
        if (((struct thread_info *)current)->preempt_count ...
    
    Evaluating current twice presently means it has to be loaded twice, and
    here gcc happens to pick a different register each time, then
    preempt_count is accessed via that base register:
    
        1058:       ld      r10,2392(r13)     <-- current
        105c:       lwz     r9,0(r10)         <-- preempt_count
        1060:       addis   r9,r9,-1
        1064:       stw     r9,0(r10)         <-- preempt_count
        1068:       ld      r9,2392(r13)      <-- current
        106c:       lwz     r9,0(r9)          <-- preempt_count
        1070:       rlwinm. r9,r9,0,11,23
        1074:       bne     1090 <irq_exit+0x60>
    
    This can frustrate store-hit-load detection heuristics and cause
    flushes. Allowing the compiler to cache current in a reigster with this
    patch results in the same base register being used for all accesses,
    which is more likely to be detected as an alias:
    
        1058:       ld      r31,2392(r13)
        ...
        1070:       lwz     r9,0(r31)
        1074:       addis   r9,r9,-1
        1078:       stw     r9,0(r31)
        107c:       lwz     r9,0(r31)
        1080:       rlwinm. r9,r9,0,11,23
        1084:       bne     10a0 <irq_exit+0x60>
    Signed-off-by: default avatarNicholas Piggin <npiggin@gmail.com>
    Signed-off-by: default avatarMichael Ellerman <mpe@ellerman.id.au>
    Link: https://lore.kernel.org/r/20190612140317.24490-1-npiggin@gmail.com
    e354d7dc
current.h 680 Bytes