• Reza Arbab's avatar
    powerpc/powernv: Fix concurrency issue with npu->mmio_atsd_usage · 9eab9901
    Reza Arbab authored
    We've encountered a performance issue when multiple processors stress
    {get,put}_mmio_atsd_reg(). These functions contend for
    mmio_atsd_usage, an unsigned long used as a bitmask.
    
    The accesses to mmio_atsd_usage are done using test_and_set_bit_lock()
    and clear_bit_unlock(). As implemented, both of these will require
    a (successful) stwcx to that same cache line.
    
    What we end up with is thread A, attempting to unlock, being slowed by
    other threads repeatedly attempting to lock. A's stwcx instructions
    fail and retry because the memory reservation is lost every time a
    different thread beats it to the punch.
    
    There may be a long-term way to fix this at a larger scale, but for
    now resolve the immediate problem by gating our call to
    test_and_set_bit_lock() with one to test_bit(), which is obviously
    implemented without using a store.
    
    Fixes: 1ab66d1f ("powerpc/powernv: Introduce address translation services for Nvlink2")
    Signed-off-by: default avatarReza Arbab <arbab@linux.ibm.com>
    Acked-by: default avatarAlistair Popple <alistair@popple.id.au>
    Signed-off-by: default avatarMichael Ellerman <mpe@ellerman.id.au>
    9eab9901
npu-dma.c 25 KB