• Mark Rutland's avatar
    mm/slub: fix lockups on PREEMPT && !SMP kernels · 859b7a0e
    Mark Rutland authored
    Commit 9aabf810 ("mm/slub: optimize alloc/free fastpath by removing
    preemption on/off") introduced an occasional hang for kernels built with
    CONFIG_PREEMPT && !CONFIG_SMP.
    
    The problem is the following loop the patch introduced to
    slab_alloc_node and slab_free:
    
        do {
            tid = this_cpu_read(s->cpu_slab->tid);
            c = raw_cpu_ptr(s->cpu_slab);
        } while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid));
    
    GCC 4.9 has been observed to hoist the load of c and c->tid above the
    loop for !SMP kernels (as in this case raw_cpu_ptr(x) is compile-time
    constant and does not force a reload).  On arm64 the generated assembly
    looks like:
    
             ldr     x4, [x0,#8]
      loop:
             ldr     x1, [x0,#8]
             cmp     x1, x4
             b.ne    loop
    
    If the thread is preempted between the load of c->tid (into x1) and tid
    (into x4), and an allocation or free occurs in another thread (bumping
    the cpu_slab's tid), the thread will be stuck in the loop until
    s->cpu_slab->tid wraps, which may be forever in the absence of
    allocations/frees on the same CPU.
    
    This patch changes the loop condition to access c->tid with READ_ONCE.
    This ensures that the value is reloaded even when the compiler would
    otherwise assume it could cache the value, and also ensures that the
    load will not be torn.
    Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
    Cc: Catalin Marinas <catalin.marinas@arm.com>
    Acked-by: default avatarChristoph Lameter <cl@linux.com>
    Cc: David Rientjes <rientjes@google.com>
    Cc: Jesper Dangaard Brouer <brouer@redhat.com>
    Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
    Cc: Pekka Enberg <penberg@kernel.org>
    Cc: Steve Capper <steve.capper@linaro.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    859b7a0e
slub.c 127 KB