• Mathieu Desnoyers's avatar
    sched/membarrier: Fix p->mm->membarrier_state racy load · 227a4aad
    Mathieu Desnoyers authored
    The membarrier_state field is located within the mm_struct, which
    is not guaranteed to exist when used from runqueue-lock-free iteration
    on runqueues by the membarrier system call.
    
    Copy the membarrier_state from the mm_struct into the scheduler runqueue
    when the scheduler switches between mm.
    
    When registering membarrier for mm, after setting the registration bit
    in the mm membarrier state, issue a synchronize_rcu() to ensure the
    scheduler observes the change. In order to take care of the case
    where a runqueue keeps executing the target mm without swapping to
    other mm, iterate over each runqueue and issue an IPI to copy the
    membarrier_state from the mm_struct into each runqueue which have the
    same mm which state has just been modified.
    
    Move the mm membarrier_state field closer to pgd in mm_struct to use
    a cache line already touched by the scheduler switch_mm.
    
    The membarrier_execve() (now membarrier_exec_mmap) hook now needs to
    clear the runqueue's membarrier state in addition to clear the mm
    membarrier state, so move its implementation into the scheduler
    membarrier code so it can access the runqueue structure.
    
    Add memory barrier in membarrier_exec_mmap() prior to clearing
    the membarrier state, ensuring memory accesses executed prior to exec
    are not reordered with the stores clearing the membarrier state.
    
    As suggested by Linus, move all membarrier.c RCU read-side locks outside
    of the for each cpu loops.
    Suggested-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    Signed-off-by: default avatarMathieu Desnoyers <mathieu.desnoyers@efficios.com>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Cc: Chris Metcalf <cmetcalf@ezchip.com>
    Cc: Christoph Lameter <cl@linux.com>
    Cc: Eric W. Biederman <ebiederm@xmission.com>
    Cc: Kirill Tkhai <tkhai@yandex.ru>
    Cc: Mike Galbraith <efault@gmx.de>
    Cc: Oleg Nesterov <oleg@redhat.com>
    Cc: Paul E. McKenney <paulmck@linux.ibm.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Russell King - ARM Linux admin <linux@armlinux.org.uk>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Link: https://lkml.kernel.org/r/20190919173705.2181-5-mathieu.desnoyers@efficios.comSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
    227a4aad
membarrier.c 11 KB