• Paul E. McKenney's avatar
    rcu-tasks: Add an RCU Tasks Trace to simplify protection of tracing hooks · d5f177d3
    Paul E. McKenney authored
    Because RCU does not watch exception early-entry/late-exit, idle-loop,
    or CPU-hotplug execution, protection of tracing and BPF operations is
    needlessly complicated.  This commit therefore adds a variant of
    Tasks RCU that:
    
    o	Has explicit read-side markers to allow finite grace periods in
    	the face of in-kernel loops for PREEMPT=n builds.  These markers
    	are rcu_read_lock_trace() and rcu_read_unlock_trace().
    
    o	Protects code in the idle loop, exception entry/exit, and
    	CPU-hotplug code paths.  In this respect, RCU-tasks trace is
    	similar to SRCU, but with lighter-weight readers.
    
    o	Avoids expensive read-side instruction, having overhead similar
    	to that of Preemptible RCU.
    
    There are of course downsides:
    
    o	The grace-period code can send IPIs to CPUs, even when those
    	CPUs are in the idle loop or in nohz_full userspace.  This is
    	mitigated by later commits.
    
    o	It is necessary to scan the full tasklist, much as for Tasks RCU.
    
    o	There is a single callback queue guarded by a single lock,
    	again, much as for Tasks RCU.  However, those early use cases
    	that request multiple grace periods in quick succession are
    	expected to do so from a single task, which makes the single
    	lock almost irrelevant.  If needed, multiple callback queues
    	can be provided using any number of schemes.
    
    Perhaps most important, this variant of RCU does not affect the vanilla
    flavors, rcu_preempt and rcu_sched.  The fact that RCU Tasks Trace
    readers can operate from idle, offline, and exception entry/exit in no
    way enables rcu_preempt and rcu_sched readers to do so.
    
    The memory ordering was outlined here:
    https://lore.kernel.org/lkml/20200319034030.GX3199@paulmck-ThinkPad-P72/
    
    This effort benefited greatly from off-list discussions of BPF
    requirements with Alexei Starovoitov and Andrii Nakryiko.  At least
    some of the on-list discussions are captured in the Link: tags below.
    In addition, KCSAN was quite helpful in finding some early bugs.
    
    Link: https://lore.kernel.org/lkml/20200219150744.428764577@infradead.org/
    Link: https://lore.kernel.org/lkml/87mu8p797b.fsf@nanos.tec.linutronix.de/
    Link: https://lore.kernel.org/lkml/20200225221305.605144982@linutronix.de/
    Cc: Alexei Starovoitov <alexei.starovoitov@gmail.com>
    Cc: Andrii Nakryiko <andriin@fb.com>
    [ paulmck: Apply feedback from Steve Rostedt and Joel Fernandes. ]
    [ paulmck: Decrement trc_n_readers_need_end upon IPI failure. ]
    [ paulmck: Fix locking issue reported by rcutorture. ]
    Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
    d5f177d3
fork.c 74.3 KB