• Paul E. McKenney's avatar
    rcu: Improve non-root rcu_cbs_completed() accuracy · 9036c2ff
    Paul E. McKenney authored
    When rcu_cbs_completed() is invoked on a non-root rcu_node structure,
    it unconditionally assumes that two grace periods must complete before
    the callbacks at hand can be invoked.  This is overly conservative because
    if that non-root rcu_node structure believes that no grace period is in
    progress, and if the corresponding rcu_state structure's ->gpnum field
    has not yet been incremented, then these callbacks may safely be invoked
    after only one grace period has completed.
    
    This change is required to permit grace-period start requests to use
    funnel locking, which is in turn permitted to reduce root rcu_node ->lock
    contention, which has been observed by Nick Piggin.  Furthermore, such
    contention will likely be increased by the merging of RCU-bh, RCU-preempt,
    and RCU-sched, so it makes sense to take steps to decrease it.
    
    This commit therefore improves the accuracy of rcu_cbs_completed() when
    invoked on a non-root rcu_node structure as described above.
    Reported-by: default avatarNicholas Piggin <npiggin@gmail.com>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    Tested-by: default avatarNicholas Piggin <npiggin@gmail.com>
    9036c2ff
tree.c 131 KB