Commit 85f69b32 authored by Paul E. McKenney's avatar Paul E. McKenney

rcu/nocb: Check for deferred nocb wakeups before nohz_full early exit

In theory, a timer is used to defer wakeups of no-CBs grace-period
kthreads when the wakeup cannot be done safely directly from the
call_rcu().  In practice, the one-jiffy delay is not always consistent
with timely callback invocation under heavy call_rcu() loads.  Therefore,
there are a number of checks for a pending deferred wakeup, including
from the scheduling-clock interrupt.  Unfortunately, this check follows
the rcu_nohz_full_cpu() early exit, which renders it useless on such CPUs.

This commit therefore moves the check for the pending deferred no-CB
wakeup to precede the rcu_nohz_full_cpu() early exit.
Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.ibm.com>
parent c00045be
...@@ -2739,6 +2739,10 @@ static int rcu_pending(void) ...@@ -2739,6 +2739,10 @@ static int rcu_pending(void)
/* Check for CPU stalls, if enabled. */ /* Check for CPU stalls, if enabled. */
check_cpu_stall(rdp); check_cpu_stall(rdp);
/* Does this CPU need a deferred NOCB wakeup? */
if (rcu_nocb_need_deferred_wakeup(rdp))
return 1;
/* Is this CPU a NO_HZ_FULL CPU that should ignore RCU? */ /* Is this CPU a NO_HZ_FULL CPU that should ignore RCU? */
if (rcu_nohz_full_cpu()) if (rcu_nohz_full_cpu())
return 0; return 0;
...@@ -2763,10 +2767,6 @@ static int rcu_pending(void) ...@@ -2763,10 +2767,6 @@ static int rcu_pending(void)
unlikely(READ_ONCE(rdp->gpwrap))) /* outside lock */ unlikely(READ_ONCE(rdp->gpwrap))) /* outside lock */
return 1; return 1;
/* Does this CPU need a deferred NOCB wakeup? */
if (rcu_nocb_need_deferred_wakeup(rdp))
return 1;
/* nothing to do */ /* nothing to do */
return 0; return 0;
} }
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment