• Joel Fernandes (Google)'s avatar
    rcu/tree: Make rcu_do_batch count how many callbacks were executed · 6bc33582
    Joel Fernandes (Google) authored
    The rcu_do_batch() function extracts the ready-to-invoke callbacks
    from the rcu_segcblist located in the ->cblist field of the current
    CPU's rcu_data structure.  These callbacks are first moved to a local
    (unsegmented) rcu_cblist.  The rcu_do_batch() function then uses this
    rcu_cblist's ->len field to count how many CBs it has invoked, but it
    does so by counting that field down from zero.  Finally, this function
    negates the value in this ->len field (resulting in a positive number)
    and subtracts the result from the ->len field of the current CPU's
    ->cblist field.
    
    Except that it is sometimes necessary for rcu_do_batch() to stop invoking
    callbacks mid-stream, despite there being more ready to invoke, for
    example, if a high-priority task wakes up.  In this case the remaining
    not-yet-invoked callbacks are requeued back onto the CPU's ->cblist,
    but remain in the ready-to-invoke segment of that list.  As above, the
    negative of the local rcu_cblist's ->len field is still subtracted from
    the ->len field of the current CPU's ->cblist field.
    
    The design of counting down from 0 is confusing and error-prone, plus
    use of a positive count will make it easier to provide a uniform and
    consistent API to deal with the per-segment counts that are added
    later in this series.  For example, rcu_segcblist_extract_done_cbs()
    can unconditionally populate the resulting unsegmented list's ->len
    field during extraction.
    
    This commit therefore explicitly counts how many callbacks were executed
    in rcu_do_batch() itself, counting up from zero, and then uses that
    to update the per-CPU segcb list's ->len field, without relying on the
    downcounting of rcl->len from zero.
    Reviewed-by: default avatarFrederic Weisbecker <frederic@kernel.org>
    Reviewed-by: default avatarNeeraj Upadhyay <neeraju@codeaurora.org>
    Signed-off-by: default avatarJoel Fernandes (Google) <joel@joelfernandes.org>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
    6bc33582
tree.c 145 KB