Commit 0d3eb744 authored by Linus Torvalds's avatar Linus Torvalds

Merge tag 'urgent-rcu.2023.04.07a' of...

Merge tag 'urgent-rcu.2023.04.07a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu

Pull RCU fix from Paul McKenney:
 "This fixes a pair of bugs in which an improbable but very real
  sequence of events can cause kfree_rcu() to be a bit too quick about
  freeing the memory passed to it.

  It turns out that this pair of bugs is about two years old, and so
  this is not a v6.3 regression. However: (1) It just started showing up
  in the wild and (2) Its consequences are dire, so its fix needs to go
  in sooner rather than later.

  Testing is of course being upgraded, and the upgraded tests detect
  this situation very quickly. But to the best of my knowledge right
  now, the tests are not particularly urgent and will thus most likely
  show up in the v6.5 merge window (the one after this coming one).

  Kudos to Ziwei Dai and his group for tracking this one down the hard
  way!"

* tag 'urgent-rcu.2023.04.07a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu:
  rcu/kvfree: Avoid freeing new kfree_rcu() memory after old grace period
parents dfc19154 5da7cb19
...@@ -3024,6 +3024,18 @@ need_offload_krc(struct kfree_rcu_cpu *krcp) ...@@ -3024,6 +3024,18 @@ need_offload_krc(struct kfree_rcu_cpu *krcp)
return !!READ_ONCE(krcp->head); return !!READ_ONCE(krcp->head);
} }
static bool
need_wait_for_krwp_work(struct kfree_rcu_cpu_work *krwp)
{
int i;
for (i = 0; i < FREE_N_CHANNELS; i++)
if (!list_empty(&krwp->bulk_head_free[i]))
return true;
return !!krwp->head_free;
}
static int krc_count(struct kfree_rcu_cpu *krcp) static int krc_count(struct kfree_rcu_cpu *krcp)
{ {
int sum = atomic_read(&krcp->head_count); int sum = atomic_read(&krcp->head_count);
...@@ -3107,15 +3119,14 @@ static void kfree_rcu_monitor(struct work_struct *work) ...@@ -3107,15 +3119,14 @@ static void kfree_rcu_monitor(struct work_struct *work)
for (i = 0; i < KFREE_N_BATCHES; i++) { for (i = 0; i < KFREE_N_BATCHES; i++) {
struct kfree_rcu_cpu_work *krwp = &(krcp->krw_arr[i]); struct kfree_rcu_cpu_work *krwp = &(krcp->krw_arr[i]);
// Try to detach bulk_head or head and attach it over any // Try to detach bulk_head or head and attach it, only when
// available corresponding free channel. It can be that // all channels are free. Any channel is not free means at krwp
// a previous RCU batch is in progress, it means that // there is on-going rcu work to handle krwp's free business.
// immediately to queue another one is not possible so if (need_wait_for_krwp_work(krwp))
// in that case the monitor work is rearmed. continue;
if ((!list_empty(&krcp->bulk_head[0]) && list_empty(&krwp->bulk_head_free[0])) ||
(!list_empty(&krcp->bulk_head[1]) && list_empty(&krwp->bulk_head_free[1])) ||
(READ_ONCE(krcp->head) && !krwp->head_free)) {
// kvfree_rcu_drain_ready() might handle this krcp, if so give up.
if (need_offload_krc(krcp)) {
// Channel 1 corresponds to the SLAB-pointer bulk path. // Channel 1 corresponds to the SLAB-pointer bulk path.
// Channel 2 corresponds to vmalloc-pointer bulk path. // Channel 2 corresponds to vmalloc-pointer bulk path.
for (j = 0; j < FREE_N_CHANNELS; j++) { for (j = 0; j < FREE_N_CHANNELS; j++) {
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment