• Marco Elver's avatar
    kfence: await for allocation using wait_event · 407f1d8c
    Marco Elver authored
    Patch series "kfence: optimize timer scheduling", v2.
    
    We have observed that mostly-idle systems with KFENCE enabled wake up
    otherwise idle CPUs, preventing such to enter a lower power state.
    Debugging revealed that KFENCE spends too much active time in
    toggle_allocation_gate().
    
    While the first version of KFENCE was using all the right bits to be
    scheduling optimal, and thus power efficient, by simply using wait_event()
    + wake_up(), that code was unfortunately removed.
    
    As KFENCE was exposed to various different configs and tests, the
    scheduling optimal code slowly disappeared.  First because of hung task
    warnings, and finally because of deadlocks when an allocation is made by
    timer code with debug objects enabled.  Clearly, the "fixes" were not too
    friendly for devices that want to be power efficient.
    
    Therefore, let's try a little harder to fix the hung task and deadlock
    problems that we have with wait_event() + wake_up(), while remaining as
    scheduling friendly and power efficient as possible.
    
    Crucially, we need to defer the wake_up() to an irq_work, avoiding any
    potential for deadlock.
    
    The result with this series is that on the devices where we observed a
    power regression, power usage returns back to baseline levels.
    
    This patch (of 3):
    
    On mostly-idle systems, we have observed that toggle_allocation_gate() is
    a cause of frequent wake-ups, preventing an otherwise idle CPU to go into
    a lower power state.
    
    A late change in KFENCE's development, due to a potential deadlock [1],
    required changing the scheduling-friendly wait_event_timeout() and
    wake_up() to an open-coded wait-loop using schedule_timeout().  [1]
    https://lkml.kernel.org/r/000000000000c0645805b7f982e4@google.com
    
    To avoid unnecessary wake-ups, switch to using wait_event_timeout().
    
    Unfortunately, we still cannot use a version with direct wake_up() in
    __kfence_alloc() due to the same potential for deadlock as in [1].
    Instead, add a level of indirection via an irq_work that is scheduled if
    we determine that the kfence_timer requires a wake_up().
    
    Link: https://lkml.kernel.org/r/20210421105132.3965998-1-elver@google.com
    Link: https://lkml.kernel.org/r/20210421105132.3965998-2-elver@google.com
    Fixes: 0ce20dd8 ("mm: add Kernel Electric-Fence infrastructure")
    Signed-off-by: default avatarMarco Elver <elver@google.com>
    Cc: Alexander Potapenko <glider@google.com>
    Cc: Dmitry Vyukov <dvyukov@google.com>
    Cc: Jann Horn <jannh@google.com>
    Cc: Mark Rutland <mark.rutland@arm.com>
    Cc: Hillf Danton <hdanton@sina.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    407f1d8c
core.c 25.7 KB