• Mathieu Desnoyers's avatar
    sched: Fix performance regression introduced by mm_cid · 223baf9d
    Mathieu Desnoyers authored
    Introduce per-mm/cpu current concurrency id (mm_cid) to fix a PostgreSQL
    sysbench regression reported by Aaron Lu.
    
    Keep track of the currently allocated mm_cid for each mm/cpu rather than
    freeing them immediately on context switch. This eliminates most atomic
    operations when context switching back and forth between threads
    belonging to different memory spaces in multi-threaded scenarios (many
    processes, each with many threads). The per-mm/per-cpu mm_cid values are
    serialized by their respective runqueue locks.
    
    Thread migration is handled by introducing invocation to
    sched_mm_cid_migrate_to() (with destination runqueue lock held) in
    activate_task() for migrating tasks. If the destination cpu's mm_cid is
    unset, and if the source runqueue is not actively using its mm_cid, then
    the source cpu's mm_cid is moved to the destination cpu on migration.
    
    Introduce a task-work executed periodically, similarly to NUMA work,
    which delays reclaim of cid values when they are unused for a period of
    time.
    
    Keep track of the allocation time for each per-cpu cid, and let the task
    work clear them when they are observed to be older than
    SCHED_MM_CID_PERIOD_NS and unused. This task work also clears all
    mm_cids which are greater or equal to the Hamming weight of the mm
    cidmask to keep concurrency ids compact.
    
    Because we want to ensure the mm_cid converges towards the smaller
    values as migrations happen, the prior optimization that was done when
    context switching between threads belonging to the same mm is removed,
    because it could delay the lazy release of the destination runqueue
    mm_cid after it has been replaced by a migration. Removing this prior
    optimization is not an issue performance-wise because the introduced
    per-mm/per-cpu mm_cid tracking also covers this more specific case.
    
    Fixes: af7f588d ("sched: Introduce per-memory-map concurrency ID")
    Reported-by: default avatarAaron Lu <aaron.lu@intel.com>
    Signed-off-by: default avatarMathieu Desnoyers <mathieu.desnoyers@efficios.com>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Tested-by: default avatarAaron Lu <aaron.lu@intel.com>
    Link: https://lore.kernel.org/lkml/20230327080502.GA570847@ziqianlu-desk2/
    223baf9d
core.c 306 KB