• Shakeel Butt's avatar
    mm: convert mm's rss stats into percpu_counter · f1a79412
    Shakeel Butt authored
    Currently mm_struct maintains rss_stats which are updated on page fault
    and the unmapping codepaths.  For page fault codepath the updates are
    cached per thread with the batch of TASK_RSS_EVENTS_THRESH which is 64. 
    The reason for caching is performance for multithreaded applications
    otherwise the rss_stats updates may become hotspot for such applications.
    
    However this optimization comes with the cost of error margin in the rss
    stats.  The rss_stats for applications with large number of threads can be
    very skewed.  At worst the error margin is (nr_threads * 64) and we have a
    lot of applications with 100s of threads, so the error margin can be very
    high.  Internally we had to reduce TASK_RSS_EVENTS_THRESH to 32.
    
    Recently we started seeing the unbounded errors for rss_stats for specific
    applications which use TCP rx0cp.  It seems like vm_insert_pages()
    codepath does not sync rss_stats at all.
    
    This patch converts the rss_stats into percpu_counter to convert the error
    margin from (nr_threads * 64) to approximately (nr_cpus ^ 2).  However
    this conversion enable us to get the accurate stats for situations where
    accuracy is more important than the cpu cost.
    
    This patch does not make such tradeoffs - we can just use
    percpu_counter_add_local() for the updates and percpu_counter_sum() (or
    percpu_counter_sync() + percpu_counter_read) for the readers.  At the
    moment the readers are either procfs interface, oom_killer and memory
    reclaim which I think are not performance critical and should be ok with
    slow read.  However I think we can make that change in a separate patch.
    
    Link: https://lkml.kernel.org/r/20221024052841.3291983-1-shakeelb@google.comSigned-off-by: default avatarShakeel Butt <shakeelb@google.com>
    Cc: Marek Szyprowski <m.szyprowski@samsung.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    f1a79412
kmem.h 8.63 KB