• Roman Gushchin's avatar
    mm: memcontrol: flush percpu vmstats before releasing memcg · c350a99e
    Roman Gushchin authored
    Percpu caching of local vmstats with the conditional propagation by the
    cgroup tree leads to an accumulation of errors on non-leaf levels.
    
    Let's imagine two nested memory cgroups A and A/B.  Say, a process
    belonging to A/B allocates 100 pagecache pages on the CPU 0.  The percpu
    cache will spill 3 times, so that 32*3=96 pages will be accounted to A/B
    and A atomic vmstat counters, 4 pages will remain in the percpu cache.
    
    Imagine A/B is nearby memory.max, so that every following allocation
    triggers a direct reclaim on the local CPU.  Say, each such attempt will
    free 16 pages on a new cpu.  That means every percpu cache will have -16
    pages, except the first one, which will have 4 - 16 = -12.  A/B and A
    atomic counters will not be touched at all.
    
    Now a user removes A/B.  All percpu caches are freed and corresponding
    vmstat numbers are forgotten.  A has 96 pages more than expected.
    
    As memory cgroups are created and destroyed, errors do accumulate.  Even
    1-2 pages differences can accumulate into large numbers.
    
    To fix this issue let's accumulate and propagate percpu vmstat values
    before releasing the memory cgroup.  At this point these numbers are
    stable and cannot be changed.
    
    Since on cpu hotplug we do flush percpu vmstats anyway, we can iterate
    only over online cpus.
    
    Link: http://lkml.kernel.org/r/20190819202338.363363-2-guro@fb.com
    Fixes: 42a30035 ("mm: memcontrol: fix recursive statistics correctness & scalabilty")
    Signed-off-by: default avatarRoman Gushchin <guro@fb.com>
    Acked-by: default avatarMichal Hocko <mhocko@suse.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
    Cc: <stable@vger.kernel.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    c350a99e
memcontrol.c 177 KB