• Qi Zheng's avatar
    mm: vmscan: make memcg slab shrink lockless · caa05325
    Qi Zheng authored
    Like global slab shrink, this commit also uses SRCU to make memcg slab
    shrink lockless.
    
    We can reproduce the down_read_trylock() hotspot through the
    following script:
    
    ```
    
    DIR="/root/shrinker/memcg/mnt"
    
    do_create()
    {
        mkdir -p /sys/fs/cgroup/memory/test
        mkdir -p /sys/fs/cgroup/perf_event/test
        echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes
        for i in `seq 0 $1`;
        do
            mkdir -p /sys/fs/cgroup/memory/test/$i;
            echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs;
            echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs;
            mkdir -p $DIR/$i;
        done
    }
    
    do_mount()
    {
        for i in `seq $1 $2`;
        do
            mount -t tmpfs $i $DIR/$i;
        done
    }
    
    do_touch()
    {
        for i in `seq $1 $2`;
        do
            echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs;
            echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs;
                dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 &
        done
    }
    
    case "$1" in
      touch)
        do_touch $2 $3
        ;;
      test)
          do_create 4000
        do_mount 0 4000
        do_touch 0 3000
        ;;
      *)
        exit 1
        ;;
    esac
    ```
    
    Save the above script, then run test and touch commands.
    Then we can use the following perf command to view hotspots:
    
    perf top -U -F 999
    
    1) Before applying this patchset:
    
      32.31%  [kernel]           [k] down_read_trylock
      19.40%  [kernel]           [k] pv_native_safe_halt
      16.24%  [kernel]           [k] up_read
      15.70%  [kernel]           [k] shrink_slab
       4.69%  [kernel]           [k] _find_next_bit
       2.62%  [kernel]           [k] shrink_node
       1.78%  [kernel]           [k] shrink_lruvec
       0.76%  [kernel]           [k] do_shrink_slab
    
    2) After applying this patchset:
    
      27.83%  [kernel]           [k] _find_next_bit
      16.97%  [kernel]           [k] shrink_slab
      15.82%  [kernel]           [k] pv_native_safe_halt
       9.58%  [kernel]           [k] shrink_node
       8.31%  [kernel]           [k] shrink_lruvec
       5.64%  [kernel]           [k] do_shrink_slab
       3.88%  [kernel]           [k] mem_cgroup_iter
    
    At the same time, we use the following perf command to capture
    IPC information:
    
    perf stat -e cycles,instructions -G test -a --repeat 5 -- sleep 10
    
    1) Before applying this patchset:
    
     Performance counter stats for 'system wide' (5 runs):
    
          454187219766      cycles                    test                    ( +-  1.84% )
           78896433101      instructions              test #    0.17  insn per cycle           ( +-  0.44% )
    
            10.0020430 +- 0.0000366 seconds time elapsed  ( +-  0.00% )
    
    2) After applying this patchset:
    
     Performance counter stats for 'system wide' (5 runs):
    
          841954709443      cycles                    test                    ( +- 15.80% )  (98.69%)
          527258677936      instructions              test #    0.63  insn per cycle           ( +- 15.11% )  (98.68%)
    
              10.01064 +- 0.00831 seconds time elapsed  ( +-  0.08% )
    
    We can see that IPC drops very seriously when calling
    down_read_trylock() at high frequency. After using SRCU,
    the IPC is at a normal level.
    
    Link: https://lkml.kernel.org/r/20230313112819.38938-4-zhengqi.arch@bytedance.comSigned-off-by: default avatarQi Zheng <zhengqi.arch@bytedance.com>
    Acked-by: default avatarKirill Tkhai <tkhai@ya.ru>
    Acked-by: default avatarVlastimil Babka <Vbabka@suse.cz>
    Acked-by: default avatarRoman Gushchin <roman.gushchin@linux.dev>
    Cc: Christian König <christian.koenig@amd.com>
    Cc: David Hildenbrand <david@redhat.com>
    Cc: Davidlohr Bueso <dave@stgolabs.net>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Michal Hocko <mhocko@kernel.org>
    Cc: Muchun Song <muchun.song@linux.dev>
    Cc: Paul E. McKenney <paulmck@kernel.org>
    Cc: Shakeel Butt <shakeelb@google.com>
    Cc: Sultan Alsawaf <sultan@kerneltoast.com>
    Cc: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
    Cc: Yang Shi <shy828301@gmail.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    caa05325
vmscan.c 221 KB