• KAMEZAWA Hiroyuki's avatar
    memcg: fix performance of mem_cgroup_begin_update_page_stat() · 4331f7d3
    KAMEZAWA Hiroyuki authored
    mem_cgroup_begin_update_page_stat() should be very fast because it's
    called very frequently.  Now, it needs to look up page_cgroup and its
    memcg....this is slow.
    
    This patch adds a global variable to check "any memcg is moving or not".
    With this, the caller doesn't need to visit page_cgroup and memcg.
    
    Here is a test result.  A test program makes page faults onto a file,
    MAP_SHARED and makes each page's page_mapcount(page) > 1, and free the
    range by madvise() and page fault again.  This program causes 26214400
    times of page fault onto a file(size was 1G.) and shows shows the cost of
    mem_cgroup_begin_update_page_stat().
    
    Before this patch for mem_cgroup_begin_update_page_stat()
    
        [kamezawa@bluextal test]$ time ./mmap 1G
    
        real    0m21.765s
        user    0m5.999s
        sys     0m15.434s
    
        27.46%     mmap  mmap               [.] reader
        21.15%     mmap  [kernel.kallsyms]  [k] page_fault
         9.17%     mmap  [kernel.kallsyms]  [k] filemap_fault
         2.96%     mmap  [kernel.kallsyms]  [k] __do_fault
         2.83%     mmap  [kernel.kallsyms]  [k] __mem_cgroup_begin_update_page_stat
    
    After this patch
    
        [root@bluextal test]# time ./mmap 1G
    
        real    0m21.373s
        user    0m6.113s
        sys     0m15.016s
    
    In usual path, calls to __mem_cgroup_begin_update_page_stat() goes away.
    
    Note: we may be able to remove this optimization in future if
          we can get pointer to memcg directly from struct page.
    
    [akpm@linux-foundation.org: don't return a void]
    Signed-off-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
    Acked-by: default avatarGreg Thelen <gthelen@google.com>
    Acked-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
    Cc: Michal Hocko <mhocko@suse.cz>
    Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
    Cc: Ying Han <yinghan@google.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    4331f7d3
memcontrol.c 145 KB