1. 06 Nov, 2021 40 commits
    • Mike Rapoport's avatar
      memblock: rename memblock_free to memblock_phys_free · 3ecc6834
      Mike Rapoport authored
      Since memblock_free() operates on a physical range, make its name
      reflect it and rename it to memblock_phys_free(), so it will be a
      logical counterpart to memblock_phys_alloc().
      
      The callers are updated with the below semantic patch:
      
          @@
          expression addr;
          expression size;
          @@
          - memblock_free(addr, size);
          + memblock_phys_free(addr, size);
      
      Link: https://lkml.kernel.org/r/20210930185031.18648-6-rppt@kernel.orgSigned-off-by: default avatarMike Rapoport <rppt@linux.ibm.com>
      Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
      Cc: Juergen Gross <jgross@suse.com>
      Cc: Shahab Vahedi <Shahab.Vahedi@synopsys.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      3ecc6834
    • Mike Rapoport's avatar
      memblock: stop aliasing __memblock_free_late with memblock_free_late · 621d9739
      Mike Rapoport authored
      memblock_free_late() is a NOP wrapper for __memblock_free_late(), there
      is no point to keep this indirection.
      
      Drop the wrapper and rename __memblock_free_late() to
      memblock_free_late().
      
      Link: https://lkml.kernel.org/r/20210930185031.18648-5-rppt@kernel.orgSigned-off-by: default avatarMike Rapoport <rppt@linux.ibm.com>
      Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
      Cc: Juergen Gross <jgross@suse.com>
      Cc: Shahab Vahedi <Shahab.Vahedi@synopsys.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      621d9739
    • Mike Rapoport's avatar
      memblock: drop memblock_free_early_nid() and memblock_free_early() · fa277171
      Mike Rapoport authored
      memblock_free_early_nid() is unused and memblock_free_early() is an
      alias for memblock_free().
      
      Replace calls to memblock_free_early() with calls to memblock_free() and
      remove memblock_free_early() and memblock_free_early_nid().
      
      Link: https://lkml.kernel.org/r/20210930185031.18648-4-rppt@kernel.orgSigned-off-by: default avatarMike Rapoport <rppt@linux.ibm.com>
      Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
      Cc: Juergen Gross <jgross@suse.com>
      Cc: Shahab Vahedi <Shahab.Vahedi@synopsys.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      fa277171
    • Mike Rapoport's avatar
      xen/x86: free_p2m_page: use memblock_free_ptr() to free a virtual pointer · c486514d
      Mike Rapoport authored
      free_p2m_page() wrongly passes a virtual pointer to memblock_free() that
      treats it as a physical address.
      
      Call memblock_free_ptr() instead that gets a virtual address to free the
      memory.
      
      Link: https://lkml.kernel.org/r/20210930185031.18648-3-rppt@kernel.orgSigned-off-by: default avatarMike Rapoport <rppt@linux.ibm.com>
      Reviewed-by: default avatarJuergen Gross <jgross@suse.com>
      Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
      Cc: Shahab Vahedi <Shahab.Vahedi@synopsys.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      c486514d
    • Mike Rapoport's avatar
      arch_numa: simplify numa_distance allocation · 5787ea5b
      Mike Rapoport authored
      Patch series "memblock: cleanup memblock_free interface", v2.
      
      This is the fix for memblock freeing APIs mismatch [1].
      
      The first patch is a cleanup of numa_distance allocation in arch_numa
      I've spotted during the conversion.  The second patch is a fix for Xen
      memory freeing on some of the error paths.
      
      [1] https://lore.kernel.org/all/CAHk-=wj9k4LZTz+svCxLYs5Y1=+yKrbAUArH1+ghyG3OLd8VVg@mail.gmail.com
      
      This patch (of 6):
      
      Memory allocation of numa_distance uses memblock_phys_alloc_range()
      without actual range limits, converts the returned physical address to
      virtual and then only uses the virtual address for further
      initialization.
      
      Simplify this by replacing memblock_phys_alloc_range() with
      memblock_alloc().
      
      Link: https://lkml.kernel.org/r/20210930185031.18648-1-rppt@kernel.org
      Link: https://lkml.kernel.org/r/20210930185031.18648-2-rppt@kernel.orgSigned-off-by: default avatarMike Rapoport <rppt@linux.ibm.com>
      Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
      Cc: Juergen Gross <jgross@suse.com>
      Cc: Shahab Vahedi <Shahab.Vahedi@synopsys.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      5787ea5b
    • Naoya Horiguchi's avatar
      tools/vm/page-types.c: print file offset in hexadecimal · 41d4613b
      Naoya Horiguchi authored
      In page list mode (with -l and -L option), virtual address and physical
      address are printed in hexadecimal, but file offset is not, which is
      confusing, so let's align it.
      
      Link: https://lkml.kernel.org/r/20211004061325.1525902-4-naoya.horiguchi@linux.devSigned-off-by: default avatarNaoya Horiguchi <naoya.horiguchi@nec.com>
      Cc: Bin Wang <wangbin224@huawei.com>
      Cc: Changbin Du <changbin.du@intel.com>
      Cc: Christian Hansen <chansen3@cisco.com>
      Cc: Konstantin Khlebnikov <koct9i@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      41d4613b
    • Naoya Horiguchi's avatar
      tools/vm/page-types.c: move show_file() to summary output · b76901db
      Naoya Horiguchi authored
      Currently file info from show_file() is printed out within page list
      like below, but this is inconvenient a little to utilize the page list
      from other scripts (maybe needs additional filtering).
      
          $ ./page-types -f page-types.c -l
          foffset offset  len     flags
          page-types.c Inode: 15108680 Size: 30953 (8 pages)
          Modify: Sat Oct  2 23:11:20 2021 (2399 seconds ago)
          Access: Sat Oct  2 23:11:28 2021 (2391 seconds ago)
          0       d9f59e  1       ___U_lA____________________________________
          1       1031eb5 1       __RU_l_____________________________________
          2       13bf717 1       __RU_l_____________________________________
          3       13ac333 1       ___U_lA____________________________________
          4       d9f59f  1       __RU_l_____________________________________
          5       183fd49 1       ___U_lA____________________________________
          6       13cbf69 1       ___U_lA____________________________________
          7       d9ef05  1       ___U_lA____________________________________
      
                       flags      page-count       MB  symbolic-flags                     long-symbolic-flags
          0x000000000000002c               3        0  __RU_l_____________________________________        referenced,uptodate,lru
          0x0000000000000068               5        0  ___U_lA____________________________________        uptodate,lru,active
                       total               8        0
      
      With this patch file info is printed out in summary part like below:
      
          $ ./page-types -f page-types.c -l
          foffset offset  len     flags
          0       d9f59e  1       ___U_lA_____________________________________
          1       1031eb5 1       __RU_l______________________________________
          2       13bf717 1       __RU_l______________________________________
          3       13ac333 1       ___U_lA_____________________________________
          4       d9f59f  1       __RU_l______________________________________
          5       183fd49 1       ___U_lA_____________________________________
          6       13cbf69 1       ___U_lA_____________________________________
      
          page-types.c Inode: 15108680 Size: 30953 (8 pages)
          Modify: Sat Oct  2 23:11:20 2021 (2435 seconds ago)
          Access: Sat Oct  2 23:11:28 2021 (2427 seconds ago)
      
                       flags      page-count       MB  symbolic-flags                     long-symbolic-flags
          0x000000000000002c               3        0  __RU_l______________________________________       referenced,uptodate,lru
          0x0000000000000068               4        0  ___U_lA_____________________________________       uptodate,lru,active
                       total               7        0
      
      Link: https://lkml.kernel.org/r/20211004061325.1525902-3-naoya.horiguchi@linux.devSigned-off-by: default avatarNaoya Horiguchi <naoya.horiguchi@nec.com>
      Cc: Bin Wang <wangbin224@huawei.com>
      Cc: Changbin Du <changbin.du@intel.com>
      Cc: Christian Hansen <chansen3@cisco.com>
      Cc: Konstantin Khlebnikov <koct9i@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b76901db
    • Naoya Horiguchi's avatar
      tools/vm/page-types.c: make walk_file() aware of address range option · a62f5ecb
      Naoya Horiguchi authored
      Patch series "tools/vm/page-types.c: a few improvements".
      
      This patchset adds some improvements on tools/vm/page-types.c.  Patch
      1/3 makes -a option (specify address range) work with -f (file cache
      mode).  Patch 2/3 and 3/3 are to fix minor formatting issues of this
      tool.  These would make life a little easier for the users of this tool.
      
      Please see individual patches for more details about specific issues.
      
      This patch (of 3):
      
      -a|--addr option is used to limit the range of address to be scanned for
      page status.  It works now for physical address space (dafult mode) or for
      virtual address space (with -p option), but not for file address space
      (with -f option).  So make walk_file() aware of -a option.
      
      Link: https://lkml.kernel.org/r/20211004061325.1525902-1-naoya.horiguchi@linux.dev
      Link: https://lkml.kernel.org/r/20211004061325.1525902-2-naoya.horiguchi@linux.devSigned-off-by: default avatarNaoya Horiguchi <naoya.horiguchi@nec.com>
      Cc: Konstantin Khlebnikov <koct9i@gmail.com>
      Cc: Christian Hansen <chansen3@cisco.com>
      Cc: Changbin Du <changbin.du@intel.com>
      Cc: Bin Wang <wangbin224@huawei.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      a62f5ecb
    • Zhenliang Wei's avatar
      tools/vm/page_owner_sort.c: count and sort by mem · f7df2b1c
      Zhenliang Wei authored
      When viewing page owner information, we may be more concerned about the
      total memory rather than the times of stack appears.  Therefore, the
      following adjustments are made:
      
      1. Added the statistics on the total number of pages.
      
      2. Added the optional parameter "-m" to configure the program to sort by
         memory (total pages).
      
      The general output of page_owner is as follows:
      
      	Page allocated via order XXX, ...
      	PFN XXX ...
      	 // Detailed stack
      
      	Page allocated via order XXX, ...
      	PFN XXX ...
      	 // Detailed stack
      
      The original page_owner_sort ignores PFN rows, puts the remaining rows
      in buf, counts the times of buf, and finally sorts them according to the
      times.  General output:
      
      	XXX times:
      	Page allocated via order XXX, ...
      	 // Detailed stack
      
      Now, we use regexp to extract the page order value from the buf, and
      count the total pages for the buf.  General output:
      
      	XXX times, XXX pages:
      	Page allocated via order XXX, ...
      	 // Detailed stack
      
      By default, it is still sorted by the times of buf; If you want to sort
      by the pages nums of buf, use the new -m parameter.
      
      Link: https://lkml.kernel.org/r/1631678242-41033-1-git-send-email-weizhenliang@huawei.comSigned-off-by: default avatarZhenliang Wei <weizhenliang@huawei.com>
      Cc: Tang Bin <tangbin@cmss.chinamobile.com>
      Cc: Zhang Shengju <zhangshengju@cmss.chinamobile.com>
      Cc: Zhenliang Wei <weizhenliang@huawei.com>
      Cc: Xiaoming Ni <nixiaoming@huawei.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      f7df2b1c
    • Yuanzheng Song's avatar
      mm/vmpressure: fix data-race with memcg->socket_pressure · 7e6ec49c
      Yuanzheng Song authored
      When reading memcg->socket_pressure in mem_cgroup_under_socket_pressure()
      and writing memcg->socket_pressure in vmpressure() at the same time, the
      following data-race occurs:
      
        BUG: KCSAN: data-race in __sk_mem_reduce_allocated / vmpressure
      
        write to 0xffff8881286f4938 of 8 bytes by task 24550 on cpu 3:
         vmpressure+0x218/0x230 mm/vmpressure.c:307
         shrink_node_memcgs+0x2b9/0x410 mm/vmscan.c:2658
         shrink_node+0x9d2/0x11d0 mm/vmscan.c:2769
         shrink_zones+0x29f/0x470 mm/vmscan.c:2972
         do_try_to_free_pages+0x193/0x6e0 mm/vmscan.c:3027
         try_to_free_mem_cgroup_pages+0x1c0/0x3f0 mm/vmscan.c:3345
         reclaim_high mm/memcontrol.c:2440 [inline]
         mem_cgroup_handle_over_high+0x18b/0x4d0 mm/memcontrol.c:2624
         tracehook_notify_resume include/linux/tracehook.h:197 [inline]
         exit_to_user_mode_loop kernel/entry/common.c:164 [inline]
         exit_to_user_mode_prepare+0x110/0x170 kernel/entry/common.c:191
         syscall_exit_to_user_mode+0x16/0x30 kernel/entry/common.c:266
         ret_from_fork+0x15/0x30 arch/x86/entry/entry_64.S:289
      
        read to 0xffff8881286f4938 of 8 bytes by interrupt on cpu 1:
         mem_cgroup_under_socket_pressure include/linux/memcontrol.h:1483 [inline]
         sk_under_memory_pressure include/net/sock.h:1314 [inline]
         __sk_mem_reduce_allocated+0x1d2/0x270 net/core/sock.c:2696
         __sk_mem_reclaim+0x44/0x50 net/core/sock.c:2711
         sk_mem_reclaim include/net/sock.h:1490 [inline]
         ......
         net_rx_action+0x17a/0x480 net/core/dev.c:6864
         __do_softirq+0x12c/0x2af kernel/softirq.c:298
         run_ksoftirqd+0x13/0x20 kernel/softirq.c:653
         smpboot_thread_fn+0x33f/0x510 kernel/smpboot.c:165
         kthread+0x1fc/0x220 kernel/kthread.c:292
         ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:296
      
      Fix it by using READ_ONCE() and WRITE_ONCE() to read and write
      memcg->socket_pressure.
      
      Link: https://lkml.kernel.org/r/20211025082843.671690-1-songyuanzheng@huawei.comSigned-off-by: default avatarYuanzheng Song <songyuanzheng@huawei.com>
      Reviewed-by: default avatarMuchun Song <songmuchun@bytedance.com>
      Cc: Shakeel Butt <shakeelb@google.com>
      Cc: Roman Gushchin <guro@fb.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
      Cc: Alex Shi <alexs@kernel.org>
      Cc: Wei Yang <richard.weiyang@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      7e6ec49c
    • Mel Gorman's avatar
      mm/vmscan: delay waking of tasks throttled on NOPROGRESS · 66ce520b
      Mel Gorman authored
      Tracing indicates that tasks throttled on NOPROGRESS are woken
      prematurely resulting in occasional massive spikes in direct reclaim
      activity.  This patch wakes tasks throttled on NOPROGRESS if reclaim
      efficiency is at least 12%.
      
      Link: https://lkml.kernel.org/r/20211022144651.19914-9-mgorman@techsingularity.netSigned-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: "Darrick J . Wong" <djwong@kernel.org>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Matthew Wilcox <willy@infradead.org>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: NeilBrown <neilb@suse.de>
      Cc: Rik van Riel <riel@surriel.com>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      66ce520b
    • Mel Gorman's avatar
      mm/vmscan: increase the timeout if page reclaim is not making progress · a19594ca
      Mel Gorman authored
      Tracing of the stutterp workload showed the following delays
      
            1 usect_delayed=124000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usect_delayed=128000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usect_delayed=176000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usect_delayed=536000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usect_delayed=544000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usect_delayed=556000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usect_delayed=624000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usect_delayed=716000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usect_delayed=772000 reason=VMSCAN_THROTTLE_NOPROGRESS
            2 usect_delayed=512000 reason=VMSCAN_THROTTLE_NOPROGRESS
           16 usect_delayed=120000 reason=VMSCAN_THROTTLE_NOPROGRESS
           53 usect_delayed=116000 reason=VMSCAN_THROTTLE_NOPROGRESS
          116 usect_delayed=112000 reason=VMSCAN_THROTTLE_NOPROGRESS
         5907 usect_delayed=108000 reason=VMSCAN_THROTTLE_NOPROGRESS
        71741 usect_delayed=104000 reason=VMSCAN_THROTTLE_NOPROGRESS
      
      All the throttling hit the full timeout and then there was wakeup delays
      meaning that the wakeups are premature as no other reclaimer such as
      kswapd has made progress.  This patch increases the maximum timeout.
      
      Link: https://lkml.kernel.org/r/20211022144651.19914-8-mgorman@techsingularity.netSigned-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: "Darrick J . Wong" <djwong@kernel.org>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Matthew Wilcox <willy@infradead.org>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: NeilBrown <neilb@suse.de>
      Cc: Rik van Riel <riel@surriel.com>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      a19594ca
    • Mel Gorman's avatar
      mm/vmscan: centralise timeout values for reclaim_throttle · c3f4a9a2
      Mel Gorman authored
      Neil Brown raised concerns about callers of reclaim_throttle specifying
      a timeout value.  The original timeout values to congestion_wait() were
      probably pulled out of thin air or copy&pasted from somewhere else.
      This patch centralises the timeout values and selects a timeout based on
      the reason for reclaim throttling.  These figures are also pulled out of
      the same thin air but better values may be derived
      
      Running a workload that is throttling for inappropriate periods and
      tracing mm_vmscan_throttled can be used to pick a more appropriate
      value.  Excessive throttling would pick a lower timeout where as
      excessive CPU usage in reclaim context would select a larger timeout.
      Ideally a large value would always be used and the wakeups would occur
      before a timeout but that requires careful testing.
      
      Link: https://lkml.kernel.org/r/20211022144651.19914-7-mgorman@techsingularity.netSigned-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: "Darrick J . Wong" <djwong@kernel.org>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Matthew Wilcox <willy@infradead.org>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: NeilBrown <neilb@suse.de>
      Cc: Rik van Riel <riel@surriel.com>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      c3f4a9a2
    • Mel Gorman's avatar
      mm/page_alloc: remove the throttling logic from the page allocator · 132b0d21
      Mel Gorman authored
      The page allocator stalls based on the number of pages that are waiting
      for writeback to start but this should now be redundant.
      shrink_inactive_list() will wake flusher threads if the LRU tail are
      unqueued dirty pages so the flusher should be active.  If it fails to
      make progress due to pages under writeback not being completed quickly
      then it should stall on VMSCAN_THROTTLE_WRITEBACK.
      
      Link: https://lkml.kernel.org/r/20211022144651.19914-6-mgorman@techsingularity.netSigned-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: "Darrick J . Wong" <djwong@kernel.org>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Matthew Wilcox <willy@infradead.org>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: NeilBrown <neilb@suse.de>
      Cc: Rik van Riel <riel@surriel.com>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      132b0d21
    • Mel Gorman's avatar
      mm/writeback: throttle based on page writeback instead of congestion · 8d58802f
      Mel Gorman authored
      do_writepages throttles on congestion if the writepages() fails due to a
      lack of memory but congestion_wait() is partially broken as the
      congestion state is not updated for all BDIs.
      
      This patch stalls waiting for a number of pages to complete writeback
      that located on the local node.  The main weakness is that there is no
      correlation between the location of the inode's pages and locality but
      that is still better than congestion_wait.
      
      Link: https://lkml.kernel.org/r/20211022144651.19914-5-mgorman@techsingularity.netSigned-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: "Darrick J . Wong" <djwong@kernel.org>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Matthew Wilcox <willy@infradead.org>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: NeilBrown <neilb@suse.de>
      Cc: Rik van Riel <riel@surriel.com>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8d58802f
    • Mel Gorman's avatar
      mm/vmscan: throttle reclaim when no progress is being made · 69392a40
      Mel Gorman authored
      Memcg reclaim throttles on congestion if no reclaim progress is made.
      This makes little sense, it might be due to writeback or a host of other
      factors.
      
      For !memcg reclaim, it's messy.  Direct reclaim primarily is throttled
      in the page allocator if it is failing to make progress.  Kswapd
      throttles if too many pages are under writeback and marked for immediate
      reclaim.
      
      This patch explicitly throttles if reclaim is failing to make progress.
      
      [vbabka@suse.cz: Remove redundant code]
      
      Link: https://lkml.kernel.org/r/20211022144651.19914-4-mgorman@techsingularity.netSigned-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: "Darrick J . Wong" <djwong@kernel.org>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Matthew Wilcox <willy@infradead.org>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: NeilBrown <neilb@suse.de>
      Cc: Rik van Riel <riel@surriel.com>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      69392a40
    • Mel Gorman's avatar
      mm/vmscan: throttle reclaim and compaction when too may pages are isolated · d818fca1
      Mel Gorman authored
      Page reclaim throttles on congestion if too many parallel reclaim
      instances have isolated too many pages.  This makes no sense, excessive
      parallelisation has nothing to do with writeback or congestion.
      
      This patch creates an additional workqueue to sleep on when too many
      pages are isolated.  The throttled tasks are woken when the number of
      isolated pages is reduced or a timeout occurs.  There may be some false
      positive wakeups for GFP_NOIO/GFP_NOFS callers but the tasks will
      throttle again if necessary.
      
      [shy828301@gmail.com: Wake up from compaction context]
      [vbabka@suse.cz: Account number of throttled tasks only for writeback]
      
      Link: https://lkml.kernel.org/r/20211022144651.19914-3-mgorman@techsingularity.netSigned-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: "Darrick J . Wong" <djwong@kernel.org>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Matthew Wilcox <willy@infradead.org>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: NeilBrown <neilb@suse.de>
      Cc: Rik van Riel <riel@surriel.com>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d818fca1
    • Mel Gorman's avatar
      mm/vmscan: throttle reclaim until some writeback completes if congested · 8cd7c588
      Mel Gorman authored
      Patch series "Remove dependency on congestion_wait in mm/", v5.
      
      This series that removes all calls to congestion_wait in mm/ and deletes
      wait_iff_congested.  It's not a clever implementation but
      congestion_wait has been broken for a long time [1].
      
      Even if congestion throttling worked, it was never a great idea.  While
      excessive dirty/writeback pages at the tail of the LRU is one
      possibility that reclaim may be slow, there is also the problem of too
      many pages being isolated and reclaim failing for other reasons
      (elevated references, too many pages isolated, excessive LRU contention
      etc).
      
      This series replaces the "congestion" throttling with 3 different types.
      
       - If there are too many dirty/writeback pages, sleep until a timeout or
         enough pages get cleaned
      
       - If too many pages are isolated, sleep until enough isolated pages are
         either reclaimed or put back on the LRU
      
       - If no progress is being made, direct reclaim tasks sleep until
         another task makes progress with acceptable efficiency.
      
      This was initially tested with a mix of workloads that used to trigger
      corner cases that no longer work.  A new test case was created called
      "stutterp" (pagereclaim-stutterp-noreaders in mmtests) using a freshly
      created XFS filesystem.  Note that it may be necessary to increase the
      timeout of ssh if executing remotely as ssh itself can get throttled and
      the connection may timeout.
      
      stutterp varies the number of "worker" processes from 4 up to NR_CPUS*4
      to check the impact as the number of direct reclaimers increase.  It has
      four types of worker.
      
       - One "anon latency" worker creates small mappings with mmap() and
         times how long it takes to fault the mapping reading it 4K at a time
      
       - X file writers which is fio randomly writing X files where the total
         size of the files add up to the allowed dirty_ratio. fio is allowed
         to run for a warmup period to allow some file-backed pages to
         accumulate. The duration of the warmup is based on the best-case
         linear write speed of the storage.
      
       - Y file readers which is fio randomly reading small files
      
       - Z anon memory hogs which continually map (100-dirty_ratio)% of memory
      
       - Total estimated WSS = (100+dirty_ration) percentage of memory
      
      X+Y+Z+1 == NR_WORKERS varying from 4 up to NR_CPUS*4
      
      The intent is to maximise the total WSS with a mix of file and anon
      memory where some anonymous memory must be swapped and there is a high
      likelihood of dirty/writeback pages reaching the end of the LRU.
      
      The test can be configured to have no background readers to stress
      dirty/writeback pages.  The results below are based on having zero
      readers.
      
      The short summary of the results is that the series works and stalls
      until some event occurs but the timeouts may need adjustment.
      
      The test results are not broken down by patch as the series should be
      treated as one block that replaces a broken throttling mechanism with a
      working one.
      
      Finally, three machines were tested but I'm reporting the worst set of
      results.  The other two machines had much better latencies for example.
      
      First the results of the "anon latency" latency
      
        stutterp
                                      5.15.0-rc1             5.15.0-rc1
                                         vanilla mm-reclaimcongest-v5r4
        Amean     mmap-4      31.4003 (   0.00%)   2661.0198 (-8374.52%)
        Amean     mmap-7      38.1641 (   0.00%)    149.2891 (-291.18%)
        Amean     mmap-12     60.0981 (   0.00%)    187.8105 (-212.51%)
        Amean     mmap-21    161.2699 (   0.00%)    213.9107 ( -32.64%)
        Amean     mmap-30    174.5589 (   0.00%)    377.7548 (-116.41%)
        Amean     mmap-48   8106.8160 (   0.00%)   1070.5616 (  86.79%)
        Stddev    mmap-4      41.3455 (   0.00%)  27573.9676 (-66591.66%)
        Stddev    mmap-7      53.5556 (   0.00%)   4608.5860 (-8505.23%)
        Stddev    mmap-12    171.3897 (   0.00%)   5559.4542 (-3143.75%)
        Stddev    mmap-21   1506.6752 (   0.00%)   5746.2507 (-281.39%)
        Stddev    mmap-30    557.5806 (   0.00%)   7678.1624 (-1277.05%)
        Stddev    mmap-48  61681.5718 (   0.00%)  14507.2830 (  76.48%)
        Max-90    mmap-4      31.4243 (   0.00%)     83.1457 (-164.59%)
        Max-90    mmap-7      41.0410 (   0.00%)     41.0720 (  -0.08%)
        Max-90    mmap-12     66.5255 (   0.00%)     53.9073 (  18.97%)
        Max-90    mmap-21    146.7479 (   0.00%)    105.9540 (  27.80%)
        Max-90    mmap-30    193.9513 (   0.00%)     64.3067 (  66.84%)
        Max-90    mmap-48    277.9137 (   0.00%)    591.0594 (-112.68%)
        Max       mmap-4    1913.8009 (   0.00%) 299623.9695 (-15555.96%)
        Max       mmap-7    2423.9665 (   0.00%) 204453.1708 (-8334.65%)
        Max       mmap-12   6845.6573 (   0.00%) 221090.3366 (-3129.64%)
        Max       mmap-21  56278.6508 (   0.00%) 213877.3496 (-280.03%)
        Max       mmap-30  19716.2990 (   0.00%) 216287.6229 (-997.00%)
        Max       mmap-48 477923.9400 (   0.00%) 245414.8238 (  48.65%)
      
      For most thread counts, the time to mmap() is unfortunately increased.
      In earlier versions of the series, this was lower but a large number of
      throttling events were reaching their timeout increasing the amount of
      inefficient scanning of the LRU.  There is no prioritisation of reclaim
      tasks making progress based on each tasks rate of page allocation versus
      progress of reclaim.  The variance is also impacted for high worker
      counts but in all cases, the differences in latency are not
      statistically significant due to very large maximum outliers.  Max-90
      shows that 90% of the stalls are comparable but the Max results show the
      massive outliers which are increased to to stalling.
      
      It is expected that this will be very machine dependant.  Due to the
      test design, reclaim is difficult so allocations stall and there are
      variances depending on whether THPs can be allocated or not.  The amount
      of memory will affect exactly how bad the corner cases are and how often
      they trigger.  The warmup period calculation is not ideal as it's based
      on linear writes where as fio is randomly writing multiple files from
      multiple tasks so the start state of the test is variable.  For example,
      these are the latencies on a single-socket machine that had more memory
      
        Amean     mmap-4      42.2287 (   0.00%)     49.6838 * -17.65%*
        Amean     mmap-7     216.4326 (   0.00%)     47.4451 *  78.08%*
        Amean     mmap-12   2412.0588 (   0.00%)     51.7497 (  97.85%)
        Amean     mmap-21   5546.2548 (   0.00%)     51.8862 (  99.06%)
        Amean     mmap-30   1085.3121 (   0.00%)     72.1004 (  93.36%)
      
      The overall system CPU usage and elapsed time is as follows
      
                          5.15.0-rc3  5.15.0-rc3
                             vanilla mm-reclaimcongest-v5r4
        Duration User        6989.03      983.42
        Duration System      7308.12      799.68
        Duration Elapsed     2277.67     2092.98
      
      The patches reduce system CPU usage by 89% as the vanilla kernel is rarely
      stalling.
      
      The high-level /proc/vmstats show
      
                                             5.15.0-rc1     5.15.0-rc1
                                                vanilla mm-reclaimcongest-v5r2
        Ops Direct pages scanned          1056608451.00   503594991.00
        Ops Kswapd pages scanned           109795048.00   147289810.00
        Ops Kswapd pages reclaimed          63269243.00    31036005.00
        Ops Direct pages reclaimed          10803973.00     6328887.00
        Ops Kswapd efficiency %                   57.62          21.07
        Ops Kswapd velocity                    48204.98       57572.86
        Ops Direct efficiency %                    1.02           1.26
        Ops Direct velocity                   463898.83      196845.97
      
      Kswapd scanned less pages but the detailed pattern is different.  The
      vanilla kernel scans slowly over time where as the patches exhibits
      burst patterns of scan activity.  Direct reclaim scanning is reduced by
      52% due to stalling.
      
      The pattern for stealing pages is also slightly different.  Both kernels
      exhibit spikes but the vanilla kernel when reclaiming shows pages being
      reclaimed over a period of time where as the patches tend to reclaim in
      spikes.  The difference is that vanilla is not throttling and instead
      scanning constantly finding some pages over time where as the patched
      kernel throttles and reclaims in spikes.
      
        Ops Percentage direct scans               90.59          77.37
      
      For direct reclaim, vanilla scanned 90.59% of pages where as with the
      patches, 77.37% were direct reclaim due to throttling
      
        Ops Page writes by reclaim           2613590.00     1687131.00
      
      Page writes from reclaim context are reduced.
      
        Ops Page writes anon                 2932752.00     1917048.00
      
      And there is less swapping.
      
        Ops Page reclaim immediate         996248528.00   107664764.00
      
      The number of pages encountered at the tail of the LRU tagged for
      immediate reclaim but still dirty/writeback is reduced by 89%.
      
        Ops Slabs scanned                     164284.00      153608.00
      
      Slab scan activity is similar.
      
      ftrace was used to gather stall activity
      
        Vanilla
        -------
            1 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=16000
            2 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=12000
            8 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=8000
           29 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=4000
        82394 writeback_wait_iff_congested: usec_timeout=100000 usec_delayed=0
      
      The fast majority of wait_iff_congested calls do not stall at all.  What
      is likely happening is that cond_resched() reschedules the task for a
      short period when the BDI is not registering congestion (which it never
      will in this test setup).
      
            1 writeback_congestion_wait: usec_timeout=100000 usec_delayed=120000
            2 writeback_congestion_wait: usec_timeout=100000 usec_delayed=132000
            4 writeback_congestion_wait: usec_timeout=100000 usec_delayed=112000
          380 writeback_congestion_wait: usec_timeout=100000 usec_delayed=108000
          778 writeback_congestion_wait: usec_timeout=100000 usec_delayed=104000
      
      congestion_wait if called always exceeds the timeout as there is no
      trigger to wake it up.
      
      Bottom line: Vanilla will throttle but it's not effective.
      
      Patch series
      ------------
      
      Kswapd throttle activity was always due to scanning pages tagged for
      immediate reclaim at the tail of the LRU
      
            1 usec_timeout=100000 usect_delayed=72000 reason=VMSCAN_THROTTLE_WRITEBACK
            4 usec_timeout=100000 usect_delayed=20000 reason=VMSCAN_THROTTLE_WRITEBACK
            5 usec_timeout=100000 usect_delayed=12000 reason=VMSCAN_THROTTLE_WRITEBACK
            6 usec_timeout=100000 usect_delayed=16000 reason=VMSCAN_THROTTLE_WRITEBACK
           11 usec_timeout=100000 usect_delayed=100000 reason=VMSCAN_THROTTLE_WRITEBACK
           11 usec_timeout=100000 usect_delayed=8000 reason=VMSCAN_THROTTLE_WRITEBACK
           94 usec_timeout=100000 usect_delayed=0 reason=VMSCAN_THROTTLE_WRITEBACK
          112 usec_timeout=100000 usect_delayed=4000 reason=VMSCAN_THROTTLE_WRITEBACK
      
      The majority of events did not stall or stalled for a short period.
      Roughly 16% of stalls reached the timeout before expiry.  For direct
      reclaim, the number of times stalled for each reason were
      
         6624 reason=VMSCAN_THROTTLE_ISOLATED
        93246 reason=VMSCAN_THROTTLE_NOPROGRESS
        96934 reason=VMSCAN_THROTTLE_WRITEBACK
      
      The most common reason to stall was due to excessive pages tagged for
      immediate reclaim at the tail of the LRU followed by a failure to make
      forward.  A relatively small number were due to too many pages isolated
      from the LRU by parallel threads
      
      For VMSCAN_THROTTLE_ISOLATED, the breakdown of delays was
      
            9 usec_timeout=20000 usect_delayed=4000 reason=VMSCAN_THROTTLE_ISOLATED
           12 usec_timeout=20000 usect_delayed=16000 reason=VMSCAN_THROTTLE_ISOLATED
           83 usec_timeout=20000 usect_delayed=20000 reason=VMSCAN_THROTTLE_ISOLATED
         6520 usec_timeout=20000 usect_delayed=0 reason=VMSCAN_THROTTLE_ISOLATED
      
      Most did not stall at all.  A small number reached the timeout.
      
      For VMSCAN_THROTTLE_NOPROGRESS, the breakdown of stalls were all over
      the map
      
            1 usec_timeout=500000 usect_delayed=324000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usec_timeout=500000 usect_delayed=332000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usec_timeout=500000 usect_delayed=348000 reason=VMSCAN_THROTTLE_NOPROGRESS
            1 usec_timeout=500000 usect_delayed=360000 reason=VMSCAN_THROTTLE_NOPROGRESS
            2 usec_timeout=500000 usect_delayed=228000 reason=VMSCAN_THROTTLE_NOPROGRESS
            2 usec_timeout=500000 usect_delayed=260000 reason=VMSCAN_THROTTLE_NOPROGRESS
            2 usec_timeout=500000 usect_delayed=340000 reason=VMSCAN_THROTTLE_NOPROGRESS
            2 usec_timeout=500000 usect_delayed=364000 reason=VMSCAN_THROTTLE_NOPROGRESS
            2 usec_timeout=500000 usect_delayed=372000 reason=VMSCAN_THROTTLE_NOPROGRESS
            2 usec_timeout=500000 usect_delayed=428000 reason=VMSCAN_THROTTLE_NOPROGRESS
            2 usec_timeout=500000 usect_delayed=460000 reason=VMSCAN_THROTTLE_NOPROGRESS
            2 usec_timeout=500000 usect_delayed=464000 reason=VMSCAN_THROTTLE_NOPROGRESS
            3 usec_timeout=500000 usect_delayed=244000 reason=VMSCAN_THROTTLE_NOPROGRESS
            3 usec_timeout=500000 usect_delayed=252000 reason=VMSCAN_THROTTLE_NOPROGRESS
            3 usec_timeout=500000 usect_delayed=272000 reason=VMSCAN_THROTTLE_NOPROGRESS
            4 usec_timeout=500000 usect_delayed=188000 reason=VMSCAN_THROTTLE_NOPROGRESS
            4 usec_timeout=500000 usect_delayed=268000 reason=VMSCAN_THROTTLE_NOPROGRESS
            4 usec_timeout=500000 usect_delayed=328000 reason=VMSCAN_THROTTLE_NOPROGRESS
            4 usec_timeout=500000 usect_delayed=380000 reason=VMSCAN_THROTTLE_NOPROGRESS
            4 usec_timeout=500000 usect_delayed=392000 reason=VMSCAN_THROTTLE_NOPROGRESS
            4 usec_timeout=500000 usect_delayed=432000 reason=VMSCAN_THROTTLE_NOPROGRESS
            5 usec_timeout=500000 usect_delayed=204000 reason=VMSCAN_THROTTLE_NOPROGRESS
            5 usec_timeout=500000 usect_delayed=220000 reason=VMSCAN_THROTTLE_NOPROGRESS
            5 usec_timeout=500000 usect_delayed=412000 reason=VMSCAN_THROTTLE_NOPROGRESS
            5 usec_timeout=500000 usect_delayed=436000 reason=VMSCAN_THROTTLE_NOPROGRESS
            6 usec_timeout=500000 usect_delayed=488000 reason=VMSCAN_THROTTLE_NOPROGRESS
            7 usec_timeout=500000 usect_delayed=212000 reason=VMSCAN_THROTTLE_NOPROGRESS
            7 usec_timeout=500000 usect_delayed=300000 reason=VMSCAN_THROTTLE_NOPROGRESS
            7 usec_timeout=500000 usect_delayed=316000 reason=VMSCAN_THROTTLE_NOPROGRESS
            7 usec_timeout=500000 usect_delayed=472000 reason=VMSCAN_THROTTLE_NOPROGRESS
            8 usec_timeout=500000 usect_delayed=248000 reason=VMSCAN_THROTTLE_NOPROGRESS
            8 usec_timeout=500000 usect_delayed=356000 reason=VMSCAN_THROTTLE_NOPROGRESS
            8 usec_timeout=500000 usect_delayed=456000 reason=VMSCAN_THROTTLE_NOPROGRESS
            9 usec_timeout=500000 usect_delayed=124000 reason=VMSCAN_THROTTLE_NOPROGRESS
            9 usec_timeout=500000 usect_delayed=376000 reason=VMSCAN_THROTTLE_NOPROGRESS
            9 usec_timeout=500000 usect_delayed=484000 reason=VMSCAN_THROTTLE_NOPROGRESS
           10 usec_timeout=500000 usect_delayed=172000 reason=VMSCAN_THROTTLE_NOPROGRESS
           10 usec_timeout=500000 usect_delayed=420000 reason=VMSCAN_THROTTLE_NOPROGRESS
           10 usec_timeout=500000 usect_delayed=452000 reason=VMSCAN_THROTTLE_NOPROGRESS
           11 usec_timeout=500000 usect_delayed=256000 reason=VMSCAN_THROTTLE_NOPROGRESS
           12 usec_timeout=500000 usect_delayed=112000 reason=VMSCAN_THROTTLE_NOPROGRESS
           12 usec_timeout=500000 usect_delayed=116000 reason=VMSCAN_THROTTLE_NOPROGRESS
           12 usec_timeout=500000 usect_delayed=144000 reason=VMSCAN_THROTTLE_NOPROGRESS
           12 usec_timeout=500000 usect_delayed=152000 reason=VMSCAN_THROTTLE_NOPROGRESS
           12 usec_timeout=500000 usect_delayed=264000 reason=VMSCAN_THROTTLE_NOPROGRESS
           12 usec_timeout=500000 usect_delayed=384000 reason=VMSCAN_THROTTLE_NOPROGRESS
           12 usec_timeout=500000 usect_delayed=424000 reason=VMSCAN_THROTTLE_NOPROGRESS
           12 usec_timeout=500000 usect_delayed=492000 reason=VMSCAN_THROTTLE_NOPROGRESS
           13 usec_timeout=500000 usect_delayed=184000 reason=VMSCAN_THROTTLE_NOPROGRESS
           13 usec_timeout=500000 usect_delayed=444000 reason=VMSCAN_THROTTLE_NOPROGRESS
           14 usec_timeout=500000 usect_delayed=308000 reason=VMSCAN_THROTTLE_NOPROGRESS
           14 usec_timeout=500000 usect_delayed=440000 reason=VMSCAN_THROTTLE_NOPROGRESS
           14 usec_timeout=500000 usect_delayed=476000 reason=VMSCAN_THROTTLE_NOPROGRESS
           16 usec_timeout=500000 usect_delayed=140000 reason=VMSCAN_THROTTLE_NOPROGRESS
           17 usec_timeout=500000 usect_delayed=232000 reason=VMSCAN_THROTTLE_NOPROGRESS
           17 usec_timeout=500000 usect_delayed=240000 reason=VMSCAN_THROTTLE_NOPROGRESS
           17 usec_timeout=500000 usect_delayed=280000 reason=VMSCAN_THROTTLE_NOPROGRESS
           18 usec_timeout=500000 usect_delayed=404000 reason=VMSCAN_THROTTLE_NOPROGRESS
           20 usec_timeout=500000 usect_delayed=148000 reason=VMSCAN_THROTTLE_NOPROGRESS
           20 usec_timeout=500000 usect_delayed=216000 reason=VMSCAN_THROTTLE_NOPROGRESS
           20 usec_timeout=500000 usect_delayed=468000 reason=VMSCAN_THROTTLE_NOPROGRESS
           21 usec_timeout=500000 usect_delayed=448000 reason=VMSCAN_THROTTLE_NOPROGRESS
           23 usec_timeout=500000 usect_delayed=168000 reason=VMSCAN_THROTTLE_NOPROGRESS
           23 usec_timeout=500000 usect_delayed=296000 reason=VMSCAN_THROTTLE_NOPROGRESS
           25 usec_timeout=500000 usect_delayed=132000 reason=VMSCAN_THROTTLE_NOPROGRESS
           25 usec_timeout=500000 usect_delayed=352000 reason=VMSCAN_THROTTLE_NOPROGRESS
           26 usec_timeout=500000 usect_delayed=180000 reason=VMSCAN_THROTTLE_NOPROGRESS
           27 usec_timeout=500000 usect_delayed=284000 reason=VMSCAN_THROTTLE_NOPROGRESS
           28 usec_timeout=500000 usect_delayed=164000 reason=VMSCAN_THROTTLE_NOPROGRESS
           29 usec_timeout=500000 usect_delayed=136000 reason=VMSCAN_THROTTLE_NOPROGRESS
           30 usec_timeout=500000 usect_delayed=200000 reason=VMSCAN_THROTTLE_NOPROGRESS
           30 usec_timeout=500000 usect_delayed=400000 reason=VMSCAN_THROTTLE_NOPROGRESS
           31 usec_timeout=500000 usect_delayed=196000 reason=VMSCAN_THROTTLE_NOPROGRESS
           32 usec_timeout=500000 usect_delayed=156000 reason=VMSCAN_THROTTLE_NOPROGRESS
           33 usec_timeout=500000 usect_delayed=224000 reason=VMSCAN_THROTTLE_NOPROGRESS
           35 usec_timeout=500000 usect_delayed=128000 reason=VMSCAN_THROTTLE_NOPROGRESS
           35 usec_timeout=500000 usect_delayed=176000 reason=VMSCAN_THROTTLE_NOPROGRESS
           36 usec_timeout=500000 usect_delayed=368000 reason=VMSCAN_THROTTLE_NOPROGRESS
           36 usec_timeout=500000 usect_delayed=496000 reason=VMSCAN_THROTTLE_NOPROGRESS
           37 usec_timeout=500000 usect_delayed=312000 reason=VMSCAN_THROTTLE_NOPROGRESS
           38 usec_timeout=500000 usect_delayed=304000 reason=VMSCAN_THROTTLE_NOPROGRESS
           40 usec_timeout=500000 usect_delayed=288000 reason=VMSCAN_THROTTLE_NOPROGRESS
           43 usec_timeout=500000 usect_delayed=408000 reason=VMSCAN_THROTTLE_NOPROGRESS
           55 usec_timeout=500000 usect_delayed=416000 reason=VMSCAN_THROTTLE_NOPROGRESS
           56 usec_timeout=500000 usect_delayed=76000 reason=VMSCAN_THROTTLE_NOPROGRESS
           58 usec_timeout=500000 usect_delayed=120000 reason=VMSCAN_THROTTLE_NOPROGRESS
           59 usec_timeout=500000 usect_delayed=208000 reason=VMSCAN_THROTTLE_NOPROGRESS
           61 usec_timeout=500000 usect_delayed=68000 reason=VMSCAN_THROTTLE_NOPROGRESS
           71 usec_timeout=500000 usect_delayed=192000 reason=VMSCAN_THROTTLE_NOPROGRESS
           71 usec_timeout=500000 usect_delayed=480000 reason=VMSCAN_THROTTLE_NOPROGRESS
           79 usec_timeout=500000 usect_delayed=60000 reason=VMSCAN_THROTTLE_NOPROGRESS
           82 usec_timeout=500000 usect_delayed=320000 reason=VMSCAN_THROTTLE_NOPROGRESS
           82 usec_timeout=500000 usect_delayed=92000 reason=VMSCAN_THROTTLE_NOPROGRESS
           85 usec_timeout=500000 usect_delayed=64000 reason=VMSCAN_THROTTLE_NOPROGRESS
           85 usec_timeout=500000 usect_delayed=80000 reason=VMSCAN_THROTTLE_NOPROGRESS
           88 usec_timeout=500000 usect_delayed=84000 reason=VMSCAN_THROTTLE_NOPROGRESS
           90 usec_timeout=500000 usect_delayed=160000 reason=VMSCAN_THROTTLE_NOPROGRESS
           90 usec_timeout=500000 usect_delayed=292000 reason=VMSCAN_THROTTLE_NOPROGRESS
           94 usec_timeout=500000 usect_delayed=56000 reason=VMSCAN_THROTTLE_NOPROGRESS
          118 usec_timeout=500000 usect_delayed=88000 reason=VMSCAN_THROTTLE_NOPROGRESS
          119 usec_timeout=500000 usect_delayed=72000 reason=VMSCAN_THROTTLE_NOPROGRESS
          126 usec_timeout=500000 usect_delayed=108000 reason=VMSCAN_THROTTLE_NOPROGRESS
          146 usec_timeout=500000 usect_delayed=52000 reason=VMSCAN_THROTTLE_NOPROGRESS
          148 usec_timeout=500000 usect_delayed=36000 reason=VMSCAN_THROTTLE_NOPROGRESS
          148 usec_timeout=500000 usect_delayed=48000 reason=VMSCAN_THROTTLE_NOPROGRESS
          159 usec_timeout=500000 usect_delayed=28000 reason=VMSCAN_THROTTLE_NOPROGRESS
          178 usec_timeout=500000 usect_delayed=44000 reason=VMSCAN_THROTTLE_NOPROGRESS
          183 usec_timeout=500000 usect_delayed=40000 reason=VMSCAN_THROTTLE_NOPROGRESS
          237 usec_timeout=500000 usect_delayed=100000 reason=VMSCAN_THROTTLE_NOPROGRESS
          266 usec_timeout=500000 usect_delayed=32000 reason=VMSCAN_THROTTLE_NOPROGRESS
          313 usec_timeout=500000 usect_delayed=24000 reason=VMSCAN_THROTTLE_NOPROGRESS
          347 usec_timeout=500000 usect_delayed=96000 reason=VMSCAN_THROTTLE_NOPROGRESS
          470 usec_timeout=500000 usect_delayed=20000 reason=VMSCAN_THROTTLE_NOPROGRESS
          559 usec_timeout=500000 usect_delayed=16000 reason=VMSCAN_THROTTLE_NOPROGRESS
          964 usec_timeout=500000 usect_delayed=12000 reason=VMSCAN_THROTTLE_NOPROGRESS
         2001 usec_timeout=500000 usect_delayed=104000 reason=VMSCAN_THROTTLE_NOPROGRESS
         2447 usec_timeout=500000 usect_delayed=8000 reason=VMSCAN_THROTTLE_NOPROGRESS
         7888 usec_timeout=500000 usect_delayed=4000 reason=VMSCAN_THROTTLE_NOPROGRESS
        22727 usec_timeout=500000 usect_delayed=0 reason=VMSCAN_THROTTLE_NOPROGRESS
        51305 usec_timeout=500000 usect_delayed=500000 reason=VMSCAN_THROTTLE_NOPROGRESS
      
      The full timeout is often hit but a large number also do not stall at
      all.  The remainder slept a little allowing other reclaim tasks to make
      progress.
      
      While this timeout could be further increased, it could also negatively
      impact worst-case behaviour when there is no prioritisation of what task
      should make progress.
      
      For VMSCAN_THROTTLE_WRITEBACK, the breakdown was
      
            1 usec_timeout=100000 usect_delayed=44000 reason=VMSCAN_THROTTLE_WRITEBACK
            2 usec_timeout=100000 usect_delayed=76000 reason=VMSCAN_THROTTLE_WRITEBACK
            3 usec_timeout=100000 usect_delayed=80000 reason=VMSCAN_THROTTLE_WRITEBACK
            5 usec_timeout=100000 usect_delayed=48000 reason=VMSCAN_THROTTLE_WRITEBACK
            5 usec_timeout=100000 usect_delayed=84000 reason=VMSCAN_THROTTLE_WRITEBACK
            6 usec_timeout=100000 usect_delayed=72000 reason=VMSCAN_THROTTLE_WRITEBACK
            7 usec_timeout=100000 usect_delayed=88000 reason=VMSCAN_THROTTLE_WRITEBACK
           11 usec_timeout=100000 usect_delayed=56000 reason=VMSCAN_THROTTLE_WRITEBACK
           12 usec_timeout=100000 usect_delayed=64000 reason=VMSCAN_THROTTLE_WRITEBACK
           16 usec_timeout=100000 usect_delayed=92000 reason=VMSCAN_THROTTLE_WRITEBACK
           24 usec_timeout=100000 usect_delayed=68000 reason=VMSCAN_THROTTLE_WRITEBACK
           28 usec_timeout=100000 usect_delayed=32000 reason=VMSCAN_THROTTLE_WRITEBACK
           30 usec_timeout=100000 usect_delayed=60000 reason=VMSCAN_THROTTLE_WRITEBACK
           30 usec_timeout=100000 usect_delayed=96000 reason=VMSCAN_THROTTLE_WRITEBACK
           32 usec_timeout=100000 usect_delayed=52000 reason=VMSCAN_THROTTLE_WRITEBACK
           42 usec_timeout=100000 usect_delayed=40000 reason=VMSCAN_THROTTLE_WRITEBACK
           77 usec_timeout=100000 usect_delayed=28000 reason=VMSCAN_THROTTLE_WRITEBACK
           99 usec_timeout=100000 usect_delayed=36000 reason=VMSCAN_THROTTLE_WRITEBACK
          137 usec_timeout=100000 usect_delayed=24000 reason=VMSCAN_THROTTLE_WRITEBACK
          190 usec_timeout=100000 usect_delayed=20000 reason=VMSCAN_THROTTLE_WRITEBACK
          339 usec_timeout=100000 usect_delayed=16000 reason=VMSCAN_THROTTLE_WRITEBACK
          518 usec_timeout=100000 usect_delayed=12000 reason=VMSCAN_THROTTLE_WRITEBACK
          852 usec_timeout=100000 usect_delayed=8000 reason=VMSCAN_THROTTLE_WRITEBACK
         3359 usec_timeout=100000 usect_delayed=4000 reason=VMSCAN_THROTTLE_WRITEBACK
         7147 usec_timeout=100000 usect_delayed=0 reason=VMSCAN_THROTTLE_WRITEBACK
        83962 usec_timeout=100000 usect_delayed=100000 reason=VMSCAN_THROTTLE_WRITEBACK
      
      The majority hit the timeout in direct reclaim context although a
      sizable number did not stall at all.  This is very different to kswapd
      where only a tiny percentage of stalls due to writeback reached the
      timeout.
      
      Bottom line, the throttling appears to work and the wakeup events may
      limit worst case stalls.  There might be some grounds for adjusting
      timeouts but it's likely futile as the worst-case scenarios depend on
      the workload, memory size and the speed of the storage.  A better
      approach to improve the series further would be to prioritise tasks
      based on their rate of allocation with the caveat that it may be very
      expensive to track.
      
      This patch (of 5):
      
      Page reclaim throttles on wait_iff_congested under the following
      conditions:
      
       - kswapd is encountering pages under writeback and marked for immediate
         reclaim implying that pages are cycling through the LRU faster than
         pages can be cleaned.
      
       - Direct reclaim will stall if all dirty pages are backed by congested
         inodes.
      
      wait_iff_congested is almost completely broken with few exceptions.
      This patch adds a new node-based workqueue and tracks the number of
      throttled tasks and pages written back since throttling started.  If
      enough pages belonging to the node are written back then the throttled
      tasks will wake early.  If not, the throttled tasks sleeps until the
      timeout expires.
      
      [neilb@suse.de: Uninterruptible sleep and simpler wakeups]
      [hdanton@sina.com: Avoid race when reclaim starts]
      [vbabka@suse.cz: vmstat irq-safe api, clarifications]
      
      Link: https://lore.kernel.org/linux-mm/45d8b7a6-8548-65f5-cccf-9f451d4ae3d4@kernel.dk/ [1]
      Link: https://lkml.kernel.org/r/20211022144651.19914-1-mgorman@techsingularity.net
      Link: https://lkml.kernel.org/r/20211022144651.19914-2-mgorman@techsingularity.netSigned-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: NeilBrown <neilb@suse.de>
      Cc: "Theodore Ts'o" <tytso@mit.edu>
      Cc: Andreas Dilger <adilger.kernel@dilger.ca>
      Cc: "Darrick J . Wong" <djwong@kernel.org>
      Cc: Matthew Wilcox <willy@infradead.org>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Rik van Riel <riel@surriel.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8cd7c588
    • Kai Song's avatar
      mm/vmscan.c: fix -Wunused-but-set-variable warning · cb75463c
      Kai Song authored
      We fix the following warning when building kernel with W=1:
      
        mm/vmscan.c:1362:6: warning: variable 'err' set but not used [-Wunused-but-set-variable]
      
      Link: https://lkml.kernel.org/r/20210924181218.21165-1-songkai01@inspur.comSigned-off-by: default avatarKai Song <songkai01@inspur.com>
      Reviewed-by: default avatarYang Shi <shy828301@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      cb75463c
    • Miaohe Lin's avatar
      mm/page_isolation: guard against possible putback unisolated page · a500cb34
      Miaohe Lin authored
      Isolating a free page in an isolated pageblock is expected to always
      work as watermarks don't apply here.
      
      But if __isolate_free_page() failed, due to condition changes, the page
      will be left on the free list.  And the page will be put back to free
      list again via __putback_isolated_page().  This may trigger
      VM_BUG_ON_PAGE() on page->flags checking in __free_one_page() if
      PageReported is set.  Or we will corrupt the free list because
      list_add() will be called for pages already on another list.
      
      Add a VM_WARN_ON() to complain about this change.
      
      Link: https://lkml.kernel.org/r/20210914114508.23725-1-linmiaohe@huawei.com
      Fixes: 3c605096 ("mm/page_alloc: restrict max order of merging on isolated pageblock")
      Signed-off-by: default avatarMiaohe Lin <linmiaohe@huawei.com>
      Reviewed-by: default avatarDavid Hildenbrand <david@redhat.com>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Cc: John Hubbard <jhubbard@nvidia.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      a500cb34
    • Miaohe Lin's avatar
      mm/page_isolation: fix potential missing call to unset_migratetype_isolate() · e1d8c966
      Miaohe Lin authored
      In start_isolate_page_range() undo path, pfn_to_online_page() just
      checks the first pfn in a pageblock while __first_valid_page() will
      traverse the pageblock until the first online pfn is found.  So we may
      miss the call to unset_migratetype_isolate() in undo path and pages will
      remain isolated unexpectedly.
      
      Fix this by calling undo_isolate_page_range() and this will also help to
      simplify the code further.  Note we shouldn't ever trigger it because
      MAX_ORDER-1 aligned pfn ranges shouldn't contain memory holes now.
      
      Link: https://lkml.kernel.org/r/20210914114348.15569-1-linmiaohe@huawei.com
      Fixes: 2ce13640 ("mm: __first_valid_page skip over offline pages")
      Signed-off-by: default avatarMiaohe Lin <linmiaohe@huawei.com>
      Reviewed-by: default avatarDavid Hildenbrand <david@redhat.com>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e1d8c966
    • Axel Rasmussen's avatar
      userfaultfd/selftests: fix calculation of expected ioctls · ad0ce23e
      Axel Rasmussen authored
      Today, we assert that the ioctls the kernel reports as supported for a
      registration match a precomputed list.  We decide which ioctls are
      supported by examining the memory type.  Then, in several locations we
      "fix up" this list by adding or removing things this initial decision
      got wrong.
      
      What ioctls the kernel reports is actually a function of several things:
      - The memory type
      - Kernel feature support (e.g., no writeprotect on aarch64)
      - The registration type (e.g., CONTINUE only supported for MINOR mode)
      
      So, we can't fully compute this at the start, in set_test_type.  It
      varies per test, depending on what registration mode(s) those tests use.
      
      Instead, introduce a new function which computes the correct list.  This
      centralizes the add/remove of ioctls depending on these function inputs
      in one place, so we don't have to repeat ourselves in various tests.
      
      Not only is the resulting code a bit shorter, but it fixes a real bug in
      the existing code: previously, we would incorrectly require the
      writeprotect ioctl to be present on aarch64, where it isn't actually
      supported.
      
      Link: https://lkml.kernel.org/r/20210930212309.4001967-4-axelrasmussen@google.comSigned-off-by: default avatarAxel Rasmussen <axelrasmussen@google.com>
      Reviewed-by: default avatarPeter Xu <peterx@redhat.com>
      Cc: Shuah Khan <shuah@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ad0ce23e
    • Axel Rasmussen's avatar
      userfaultfd/selftests: fix feature support detection · 1042a53d
      Axel Rasmussen authored
      Before any tests are run, in set_test_type, we decide what feature(s) we
      are going to be testing, based upon our command line arguments.
      However, the supported features are not just a function of the memory
      type being used, so this is broken.
      
      For instance, consider writeprotect support.  It is "normally" supported
      for anonymous memory, but furthermore it requires that the kernel has
      CONFIG_HAVE_ARCH_USERFAULTFD_WP.  So, it is *not* supported at all on
      aarch64, for example.
      
      So, this fixes this by querying the kernel for the set of features it
      supports in set_test_type, by opening a userfaultfd and issuing a
      UFFDIO_API ioctl.  Based upon the reported features, we toggle what
      tests are enabled.
      
      Link: https://lkml.kernel.org/r/20210930212309.4001967-3-axelrasmussen@google.comSigned-off-by: default avatarAxel Rasmussen <axelrasmussen@google.com>
      Reviewed-by: default avatarPeter Xu <peterx@redhat.com>
      Cc: Shuah Khan <shuah@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      1042a53d
    • Axel Rasmussen's avatar
      userfaultfd/selftests: don't rely on GNU extensions for random numbers · 1c10e674
      Axel Rasmussen authored
      Patch series "Small userfaultfd selftest fixups", v2.
      
      This patch (of 3):
      
      Two arguments for doing this:
      
      First, and maybe most importantly, the resulting code is significantly
      shorter / simpler.
      
      Then, we avoid using GNU libc extensions.  Why does this matter? It
      makes testing userfaultfd with the selftest easier e.g.  on distros
      which use something other than glibc (e.g., Alpine, which uses musl);
      basically, it makes the test more portable.
      
      Link: https://lkml.kernel.org/r/20210930212309.4001967-2-axelrasmussen@google.comSigned-off-by: default avatarAxel Rasmussen <axelrasmussen@google.com>
      Reviewed-by: default avatarPeter Xu <peterx@redhat.com>
      Cc: Shuah Khan <shuah@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      1c10e674
    • Mike Kravetz's avatar
      hugetlb: remove unnecessary set_page_count in prep_compound_gigantic_page · 2c0078a7
      Mike Kravetz authored
      In commit 7118fc29 ("hugetlb: address ref count racing in
      prep_compound_gigantic_page"), page_ref_freeze is used to atomically
      zero the ref count of tail pages iff they are 1.  The unconditional call
      to set_page_count(0) was left in the code.  This call is after
      page_ref_freeze so it is really a noop.
      
      Remove redundant and unnecessary set_page_count call.
      
      Link: https://lkml.kernel.org/r/20211026220635.35187-1-mike.kravetz@oracle.com
      Fixes: 7118fc29 ("hugetlb: address ref count racing in prep_compound_gigantic_page")
      Signed-off-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Suggested-by: default avatarPasha Tatashin <pasha.tatashin@soleen.com>
      Reviewed-by: default avatarPasha Tatashin <pasha.tatashin@soleen.com>
      Reviewed-by: default avatarMatthew Wilcox (Oracle) <willy@infradead.org>
      Reviewed-by: default avatarOscar Salvador <osalvador@suse.de>
      Reviewed-by: default avatarMuchun Song <songmuchun@bytedance.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      2c0078a7
    • Baolin Wang's avatar
      hugetlb: remove redundant VM_BUG_ON() in add_reservation_in_range() · 76efc67a
      Baolin Wang authored
      When calling hugetlb_resv_map_add(), we've guaranteed that the parameter
      'to' is always larger than 'from', so it never returns a negative value
      from hugetlb_resv_map_add().  Thus remove the redundant VM_BUG_ON().
      
      Link: https://lkml.kernel.org/r/2b565552f3d06753da1e8dda439c0d96d6d9a5a3.1634797639.git.baolin.wang@linux.alibaba.comSigned-off-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
      Reviewed-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Cc: Michal Hocko <mhocko@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      76efc67a
    • Baolin Wang's avatar
      hugetlb: remove redundant validation in has_same_uncharge_info() · 0739eb43
      Baolin Wang authored
      The callers of has_same_uncharge_info() has accessed the original
      file_region and new file_region, and they are impossible to be NULL now.
      
      So we can remove the file_region validation in has_same_uncharge_info()
      to simplify the code.
      
      Link: https://lkml.kernel.org/r/97fc68d3f8d34f63c204645e10d7a718997e50b7.1634797639.git.baolin.wang@linux.alibaba.comSigned-off-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
      Reviewed-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Cc: Michal Hocko <mhocko@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      0739eb43
    • Baolin Wang's avatar
      hugetlb: replace the obsolete hugetlb_instantiation_mutex in the comments · aa6d2e8c
      Baolin Wang authored
      After commit 8382d914 ("mm, hugetlb: improve page-fault
      scalability"), the hugetlb_instantiation_mutex lock had been replaced by
      hugetlb_fault_mutex_table to serializes faults on the same logical page.
      
      Thus update the obsolete hugetlb_instantiation_mutex related comments.
      
      Link: https://lkml.kernel.org/r/4b3febeae37455ff7b74aa0aad16cc6909cf0926.1634797639.git.baolin.wang@linux.alibaba.comSigned-off-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
      Reviewed-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Cc: Michal Hocko <mhocko@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      aa6d2e8c
    • Baolin Wang's avatar
      hugetlb_cgroup: remove unused hugetlb_cgroup_from_counter macro · df8931c8
      Baolin Wang authored
      Patch series "Some cleanups and improvements for hugetlb".
      
      This patchset does some cleanups and improvements for hugetlb and
      hugetlb_cgroup.
      
      This patch (of 4):
      
      Since commit 726b7bbe ("hugetlb_cgroup: fix illegal access to
      memory"), the hugetlb_cgroup_from_counter() macro is not used any more,
      remove it.
      
      Link: https://lkml.kernel.org/r/cover.1634797639.git.baolin.wang@linux.alibaba.com
      Link: https://lkml.kernel.org/r/f03b29b801fa9942466ab15334ec09988e124ae6.1634797639.git.baolin.wang@linux.alibaba.comSigned-off-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
      Reviewed-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Cc: Michal Hocko <mhocko@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      df8931c8
    • Ran Jianping's avatar
      mm: remove duplicate include in hugepage-mremap.c · b65c23f7
      Ran Jianping authored
      Remove duplicate includes 'unistd.h' included in
       '/tools/testing/selftests/vm/hugepage-mremap.c'  is duplicated.It is also
       included on 23 line.
      
      Link: https://lkml.kernel.org/r/20211018102336.869726-1-ran.jianping@zte.com.cnSigned-off-by: default avatarRan Jianping <ran.jianping@zte.com.cn>
      Reported-by: default avatarZeal Robot <zealci@zte.com.cn>
      Cc: Shuah Khan <shuah@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b65c23f7
    • Baolin Wang's avatar
      hugetlb: support node specified when using cma for gigantic hugepages · 38e719ab
      Baolin Wang authored
      Now the size of CMA area for gigantic hugepages runtime allocation is
      balanced for all online nodes, but we also want to specify the size of
      CMA per-node, or only one node in some cases, which are similar with
      patch [1].
      
      For example, on some multi-nodes systems, each node's memory can be
      different, allocating the same size of CMA for each node is not suitable
      for the low-memory nodes.  Meanwhile some workloads like DPDK mentioned
      by Zhenguo in patch [1] only need hugepages in one node.
      
      On the other hand, we have some machines with multiple types of memory,
      like DRAM and PMEM (persistent memory).  On this system, we may want to
      specify all the hugepages only on DRAM node, or specify the proportion
      of DRAM node and PMEM node, to tuning the performance of the workloads.
      
      Thus this patch adds node format for 'hugetlb_cma' parameter to support
      specifying the size of CMA per-node.  An example is as follows:
      
        hugetlb_cma=0:5G,2:5G
      
      which means allocating 5G size of CMA area on node 0 and node 2
      respectively.  And the users should use the node specific sysfs file to
      allocate the gigantic hugepages if specified the CMA size on that node.
      
      Link: https://lkml.kernel.org/r/20211005054729.86457-1-yaozhenguo1@gmail.com [1]
      Link: https://lkml.kernel.org/r/bb790775ca60bb8f4b26956bb3f6988f74e075c7.1634261144.git.baolin.wang@linux.alibaba.comSigned-off-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
      Reviewed-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Cc: Michal Hocko <mhocko@kernel.org>
      Cc: Roman Gushchin <guro@fb.com>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      38e719ab
    • Mina Almasry's avatar
      mm, hugepages: add hugetlb vma mremap() test · 12b61320
      Mina Almasry authored
      [almasrymina@google.com: v8]
        Link: https://lkml.kernel.org/r/20211014200542.4126947-2-almasrymina@google.com
      [wanjiabing@vivo.com: remove duplicated include in hugepage-mremap]
        Link: https://lkml.kernel.org/r/20211021122944.8857-1-wanjiabing@vivo.com
      
      Link: https://lkml.kernel.org/r/20211013195825.3058275-2-almasrymina@google.comSigned-off-by: default avatarMina Almasry <almasrymina@google.com>
      Signed-off-by: default avatarWan Jiabing <wanjiabing@vivo.com>
      Acked-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Cc: Ken Chen <kenchen@google.com>
      Cc: Chris Kennelly <ckennelly@google.com>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Cc: Kirill Shutemov <kirill@shutemov.name>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      12b61320
    • Mina Almasry's avatar
      mm, hugepages: add mremap() support for hugepage backed vma · 550a7d60
      Mina Almasry authored
      Support mremap() for hugepage backed vma segment by simply repositioning
      page table entries.  The page table entries are repositioned to the new
      virtual address on mremap().
      
      Hugetlb mremap() support is of course generic; my motivating use case is
      a library (hugepage_text), which reloads the ELF text of executables in
      hugepages.  This significantly increases the execution performance of
      said executables.
      
      Restrict the mremap operation on hugepages to up to the size of the
      original mapping as the underlying hugetlb reservation is not yet
      capable of handling remapping to a larger size.
      
      During the mremap() operation we detect pmd_share'd mappings and we
      unshare those during the mremap().  On access and fault the sharing is
      established again.
      
      Link: https://lkml.kernel.org/r/20211013195825.3058275-1-almasrymina@google.comSigned-off-by: default avatarMina Almasry <almasrymina@google.com>
      Reviewed-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Cc: Ken Chen <kenchen@google.com>
      Cc: Chris Kennelly <ckennelly@google.com>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Cc: Kirill Shutemov <kirill@shutemov.name>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      550a7d60
    • Liangcai Fan's avatar
      mm: khugepaged: recalculate min_free_kbytes after stopping khugepaged · bd3400ea
      Liangcai Fan authored
      When initializing transparent huge pages, min_free_kbytes would be
      calculated according to what khugepaged expected.
      
      So when transparent huge pages get disabled, min_free_kbytes should be
      recalculated instead of the higher value set by khugepaged.
      
      Link: https://lkml.kernel.org/r/1633937809-16558-1-git-send-email-liangcaifan19@gmail.comSigned-off-by: default avatarLiangcai Fan <liangcaifan19@gmail.com>
      Signed-off-by: default avatarChunyan Zhang <zhang.lyra@gmail.com>
      Cc: Mike Kravetz <mike.kravetz@oracle.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      bd3400ea
    • Mike Kravetz's avatar
      hugetlb: add hugetlb demote page support · 8531fc6f
      Mike Kravetz authored
      Demote page functionality will split a huge page into a number of huge
      pages of a smaller size.  For example, on x86 a 1GB huge page can be
      demoted into 512 2M huge pages.  Demotion is done 'in place' by simply
      splitting the huge page.
      
      Added '*_for_demote' wrappers for remove_hugetlb_page,
      destroy_compound_hugetlb_page and prep_compound_gigantic_page for use by
      demote code.
      
      [mike.kravetz@oracle.com: v4]
        Link: https://lkml.kernel.org/r/6ca29b8e-527c-d6ec-900e-e6a43e4f8b73@oracle.com
      
      Link: https://lkml.kernel.org/r/20211007181918.136982-6-mike.kravetz@oracle.comSigned-off-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Reviewed-by: default avatarOscar Salvador <osalvador@suse.de>
      Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
      Cc: David Hildenbrand <david@redhat.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Muchun Song <songmuchun@bytedance.com>
      Cc: Naoya Horiguchi <naoya.horiguchi@linux.dev>
      Cc: Nghia Le <nghialm78@gmail.com>
      Cc: Zi Yan <ziy@nvidia.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8531fc6f
    • Mike Kravetz's avatar
      hugetlb: add demote bool to gigantic page routines · 34d9e35b
      Mike Kravetz authored
      The routines remove_hugetlb_page and destroy_compound_gigantic_page will
      remove a gigantic page and make the set of base pages ready to be
      returned to a lower level allocator.  In the process of doing this, they
      make all base pages reference counted.
      
      The routine prep_compound_gigantic_page creates a gigantic page from a
      set of base pages.  It assumes that all these base pages are reference
      counted.
      
      During demotion, a gigantic page will be split into huge pages of a
      smaller size.  This logically involves use of the routines,
      remove_hugetlb_page, and destroy_compound_gigantic_page followed by
      prep_compound*_page for each smaller huge page.
      
      When pages are reference counted (ref count >= 0), additional
      speculative ref counts could be taken as described in previous commits
      [1] and [2].  This could result in errors while demoting a huge page.
      Quite a bit of code would need to be created to handle all possible
      issues.
      
      Instead of dealing with the possibility of speculative ref counts, avoid
      the possibility by keeping ref counts at zero during the demote process.
      Add a boolean 'demote' to the routines remove_hugetlb_page,
      destroy_compound_gigantic_page and prep_compound_gigantic_page.  If the
      boolean is set, the remove and destroy routines will not reference count
      pages and the prep routine will not expect reference counted pages.
      
      '*_for_demote' wrappers of the routines will be added in a subsequent
      patch where this functionality is used.
      
      [1] https://lore.kernel.org/linux-mm/20210622021423.154662-3-mike.kravetz@oracle.com/
      [2] https://lore.kernel.org/linux-mm/20210809184832.18342-3-mike.kravetz@oracle.com/
      
      Link: https://lkml.kernel.org/r/20211007181918.136982-5-mike.kravetz@oracle.comSigned-off-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Reviewed-by: default avatarOscar Salvador <osalvador@suse.de>
      Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
      Cc: David Hildenbrand <david@redhat.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Muchun Song <songmuchun@bytedance.com>
      Cc: Naoya Horiguchi <naoya.horiguchi@linux.dev>
      Cc: Nghia Le <nghialm78@gmail.com>
      Cc: Zi Yan <ziy@nvidia.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      34d9e35b
    • Mike Kravetz's avatar
      hugetlb: be sure to free demoted CMA pages to CMA · a01f4390
      Mike Kravetz authored
      When huge page demotion is fully implemented, gigantic pages can be
      demoted to a smaller huge page size.  For example, on x86 a 1G page can
      be demoted to 512 2M pages.  However, gigantic pages can potentially be
      allocated from CMA.  If a gigantic page which was allocated from CMA is
      demoted, the corresponding demoted pages needs to be returned to CMA.
      
      Use the new interface cma_pages_valid() to determine if a non-gigantic
      hugetlb page should be freed to CMA.  Also, clear mapping field of these
      pages as expected by cma_release.
      
      This also requires a change to CMA region creation for gigantic pages.
      CMA uses a per-region bit map to track allocations.  When setting up the
      region, you specify how many pages each bit represents.  Currently, only
      gigantic pages are allocated/freed from CMA so the region is set up such
      that one bit represents a gigantic page size allocation.
      
      With demote, a gigantic page (allocation) could be split into smaller
      size pages.  And, these smaller size pages will be freed to CMA.  So,
      since the per-region bit map needs to be set up to represent the
      smallest allocation/free size, it now needs to be set to the smallest
      huge page size which can be freed to CMA.
      
      Unfortunately, we set up the CMA region for huge pages before we set up
      huge pages sizes (hstates).  So, technically we do not know the smallest
      huge page size as this can change via command line options and
      architecture specific code.  Therefore, at region setup time we use
      HUGETLB_PAGE_ORDER as the smallest possible huge page size that can be
      given back to CMA.  It is possible that this value is sub-optimal for
      some architectures/config options.  If needed, this can be addressed in
      follow on work.
      
      Link: https://lkml.kernel.org/r/20211007181918.136982-4-mike.kravetz@oracle.comSigned-off-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
      Cc: David Hildenbrand <david@redhat.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Muchun Song <songmuchun@bytedance.com>
      Cc: Naoya Horiguchi <naoya.horiguchi@linux.dev>
      Cc: Nghia Le <nghialm78@gmail.com>
      Cc: Oscar Salvador <osalvador@suse.de>
      Cc: Zi Yan <ziy@nvidia.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      a01f4390
    • Mike Kravetz's avatar
      mm/cma: add cma_pages_valid to determine if pages are in CMA · 9871e2de
      Mike Kravetz authored
      Add new interface cma_pages_valid() which indicates if the specified
      pages are part of a CMA region.  This interface will be used in a
      subsequent patch by hugetlb code.
      
      In order to keep the same amount of DEBUG information, a pr_debug() call
      was added to cma_pages_valid().  In the case where the page passed to
      cma_release is not in cma region, the debug message will be printed from
      cma_pages_valid as opposed to cma_release.
      
      Link: https://lkml.kernel.org/r/20211007181918.136982-3-mike.kravetz@oracle.comSigned-off-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Acked-by: default avatarDavid Hildenbrand <david@redhat.com>
      Reviewed-by: default avatarOscar Salvador <osalvador@suse.de>
      Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Muchun Song <songmuchun@bytedance.com>
      Cc: Naoya Horiguchi <naoya.horiguchi@linux.dev>
      Cc: Nghia Le <nghialm78@gmail.com>
      Cc: Zi Yan <ziy@nvidia.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      9871e2de
    • Mike Kravetz's avatar
      hugetlb: add demote hugetlb page sysfs interfaces · 79dfc695
      Mike Kravetz authored
      Patch series "hugetlb: add demote/split page functionality", v4.
      
      The concurrent use of multiple hugetlb page sizes on a single system is
      becoming more common.  One of the reasons is better TLB support for
      gigantic page sizes on x86 hardware.  In addition, hugetlb pages are
      being used to back VMs in hosting environments.
      
      When using hugetlb pages to back VMs, it is often desirable to
      preallocate hugetlb pools.  This avoids the delay and uncertainty of
      allocating hugetlb pages at VM startup.  In addition, preallocating huge
      pages minimizes the issue of memory fragmentation that increases the
      longer the system is up and running.
      
      In such environments, a combination of larger and smaller hugetlb pages
      are preallocated in anticipation of backing VMs of various sizes.  Over
      time, the preallocated pool of smaller hugetlb pages may become depleted
      while larger hugetlb pages still remain.  In such situations, it is
      desirable to convert larger hugetlb pages to smaller hugetlb pages.
      
      Converting larger to smaller hugetlb pages can be accomplished today by
      first freeing the larger page to the buddy allocator and then allocating
      the smaller pages.  For example, to convert 50 GB pages on x86:
      
        gb_pages=`cat .../hugepages-1048576kB/nr_hugepages`
        m2_pages=`cat .../hugepages-2048kB/nr_hugepages`
        echo $(($gb_pages - 50)) > .../hugepages-1048576kB/nr_hugepages
        echo $(($m2_pages + 25600)) > .../hugepages-2048kB/nr_hugepages
      
      On an idle system this operation is fairly reliable and results are as
      expected.  The number of 2MB pages is increased as expected and the time
      of the operation is a second or two.
      
      However, when there is activity on the system the following issues
      arise:
      
      1) This process can take quite some time, especially if allocation of
         the smaller pages is not immediate and requires migration/compaction.
      
      2) There is no guarantee that the total size of smaller pages allocated
         will match the size of the larger page which was freed. This is
         because the area freed by the larger page could quickly be
         fragmented.
      
      In a test environment with a load that continually fills the page cache
      with clean pages, results such as the following can be observed:
      
        Unexpected number of 2MB pages allocated: Expected 25600, have 19944
        real    0m42.092s
        user    0m0.008s
        sys     0m41.467s
      
      To address these issues, introduce the concept of hugetlb page demotion.
      Demotion provides a means of 'in place' splitting of a hugetlb page to
      pages of a smaller size.  This avoids freeing pages to buddy and then
      trying to allocate from buddy.
      
      Page demotion is controlled via sysfs files that reside in the per-hugetlb
      page size and per node directories.
      
       - demote_size
              Target page size for demotion, a smaller huge page size. File
              can be written to chose a smaller huge page size if multiple are
              available.
      
       - demote
              Writable number of hugetlb pages to be demoted
      
      To demote 50 GB huge pages, one would:
      
        cat .../hugepages-1048576kB/free_hugepages   /* optional, verify free pages */
        cat .../hugepages-1048576kB/demote_size      /* optional, verify target size */
        echo 50 > .../hugepages-1048576kB/demote
      
      Only hugetlb pages which are free at the time of the request can be
      demoted.  Demotion does not add to the complexity of surplus pages and
      honors reserved huge pages.  Therefore, when a value is written to the
      sysfs demote file, that value is only the maximum number of pages which
      will be demoted.  It is possible fewer will actually be demoted.  The
      recently introduced per-hstate mutex is used to synchronize demote
      operations with other operations that modify hugetlb pools.
      
      Real world use cases
      --------------------
      The above scenario describes a real world use case where hugetlb pages
      are used to back VMs on x86.  Both issues of long allocation times and
      not necessarily getting the expected number of smaller huge pages after
      a free and allocate cycle have been experienced.  The occurrence of
      these issues is dependent on other activity within the host and can not
      be predicted.
      
      This patch (of 5):
      
      Two new sysfs files are added to demote hugtlb pages.  These files are
      both per-hugetlb page size and per node.  Files are:
      
        demote_size - The size in Kb that pages are demoted to. (read-write)
        demote - The number of huge pages to demote. (write-only)
      
      By default, demote_size is the next smallest huge page size.  Valid huge
      page sizes less than huge page size may be written to this file.  When
      huge pages are demoted, they are demoted to this size.
      
      Writing a value to demote will result in an attempt to demote that
      number of hugetlb pages to an appropriate number of demote_size pages.
      
      NOTE: Demote interfaces are only provided for huge page sizes if there
      is a smaller target demote huge page size.  For example, on x86 1GB huge
      pages will have demote interfaces.  2MB huge pages will not have demote
      interfaces.
      
      This patch does not provide full demote functionality.  It only provides
      the sysfs interfaces.
      
      It also provides documentation for the new interfaces.
      
      [mike.kravetz@oracle.com: n_mask initialization does not need to be protected by the mutex]
        Link: https://lkml.kernel.org/r/0530e4ef-2492-5186-f919-5db68edea654@oracle.com
      
      Link: https://lkml.kernel.org/r/20211007181918.136982-2-mike.kravetz@oracle.comSigned-off-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Reviewed-by: default avatarOscar Salvador <osalvador@suse.de>
      Cc: David Hildenbrand <david@redhat.com>
      Cc: Michal Hocko <mhocko@suse.com>
      Cc: Zi Yan <ziy@nvidia.com>
      Cc: Muchun Song <songmuchun@bytedance.com>
      Cc: Naoya Horiguchi <naoya.horiguchi@linux.dev>
      Cc: David Rientjes <rientjes@google.com>
      Cc: "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>
      Cc: Nghia Le <nghialm78@gmail.com>
      Cc: Mike Kravetz <mike.kravetz@oracle.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      79dfc695
    • Peter Xu's avatar
      mm/hugetlb: drop __unmap_hugepage_range definition from hugetlb.h · 73c54763
      Peter Xu authored
      Remove __unmap_hugepage_range() from the header file, because it is only
      used in hugetlb.c.
      
      Link: https://lkml.kernel.org/r/20210917165108.9341-1-peterx@redhat.comSigned-off-by: default avatarPeter Xu <peterx@redhat.com>
      Suggested-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Reviewed-by: default avatarMike Kravetz <mike.kravetz@oracle.com>
      Reviewed-by: default avatarJohn Hubbard <jhubbard@nvidia.com>
      Reviewed-by: default avatarMuchun Song <songmuchun@bytedance.com>
      Reviewed-by: default avatarDavid Hildenbrand <david@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      73c54763