• Joonsoo Kim's avatar
    mm/compaction: speed up pageblock_pfn_to_page() when zone is contiguous · 7cf91a98
    Joonsoo Kim authored
    There is a performance drop report due to hugepage allocation and in
    there half of cpu time are spent on pageblock_pfn_to_page() in
    compaction [1].
    
    In that workload, compaction is triggered to make hugepage but most of
    pageblocks are un-available for compaction due to pageblock type and
    skip bit so compaction usually fails.  Most costly operations in this
    case is to find valid pageblock while scanning whole zone range.  To
    check if pageblock is valid to compact, valid pfn within pageblock is
    required and we can obtain it by calling pageblock_pfn_to_page().  This
    function checks whether pageblock is in a single zone and return valid
    pfn if possible.  Problem is that we need to check it every time before
    scanning pageblock even if we re-visit it and this turns out to be very
    expensive in this workload.
    
    Although we have no way to skip this pageblock check in the system where
    hole exists at arbitrary position, we can use cached value for zone
    continuity and just do pfn_to_page() in the system where hole doesn't
    exist.  This optimization considerably speeds up in above workload.
    
    Before vs After
      Max: 1096 MB/s vs 1325 MB/s
      Min: 635 MB/s 1015 MB/s
      Avg: 899 MB/s 1194 MB/s
    
    Avg is improved by roughly 30% [2].
    
    [1]: http://www.spinics.net/lists/linux-mm/msg97378.html
    [2]: https://lkml.org/lkml/2015/12/9/23
    
    [akpm@linux-foundation.org: don't forget to restore zone->contiguous on error path, per Vlastimil]
    Signed-off-by: default avatarJoonsoo Kim <iamjoonsoo.kim@lge.com>
    Reported-by: default avatarAaron Lu <aaron.lu@intel.com>
    Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Tested-by: default avatarAaron Lu <aaron.lu@intel.com>
    Cc: Mel Gorman <mgorman@suse.de>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: David Rientjes <rientjes@google.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    7cf91a98
memory_hotplug.c 53.7 KB