• Yu Zhao's avatar
    mm/contig_alloc: support __GFP_COMP · e98337d1
    Yu Zhao authored
    Patch series "mm/hugetlb: alloc/free gigantic folios", v2.
    
    Use __GFP_COMP for gigantic folios can greatly reduce not only the amount
    of code but also the allocation and free time.
    
    Approximate LOC to mm/hugetlb.c: +60, -240
    
    Allocate and free 500 1GB hugeTLB memory without HVO by:
      time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
      time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
    
           Before  After
    Alloc  ~13s    ~10s
    Free   ~15s    <1s
    
    The above magnitude generally holds for multiple x86 and arm64 CPU
    models.
    
    Perf profile before:
      Alloc
        - 99.99% alloc_pool_huge_folio
           - __alloc_fresh_hugetlb_folio
              - 83.23% alloc_contig_pages_noprof
                 - 47.46% alloc_contig_range_noprof
                    - 20.96% isolate_freepages_range
                         16.10% split_page
                    - 14.10% start_isolate_page_range
                    - 12.02% undo_isolate_page_range
    
      Free
        - update_and_free_pages_bulk
           - 87.71% free_contig_range
              - 76.02% free_unref_page
                 - 41.30% free_unref_page_commit
                    - 32.58% free_pcppages_bulk
                       - 24.75% __free_one_page
                   13.96% _raw_spin_trylock
             12.27% __update_and_free_hugetlb_folio
    
    Perf profile after:
      Alloc
        - 99.99% alloc_pool_huge_folio
             alloc_gigantic_folio
           - alloc_contig_pages_noprof
              - 59.15% alloc_contig_range_noprof
                 - 20.72% start_isolate_page_range
                   20.64% prep_new_page
                 - 17.13% undo_isolate_page_range
    
      Free
        - update_and_free_pages_bulk
           - __folio_put
           - __free_pages_ok
                7.46% free_tail_page_prepare
              - 1.97% free_one_page
                   1.86% __free_one_page
    
    This patch (of 3):
    
    Support __GFP_COMP in alloc_contig_range().  When the flag is set, upon
    success the function returns a large folio prepared by prep_new_page(),
    rather than a range of order-0 pages prepared by split_free_pages() (which
    is renamed from split_map_pages()).
    
    alloc_contig_range() can be used to allocate folios larger than
    MAX_PAGE_ORDER, e.g., gigantic hugeTLB folios.  So on the free path,
    free_one_page() needs to handle that by split_large_buddy().
    
    [akpm@linux-foundation.org: fix folio_alloc_gigantic_noprof() WARN expression, per Yu Liao]
    Link: https://lkml.kernel.org/r/20240814035451.773331-1-yuzhao@google.com
    Link: https://lkml.kernel.org/r/20240814035451.773331-2-yuzhao@google.comSigned-off-by: default avatarYu Zhao <yuzhao@google.com>
    Acked-by: default avatarZi Yan <ziy@nvidia.com>
    Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
    Cc: Muchun Song <muchun.song@linux.dev>
    Cc: Frank van der Linden <fvdl@google.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    e98337d1
compaction.c 92.5 KB