• Peter Xu's avatar
    mm/gup: handle hugetlb in the generic follow_page_mask code · 9cb28da5
    Peter Xu authored
    Now follow_page() is ready to handle hugetlb pages in whatever form, and
    over all architectures.  Switch to the generic code path.
    
    Time to retire hugetlb_follow_page_mask(), following the previous
    retirement of follow_hugetlb_page() in 48498071.
    
    There may be a slight difference of how the loops run when processing slow
    GUP over a large hugetlb range on cont_pte/cont_pmd supported archs: each
    loop of __get_user_pages() will resolve one pgtable entry with the patch
    applied, rather than relying on the size of hugetlb hstate, the latter may
    cover multiple entries in one loop.
    
    A quick performance test on an aarch64 VM on M1 chip shows 15% degrade
    over a tight loop of slow gup after the path switched.  That shouldn't be
    a problem because slow-gup should not be a hot path for GUP in general:
    when page is commonly present, fast-gup will already succeed, while when
    the page is indeed missing and require a follow up page fault, the slow
    gup degrade will probably buried in the fault paths anyway.  It also
    explains why slow gup for THP used to be very slow before 57edfcfd
    ("mm/gup: accelerate thp gup even for "pages != NULL"") lands, the latter
    not part of a performance analysis but a side benefit.  If the performance
    will be a concern, we can consider handle CONT_PTE in follow_page().
    
    Before that is justified to be necessary, keep everything clean and simple.
    
    Link: https://lkml.kernel.org/r/20240327152332.950956-14-peterx@redhat.comSigned-off-by: default avatarPeter Xu <peterx@redhat.com>
    Reviewed-by: default avatarJason Gunthorpe <jgg@nvidia.com>
    Tested-by: default avatarRyan Roberts <ryan.roberts@arm.com>
    Cc: Andrea Arcangeli <aarcange@redhat.com>
    Cc: Andrew Jones <andrew.jones@linux.dev>
    Cc: Aneesh Kumar K.V (IBM) <aneesh.kumar@kernel.org>
    Cc: Axel Rasmussen <axelrasmussen@google.com>
    Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
    Cc: Christoph Hellwig <hch@infradead.org>
    Cc: David Hildenbrand <david@redhat.com>
    Cc: James Houghton <jthoughton@google.com>
    Cc: John Hubbard <jhubbard@nvidia.com>
    Cc: Kirill A. Shutemov <kirill@shutemov.name>
    Cc: Lorenzo Stoakes <lstoakes@gmail.com>
    Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
    Cc: Michael Ellerman <mpe@ellerman.id.au>
    Cc: "Mike Rapoport (IBM)" <rppt@kernel.org>
    Cc: Muchun Song <muchun.song@linux.dev>
    Cc: Rik van Riel <riel@surriel.com>
    Cc: Vlastimil Babka <vbabka@suse.cz>
    Cc: Yang Shi <shy828301@gmail.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    9cb28da5
gup.c 103 KB