• Steven Capper's avatar
    ARM: 7858/1: mm: make UACCESS_WITH_MEMCPY huge page aware · a3a9ea65
    Steven Capper authored
    The memory pinning code in uaccess_with_memcpy.c does not check
    for HugeTLB or THP pmds, and will enter an infinite loop should
    a __copy_to_user or __clear_user occur against a huge page.
    
    This patch adds detection code for huge pages to pin_page_for_write.
    As this code can be executed in a fast path it refers to the actual
    pmds rather than the vma. If a HugeTLB or THP is found (they have
    the same pmd representation on ARM), the page table spinlock is
    taken to prevent modification whilst the page is pinned.
    
    On ARM, huge pages are only represented as pmds, thus no huge pud
    checks are performed. (For huge puds one would lock the page table
    in a similar manner as in the pmd case).
    
    Two helper functions are introduced; pmd_thp_or_huge will check
    whether or not a page is huge or transparent huge (which have the
    same pmd layout on ARM), and pmd_hugewillfault will detect whether
    or not a page fault will occur on write to the page.
    
    Running the following test (with the chunking from read_zero
    removed):
     $ dd if=/dev/zero of=/dev/null bs=10M count=1024
    Gave:  2.3 GB/s backed by normal pages,
           2.9 GB/s backed by huge pages,
           5.1 GB/s backed by huge pages, with page mask=HPAGE_MASK.
    
    After some discussion, it was decided not to adopt the HPAGE_MASK,
    as this would have a significant detrimental effect on the overall
    system latency due to page_table_lock being held for too long.
    This could be revisited if split huge page locks are adopted.
    Signed-off-by: default avatarSteve Capper <steve.capper@linaro.org>
    Reviewed-by: default avatarNicolas Pitre <nico@linaro.org>
    Signed-off-by: default avatarRussell King <rmk+kernel@arm.linux.org.uk>
    a3a9ea65
pgtable-2level.h 6.93 KB