Commit 3faa52c0 authored by John Hubbard's avatar John Hubbard Committed by Linus Torvalds

mm/gup: track FOLL_PIN pages

Add tracking of pages that were pinned via FOLL_PIN.  This tracking is
implemented via overloading of page->_refcount: pins are added by adding
GUP_PIN_COUNTING_BIAS (1024) to the refcount.  This provides a fuzzy
indication of pinning, and it can have false positives (and that's OK).
Please see the pre-existing Documentation/core-api/pin_user_pages.rst for
details.

As mentioned in pin_user_pages.rst, callers who effectively set FOLL_PIN
(typically via pin_user_pages*()) are required to ultimately free such
pages via unpin_user_page().

Please also note the limitation, discussed in pin_user_pages.rst under the
"TODO: for 1GB and larger huge pages" section.  (That limitation will be
removed in a following patch.)

The effect of a FOLL_PIN flag is similar to that of FOLL_GET, and may be
thought of as "FOLL_GET for DIO and/or RDMA use".

Pages that have been pinned via FOLL_PIN are identifiable via a new
function call:

   bool page_maybe_dma_pinned(struct page *page);

What to do in response to encountering such a page, is left to later
patchsets. There is discussion about this in [1], [2], [3], and [4].

This also changes a BUG_ON(), to a WARN_ON(), in follow_page_mask().

[1] Some slow progress on get_user_pages() (Apr 2, 2019):
    https://lwn.net/Articles/784574/
[2] DMA and get_user_pages() (LPC: Dec 12, 2018):
    https://lwn.net/Articles/774411/
[3] The trouble with get_user_pages() (Apr 30, 2018):
    https://lwn.net/Articles/753027/
[4] LWN kernel index: get_user_pages():
    https://lwn.net/Kernel/Index/#Memory_management-get_user_pages

[jhubbard@nvidia.com: add kerneldoc]
  Link: http://lkml.kernel.org/r/20200307021157.235726-1-jhubbard@nvidia.com
[imbrenda@linux.ibm.com: if pin fails, we need to unpin, a simple put_page will not be enough]
  Link: http://lkml.kernel.org/r/20200306132537.783769-2-imbrenda@linux.ibm.com
[akpm@linux-foundation.org: fix put_compound_head defined but not used]
Suggested-by: default avatarJan Kara <jack@suse.cz>
Suggested-by: default avatarJérôme Glisse <jglisse@redhat.com>
Signed-off-by: default avatarJohn Hubbard <jhubbard@nvidia.com>
Signed-off-by: default avatarClaudio Imbrenda <imbrenda@linux.ibm.com>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Reviewed-by: default avatarJan Kara <jack@suse.cz>
Acked-by: default avatarKirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Ira Weiny <ira.weiny@intel.com>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Link: http://lkml.kernel.org/r/20200211001536.1027652-7-jhubbard@nvidia.comSigned-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 94202f12
...@@ -173,8 +173,8 @@ CASE 4: Pinning for struct page manipulation only ...@@ -173,8 +173,8 @@ CASE 4: Pinning for struct page manipulation only
------------------------------------------------- -------------------------------------------------
Here, normal GUP calls are sufficient, so neither flag needs to be set. Here, normal GUP calls are sufficient, so neither flag needs to be set.
page_dma_pinned(): the whole point of pinning page_maybe_dma_pinned(): the whole point of pinning
============================================= ===================================================
The whole point of marking pages as "DMA-pinned" or "gup-pinned" is to be able The whole point of marking pages as "DMA-pinned" or "gup-pinned" is to be able
to query, "is this page DMA-pinned?" That allows code such as page_mkclean() to query, "is this page DMA-pinned?" That allows code such as page_mkclean()
...@@ -186,7 +186,7 @@ and debates (see the References at the end of this document). It's a TODO item ...@@ -186,7 +186,7 @@ and debates (see the References at the end of this document). It's a TODO item
here: fill in the details once that's worked out. Meanwhile, it's safe to say here: fill in the details once that's worked out. Meanwhile, it's safe to say
that having this available: :: that having this available: ::
static inline bool page_dma_pinned(struct page *page) static inline bool page_maybe_dma_pinned(struct page *page)
...is a prerequisite to solving the long-running gup+DMA problem. ...is a prerequisite to solving the long-running gup+DMA problem.
......
...@@ -1001,6 +1001,8 @@ static inline void get_page(struct page *page) ...@@ -1001,6 +1001,8 @@ static inline void get_page(struct page *page)
page_ref_inc(page); page_ref_inc(page);
} }
bool __must_check try_grab_page(struct page *page, unsigned int flags);
static inline __must_check bool try_get_page(struct page *page) static inline __must_check bool try_get_page(struct page *page)
{ {
page = compound_head(page); page = compound_head(page);
...@@ -1029,29 +1031,79 @@ static inline void put_page(struct page *page) ...@@ -1029,29 +1031,79 @@ static inline void put_page(struct page *page)
__put_page(page); __put_page(page);
} }
/** /*
* unpin_user_page() - release a gup-pinned page * GUP_PIN_COUNTING_BIAS, and the associated functions that use it, overload
* @page: pointer to page to be released * the page's refcount so that two separate items are tracked: the original page
* reference count, and also a new count of how many pin_user_pages() calls were
* made against the page. ("gup-pinned" is another term for the latter).
*
* With this scheme, pin_user_pages() becomes special: such pages are marked as
* distinct from normal pages. As such, the unpin_user_page() call (and its
* variants) must be used in order to release gup-pinned pages.
*
* Choice of value:
*
* By making GUP_PIN_COUNTING_BIAS a power of two, debugging of page reference
* counts with respect to pin_user_pages() and unpin_user_page() becomes
* simpler, due to the fact that adding an even power of two to the page
* refcount has the effect of using only the upper N bits, for the code that
* counts up using the bias value. This means that the lower bits are left for
* the exclusive use of the original code that increments and decrements by one
* (or at least, by much smaller values than the bias value).
* *
* Pages that were pinned via pin_user_pages*() must be released via either * Of course, once the lower bits overflow into the upper bits (and this is
* unpin_user_page(), or one of the unpin_user_pages*() routines. This is so * OK, because subtraction recovers the original values), then visual inspection
* that eventually such pages can be separately tracked and uniquely handled. In * no longer suffices to directly view the separate counts. However, for normal
* particular, interactions with RDMA and filesystems need special handling. * applications that don't have huge page reference counts, this won't be an
* issue.
* *
* unpin_user_page() and put_page() are not interchangeable, despite this early * Locking: the lockless algorithm described in page_cache_get_speculative()
* implementation that makes them look the same. unpin_user_page() calls must * and page_cache_gup_pin_speculative() provides safe operation for
* be perfectly matched up with pin*() calls. * get_user_pages and page_mkclean and other calls that race to set up page
* table entries.
*/ */
static inline void unpin_user_page(struct page *page) #define GUP_PIN_COUNTING_BIAS (1U << 10)
{
put_page(page);
}
void unpin_user_page(struct page *page);
void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages,
bool make_dirty); bool make_dirty);
void unpin_user_pages(struct page **pages, unsigned long npages); void unpin_user_pages(struct page **pages, unsigned long npages);
/**
* page_maybe_dma_pinned() - report if a page is pinned for DMA.
*
* This function checks if a page has been pinned via a call to
* pin_user_pages*().
*
* For non-huge pages, the return value is partially fuzzy: false is not fuzzy,
* because it means "definitely not pinned for DMA", but true means "probably
* pinned for DMA, but possibly a false positive due to having at least
* GUP_PIN_COUNTING_BIAS worth of normal page references".
*
* False positives are OK, because: a) it's unlikely for a page to get that many
* refcounts, and b) all the callers of this routine are expected to be able to
* deal gracefully with a false positive.
*
* For more information, please see Documentation/vm/pin_user_pages.rst.
*
* @page: pointer to page to be queried.
* @Return: True, if it is likely that the page has been "dma-pinned".
* False, if the page is definitely not dma-pinned.
*/
static inline bool page_maybe_dma_pinned(struct page *page)
{
/*
* page_ref_count() is signed. If that refcount overflows, then
* page_ref_count() returns a negative value, and callers will avoid
* further incrementing the refcount.
*
* Here, for that overflow case, use the signed bit to count a little
* bit higher via unsigned math, and thus still get an accurate result.
*/
return ((unsigned int)page_ref_count(compound_head(page))) >=
GUP_PIN_COUNTING_BIAS;
}
#if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP) #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)
#define SECTION_IN_PAGE_FLAGS #define SECTION_IN_PAGE_FLAGS
#endif #endif
......
This diff is collapsed.
...@@ -958,6 +958,11 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, ...@@ -958,6 +958,11 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,
*/ */
WARN_ONCE(flags & FOLL_COW, "mm: In follow_devmap_pmd with FOLL_COW set"); WARN_ONCE(flags & FOLL_COW, "mm: In follow_devmap_pmd with FOLL_COW set");
/* FOLL_GET and FOLL_PIN are mutually exclusive. */
if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==
(FOLL_PIN | FOLL_GET)))
return NULL;
if (flags & FOLL_WRITE && !pmd_write(*pmd)) if (flags & FOLL_WRITE && !pmd_write(*pmd))
return NULL; return NULL;
...@@ -973,7 +978,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, ...@@ -973,7 +978,7 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,
* device mapped pages can only be returned if the * device mapped pages can only be returned if the
* caller will manage the page reference count. * caller will manage the page reference count.
*/ */
if (!(flags & FOLL_GET)) if (!(flags & (FOLL_GET | FOLL_PIN)))
return ERR_PTR(-EEXIST); return ERR_PTR(-EEXIST);
pfn += (addr & ~PMD_MASK) >> PAGE_SHIFT; pfn += (addr & ~PMD_MASK) >> PAGE_SHIFT;
...@@ -981,7 +986,8 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, ...@@ -981,7 +986,8 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,
if (!*pgmap) if (!*pgmap)
return ERR_PTR(-EFAULT); return ERR_PTR(-EFAULT);
page = pfn_to_page(pfn); page = pfn_to_page(pfn);
get_page(page); if (!try_grab_page(page, flags))
page = ERR_PTR(-ENOMEM);
return page; return page;
} }
...@@ -1101,6 +1107,11 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, ...@@ -1101,6 +1107,11 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr,
if (flags & FOLL_WRITE && !pud_write(*pud)) if (flags & FOLL_WRITE && !pud_write(*pud))
return NULL; return NULL;
/* FOLL_GET and FOLL_PIN are mutually exclusive. */
if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==
(FOLL_PIN | FOLL_GET)))
return NULL;
if (pud_present(*pud) && pud_devmap(*pud)) if (pud_present(*pud) && pud_devmap(*pud))
/* pass */; /* pass */;
else else
...@@ -1112,8 +1123,10 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, ...@@ -1112,8 +1123,10 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr,
/* /*
* device mapped pages can only be returned if the * device mapped pages can only be returned if the
* caller will manage the page reference count. * caller will manage the page reference count.
*
* At least one of FOLL_GET | FOLL_PIN must be set, so assert that here:
*/ */
if (!(flags & FOLL_GET)) if (!(flags & (FOLL_GET | FOLL_PIN)))
return ERR_PTR(-EEXIST); return ERR_PTR(-EEXIST);
pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT;
...@@ -1121,7 +1134,8 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, ...@@ -1121,7 +1134,8 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr,
if (!*pgmap) if (!*pgmap)
return ERR_PTR(-EFAULT); return ERR_PTR(-EFAULT);
page = pfn_to_page(pfn); page = pfn_to_page(pfn);
get_page(page); if (!try_grab_page(page, flags))
page = ERR_PTR(-ENOMEM);
return page; return page;
} }
...@@ -1497,8 +1511,13 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, ...@@ -1497,8 +1511,13 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
page = pmd_page(*pmd); page = pmd_page(*pmd);
VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page);
if (!try_grab_page(page, flags))
return ERR_PTR(-ENOMEM);
if (flags & FOLL_TOUCH) if (flags & FOLL_TOUCH)
touch_pmd(vma, addr, pmd, flags); touch_pmd(vma, addr, pmd, flags);
if ((flags & FOLL_MLOCK) && (vma->vm_flags & VM_LOCKED)) { if ((flags & FOLL_MLOCK) && (vma->vm_flags & VM_LOCKED)) {
/* /*
* We don't mlock() pte-mapped THPs. This way we can avoid * We don't mlock() pte-mapped THPs. This way we can avoid
...@@ -1535,8 +1554,6 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, ...@@ -1535,8 +1554,6 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
skip_mlock: skip_mlock:
page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page); VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page);
if (flags & FOLL_GET)
get_page(page);
out: out:
return page; return page;
......
...@@ -4375,19 +4375,6 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, ...@@ -4375,19 +4375,6 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT;
page = pte_page(huge_ptep_get(pte)); page = pte_page(huge_ptep_get(pte));
/*
* Instead of doing 'try_get_page()' below in the same_page
* loop, just check the count once here.
*/
if (unlikely(page_count(page) <= 0)) {
if (pages) {
spin_unlock(ptl);
remainder = 0;
err = -ENOMEM;
break;
}
}
/* /*
* If subpage information not requested, update counters * If subpage information not requested, update counters
* and skip the same_page loop below. * and skip the same_page loop below.
...@@ -4405,7 +4392,22 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, ...@@ -4405,7 +4392,22 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
same_page: same_page:
if (pages) { if (pages) {
pages[i] = mem_map_offset(page, pfn_offset); pages[i] = mem_map_offset(page, pfn_offset);
get_page(pages[i]); /*
* try_grab_page() should always succeed here, because:
* a) we hold the ptl lock, and b) we've just checked
* that the huge page is present in the page tables. If
* the huge page is present, then the tail pages must
* also be present. The ptl prevents the head page and
* tail pages from being rearranged in any way. So this
* page must be available at this point, unless the page
* refcount overflowed:
*/
if (WARN_ON_ONCE(!try_grab_page(pages[i], flags))) {
spin_unlock(ptl);
remainder = 0;
err = -ENOMEM;
break;
}
} }
if (vmas) if (vmas)
...@@ -4965,6 +4967,12 @@ follow_huge_pmd(struct mm_struct *mm, unsigned long address, ...@@ -4965,6 +4967,12 @@ follow_huge_pmd(struct mm_struct *mm, unsigned long address,
struct page *page = NULL; struct page *page = NULL;
spinlock_t *ptl; spinlock_t *ptl;
pte_t pte; pte_t pte;
/* FOLL_GET and FOLL_PIN are mutually exclusive. */
if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==
(FOLL_PIN | FOLL_GET)))
return NULL;
retry: retry:
ptl = pmd_lockptr(mm, pmd); ptl = pmd_lockptr(mm, pmd);
spin_lock(ptl); spin_lock(ptl);
...@@ -4977,8 +4985,18 @@ follow_huge_pmd(struct mm_struct *mm, unsigned long address, ...@@ -4977,8 +4985,18 @@ follow_huge_pmd(struct mm_struct *mm, unsigned long address,
pte = huge_ptep_get((pte_t *)pmd); pte = huge_ptep_get((pte_t *)pmd);
if (pte_present(pte)) { if (pte_present(pte)) {
page = pmd_page(*pmd) + ((address & ~PMD_MASK) >> PAGE_SHIFT); page = pmd_page(*pmd) + ((address & ~PMD_MASK) >> PAGE_SHIFT);
if (flags & FOLL_GET) /*
get_page(page); * try_grab_page() should always succeed here, because: a) we
* hold the pmd (ptl) lock, and b) we've just checked that the
* huge pmd (head) page is present in the page tables. The ptl
* prevents the head page and tail pages from being rearranged
* in any way. So this page must be available at this point,
* unless the page refcount overflowed:
*/
if (WARN_ON_ONCE(!try_grab_page(page, flags))) {
page = NULL;
goto out;
}
} else { } else {
if (is_hugetlb_entry_migration(pte)) { if (is_hugetlb_entry_migration(pte)) {
spin_unlock(ptl); spin_unlock(ptl);
...@@ -4999,7 +5017,7 @@ struct page * __weak ...@@ -4999,7 +5017,7 @@ struct page * __weak
follow_huge_pud(struct mm_struct *mm, unsigned long address, follow_huge_pud(struct mm_struct *mm, unsigned long address,
pud_t *pud, int flags) pud_t *pud, int flags)
{ {
if (flags & FOLL_GET) if (flags & (FOLL_GET | FOLL_PIN))
return NULL; return NULL;
return pte_page(*(pte_t *)pud) + ((address & ~PUD_MASK) >> PAGE_SHIFT); return pte_page(*(pte_t *)pud) + ((address & ~PUD_MASK) >> PAGE_SHIFT);
...@@ -5008,7 +5026,7 @@ follow_huge_pud(struct mm_struct *mm, unsigned long address, ...@@ -5008,7 +5026,7 @@ follow_huge_pud(struct mm_struct *mm, unsigned long address,
struct page * __weak struct page * __weak
follow_huge_pgd(struct mm_struct *mm, unsigned long address, pgd_t *pgd, int flags) follow_huge_pgd(struct mm_struct *mm, unsigned long address, pgd_t *pgd, int flags)
{ {
if (flags & FOLL_GET) if (flags & (FOLL_GET | FOLL_PIN))
return NULL; return NULL;
return pte_page(*(pte_t *)pgd) + ((address & ~PGDIR_MASK) >> PAGE_SHIFT); return pte_page(*(pte_t *)pgd) + ((address & ~PGDIR_MASK) >> PAGE_SHIFT);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment