Commit b104a35d authored by David Rientjes's avatar David Rientjes Committed by Linus Torvalds

mm, thp: do not allow thp faults to avoid cpuset restrictions

The page allocator relies on __GFP_WAIT to determine if ALLOC_CPUSET
should be set in allocflags.  ALLOC_CPUSET controls if a page allocation
should be restricted only to the set of allowed cpuset mems.

Transparent hugepages clears __GFP_WAIT when defrag is disabled to prevent
the fault path from using memory compaction or direct reclaim.  Thus, it
is unfairly able to allocate outside of its cpuset mems restriction as a
side-effect.

This patch ensures that ALLOC_CPUSET is only cleared when the gfp mask is
truly GFP_ATOMIC by verifying it is also not a thp allocation.
Signed-off-by: default avatarDavid Rientjes <rientjes@google.com>
Reported-by: default avatarAlex Thorlton <athorlton@sgi.com>
Tested-by: default avatarAlex Thorlton <athorlton@sgi.com>
Cc: Bob Liu <lliubbo@gmail.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hedi Berriche <hedi@sgi.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Rik van Riel <riel@redhat.com>
Cc: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent f6789593
...@@ -2447,7 +2447,7 @@ static inline int ...@@ -2447,7 +2447,7 @@ static inline int
gfp_to_alloc_flags(gfp_t gfp_mask) gfp_to_alloc_flags(gfp_t gfp_mask)
{ {
int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET; int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET;
const gfp_t wait = gfp_mask & __GFP_WAIT; const bool atomic = !(gfp_mask & (__GFP_WAIT | __GFP_NO_KSWAPD));
/* __GFP_HIGH is assumed to be the same as ALLOC_HIGH to save a branch. */ /* __GFP_HIGH is assumed to be the same as ALLOC_HIGH to save a branch. */
BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH); BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH);
...@@ -2456,20 +2456,20 @@ gfp_to_alloc_flags(gfp_t gfp_mask) ...@@ -2456,20 +2456,20 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
* The caller may dip into page reserves a bit more if the caller * The caller may dip into page reserves a bit more if the caller
* cannot run direct reclaim, or if the caller has realtime scheduling * cannot run direct reclaim, or if the caller has realtime scheduling
* policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will
* set both ALLOC_HARDER (!wait) and ALLOC_HIGH (__GFP_HIGH). * set both ALLOC_HARDER (atomic == true) and ALLOC_HIGH (__GFP_HIGH).
*/ */
alloc_flags |= (__force int) (gfp_mask & __GFP_HIGH); alloc_flags |= (__force int) (gfp_mask & __GFP_HIGH);
if (!wait) { if (atomic) {
/* /*
* Not worth trying to allocate harder for * Not worth trying to allocate harder for __GFP_NOMEMALLOC even
* __GFP_NOMEMALLOC even if it can't schedule. * if it can't schedule.
*/ */
if (!(gfp_mask & __GFP_NOMEMALLOC)) if (!(gfp_mask & __GFP_NOMEMALLOC))
alloc_flags |= ALLOC_HARDER; alloc_flags |= ALLOC_HARDER;
/* /*
* Ignore cpuset if GFP_ATOMIC (!wait) rather than fail alloc. * Ignore cpuset mems for GFP_ATOMIC rather than fail, see the
* See also cpuset_zone_allowed() comment in kernel/cpuset.c. * comment for __cpuset_node_allowed_softwall().
*/ */
alloc_flags &= ~ALLOC_CPUSET; alloc_flags &= ~ALLOC_CPUSET;
} else if (unlikely(rt_task(current)) && !in_interrupt()) } else if (unlikely(rt_task(current)) && !in_interrupt())
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment