Commit 0a5f4e5b authored by Aaron Lu's avatar Aaron Lu Committed by Linus Torvalds

mm/free_pcppages_bulk: do not hold lock when picking pages to free

When freeing a batch of pages from Per-CPU-Pages(PCP) back to buddy, the
zone->lock is held and then pages are chosen from PCP's migratetype
list.  While there is actually no need to do this 'choose part' under
lock since it's PCP pages, the only CPU that can touch them is us and
irq is also disabled.

Moving this part outside could reduce lock held time and improve
performance.  Test with will-it-scale/page_fault1 full load:

  kernel      Broadwell(2S)  Skylake(2S)   Broadwell(4S)  Skylake(4S)
  v4.16-rc2+  9034215        7971818       13667135       15677465
  this patch  95363747 +5.6%  8314710 +4.3% 14070408 +3.0% 16675866 +6.4%

What the test does is: starts $nr_cpu processes and each will repeatedly
do the following for 5 minutes:

 - mmap 128M anonymouse space

 - write access to that space

 - munmap.

The score is the aggregated iteration.

https://github.com/antonblanchard/will-it-scale/blob/master/tests/page_fault1.c

Link: http://lkml.kernel.org/r/20180301062845.26038-3-aaron.lu@intel.comSigned-off-by: default avatarAaron Lu <aaron.lu@intel.com>
Acked-by: default avatarMel Gorman <mgorman@techsingularity.net>
Acked-by: default avatarMichal Hocko <mhocko@suse.com>
Reviewed-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Huang Ying <ying.huang@intel.com>
Cc: Kemi Wang <kemi.wang@intel.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 77ba9062
...@@ -1080,12 +1080,10 @@ static void free_pcppages_bulk(struct zone *zone, int count, ...@@ -1080,12 +1080,10 @@ static void free_pcppages_bulk(struct zone *zone, int count,
int migratetype = 0; int migratetype = 0;
int batch_free = 0; int batch_free = 0;
bool isolated_pageblocks; bool isolated_pageblocks;
struct page *page, *tmp;
spin_lock(&zone->lock); LIST_HEAD(head);
isolated_pageblocks = has_isolate_pageblock(zone);
while (count) { while (count) {
struct page *page;
struct list_head *list; struct list_head *list;
/* /*
...@@ -1107,26 +1105,35 @@ static void free_pcppages_bulk(struct zone *zone, int count, ...@@ -1107,26 +1105,35 @@ static void free_pcppages_bulk(struct zone *zone, int count,
batch_free = count; batch_free = count;
do { do {
int mt; /* migratetype of the to-be-freed page */
page = list_last_entry(list, struct page, lru); page = list_last_entry(list, struct page, lru);
/* must delete as __free_one_page list manipulates */ /* must delete to avoid corrupting pcp list */
list_del(&page->lru); list_del(&page->lru);
pcp->count--; pcp->count--;
mt = get_pcppage_migratetype(page); if (bulkfree_pcp_prepare(page))
continue;
list_add_tail(&page->lru, &head);
} while (--count && --batch_free && !list_empty(list));
}
spin_lock(&zone->lock);
isolated_pageblocks = has_isolate_pageblock(zone);
/*
* Use safe version since after __free_one_page(),
* page->lru.next will not point to original list.
*/
list_for_each_entry_safe(page, tmp, &head, lru) {
int mt = get_pcppage_migratetype(page);
/* MIGRATE_ISOLATE page should not go to pcplists */ /* MIGRATE_ISOLATE page should not go to pcplists */
VM_BUG_ON_PAGE(is_migrate_isolate(mt), page); VM_BUG_ON_PAGE(is_migrate_isolate(mt), page);
/* Pageblock could have been isolated meanwhile */ /* Pageblock could have been isolated meanwhile */
if (unlikely(isolated_pageblocks)) if (unlikely(isolated_pageblocks))
mt = get_pageblock_migratetype(page); mt = get_pageblock_migratetype(page);
if (bulkfree_pcp_prepare(page))
continue;
__free_one_page(page, page_to_pfn(page), zone, 0, mt); __free_one_page(page, page_to_pfn(page), zone, 0, mt);
trace_mm_page_pcpu_drain(page, 0, mt); trace_mm_page_pcpu_drain(page, 0, mt);
} while (--count && --batch_free && !list_empty(list));
} }
spin_unlock(&zone->lock); spin_unlock(&zone->lock);
} }
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment