• Yosry Ahmed's avatar
    mm: page_alloc: fix missed updates of PGFREE in free_unref_{page/folios} · ec867977
    Yosry Ahmed authored
    PGFREE is currently updated in two code paths:
    
    - __free_pages_ok(): for pages freed to the buddy allocator.
    - free_unref_page_commit(): for pages freed to the pcplists.
    
    Before commit df1acc85 ("mm/page_alloc: avoid conflating IRQs disabled
    with zone->lock"), free_unref_page_commit() used to fallback to freeing
    isolated pages directly to the buddy allocator through free_one_page(). 
    This was done _after_ updating PGFREE, so the counter was correctly
    updated.
    
    However, that commit moved the fallback logic to its callers (now called
    free_unref_page() and free_unref_folios()), so PGFREE was no longer
    updated in this fallback case.
    
    Now that the code has developed, there are more cases in free_unref_page()
    and free_unref_folios() where we fallback to calling free_one_page() (e.g.
    !pcp_allowed_order(), pcp_spin_trylock() fails).  These cases also miss
    updating PGFREE.
    
    To make sure PGFREE is updated in all cases where pages are freed to the
    buddy allocator, move the update down the stack to free_one_page().
    
    This was noticed through code inspection, although it should be noticeable
    at runtime (at least with some workloads).
    
    Link: https://lkml.kernel.org/r/20240904205419.821776-1-yosryahmed@google.com
    Fixes: df1acc85 ("mm/page_alloc: avoid conflating IRQs disabled with zone->lock")
    Signed-off-by: default avatarYosry Ahmed <yosryahmed@google.com>
    Cc: Brendan Jackman <jackmanb@google.com>
    Cc: Mel Gorman <mgorman@techsingularity.net>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Vlastimil Babka <vbabka@suse.cz>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    ec867977
page_alloc.c 197 KB