• Johannes Weiner's avatar
    mm: page_alloc: close migratetype race between freeing and stealing · 55612e80
    Johannes Weiner authored
    There are three freeing paths that read the page's migratetype
    optimistically before grabbing the zone lock.  When this races with block
    stealing, those pages go on the wrong freelist.
    
    The paths in question are:
    - when freeing >costly orders that aren't THP
    - when freeing pages to the buddy upon pcp lock contention
    - when freeing pages that are isolated
    - when freeing pages initially during boot
    - when freeing the remainder in alloc_pages_exact()
    - when "accepting" unaccepted VM host memory before first use
    - when freeing pages during unpoisoning
    
    None of these are so hot that they would need this optimization at the
    cost of hampering defrag efforts.  Especially when contrasted with the
    fact that the most common buddy freeing path - free_pcppages_bulk - is
    checking the migratetype under the zone->lock just fine.
    
    In addition, isolated pages need to look up the migratetype under the lock
    anyway, which adds branches to the locked section, and results in a double
    lookup when the pages are in fact isolated.
    
    Move the lookups into the lock.
    
    Link: https://lkml.kernel.org/r/20240320180429.678181-8-hannes@cmpxchg.orgSigned-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
    Reported-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Reviewed-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Tested-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
    Cc: David Hildenbrand <david@redhat.com>
    Cc: "Huang, Ying" <ying.huang@intel.com>
    Cc: Mel Gorman <mgorman@techsingularity.net>
    Cc: Zi Yan <ziy@nvidia.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    55612e80
page_alloc.c 193 KB