• Dave Hansen's avatar
    mm/slub.c: fix page->_count corruption (again) · a0320865
    Dave Hansen authored
    Commit abca7c49 ("mm: fix slab->page _count corruption when using
    slub") notes that we can not _set_ a page->counters directly, except
    when using a real double-cmpxchg.  Doing so can lose updates to
    ->_count.
    
    That is an absolute rule:
    
            You may not *set* page->counters except via a cmpxchg.
    
    Commit abca7c49 fixed this for the folks who have the slub
    cmpxchg_double code turned off at compile time, but it left the bad case
    alone.  It can still be reached, and the same bug triggered in two
    cases:
    
    1. Turning on slub debugging at runtime, which is available on
       the distro kernels that I looked at.
    2. On 64-bit CPUs with no CMPXCHG16B (some early AMD x86-64
       cpus, evidently)
    
    There are at least 3 ways we could fix this:
    
    1. Take all of the exising calls to cmpxchg_double_slab() and
       __cmpxchg_double_slab() and convert them to take an old, new
       and target 'struct page'.
    2. Do (1), but with the newly-introduced 'slub_data'.
    3. Do some magic inside the two cmpxchg...slab() functions to
       pull the counters out of new_counters and only set those
       fields in page->{inuse,frozen,objects}.
    
    I've done (2) as well, but it's a bunch more code.  This patch is an
    attempt at (3).  This was the most straightforward and foolproof way
    that I could think to do this.
    
    This would also technically allow us to get rid of the ugly
    
    #if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
           defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
    
    in 'struct page', but leaving it alone has the added benefit that
    'counters' stays 'unsigned' instead of 'unsigned long', so all the
    copies that the slub code does stay a bit smaller.
    Signed-off-by: default avatarDave Hansen <dave.hansen@linux.intel.com>
    Cc: Christoph Lameter <cl@linux-foundation.org>
    Cc: Pekka Enberg <penberg@kernel.org>
    Cc: Matt Mackall <mpm@selenic.com>
    Cc: Pravin B Shelar <pshelar@nicira.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    a0320865
slub.c 126 KB