• Vlastimil Babka's avatar
    mm/sl[au]b: rearrange struct slab fields to allow larger rcu_head · 130d4df5
    Vlastimil Babka authored
    Joel reports [1] that increasing the rcu_head size for debugging
    purposes used to work before struct slab was split from struct page, but
    now runs into the various SLAB_MATCH() sanity checks of the layout.
    
    This is because the rcu_head in struct page is in union with large
    sub-structures and has space to grow without exceeding their size, while
    in struct slab (for SLAB and SLUB) it's in union only with a list_head.
    
    On closer inspection (and after the previous patch) we can put all
    fields except slab_cache to a union with rcu_head, as slab_cache is
    sufficient for the rcu freeing callbacks to work and the rest can be
    overwritten by rcu_head without causing issues.
    
    This is only somewhat complicated by the need to keep SLUB's
    freelist+counters aligned for cmpxchg_double. As a result the fields
    need to be reordered so that slab_cache is first (after page flags) and
    the union with rcu_head follows. For consistency, do that for SLAB as
    well, although not necessary there.
    
    As a result, the rcu_head field in struct page and struct slab is no
    longer at the same offset, but that doesn't matter as there is no
    casting that would rely on that in the slab freeing callbacks, so we can
    just drop the respective SLAB_MATCH() check.
    
    Also we need to update the SLAB_MATCH() for compound_head to reflect the
    new ordering.
    
    While at it, also add a static_assert to check the alignment needed for
    cmpxchg_double so mistakes are found sooner than a runtime GPF.
    
    [1] https://lore.kernel.org/all/85afd876-d8bb-0804-b2c5-48ed3055e702@joelfernandes.org/Reported-by: default avatarJoel Fernandes <joel@joelfernandes.org>
    Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Acked-by: default avatarHyeonggon Yoo <42.hyeyoo@gmail.com>
    130d4df5
slab.h 23.1 KB