• Christoph Lameter's avatar
    SLUB: Place kmem_cache_cpu structures in a NUMA aware way · 4c93c355
    Christoph Lameter authored
    The kmem_cache_cpu structures introduced are currently an array placed in the
    kmem_cache struct. Meaning the kmem_cache_cpu structures are overwhelmingly
    on the wrong node for systems with a higher amount of nodes. These are
    performance critical structures since the per node information has
    to be touched for every alloc and free in a slab.
    
    In order to place the kmem_cache_cpu structure optimally we put an array
    of pointers to kmem_cache_cpu structs in kmem_cache (similar to SLAB).
    
    However, the kmem_cache_cpu structures can now be allocated in a more
    intelligent way.
    
    We would like to put per cpu structures for the same cpu but different
    slab caches in cachelines together to save space and decrease the cache
    footprint. However, the slab allocators itself control only allocations
    per node. We set up a simple per cpu array for every processor with
    100 per cpu structures which is usually enough to get them all set up right.
    If we run out then we fall back to kmalloc_node. This also solves the
    bootstrap problem since we do not have to use slab allocator functions
    early in boot to get memory for the small per cpu structures.
    
    Pro:
    	- NUMA aware placement improves memory performance
    	- All global structures in struct kmem_cache become readonly
    	- Dense packing of per cpu structures reduces cacheline
    	  footprint in SMP and NUMA.
    	- Potential avoidance of exclusive cacheline fetches
    	  on the free and alloc hotpath since multiple kmem_cache_cpu
    	  structures are in one cacheline. This is particularly important
    	  for the kmalloc array.
    
    Cons:
    	- Additional reference to one read only cacheline (per cpu
    	  array of pointers to kmem_cache_cpu) in both slab_alloc()
    	  and slab_free().
    
    [akinobu.mita@gmail.com: fix cpu hotplug offline/online path]
    Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
    Cc: "Pekka Enberg" <penberg@cs.helsinki.fi>
    Cc: Akinobu Mita <akinobu.mita@gmail.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    4c93c355
slub.c 92.9 KB