• Christoph Lameter's avatar
    SLUB: Alternate fast paths using cmpxchg_local · 1f84260c
    Christoph Lameter authored
    Provide an alternate implementation of the SLUB fast paths for alloc
    and free using cmpxchg_local. The cmpxchg_local fast path is selected
    for arches that have CONFIG_FAST_CMPXCHG_LOCAL set. An arch should only
    set CONFIG_FAST_CMPXCHG_LOCAL if the cmpxchg_local is faster than an
    interrupt enable/disable sequence. This is known to be true for both
    x86 platforms so set FAST_CMPXCHG_LOCAL for both arches.
    
    Currently another requirement for the fastpath is that the kernel is
    compiled without preemption. The restriction will go away with the
    introduction of a new per cpu allocator and new per cpu operations.
    
    The advantages of a cmpxchg_local based fast path are:
    
    1. Potentially lower cycle count (30%-60% faster)
    
    2. There is no need to disable and enable interrupts on the fast path.
       Currently interrupts have to be disabled and enabled on every
       slab operation. This is likely avoiding a significant percentage
       of interrupt off / on sequences in the kernel.
    
    3. The disposal of freed slabs can occur with interrupts enabled.
    
    The alternate path is realized using #ifdef's. Several attempts to do the
    same with macros and inline functions resulted in a mess (in particular due
    to the strange way that local_interrupt_save() handles its argument and due
    to the need to define macros/functions that sometimes disable interrupts
    and sometimes do something else).
    
    [clameter: Stripped preempt bits and disabled fastpath if preempt is enabled]
    Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
    Reviewed-by: default avatarPekka Enberg <penberg@cs.helsinki.fi>
    Cc: <linux-arch@vger.kernel.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    1f84260c
slub.c 100 KB