• Ingo Molnar's avatar
    [PATCH] notifiers: fix blocking_notifier_call_chain() scalability · 1b5180b6
    Ingo Molnar authored
    while lock-profiling the -rt kernel i noticed weird contention during
    mmap-intense workloads, and the tracer showed the following gem, in one
    of our MM hotpaths:
    
     threaded-2771  1....   65us : sys_munmap (sysenter_do_call)
     threaded-2771  1....   66us : profile_munmap (sys_munmap)
     threaded-2771  1....   66us : blocking_notifier_call_chain (profile_munmap)
     threaded-2771  1....   66us : rt_down_read (blocking_notifier_call_chain)
    
    ouch! a global rw-semaphore taken in one of the most performance-
    sensitive codepaths of the kernel.  And i dont even have oprofile
    enabled! All distro kernels have CONFIG_PROFILING enabled, so this
    scalability problem affects the majority of Linux users.
    
    The fix is to enhance blocking_notifier_call_chain() to only take the
    lock if there appears to be work on the call-chain.
    
    With this patch applied i get nicely saturated system, and much higher
    munmap performance, on SMP systems.
    
    And as a bonus this also fixes a similar scalability bottleneck in the
    thread-exit codepath: profile_task_exit() ...
    Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
    Acked-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
    Acked-by: default avatarNick Piggin <nickpiggin@yahoo.com.au>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    1b5180b6
sys.c 53.5 KB