• Jens Axboe's avatar
    task_work: Use TIF_NOTIFY_SIGNAL if available · 114518eb
    Jens Axboe authored
    If the arch supports TIF_NOTIFY_SIGNAL, then use that for TWA_SIGNAL as
    it's more efficient than using the signal delivery method. This is
    especially true on threaded applications, where ->sighand is shared across
    threads, but it's also lighter weight on non-shared cases.
    
    io_uring is a heavy consumer of TWA_SIGNAL based task_work. A test with
    threads shows a nice improvement running an io_uring based echo server.
    
    stock kernel:
    0.01% <= 0.1 milliseconds
    95.86% <= 0.2 milliseconds
    98.27% <= 0.3 milliseconds
    99.71% <= 0.4 milliseconds
    100.00% <= 0.5 milliseconds
    100.00% <= 0.6 milliseconds
    100.00% <= 0.7 milliseconds
    100.00% <= 0.8 milliseconds
    100.00% <= 0.9 milliseconds
    100.00% <= 1.0 milliseconds
    100.00% <= 1.1 milliseconds
    100.00% <= 2 milliseconds
    100.00% <= 3 milliseconds
    100.00% <= 3 milliseconds
    1378930.00 requests per second
    ~1600% CPU
    
    1.38M requests/second, and all 16 CPUs are maxed out.
    
    patched kernel:
    0.01% <= 0.1 milliseconds
    98.24% <= 0.2 milliseconds
    99.47% <= 0.3 milliseconds
    99.99% <= 0.4 milliseconds
    100.00% <= 0.5 milliseconds
    100.00% <= 0.6 milliseconds
    100.00% <= 0.7 milliseconds
    100.00% <= 0.8 milliseconds
    100.00% <= 0.9 milliseconds
    100.00% <= 1.2 milliseconds
    1666111.38 requests per second
    ~1450% CPU
    
    1.67M requests/second, and we're no longer just hammering on the sighand
    lock. The original reporter states:
    
    "For 5.7.15 my benchmark achieves 1.6M qps and system cpu is at ~80%.
     for 5.7.16 or later it achieves only 1M qps and the system cpu is is
     at ~100%"
    
    with the only difference there being that TWA_SIGNAL is used
    unconditionally in 5.7.16, since it's required to be able to handle the
    inability to run task_work if the application is waiting in the kernel
    already on an event that needs task_work run to be satisfied. Also see
    commit 0ba9c9ed.
    Reported-by: default avatarRoman Gershman <romger@amazon.com>
    Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
    Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
    Reviewed-by: default avatarOleg Nesterov <oleg@redhat.com>
    Link: https://lore.kernel.org/r/20201026203230.386348-5-axboe@kernel.dk
    114518eb
task_work.c 4.35 KB