• Jens Axboe's avatar
    io_uring: switch to atomic_t for io_kiocb reference count · abc54d63
    Jens Axboe authored
    io_uring manipulates references twice for each request, and hence is very
    sensitive to performance of the reference count. This commit borrows a
    trick from:
    
    commit f958d7b5
    Author: Linus Torvalds <torvalds@linux-foundation.org>
    Date:   Thu Apr 11 10:06:20 2019 -0700
    
        mm: make page ref count overflow check tighter and more explicit
    
    and switches to atomic_t for references, while still retaining overflow
    and underflow checks.
    
    This is good for a 2-3% increase in peak IOPS on a single core. Before:
    
    IOPS=2970879, IOS/call=31/31, inflight=128 (128)
    IOPS=2952597, IOS/call=31/31, inflight=128 (128)
    IOPS=2943904, IOS/call=31/31, inflight=128 (128)
    IOPS=2930006, IOS/call=31/31, inflight=96 (96)
    
    and after:
    
    IOPS=3054354, IOS/call=31/31, inflight=128 (128)
    IOPS=3059038, IOS/call=31/31, inflight=128 (128)
    IOPS=3060320, IOS/call=31/31, inflight=128 (128)
    IOPS=3068256, IOS/call=31/31, inflight=96 (96)
    Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
    abc54d63
io_uring.c 242 KB