• Yamin Friedman's avatar
    RDMA/core: Provide RDMA DIM support for ULPs · da662979
    Yamin Friedman authored
    Added the interface in the infiniband driver that applies the rdma_dim
    adaptive moderation. There is now a special function for allocating an
    ib_cq that uses rdma_dim.
    
    Performance improvement (ConnectX-5 100GbE, x86) running FIO benchmark over
    NVMf between two equal end-hosts with 56 cores across a Mellanox switch
    using null_blk device:
    
    READS without DIM:
    blk size | BW       | IOPS | 99th percentile latency  | 99.99th latency
    512B     | 3.8GiB/s | 7.7M | 1401  usec               | 2442  usec
    4k       | 7.0GiB/s | 1.8M | 4817  usec               | 6587  usec
    64k      | 10.7GiB/s| 175k | 9896  usec               | 10028 usec
    
    IO WRITES without DIM:
    blk size | BW       | IOPS | 99th percentile latency  | 99.99th latency
    512B     | 3.6GiB/s | 7.5M | 1434  usec               | 2474  usec
    4k       | 6.3GiB/s | 1.6M | 938   usec               | 1221  usec
    64k      | 10.7GiB/s| 175k | 8979  usec               | 12780 usec
    
    IO READS with DIM:
    blk size | BW       | IOPS | 99th percentile latency  | 99.99th latency
    512B     | 4GiB/s   | 8.2M | 816    usec              | 889   usec
    4k       | 10.1GiB/s| 2.65M| 3359   usec              | 5080  usec
    64k      | 10.7GiB/s| 175k | 9896   usec              | 10028 usec
    
    IO WRITES with DIM:
    blk size | BW       | IOPS  | 99th percentile latency | 99.99th latency
    512B     | 3.9GiB/s | 8.1M  | 799   usec              | 922   usec
    4k       | 9.6GiB/s | 2.5M  | 717   usec              | 1004  usec
    64k      | 10.7GiB/s| 176k  | 8586  usec              | 12256 usec
    
    The rdma_dim algorithm was designed to measure the effectiveness of
    moderation on the flow in a general way and thus should be appropriate
    for all RDMA storage protocols.
    
    rdma_dim is configured to be the default option based on performance
    improvement seen after extensive tests.
    Signed-off-by: default avatarYamin Friedman <yaminf@mellanox.com>
    Reviewed-by: default avatarMax Gurtovoy <maxg@mellanox.com>
    Reviewed-by: default avatarSagi Grimberg <sagi@grimberg.me>
    Signed-off-by: default avatarSaeed Mahameed <saeedm@mellanox.com>
    Signed-off-by: default avatarLeon Romanovsky <leonro@mellanox.com>
    Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
    da662979
ib_verbs.h 133 KB