• Peter-Jan Gootzen's avatar
    virtio-fs: add multi-queue support · 529395d2
    Peter-Jan Gootzen authored
    This commit creates a multi-queue mapping at device bring-up.
    The driver first attempts to use the existing MSI-X interrupt
    affinities (previously disabled), and if not present, will distribute
    the request queues evenly over the CPUs.
    If the latter fails as well, all CPUs are mapped to request queue zero.
    
    When a request is handed from FUSE to the virtio-fs device driver, the
    driver will use the current CPU to index into the multi-queue mapping
    and determine the optimal request queue to use.
    
    We measured the performance of this patch with the fio benchmarking
    tool, increasing the number of queues results in a significant speedup
    for both read and write operations, demonstrating the effectiveness
    of multi-queue support.
    
    Host:
      - Dell PowerEdge R760
      - CPU: Intel(R) Xeon(R) Gold 6438M, 128 cores
      - VM: KVM with 32 cores
    Virtio-fs device:
      - BlueField-3 DPU
      - CPU: ARM Cortex-A78AE, 16 cores
      - One thread per queue, each busy polling on one request queue
      - Each queue is 1024 descriptors deep
    Workload:
      - fio, sequential read or write, ioengine=libaio, numjobs=32,
        4GiB file per job, iodepth=8, bs=256KiB, runtime=30s
    Performance Results:
    +===========================+==========+===========+
    |     Number of queues      | Fio read | Fio write |
    +===========================+==========+===========+
    | 1 request queue (GiB/s)   | 6.1     | 4.6        |
    +---------------------------+----------+-----------+
    | 8 request queues (GiB/s)  | 25.8    | 10.3       |
    +---------------------------+----------+-----------+
    | 16 request queues (GiB/s) | 30.9    | 19.5       |
    +---------------------------+----------+-----------+
    | 32 request queue (GiB/s)  | 33.2    | 22.6       |
    +---------------------------+----------+-----------+
    | Speedup                   | 5.5x    | 5x         |
    +---------------=-----------+----------+-----------+
    Signed-off-by: default avatarPeter-Jan Gootzen <pgootzen@nvidia.com>
    Signed-off-by: default avatarYoray Zack <yorayz@nvidia.com>
    Signed-off-by: default avatarMax Gurtovoy <mgurtovoy@nvidia.com>
    Reviewed-by: default avatarStefan Hajnoczi <stefanha@redhat.com>
    Acked-by: default avatarMichael S. Tsirkin <mst@redhat.com>
    Signed-off-by: default avatarMiklos Szeredi <mszeredi@redhat.com>
    529395d2
virtio_fs.c 41.4 KB