• Jason Wang's avatar
    vhost: access vq metadata through kernel virtual address · 7f466032
    Jason Wang authored
    It was noticed that the copy_to/from_user() friends that was used to
    access virtqueue metdata tends to be very expensive for dataplane
    implementation like vhost since it involves lots of software checks,
    speculation barriers, hardware feature toggling (e.g SMAP). The
    extra cost will be more obvious when transferring small packets since
    the time spent on metadata accessing become more significant.
    
    This patch tries to eliminate those overheads by accessing them
    through direct mapping of those pages. Invalidation callbacks is
    implemented for co-operation with general VM management (swap, KSM,
    THP or NUMA balancing). We will try to get the direct mapping of vq
    metadata before each round of packet processing if it doesn't
    exist. If we fail, we will simplely fallback to copy_to/from_user()
    friends.
    
    This invalidation and direct mapping access are synchronized through
    spinlock and RCU. All matedata accessing through direct map is
    protected by RCU, and the setup or invalidation are done under
    spinlock.
    
    This method might does not work for high mem page which requires
    temporary mapping so we just fallback to normal
    copy_to/from_user() and may not for arch that has virtual tagged cache
    since extra cache flushing is needed to eliminate the alias. This will
    result complex logic and bad performance. For those archs, this patch
    simply go for copy_to/from_user() friends. This is done by ruling out
    kernel mapping codes through ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE.
    
    Note that this is only done when device IOTLB is not enabled. We
    could use similar method to optimize IOTLB in the future.
    
    Tests shows at most about 23% improvement on TX PPS when using
    virtio-user + vhost_net + xdp1 + TAP on 2.6GHz Broadwell:
    
            SMAP on | SMAP off
    Before: 5.2Mpps | 7.1Mpps
    After:  6.4Mpps | 8.2Mpps
    
    Cc: Andrea Arcangeli <aarcange@redhat.com>
    Cc: James Bottomley <James.Bottomley@hansenpartnership.com>
    Cc: Christoph Hellwig <hch@infradead.org>
    Cc: David Miller <davem@davemloft.net>
    Cc: Jerome Glisse <jglisse@redhat.com>
    Cc: linux-mm@kvack.org
    Cc: linux-arm-kernel@lists.infradead.org
    Cc: linux-parisc@vger.kernel.org
    Signed-off-by: default avatarJason Wang <jasowang@redhat.com>
    Signed-off-by: default avatarMichael S. Tsirkin <mst@redhat.com>
    7f466032
vhost.c 72.7 KB