• Bart Van Assche's avatar
    IB/srpt: Reduce QP buffer size · c0cf4512
    Bart Van Assche authored
    The memory needed for the send and receive queues associated with
    a QP is proportional to the max_sge parameter. The current value
    of that parameter is such that with an mlx4 HCA the QP buffer size
    is 8 MB. Since DMA is used for communication between HCA and CPU
    that buffer either has to be allocated coherently or map_single()
    must succeed for that buffer. Since large contiguous allocations
    are fragile and since the maximum segment size for e.g. swiotlb
    is 256 KB, reduce the max_sge parameter. This patch avoids that
    the following text appears on the console after SRP logout and
    relogin on a system equipped with multiple IB HCAs:
    
    mlx4_core 0000:05:00.0: swiotlb buffer is full (sz: 8388608 bytes)
    swiotlb: coherent allocation failed for device 0000:05:00.0 size=8388608
    CPU: 11 PID: 148 Comm: kworker/11:1 Not tainted 4.7.0-rc4-dbg+ #1
    Call Trace:
     [<ffffffff812c6d35>] dump_stack+0x67/0x92
     [<ffffffff812efe71>] swiotlb_alloc_coherent+0x141/0x150
     [<ffffffff810458be>] x86_swiotlb_alloc_coherent+0x3e/0x50
     [<ffffffffa03861fa>] mlx4_buf_direct_alloc.isra.5+0x9a/0x120 [mlx4_core]
     [<ffffffffa0386545>] mlx4_buf_alloc+0x165/0x1a0 [mlx4_core]
     [<ffffffffa035053d>] create_qp_common.isra.29+0x57d/0xff0 [mlx4_ib]
     [<ffffffffa03510da>] mlx4_ib_create_qp+0x12a/0x3f0 [mlx4_ib]
     [<ffffffffa031154a>] ib_create_qp+0x3a/0x250 [ib_core]
     [<ffffffffa055dd4b>] srpt_cm_handler+0x4bb/0xcad [ib_srpt]
     [<ffffffffa02c1ab0>] cm_process_work+0x20/0xf0 [ib_cm]
     [<ffffffffa02c3640>] cm_work_handler+0x1ac0/0x2059 [ib_cm]
     [<ffffffff810737ed>] process_one_work+0x19d/0x490
     [<ffffffff81073b29>] worker_thread+0x49/0x490
     [<ffffffff8107a0ea>] kthread+0xea/0x100
     [<ffffffff815b25af>] ret_from_fork+0x1f/0x40
    
    Fixes: b99f8e4d ("IB/srpt: convert to the generic RDMA READ/WRITE API")
    Signed-off-by: default avatarBart Van Assche <bart.vanassche@sandisk.com>
    Cc: Laurence Oberman <loberman@redhat.com>
    Cc: Christoph Hellwig <hch@lst.de>
    Cc: Sagi Grimberg <sagi@grimberg.me>
    Signed-off-by: default avatarDoug Ledford <dledford@redhat.com>
    c0cf4512
ib_srpt.h 11.7 KB