• Kaike Wan's avatar
    IB/hfi1: Failed to drain send queue when QP is put into error state · 662d6646
    Kaike Wan authored
    When a QP is put into error state, all pending requests in the send work
    queue should be drained. The following sequence of events could lead to a
    failure, causing a request to hang:
    
    (1) The QP builds a packet and tries to send through SDMA engine.
        However, PIO engine is still busy. Consequently, this packet is put on
        the QP's tx list and the QP is put on the PIO waiting list. The field
        qp->s_flags is set with HFI1_S_WAIT_PIO_DRAIN;
    
    (2) The QP is put into error state by the user application and
        notify_error_qp() is called, which removes the QP from the PIO waiting
        list and the packet from the QP's tx list. In addition, qp->s_flags is
        cleared of RVT_S_ANY_WAIT_IO bits, which does not include
        HFI1_S_WAIT_PIO_DRAIN bit;
    
    (3) The hfi1_schdule_send() function is called to drain the QP's send
        queue. Subsequently, hfi1_do_send() is called. Since the flag bit
        HFI1_S_WAIT_PIO_DRAIN is set in qp->s_flags, hfi1_send_ok() fails.  As
        a result, hfi1_do_send() bails out without draining any request from
        the send queue;
    
    (4) The PIO engine completes the sending and tries to wake up any QP on
        its waiting list. But the QP has been removed from the PIO waiting
        list and therefore is kept in sleep forever.
    
    The fix is to clear qp->s_flags of HFI1_S_ANY_WAIT_IO bits in step (2).
    HFI1_S_ANY_WAIT_IO includes RVT_S_ANY_WAIT_IO and HFI1_S_WAIT_PIO_DRAIN.
    
    Fixes: 2e2ba09e ("IB/rdmavt, IB/hfi1: Create device dependent s_flags")
    Cc: <stable@vger.kernel.org> # 4.19.x+
    Reviewed-by: default avatarMike Marciniszyn <mike.marciniszyn@intel.com>
    Reviewed-by: default avatarAlex Estrin <alex.estrin@intel.com>
    Signed-off-by: default avatarKaike Wan <kaike.wan@intel.com>
    Signed-off-by: default avatarDennis Dalessandro <dennis.dalessandro@intel.com>
    Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
    662d6646
qp.c 25.4 KB