• Or Gerlitz's avatar
    IB/iser: Enhance disconnection logic for multi-pathing · 39ff05db
    Or Gerlitz authored
    The iser connection teardown flow isn't over until the underlying
    Connection Manager (e.g the IB CM) delivers a disconnected or timeout
    event through the RDMA-CM.  When the remote (target) side isn't
    reachable, e.g when some HW e.g port/hca/switch isn't functioning or
    taken down administratively, the CM timeout flow is used and the event
    may be generated only after relatively long time -- on the order of
    tens of seconds.
    
    The current iser code exposes this possibly long delay to higher
    layers, specifically to the iscsid daemon and iscsi kernel stack. As a
    result, the iscsi stack doesn't respond well: this low-level CM delay
    is added to the fail-over time under HA schemes such as the one
    provided by DM multipath through the multipathd(8) service.
    
    This patch enhances the reference counting scheme on iser's IB
    connections so that the disconnect flow initiated by iscsid from user
    space (ep_disconnect) doesn't wait for the CM to deliver the
    disconnect/timeout event.  (The connection teardown isn't done from
    iser's view point until the event is delivered)
    
    The iser ib (rdma) connection object is destroyed when its reference
    count reaches zero.  When this happens on the RDMA-CM callback
    context, extra care is taken so that the RDMA-CM does the actual
    destroying of the associated ID, since doing it in the callback is
    prohibited.
    
    The reference count of iser ib connection normally reaches three,
    where the <ref, deref> relations are
    
     1. conn <init, terminate>
     2. conn <bind, stop/destroy>
     3. cma id <create, disconnect/error/timeout callbacks>
    
    With this patch, multipath fail-over time is about 30 seconds, while
    without this patch, multipath fail-over time is about 130 seconds.
    Signed-off-by: default avatarOr Gerlitz <ogerlitz@voltaire.com>
    Signed-off-by: default avatarRoland Dreier <rolandd@cisco.com>
    39ff05db
iscsi_iser.h 12 KB