• Anton Eidelman's avatar
    nvme-multipath: fix possible io hang after ctrl reconnect · af8fd042
    Anton Eidelman authored
    The following scenario results in an IO hang:
    1) ctrl completes a request with NVME_SC_ANA_TRANSITION.
       NVME_NS_ANA_PENDING bit in ns->flags is set and ana_work is triggered.
    2) ana_work: nvme_read_ana_log() tries to get the ANA log page from the ctrl.
       This fails because ctrl disconnects.
       Therefore nvme_update_ns_ana_state() is not called
       and NVME_NS_ANA_PENDING bit in ns->flags is not cleared.
    3) ctrl reconnects: nvme_mpath_init(ctrl,...) calls
       nvme_read_ana_log(ctrl, groups_only=true).
       However, nvme_update_ana_state() does not update namespaces
       because nr_nsids = 0 (due to groups_only mode).
    4) scan_work calls nvme_validate_ns() finds the ns and re-validates OK.
    
    Result:
    The ctrl is now live but NVME_NS_ANA_PENDING bit in ns->flags is still set.
    Consequently ctrl will never be considered a viable path by __nvme_find_path().
    IO will hang if ctrl is the only or the last path to the namespace.
    
    More generally, while ctrl is reconnecting, its ANA state may change.
    And because nvme_mpath_init() requests ANA log in groups_only mode,
    these changes are not propagated to the existing ctrl namespaces.
    This may result in a mal-function or an IO hang.
    
    Solution:
    nvme_mpath_init() will nvme_read_ana_log() with groups_only set to false.
    This will not harm the new ctrl case (no namespaces present),
    and will make sure the ANA state of namespaces gets updated after reconnect.
    
    Note: Another option would be for nvme_mpath_init() to invoke
    nvme_parse_ana_log(..., nvme_set_ns_ana_state) for each existing namespace.
    Reviewed-by: default avatarSagi Grimberg <sagi@grimberg.me>
    Signed-off-by: default avatarAnton Eidelman <anton@lightbitslabs.com>
    Signed-off-by: default avatarKeith Busch <kbusch@kernel.org>
    Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
    af8fd042
multipath.c 18.6 KB