• Alexander Aring's avatar
    dlm: fix race between final callback and remove · 986ae3c2
    Alexander Aring authored
    This patch fixes the following issue:
    
    node 1 is dir
    node 2 is master
    node 3 is other
    
    1->2: unlock
    2: put final lkb, rsb moved to toss
    2->1: unlock_reply
    1: queue lkb callback with EUNLOCK
    2->1: remove
    1: receive_remove ignored (rsb on keep because of queued lkb callback)
    1: complete lkb callback, put_lkb, move rsb to toss
    3->1: lookup
    1->3: lookup_reply master=2
    3->2: request
    2->3: request_reply EBADR
    
    In summary:
    An unexpected lkb reference causes the rsb to remain on the wrong list.
    The rsb being on the wrong list causes receive_remove to be ignored.
    An ignored receive_remove causes inconsistent dir and master state.
    
    This sequence requires an unusually long delay in delivering the unlock
    callback, because the remove message from 2->1 usually happens after
    some seconds.  So, it's not known exactly how frequently this sequence
    occurs in pratice.  It's possible that the same end result could also
    have another unknown cause.
    
    The solution for this issue is to further separate callback state
    from the lkb, so that an lkb reference (and from that, an rsb ref)
    are not held while a callback remains queued.  Then, within the
    unlock_reply, the lkb will be freed and the rsb moved to the toss
    list. So, the receive_remove will not be ignored.
    Signed-off-by: default avatarAlexander Aring <aahringo@redhat.com>
    Signed-off-by: default avatarDavid Teigland <teigland@redhat.com>
    986ae3c2
dlm_internal.h 22.4 KB