• Kirill Smelkov's avatar
    Y wcfs: Fix and enhance `wcfs stop` to be more reliable · 4095241e
    Kirill Smelkov authored
    Since wcfs beginning - since e3f2ee2d (wcfs: Initial implementation of
    basic filesystem) `wcfs stop` was implemented as just `fusermount -u`.
    That, however, turned out to be not robust because if wcfs is
    deadlocked, unmounting hangs, and if wcfs server is crashed, but there
    are still running client processes, unmount will fail with "Device or
    resource busy" error.
    
    For the deadlocked case we often see a situation where both wcfs and
    client zope processes are hung, kill -9 does not work on them (they
    still remain hung) and there is no easy way to do the unmount and
    restart wcfs.
    
    -> Fix `wcfs stop` to do that by first breaking the deadlock via
    /sys/fs/fuse/connection/<X>/abort and making sure that:
    
    1) wcfs.go is not running,
    2) all left clients are terminated, and
    3) the mount is also gone
    
    In many ways this coincides with what Server.stop was already doing, so
    here we teach `wcfs stop` to work via that Server.stop codepath and
    adjust the latter to work ok if Server._proc is not only
    subprocess.Popen that current process spawned, but also an xos.Proc,
    that `wcfs stop` discovered. Which can be also None if wcfs.go crashed
    by itself.
    
    As explained in the comments I took the decision to kill client
    processes instead of doing the final unmount try lazily because
    
        # NOTE if we do `fusermount -uz` (lazy unmount = MNT_DETACH), we will
        #      remove the mount from filesystem tree and /proc/mounts, but the
        #      clients will be left alive and using the old filesystem which is
        #      left in a bad ENOTCONN state. From this point of view restart of
        #      the clients is more preferred compared to leaving them running
        #      but actually disconnected from the data.
        #
        # TODO try to teach wcfs clients to detect filesystem in ENOTCONN state
        #      and reconnect automatically instead of being killed. Then we could
        #      use MNT_DETACH.
    4095241e
__init__.py 39.2 KB