1. 18 Jun, 2021 7 commits
    • Kirill Smelkov's avatar
      X wcfs: Fix ZSync to close wconn on zdb.close, even if zconn stays alive · 7203d7ab
      Kirill Smelkov authored
      Even if ZODB DB is closed and releases zconn from its .pool, zconn can
      still stay alive being referenced from any other live python object -
      e.g. from a frame, a traceback etc. This situation in particular happens
      under ERP5's runUnitTest.
      
      As the result, if we don't hook into DB.close and close wconn only on
      zconn GC, files opened on WCFS filesystem stay opened, and WCFS server
      cannot be cleanly unmounted when test run completes.
      
      -> Fix it.
      
      P.S. ZODB@bbd03b3a is good, but does
      not really solve this problem because, once again, zconn stays
      referenced from objects besides DB.
      7203d7ab
    • Kirill Smelkov's avatar
      X lib/zodb: Connection += onShutdownCallback · a26d9659
      Kirill Smelkov authored
      In the next patch we'll need this functionality to subscribe ZSync into
      db.close to know when ZODB Connection is shut down even if it stays
      alive referenced by some other objects.
      a26d9659
    • Kirill Smelkov's avatar
      . · 8dcaa5de
      Kirill Smelkov authored
      8dcaa5de
    • Kirill Smelkov's avatar
      X wcfs: lsof +D misbehaves - don't use it · 3244f3a6
      Kirill Smelkov authored
      For example even with live wcfs lsof +D output differes from regular lsof and is misleading:
      
          LSOF /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2:
      
          COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
          runUnitTe 29137 slapuser34   24r   REG   0,48      111    4 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/.wcfs/zurl
          runUnitTe 29137 slapuser34   25u   REG   0,48        0    7 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/head/watch
          runUnitTe 29137 slapuser34   26r   REG   0,48  2097152    9 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/head/bigfile/00000000000077e2
      
          LSOF +D /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2:
      
          COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
          runUnitTe 29137 slapuser34   24r   REG   0,48      111 4294967295 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/.wcfs/zurl
          runUnitTe 29137 slapuser34   25u   REG   0,48        0 4294967295 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/.wcfs/zurl
          runUnitTe 29137 slapuser34   26r   REG   0,48  2097152 4294967295 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/.wcfs/zurl
      3244f3a6
    • Kirill Smelkov's avatar
      X wcfs: _fuse_unmount: Try first `kill -TERM` before `kill -QUIT` wcfs · 153c002a
      Kirill Smelkov authored
      Just aborting the FUSE connection does not make WCFS to exit. This abort
      is needed to avoid deadlocks, but we should also signal WCFS it should
      stop. Not doing so will lead to timeout in wait after FUSE connection
      abort and `kill -QUIT` triggered, which makes WCFS crash and print lots
      of traceback to stderr before exiting.
      153c002a
    • Kirill Smelkov's avatar
      X wcfs: Tune _fuse_unmount to include `fusermount -u` error message into raised exception · 15389db0
      Kirill Smelkov authored
      Previously that message was printed to stderr and it was very confusing:
      we had something in the exception message, something in the log and
      something in stderr.
      
      Now exception message is self-sufficient and log provides more details.
      15389db0
    • Kirill Smelkov's avatar
      X wcfs: Teach start to start successfully even after unclean wcfs shutdown · 6b22f8c4
      Kirill Smelkov authored
      This patch does to start what b0ca031f did for join/serve.
      
      Stop duplicating the code and factor logic to "attach to wcsrv, or
      prepare to start" into _try_attach_wcsrv function.
      6b22f8c4
  2. 17 Jun, 2021 2 commits
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
      X wcfs: Run fusermount and friends with /bin:/usr/bin always on path · 5fcec261
      Kirill Smelkov authored
      For example runUnitTest in ERP5 runs test program with $PATH set without those
      directores. This leads to failures like
      
            File "/srv/slapgrid/slappart16/srv/testnode/djk/soft/950bed0366ff018b829e9bbb7dc3326f/parts/wendelin.core/wcfs/__init__.py", line 533, in _is_mountpoint
              mounted = (0 == subprocess.call(["mountpoint", "-q", path]))
            File "/srv/slapgrid/slappart16/srv/testnode/djk/shared/python2.7/a8bdae732e550b385eb04a06ed09d383/lib/python2.7/subprocess.py", line 172, in call
              return Popen(*popenargs, **kwargs).wait()
            File "/srv/slapgrid/slappart16/srv/testnode/djk/shared/python2.7/a8bdae732e550b385eb04a06ed09d383/lib/python2.7/subprocess.py", line 394, in __init__
              errread, errwrite)
            File "/srv/slapgrid/slappart16/srv/testnode/djk/shared/python2.7/a8bdae732e550b385eb04a06ed09d383/lib/python2.7/subprocess.py", line 1047, in _execute_child
              raise child_exception
          OSError: [Errno 2] No such file or directory
      
      where those system programs are not found.
      5fcec261
  3. 16 Jun, 2021 1 commit
  4. 14 Jun, 2021 1 commit
  5. 11 Jun, 2021 11 commits
  6. 10 Jun, 2021 1 commit
  7. 07 Jun, 2021 17 commits
    • Kirill Smelkov's avatar
      X wcfs: Goodby holeIdx · 423f77be
      Kirill Smelkov authored
      As a consequence rework treediff to work on key ranges instead of
      individial keys.
      
      Adjacency coverage gets wider, but that is ok, because in practice if
      keys are added to a BTree - that means that something is added to
      ZBigFile, and it is very likely that just added data will be accessed
      again. In other words it means that wcfs will very likely track those
      data on nearby future read access, so we don't loose anything if we
      start to track those keys right at the beginning.
      
      * t2: (74 commits)
        .
        .
        .
        .
        .
        .
        X kadj must be taken into account as kadj^δZ
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        ...
      423f77be
    • Kirill Smelkov's avatar
      . · 006f001a
      Kirill Smelkov authored
      006f001a
    • Kirill Smelkov's avatar
      . · b6e8c029
      Kirill Smelkov authored
      b6e8c029
    • Kirill Smelkov's avatar
      . · 8867a78d
      Kirill Smelkov authored
      8867a78d
    • Kirill Smelkov's avatar
      . · 2310a8d0
      Kirill Smelkov authored
      2310a8d0
    • Kirill Smelkov's avatar
      . · e3d1d90e
      Kirill Smelkov authored
      e3d1d90e
    • Kirill Smelkov's avatar
      . · 27e68642
      Kirill Smelkov authored
      27e68642
    • Kirill Smelkov's avatar
      0fa06cbd
    • Kirill Smelkov's avatar
      . · 8a14e5c7
      Kirill Smelkov authored
      8a14e5c7
    • Kirill Smelkov's avatar
      . · 4c6bd292
      Kirill Smelkov authored
      4c6bd292
    • Kirill Smelkov's avatar
      . · c7b5c7e0
      Kirill Smelkov authored
      c7b5c7e0
    • Kirill Smelkov's avatar
      . · 6210eec5
      Kirill Smelkov authored
      6210eec5
    • Kirill Smelkov's avatar
      . · fdcd011f
      Kirill Smelkov authored
      fdcd011f
    • Kirill Smelkov's avatar
      . · b0e11199
      Kirill Smelkov authored
      b0e11199
    • Kirill Smelkov's avatar
      . · b1fffe1f
      Kirill Smelkov authored
      b1fffe1f
    • Kirill Smelkov's avatar
      . · a510307c
      Kirill Smelkov authored
      a510307c
    • Kirill Smelkov's avatar
      . · a9a56005
      Kirill Smelkov authored
      a9a56005