1. 23 Jun, 2021 10 commits
    • Kirill Smelkov's avatar
      X Fix rebuild with ø @at2 · 450ba707
      Kirill Smelkov authored
      450ba707
    • Kirill Smelkov's avatar
      X Fix bug in PPTreeSubSet.Difference - it was always leaving root node alive · 66bc41ce
      Kirill Smelkov authored
      So that even A \ A was not ø.
      66bc41ce
    • Kirill Smelkov's avatar
      . · 7533257b
      Kirill Smelkov authored
      7533257b
    • Kirill Smelkov's avatar
      . · 6eeaff06
      Kirill Smelkov authored
      6eeaff06
    • Kirill Smelkov's avatar
      X found why TestΔBTailAllStructs was not effective to find δtkeycov bugs · 502e05c2
      Kirill Smelkov authored
      It was due to thinko in iterating through keys set - we were using key
      ordinals instead of keys themselvs. Now it find such bugs:
      
      === RUN   TestΔBTailAllStructs/Update/T1,3/B0:d-B1:a-B3:d,4:g→T1,4/B0:e-B1:a-B4:c
          δbtail_test.go:839: 2: trackSet:
                      have: map[000000000000000c:{pffffffffffffffff c2} 00000000000005c2:{p000000000000000c c0} 0000000000000618:{p000000000000000c c0}]
                      want: map[000000000000000c:{pffffffffffffffff c3} 00000000000005c2:{p000000000000000c c0} 0000000000000618:{p000000000000000c c0} 0000000000000620:{p000000000000000c c0}]
          δbtail_test.go:708: T1,3/B0:d-B1:a-B3:d,4:g -> T1,4/B0:e-B1:a-B4:c  ; tracked={0 1} :
      
              d12:  map[0:{d e} 3:{d ø} 4:{g c}]
              δTok: map[0:{d e} 3:{d ø} 4:{g c}]
              δT:   map[0:{d e}]
      
              δZ:               {000000000000000c 0000000000000618 0000000000000620}
              Tracked^δZ:       {0 1}
              kadj[Tracked^δZ]: {0 1 3 4 9223372036854775807}
              kadj: map[0:{0} 1:{1 3 4 9223372036854775807} 3:{1 3 4 9223372036854775807} 4:{1 3 4 9223372036854775807} 9223372036854775807:{1 3 4 9223372036854775807}]
      
              δtkeycov wrong:
              have: {}
              want: {[3,∞)}
              δT    ∌  δTok[3]
              δT    ∌  δTok[4]
      502e05c2
    • Kirill Smelkov's avatar
      . · 35bbf9b9
      Kirill Smelkov authored
      35bbf9b9
    • Kirill Smelkov's avatar
      . · 7a7c9756
      Kirill Smelkov authored
      7a7c9756
    • Kirill Smelkov's avatar
      X wcfs: assert that keycov only grow · 0917380e
      Kirill Smelkov authored
      Currently fails with e.g.:
      
      (neo) (z-dev) (g.env) kirr@deca:~/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree$ go test -short -v -failfast -run 'TestΔBTail$/Update/T3/B1:a-B8:c→T7/B1:a-B8:c'
      === RUN   TestΔBTail
      2021/06/23 14:08:45 zodb: FIXME: open /tmp/δBTail926507346/1.fs: raw cache is not ready for invalidations -> NoCache forced
      === RUN   TestΔBTail/Update/T3/B1:a-B8:c→T7/B1:a-B8:c
          δbtail_test.go:812: BUG: tkeyCov1 ⊄ tkeyCov2:
                      tkeyCov1: {[3,∞)}
                      tkeyCov2: {[7,∞)}
                      tkeyCov1 \ tkeyCov2: {[3,7)}
          δbtail_test.go:708: T3/B1:a-B8:c -> T7/B1:a-B8:c  ; tracked={8} :
      
              d12:  map[]
              δTok: map[]
              δT:   map[]
      
              δZ:               {000000000000000c}
              Tracked^δZ:       {}
              kadj[Tracked^δZ]: {}
              kadj: map[1:{1 8 9223372036854775807} 8:{1 8 9223372036854775807} 9223372036854775807:{1 8 9223372036854775807}]
      
      --- FAIL: TestΔBTail (0.84s)
          --- FAIL: TestΔBTail/Update/T3/B1:a-B8:c→T7/B1:a-B8:c (0.00s)
      0917380e
    • Kirill Smelkov's avatar
      . · 595afd6a
      Kirill Smelkov authored
      595afd6a
    • Kirill Smelkov's avatar
      X another bug in δtkeyconv computation · f30826a6
      Kirill Smelkov authored
      === RUN   TestΔBTail/Update/T7/B1:a,2:b-B8:c,9:d→T3/T-T/B1:a,2:b-B8:c,9:d
          δbtail_test.go:832: 2: trackSet:
                      have: map[000000000000000c:{pffffffffffffffff c2} 0000000000000041:{p0000000000000043 c0} 0000000000000042:{p0000000000000044 c0} 0000000000000043:{p000000000000000c c1} 0000000000000044:{p000000000000000c c1}]
                      want: map[000000000000000c:{pffffffffffffffff c1} 0000000000000041:{p0000000000000043 c0} 0000000000000043:{p000000000000000c c1}]
          δbtail_test.go:708: T7/B1:a,2:b-B8:c,9:d -> T3/T-T/B1:a,2:b-B8:c,9:d  ; tracked={1} :
      
              d12:  map[]
              δTok: map[]
              δT:   map[]
      
              δZ:               {000000000000000c 0000000000000043 0000000000000044}
              Tracked^δZ:       {}
              kadj[Tracked^δZ]: {}
              kadj: map[1:{1 2 8 9 9223372036854775807} 2:{1 2 8 9 9223372036854775807} 8:{1 2 8 9 9223372036854775807} 9:{1 2 8 9 9223372036854775807} 9223372036854775807:{1 2 8 9 9223372036854775807}]
      
              δtkeycov wrong:
              have: {[7,∞)}
              want: {}
      f30826a6
  2. 22 Jun, 2021 5 commits
  3. 21 Jun, 2021 15 commits
    • Kirill Smelkov's avatar
      . · 7e7f0f94
      Kirill Smelkov authored
      7e7f0f94
    • Kirill Smelkov's avatar
      . · f428df95
      Kirill Smelkov authored
      f428df95
    • Kirill Smelkov's avatar
      . · c13a7bc0
      Kirill Smelkov authored
      c13a7bc0
    • Kirill Smelkov's avatar
      . · d741c8c7
      Kirill Smelkov authored
      d741c8c7
    • Kirill Smelkov's avatar
      . · e09b503e
      Kirill Smelkov authored
      e09b503e
    • Kirill Smelkov's avatar
      . · 2f9a46eb
      Kirill Smelkov authored
      2f9a46eb
    • Kirill Smelkov's avatar
      . · 69449694
      Kirill Smelkov authored
      69449694
    • Kirill Smelkov's avatar
      . · 8eeef71a
      Kirill Smelkov authored
      8eeef71a
    • Kirill Smelkov's avatar
      . · 7d2e4509
      Kirill Smelkov authored
      7d2e4509
    • Kirill Smelkov's avatar
      . · e8d30e1d
      Kirill Smelkov authored
      e8d30e1d
    • Kirill Smelkov's avatar
      . · 6c0c6755
      Kirill Smelkov authored
      6c0c6755
    • Kirill Smelkov's avatar
      . · 364c2727
      Kirill Smelkov authored
      364c2727
    • Kirill Smelkov's avatar
      . · 09489db6
      Kirill Smelkov authored
      09489db6
    • Kirill Smelkov's avatar
      . · cb2fae27
      Kirill Smelkov authored
      cb2fae27
    • Kirill Smelkov's avatar
      Merge branch 't' into t2 · b221b0b6
      Kirill Smelkov authored
      * t:
        .
        X wcfs: Fix ZSync to close wconn on zdb.close, even if zconn stays alive
        X lib/zodb: Connection += onShutdownCallback
        .
        X wcfs: lsof +D misbehaves - don't use it
        X wcfs: _fuse_unmount: Try first `kill -TERM` before `kill -QUIT` wcfs
        X wcfs: Tune _fuse_unmount to include `fusermount -u` error message into raised exception
        X wcfs: Teach start to start successfully even after unclean wcfs shutdown
        fixup! X wcfs: Run fusermount and friends with /bin:/usr/bin always on path
        X wcfs: Run fusermount and friends with /bin:/usr/bin always on path
        X wcfs: Add start to spawn a Server that can be later stopped  (draft)
      b221b0b6
  4. 20 Jun, 2021 1 commit
  5. 18 Jun, 2021 7 commits
    • Kirill Smelkov's avatar
      X wcfs: Fix ZSync to close wconn on zdb.close, even if zconn stays alive · 7203d7ab
      Kirill Smelkov authored
      Even if ZODB DB is closed and releases zconn from its .pool, zconn can
      still stay alive being referenced from any other live python object -
      e.g. from a frame, a traceback etc. This situation in particular happens
      under ERP5's runUnitTest.
      
      As the result, if we don't hook into DB.close and close wconn only on
      zconn GC, files opened on WCFS filesystem stay opened, and WCFS server
      cannot be cleanly unmounted when test run completes.
      
      -> Fix it.
      
      P.S. ZODB@bbd03b3a is good, but does
      not really solve this problem because, once again, zconn stays
      referenced from objects besides DB.
      7203d7ab
    • Kirill Smelkov's avatar
      X lib/zodb: Connection += onShutdownCallback · a26d9659
      Kirill Smelkov authored
      In the next patch we'll need this functionality to subscribe ZSync into
      db.close to know when ZODB Connection is shut down even if it stays
      alive referenced by some other objects.
      a26d9659
    • Kirill Smelkov's avatar
      . · 8dcaa5de
      Kirill Smelkov authored
      8dcaa5de
    • Kirill Smelkov's avatar
      X wcfs: lsof +D misbehaves - don't use it · 3244f3a6
      Kirill Smelkov authored
      For example even with live wcfs lsof +D output differes from regular lsof and is misleading:
      
          LSOF /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2:
      
          COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
          runUnitTe 29137 slapuser34   24r   REG   0,48      111    4 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/.wcfs/zurl
          runUnitTe 29137 slapuser34   25u   REG   0,48        0    7 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/head/watch
          runUnitTe 29137 slapuser34   26r   REG   0,48  2097152    9 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/head/bigfile/00000000000077e2
      
          LSOF +D /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2:
      
          COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
          runUnitTe 29137 slapuser34   24r   REG   0,48      111 4294967295 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/.wcfs/zurl
          runUnitTe 29137 slapuser34   25u   REG   0,48        0 4294967295 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/.wcfs/zurl
          runUnitTe 29137 slapuser34   26r   REG   0,48  2097152 4294967295 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/.wcfs/zurl
      3244f3a6
    • Kirill Smelkov's avatar
      X wcfs: _fuse_unmount: Try first `kill -TERM` before `kill -QUIT` wcfs · 153c002a
      Kirill Smelkov authored
      Just aborting the FUSE connection does not make WCFS to exit. This abort
      is needed to avoid deadlocks, but we should also signal WCFS it should
      stop. Not doing so will lead to timeout in wait after FUSE connection
      abort and `kill -QUIT` triggered, which makes WCFS crash and print lots
      of traceback to stderr before exiting.
      153c002a
    • Kirill Smelkov's avatar
      X wcfs: Tune _fuse_unmount to include `fusermount -u` error message into raised exception · 15389db0
      Kirill Smelkov authored
      Previously that message was printed to stderr and it was very confusing:
      we had something in the exception message, something in the log and
      something in stderr.
      
      Now exception message is self-sufficient and log provides more details.
      15389db0
    • Kirill Smelkov's avatar
      X wcfs: Teach start to start successfully even after unclean wcfs shutdown · 6b22f8c4
      Kirill Smelkov authored
      This patch does to start what b0ca031f did for join/serve.
      
      Stop duplicating the code and factor logic to "attach to wcsrv, or
      prepare to start" into _try_attach_wcsrv function.
      6b22f8c4
  6. 17 Jun, 2021 2 commits
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
      X wcfs: Run fusermount and friends with /bin:/usr/bin always on path · 5fcec261
      Kirill Smelkov authored
      For example runUnitTest in ERP5 runs test program with $PATH set without those
      directores. This leads to failures like
      
            File "/srv/slapgrid/slappart16/srv/testnode/djk/soft/950bed0366ff018b829e9bbb7dc3326f/parts/wendelin.core/wcfs/__init__.py", line 533, in _is_mountpoint
              mounted = (0 == subprocess.call(["mountpoint", "-q", path]))
            File "/srv/slapgrid/slappart16/srv/testnode/djk/shared/python2.7/a8bdae732e550b385eb04a06ed09d383/lib/python2.7/subprocess.py", line 172, in call
              return Popen(*popenargs, **kwargs).wait()
            File "/srv/slapgrid/slappart16/srv/testnode/djk/shared/python2.7/a8bdae732e550b385eb04a06ed09d383/lib/python2.7/subprocess.py", line 394, in __init__
              errread, errwrite)
            File "/srv/slapgrid/slappart16/srv/testnode/djk/shared/python2.7/a8bdae732e550b385eb04a06ed09d383/lib/python2.7/subprocess.py", line 1047, in _execute_child
              raise child_exception
          OSError: [Errno 2] No such file or directory
      
      where those system programs are not found.
      5fcec261