1. 15 Jan, 2021 18 commits
  2. 14 Jan, 2021 4 commits
    • Kirill Smelkov's avatar
      go/neo/neonet: Kill pktBuf.Dump · dfec9278
      Kirill Smelkov authored
      It is not used anywhere and would have to be reworked with upcoming
      introduction of msgpack encoding. -> Simply remove that.
      dfec9278
    • Kirill Smelkov's avatar
      go/neo/neonet: msgPack -> pktEncode · 47e3cdd9
      Kirill Smelkov authored
      Just renaming.
      47e3cdd9
    • Kirill Smelkov's avatar
      go/neo/neonet: tests: Introduce T · 41a120c1
      Kirill Smelkov authored
      This represents a neonet testing environment and in the future will be
      used to run tests under different protocol versions, protocol encodings, etc.
      
      For now it is noop wrapper around testing.T + t.bin that wraps []byte
      and will be amended to return something different depending on t's
      encoding.
      41a120c1
    • Kirill Smelkov's avatar
      go/neo/neonet: Rework handshake to differentiate client and server parts · e407f725
      Kirill Smelkov authored
      Previously we were doing handshake symmetrically: both client and server
      were transmitting hello and receiving peer's hello simultaneously.
      However this does not allow server to adjust its behaviour depending on
      which client (protocol version, protocol encoding, ...) is connecting to it.
      
      -> Rework handshake so that client always sends its hello first, and
      only then the server side replies. This matches actual NEO/py behaviour:
      
      https://lab.nexedi.com/nexedi/neoppod/blob/v1.12-67-g261dd4b4/neo/lib/connector.py#L293-294
      
      even though the "NEO protocol" states that
      
      	Handshake transmissions are not ordered with respect to each other and can go in parallel.
      
      	( https://neo.nexedi.com/P-NEO-Protocol.Specification.2019?portal_skin=CI_slideshow#/9/2 )
      
      If I recall correctly that sentence was authored by me in 2018 based on
      previous understanding of should-be full symmetry in-between client and
      server.
      
      However soon we are going to teach server sides to autodetect client
      encoding and adjust server to talk to client via its preferred way.
      This needs handshake for client and server to be differentiated.
      
      The protocol needs to be adjusted as well. However I'm not sure it is
      going to happen...
      e407f725
  3. 13 Jan, 2021 1 commit
  4. 12 Jan, 2021 1 commit
    • Kirill Smelkov's avatar
      go/neo/neonet: Fix thinko in recvPkt · 4f6adb26
      Kirill Smelkov authored
      We were returning packet with tail read for next packet data.
      Previously decoding was forgiving, but upcoming rework for msgpack
      support will add check to catch packets with overlong payload, e.g.
      
          === RUN   TestEmptyDB/py/!ssl
          I: runneo.py: /tmp/neo214694750/1 !ssl: started master(s): 127.0.0.1:30621
          127.0.0.1:42266 > 127.0.0.1:30621: .1 RequestIdentification &{CLIENT ?(0)0  1 ø [] []}
          127.0.0.1:42266 < 127.0.0.1:30621: (N: decode header: len(payload) != msgLen) 00 00 00 01 80 01 00 00 00 09 00 f0 00 00 01 e0 00 00 01 00 00 00 00 00 06 00 00 00 58 41 d7 ff 69 82 0a 91 25 00 00 00 03 02 00 00 00 00 e0 00 00 01 02 41 d7 ff 69 82 0a 78 ff 01 00 00 00 09 31 32 37 2e 30 2e 30 2e 31 96 f7 00 00 00 01 02 41 d7 ff 69 81 c6 65 f5 00 00 00 00 09 31 32 37 2e 30 2e 30 2e 31 77 9d f0 00 00 01 02 ff ff ff ff ff ff ff ff 00 00 00 02 00 0a 00 00 00 19 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 01 01
      
      -> Fix is: pkt.data should be data[:pktLen], not data[:n] since n is how
      much we have read at all.
      4f6adb26
  5. 29 Dec, 2020 3 commits
  6. 18 Dec, 2020 1 commit
  7. 17 Dec, 2020 1 commit
  8. 30 Nov, 2020 4 commits
    • Kirill Smelkov's avatar
      go/internal/xtesting: DrvTestWatch: Verify that watching works with empty transactions · 463ef9ad
      Kirill Smelkov authored
      It was not the case with ZEO4 server because we did not had
      ZEO@bf80d23d yet.
      463ef9ad
    • Kirill Smelkov's avatar
      go/zeo: tests: Terminate spawned processses with SIGTERM instead of SIGKILL · b516190c
      Kirill Smelkov authored
      For ZEO it is not strictly required, but for upcoming NEO for example
      NEOCluster needs to shutdown gracefully, else there are processes left for e.g.
      storage nodes and they dump somthing as below on the terminal after tests
      completion:
      
          === RUN   TestLoad
          2020/10/21 14:33:00 zodb: FIXME: open ../zodb/storage/fs1/testdata/1.fs: raw cache is not ready for invalidations -> NoCache forced
          === RUN   TestLoad/py
          I: runneo.py: /tmp/neo445013868/1: Started master(s): 127.0.0.1:24661
          WARNING: This is not the recommended way to import data to NEO: you should use the Importer backend instead.
          NEO also does not implement IStorageRestoreable interface, which means that undo information is not preserved when using this tool: conflict resolution could happen when undoing an old transaction.
          Migrating from ../zodb/storage/fs1/testdata/1.fs to 127.0.0.1:24661
          Migration done in 0.19877
          --- PASS: TestLoad (0.75s)
              --- PASS: TestLoad/py (0.74s)
          PASS
          ok      lab.nexedi.com/kirr/neo/go/neo  0.749s
          (neo) (z-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/kirr/neo/go/neo$ Traceback (most recent call last):
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/tests/functional/__init__.py", line 182, in start
              getattr(neo.scripts,  command).main()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/scripts/neostorage.py", line 66, in main
              app.run()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 147, in run
              self._run()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 178, in _run
              self.doOperation()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 266, in doOperation
              poll()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 87, in _poll
              self.em.poll(1)
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 155, in poll
              self._poll(blocking)
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 253, in _poll
              timeout_object.onTimeout()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 259, in onTimeout
              on_timeout()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 207, in _deferredCommit
              self.commit()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 193, in commit
              self._commit()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 90, in _commit
              retry_if_locked(self.conn.commit)
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 45, in retry_if_locked
              return f(*args)
          OperationalError: disk I/O error
      b516190c
    • Kirill Smelkov's avatar
      go/zodb/zeo: Factor-out verifying empty database into xtesting.DrvTestEmptyDB · 8fafde00
      Kirill Smelkov authored
      And use it to verify other storages.
      8fafde00
    • Kirill Smelkov's avatar
  9. 05 Nov, 2020 4 commits
  10. 04 Nov, 2020 3 commits
    • Kirill Smelkov's avatar
      go/neo/neonet: Increase signal/noise in dump of packets with decode problems · 774f12c3
      Kirill Smelkov authored
      We recently hit "decode: buffer overflow" errors due to mismatch in
      between NEO/go and NEO/py (see previous patch). With dumpio=true that
      problem was showing itself as
      
          127.0.0.1:50624 > 127.0.0.1:41863: .5 GetObject &{0000000000000000 ffffffffffffffff 0285cbac258bf266}
          127.0.0.1:50624 < 127.0.0.1:41863: .5 (proto.Error) decode: buffer overflow; #24 [24]: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 Error ACK ;  [24]tail: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:24078: load 0285cbac258bf265:0000000000000000: 127.0.0.1:50624 - 127.0.0.1:41863 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:24078: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
      
      Here after printing error and dumping all bytes from packet payload, it
      was printing again message type, message value from to-be-decoded place
      (which is zero-initialized) and data bytes again.
      -> Fix it not to print anything after dump of payload data:
      
          127.0.0.1:60518 > 127.0.0.1:46719: .5 GetObject &{0000000000000000 ffffffffffffffff 0285cbac258bf266}
          127.0.0.1:60518 < 127.0.0.1:46719: .5 (Error) decode: buffer overflow; #24 [24]: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:27853: load 0285cbac258bf265:0000000000000000: 127.0.0.1:60518 - 127.0.0.1:46719 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:27853: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
      774f12c3
    • Kirill Smelkov's avatar
      fixup! go/neo/proto: Switch enum encoding from int32 to int8 · 05463172
      Kirill Smelkov authored
      In commit 5f13cc85 I changed enum encodings from int32 to int8, but did
      not noticed that NEO/py commit 52db5607 ("protocol: a single byte is
      more than enough to encode enums") despite specified intent and
      ErrorCodes being marked with @Enum, changed encoding only for fields
      that are marked as PEnum in structures. In NEO/py the Error.code field
      is still marked as PNumber which encodes as 32-bit integer on the wire.
      -> Fix it back.
      
      With recent xtesting.DrvTestLoad update this error was manifesting itself as (on @t branch):
      
      --- FAIL: TestLoad (2.08s)
          --- FAIL: TestLoad/py (2.07s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac258bf265:0000000000000000: 127.0.0.1:42288 - 127.0.0.1:37109 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac3d0369e5:0000000000000001: 127.0.0.1:42288 - 127.0.0.1:37109 .13: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: object was not yet created
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac41b4e832:0000000000000002: 127.0.0.1:42288 - 127.0.0.1:37109 .21: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac41b4e832:0000000000000002: 0000000000000002: object was not yet created
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac4666667f:0000000000000003: 127.0.0.1:42288 - 127.0.0.1:37109 .29: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac4666667f:0000000000000003: 0000000000000003: object was not yet created
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac4fc96318:0000000000000004: 127.0.0.1:42288 - 127.0.0.1:37109 .41: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac4fc96318:0000000000000004: 0000000000000004: object was not yet created
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac547ae165:0000000000000005: 127.0.0.1:42288 - 127.0.0.1:37109 .49: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac547ae165:0000000000000005: 0000000000000005: object was not yet created
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac628f5c4b:0000000000000006: 127.0.0.1:42288 - 127.0.0.1:37109 .65: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: object was not yet created
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbaca444447f:0000000000000007: 127.0.0.1:42288 - 127.0.0.1:37109 .125: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbaca444447f:0000000000000007: 0000000000000007: object was not yet created
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbacbbbbbbff:0000000000000008: 127.0.0.1:42288 - 127.0.0.1:37109 .149: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: object was not yet created
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbad80da7498:0000000000000009: 127.0.0.1:42288 - 127.0.0.1:37109 .269: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbad80da7498:0000000000000009: 0000000000000009: object was not yet created
              xtesting.go:331: load 7fffffffffffffff:000000000000000a: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 7fffffffffffffff:000000000000000a: 127.0.0.1:42288 - 127.0.0.1:37109 .295: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 7fffffffffffffff:000000000000000a: 000000000000000a: no such object
      05463172
    • Kirill Smelkov's avatar
      go/zodb/zeo: Load: loadBefore RPC can return None to indicate non-existing object · f8696942
      Kirill Smelkov authored
      Before this patch and with updated DrvTestLoad (see previous patch) it was failing as:
      
      --- FAIL: TestLoad (0.52s)
          --- FAIL: TestLoad/py/msgpack=false (0.25s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: 0000000000000000: no such object
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: no such object
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: 0000000000000002: no such object
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: 0000000000000003: no such object
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: 0000000000000004: no such object
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: 0000000000000005: no such object
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: no such object
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: 0000000000000007: no such object
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: no such object
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: 0000000000000009: no such object
          --- FAIL: TestLoad/py/msgpack=true (0.26s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: 0000000000000000: no such object
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: no such object
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: 0000000000000002: no such object
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: 0000000000000003: no such object
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: 0000000000000004: no such object
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: 0000000000000005: no such object
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: no such object
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: 0000000000000007: no such object
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: no such object
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: 0000000000000009: no such object
      f8696942