1. 05 Nov, 2020 9 commits
  2. 04 Nov, 2020 10 commits
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · 58689cb6
      Kirill Smelkov authored
      * master:
        go/neo/neonet: Increase signal/noise in dump of packets with decode problems
        fixup! go/neo/proto: Switch enum encoding from int32 to int8
        go/zodb/zeo: Load: loadBefore RPC can return None to indicate non-existing object
        go/internal/xtesting: DrvTestLoad: Verify load of non-existing and not-yet-created objects
      58689cb6
    • Kirill Smelkov's avatar
      go/neo/neonet: Increase signal/noise in dump of packets with decode problems · 774f12c3
      Kirill Smelkov authored
      We recently hit "decode: buffer overflow" errors due to mismatch in
      between NEO/go and NEO/py (see previous patch). With dumpio=true that
      problem was showing itself as
      
          127.0.0.1:50624 > 127.0.0.1:41863: .5 GetObject &{0000000000000000 ffffffffffffffff 0285cbac258bf266}
          127.0.0.1:50624 < 127.0.0.1:41863: .5 (proto.Error) decode: buffer overflow; #24 [24]: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 Error ACK ;  [24]tail: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:24078: load 0285cbac258bf265:0000000000000000: 127.0.0.1:50624 - 127.0.0.1:41863 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:24078: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
      
      Here after printing error and dumping all bytes from packet payload, it
      was printing again message type, message value from to-be-decoded place
      (which is zero-initialized) and data bytes again.
      -> Fix it not to print anything after dump of payload data:
      
          127.0.0.1:60518 > 127.0.0.1:46719: .5 GetObject &{0000000000000000 ffffffffffffffff 0285cbac258bf266}
          127.0.0.1:60518 < 127.0.0.1:46719: .5 (Error) decode: buffer overflow; #24 [24]: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:27853: load 0285cbac258bf265:0000000000000000: 127.0.0.1:60518 - 127.0.0.1:46719 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:27853: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
      774f12c3
    • Kirill Smelkov's avatar
      fixup! go/neo/proto: Switch enum encoding from int32 to int8 · 05463172
      Kirill Smelkov authored
      In commit 5f13cc85 I changed enum encodings from int32 to int8, but did
      not noticed that NEO/py commit 52db5607 ("protocol: a single byte is
      more than enough to encode enums") despite specified intent and
      ErrorCodes being marked with @Enum, changed encoding only for fields
      that are marked as PEnum in structures. In NEO/py the Error.code field
      is still marked as PNumber which encodes as 32-bit integer on the wire.
      -> Fix it back.
      
      With recent xtesting.DrvTestLoad update this error was manifesting itself as (on @t branch):
      
      --- FAIL: TestLoad (2.08s)
          --- FAIL: TestLoad/py (2.07s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac258bf265:0000000000000000: 127.0.0.1:42288 - 127.0.0.1:37109 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac3d0369e5:0000000000000001: 127.0.0.1:42288 - 127.0.0.1:37109 .13: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: object was not yet created
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac41b4e832:0000000000000002: 127.0.0.1:42288 - 127.0.0.1:37109 .21: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac41b4e832:0000000000000002: 0000000000000002: object was not yet created
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac4666667f:0000000000000003: 127.0.0.1:42288 - 127.0.0.1:37109 .29: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac4666667f:0000000000000003: 0000000000000003: object was not yet created
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac4fc96318:0000000000000004: 127.0.0.1:42288 - 127.0.0.1:37109 .41: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac4fc96318:0000000000000004: 0000000000000004: object was not yet created
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac547ae165:0000000000000005: 127.0.0.1:42288 - 127.0.0.1:37109 .49: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac547ae165:0000000000000005: 0000000000000005: object was not yet created
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac628f5c4b:0000000000000006: 127.0.0.1:42288 - 127.0.0.1:37109 .65: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: object was not yet created
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbaca444447f:0000000000000007: 127.0.0.1:42288 - 127.0.0.1:37109 .125: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbaca444447f:0000000000000007: 0000000000000007: object was not yet created
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbacbbbbbbff:0000000000000008: 127.0.0.1:42288 - 127.0.0.1:37109 .149: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: object was not yet created
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbad80da7498:0000000000000009: 127.0.0.1:42288 - 127.0.0.1:37109 .269: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbad80da7498:0000000000000009: 0000000000000009: object was not yet created
              xtesting.go:331: load 7fffffffffffffff:000000000000000a: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 7fffffffffffffff:000000000000000a: 127.0.0.1:42288 - 127.0.0.1:37109 .295: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 7fffffffffffffff:000000000000000a: 000000000000000a: no such object
      05463172
    • Kirill Smelkov's avatar
      go/zodb/zeo: Load: loadBefore RPC can return None to indicate non-existing object · f8696942
      Kirill Smelkov authored
      Before this patch and with updated DrvTestLoad (see previous patch) it was failing as:
      
      --- FAIL: TestLoad (0.52s)
          --- FAIL: TestLoad/py/msgpack=false (0.25s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: 0000000000000000: no such object
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: no such object
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: 0000000000000002: no such object
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: 0000000000000003: no such object
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: 0000000000000004: no such object
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: 0000000000000005: no such object
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: no such object
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: 0000000000000007: no such object
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: no such object
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: 0000000000000009: no such object
          --- FAIL: TestLoad/py/msgpack=true (0.26s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: 0000000000000000: no such object
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: no such object
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: 0000000000000002: no such object
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: 0000000000000003: no such object
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: 0000000000000004: no such object
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: 0000000000000005: no such object
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: no such object
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: 0000000000000007: no such object
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: no such object
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: 0000000000000009: no such object
      f8696942
    • Kirill Smelkov's avatar
      go/internal/xtesting: DrvTestLoad: Verify load of non-existing and not-yet-created objects · 4acadb94
      Kirill Smelkov authored
      This currently fails on ZEO and NEO. Those storages will be fixed in the
      following patches (NEO only on @t branch for now).
      4acadb94
    • Kirill Smelkov's avatar
      go/neo/proto: Don't require deletedAt to be present in OID_NOT_FOUND · a2b8516c
      Kirill Smelkov authored
      Providing deletedAt in addition to oid in OID_NOT_FOUND error is NEO/go extension.
      NEO/py sends only OidNotFound(oid):
      
      https://lab.nexedi.com/nexedi/neoppod/blob/v1.12-0-g6332112c/neo/storage/handlers/client.py#L70-72
      
      -> decode deletedAt only if it is present.
      
      Since inspecting NEO/py codebase turned out that message always comes
      with oid, remove the "abusing" words about this approach to convey oid
      via error message for OID_NOT_FOUND and OID_DOES_NOT_EXIST.
      
      Without this patch TestLoad was failing as
      
          === RUN   TestLoad/py
          I: runneo.py: /tmp/neo124851059/1: started master(s): 127.0.0.1:19681
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbac258bf265:0000000000000000: OID_NOT_FOUND: 0000000000000000
                  want: neo://1@127.0.0.1:19681: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbac3d0369e5:0000000000000001: OID_NOT_FOUND: 0000000000000001
                  want: neo://1@127.0.0.1:19681: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: object was not yet created
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbac41b4e832:0000000000000002: OID_NOT_FOUND: 0000000000000002
                  want: neo://1@127.0.0.1:19681: load 0285cbac41b4e832:0000000000000002: 0000000000000002: object was not yet created
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbac4666667f:0000000000000003: OID_NOT_FOUND: 0000000000000003
                  want: neo://1@127.0.0.1:19681: load 0285cbac4666667f:0000000000000003: 0000000000000003: object was not yet created
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbac4fc96318:0000000000000004: OID_NOT_FOUND: 0000000000000004
                  want: neo://1@127.0.0.1:19681: load 0285cbac4fc96318:0000000000000004: 0000000000000004: object was not yet created
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbac547ae165:0000000000000005: OID_NOT_FOUND: 0000000000000005
                  want: neo://1@127.0.0.1:19681: load 0285cbac547ae165:0000000000000005: 0000000000000005: object was not yet created
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbac628f5c4b:0000000000000006: OID_NOT_FOUND: 0000000000000006
                  want: neo://1@127.0.0.1:19681: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: object was not yet created
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbaca444447f:0000000000000007: OID_NOT_FOUND: 0000000000000007
                  want: neo://1@127.0.0.1:19681: load 0285cbaca444447f:0000000000000007: 0000000000000007: object was not yet created
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbacbbbbbbff:0000000000000008: OID_NOT_FOUND: 0000000000000008
                  want: neo://1@127.0.0.1:19681: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: object was not yet created
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: neo://1@127.0.0.1:19681: load 0285cbad80da7498:0000000000000009: OID_NOT_FOUND: 0000000000000009
                  want: neo://1@127.0.0.1:19681: load 0285cbad80da7498:0000000000000009: 0000000000000009: object was not yet created
      a2b8516c
    • Kirill Smelkov's avatar
      go/neo/neonet: Increase signal/noise in dump of packets with decode problems · dacce3d1
      Kirill Smelkov authored
      We recently hit "decode: buffer overflow" errors due to mismatch in
      between NEO/go and NEO/py (see previous patch). With dumpio=true that
      problem was showing itself as
      
          127.0.0.1:50624 > 127.0.0.1:41863: .5 GetObject &{0000000000000000 ffffffffffffffff 0285cbac258bf266}
          127.0.0.1:50624 < 127.0.0.1:41863: .5 (proto.Error) decode: buffer overflow; #24 [24]: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 Error ACK ;  [24]tail: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:24078: load 0285cbac258bf265:0000000000000000: 127.0.0.1:50624 - 127.0.0.1:41863 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:24078: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
      
      Here after printing error and dumping all bytes from packet payload, it
      was printing again message type, message value from to-be-decoded place
      (which is zero-initialized) and data bytes again.
      -> Fix it not to print anything after dump of payload data:
      
          127.0.0.1:60518 > 127.0.0.1:46719: .5 GetObject &{0000000000000000 ffffffffffffffff 0285cbac258bf266}
          127.0.0.1:60518 < 127.0.0.1:46719: .5 (Error) decode: buffer overflow; #24 [24]: 00 00 00 03 00 00 00 10 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:27853: load 0285cbac258bf265:0000000000000000: 127.0.0.1:60518 - 127.0.0.1:46719 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:27853: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
      dacce3d1
    • Kirill Smelkov's avatar
      fixup! go/neo/proto: Switch enum encoding from int32 to int8 · 52cd5b4f
      Kirill Smelkov authored
      In commit 144dafa0 I changed enum encodings from int32 to int8, but did
      not noticed that NEO/py commit 52db5607 ("protocol: a single byte is
      more than enough to encode enums") despite specified intent and
      ErrorCodes being marked with @Enum, changed encoding only for fields
      that are marked as PEnum in structures. In NEO/py the Error.code field
      is still marked as PNumber which encodes as 32-bit integer on the wire.
      -> Fix it back.
      
      With recent xtesting.DrvTestLoad update this error was manifesting itself as
      
      --- FAIL: TestLoad (2.08s)
          --- FAIL: TestLoad/py (2.07s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac258bf265:0000000000000000: 127.0.0.1:42288 - 127.0.0.1:37109 .5: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac258bf265:0000000000000000: 0000000000000000: object was not yet created
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac3d0369e5:0000000000000001: 127.0.0.1:42288 - 127.0.0.1:37109 .13: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: object was not yet created
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac41b4e832:0000000000000002: 127.0.0.1:42288 - 127.0.0.1:37109 .21: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac41b4e832:0000000000000002: 0000000000000002: object was not yet created
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac4666667f:0000000000000003: 127.0.0.1:42288 - 127.0.0.1:37109 .29: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac4666667f:0000000000000003: 0000000000000003: object was not yet created
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac4fc96318:0000000000000004: 127.0.0.1:42288 - 127.0.0.1:37109 .41: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac4fc96318:0000000000000004: 0000000000000004: object was not yet created
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac547ae165:0000000000000005: 127.0.0.1:42288 - 127.0.0.1:37109 .49: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac547ae165:0000000000000005: 0000000000000005: object was not yet created
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbac628f5c4b:0000000000000006: 127.0.0.1:42288 - 127.0.0.1:37109 .65: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: object was not yet created
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbaca444447f:0000000000000007: 127.0.0.1:42288 - 127.0.0.1:37109 .125: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbaca444447f:0000000000000007: 0000000000000007: object was not yet created
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbacbbbbbbff:0000000000000008: 127.0.0.1:42288 - 127.0.0.1:37109 .149: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: object was not yet created
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 0285cbad80da7498:0000000000000009: 127.0.0.1:42288 - 127.0.0.1:37109 .269: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 0285cbad80da7498:0000000000000009: 0000000000000009: object was not yet created
              xtesting.go:331: load 7fffffffffffffff:000000000000000a: returned err unexpected:
                  have: neo://1@127.0.0.1:32731: load 7fffffffffffffff:000000000000000a: 127.0.0.1:42288 - 127.0.0.1:37109 .295: decode: decode: buffer overflow
                  want: neo://1@127.0.0.1:32731: load 7fffffffffffffff:000000000000000a: 000000000000000a: no such object
      52cd5b4f
    • Kirill Smelkov's avatar
      go/zodb/zeo: Load: loadBefore RPC can return None to indicate non-existing object · 080d5450
      Kirill Smelkov authored
      Before this patch and with updated DrvTestLoad (see previous patch) it was failing as:
      
      --- FAIL: TestLoad (0.52s)
          --- FAIL: TestLoad/py/msgpack=false (0.25s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: 0000000000000000: no such object
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: no such object
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: 0000000000000002: no such object
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: 0000000000000003: no such object
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: 0000000000000004: no such object
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: 0000000000000005: no such object
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: no such object
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: 0000000000000007: no such object
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: no such object
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: /tmp/zeo535364855/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: /tmp/zeo535364855/1.fs.zeosock: call loadBefore: unexpected reply: got ogórek.None{}; expect 3-tuple
                  want: /tmp/zeo535364855/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: 0000000000000009: no such object
          --- FAIL: TestLoad/py/msgpack=true (0.26s)
              xtesting.go:306: load 0285cbac258bf265:0000000000000000: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac258bf265:0000000000000000: 0000000000000000: no such object
              xtesting.go:306: load 0285cbac3d0369e5:0000000000000001: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac3d0369e5:0000000000000001: 0000000000000001: no such object
              xtesting.go:306: load 0285cbac41b4e832:0000000000000002: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac41b4e832:0000000000000002: 0000000000000002: no such object
              xtesting.go:306: load 0285cbac4666667f:0000000000000003: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4666667f:0000000000000003: 0000000000000003: no such object
              xtesting.go:306: load 0285cbac4fc96318:0000000000000004: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac4fc96318:0000000000000004: 0000000000000004: no such object
              xtesting.go:306: load 0285cbac547ae165:0000000000000005: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac547ae165:0000000000000005: 0000000000000005: no such object
              xtesting.go:306: load 0285cbac628f5c4b:0000000000000006: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbac628f5c4b:0000000000000006: 0000000000000006: no such object
              xtesting.go:306: load 0285cbaca444447f:0000000000000007: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbaca444447f:0000000000000007: 0000000000000007: no such object
              xtesting.go:306: load 0285cbacbbbbbbff:0000000000000008: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbacbbbbbbff:0000000000000008: 0000000000000008: no such object
              xtesting.go:306: load 0285cbad80da7498:0000000000000009: returned err unexpected:
                  have: /tmp/zeo794664426/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: /tmp/zeo794664426/1.fs.zeosock: call loadBefore: zlink is closed
                  want: /tmp/zeo794664426/1.fs.zeosock: load 0285cbad80da7498:0000000000000009: 0000000000000009: no such object
      080d5450
    • Kirill Smelkov's avatar
      go/internal/xtesting: DrvTestLoad: Verify load of non-existing and not-yet-created objects · 523298bc
      Kirill Smelkov authored
      This currently fails on ZEO and NEO. Those storages will be fixed in the
      following patches.
      523298bc
  3. 03 Nov, 2020 6 commits
  4. 01 Nov, 2020 15 commits
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · a179fa40
      Kirill Smelkov authored
      * master:
      a179fa40
    • Kirill Smelkov's avatar
      Sync with NEO/py v1.12-13-gf2ea4be2 (oldproto branch) · d14040ac
      Kirill Smelkov authored
      * origin/old-proto:
        qa: skip broken ZODB test
        client: fix race with invalidations when starting a new transaction on ZODB 5
        Code clean-up, comment fixes
        master: fix crash in STARTING_BACKUP when connecting to an upstream secondary master
        mysql: workaround for MDEV-20693
        client: inline Application._loadFromCache
        client: replace global load lock by a per-oid one
        client: unindent code
        client: remove load lock in tpc_finish
        qa: check cache in testExternalInvalidation
        qa: comment testExternalInvalidation2
      d14040ac
    • Kirill Smelkov's avatar
      Sync with NEO/py v1.12-13-gf2ea4be2 (oldproto branch) · 3a6a3475
      Kirill Smelkov authored
      * origin/old-proto:
        qa: skip broken ZODB test
        client: fix race with invalidations when starting a new transaction on ZODB 5
        Code clean-up, comment fixes
        master: fix crash in STARTING_BACKUP when connecting to an upstream secondary master
        mysql: workaround for MDEV-20693
        client: inline Application._loadFromCache
        client: replace global load lock by a per-oid one
        client: unindent code
        client: remove load lock in tpc_finish
        qa: check cache in testExternalInvalidation
        qa: comment testExternalInvalidation2
      3a6a3475
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · a4bf053e
      Kirill Smelkov authored
      * master:
        go/neo/proto: Version 5 -> 6
        go/neo/proto: tweak += dry-run
        go/neo/proto: ErrorCode += DENIED
        go/neo/proto: AnswerPartitionList += NumReplicas
        go/neo/proto: Sync with NEO/py on changes to make the number of replicas modifiable at runtime
        go/neo/proto: RequestIdentification += NewNID; Move .DevPath
      a4bf053e
    • Kirill Smelkov's avatar
      Sync with NEO/py v1.12 · 8ea85c71
      Kirill Smelkov authored
      8ea85c71
    • Kirill Smelkov's avatar
      go/neo/proto: Version 5 -> 6 · d4c2253f
      Kirill Smelkov authored
      This protocol version corresponds to protocol version used by NEO/py
      v1.12 and was set in NEO/py commit c6453626 (Bump protocol version).
      
      The protocol definition was updated to match that NEO/py release in
      the previous patches.
      d4c2253f
    • Kirill Smelkov's avatar
      go/neo/proto: tweak += dry-run · 595052c2
      Kirill Smelkov authored
      This corresponds to NEO/py commit 2a27239d (tweak: add option to
      simulate).
      595052c2
    • Kirill Smelkov's avatar
      go/neo/proto: ErrorCode += DENIED · 484c34a5
      Kirill Smelkov authored
      This corresponds to NEO/py commit c2c9e99d (Better error reporting from
      the master to neoctl for denied requests).
      484c34a5
    • Kirill Smelkov's avatar
      go/neo/proto: AnswerPartitionList += NumReplicas · 56835167
      Kirill Smelkov authored
      This corresponds to NEO/py commit 21190ee7 (Make 'neoctl print pt'
      report the number of replicas).
      56835167
    • Kirill Smelkov's avatar
      go/neo/proto: Sync with NEO/py on changes to make the number of replicas modifiable at runtime · 0d75e144
      Kirill Smelkov authored
      This corresponds to NEO/py commit ef5fc508 (Make the number of replicas
      modifiable when the cluster is running).
      
      One important change in the protocol is that Client no longer queries
      Master for partition table - instead M pushed partTab to C right after
      identification (after pushing nodeTab).
      
      See also: https://neo.nexedi.com/P-NEO-Protocol.Specification.2019?portal_skin=CI_slideshow#/9/5
      0d75e144
    • Kirill Smelkov's avatar
      go/neo/proto: RequestIdentification += NewNID; Move .DevPath · 98869a9b
      Kirill Smelkov authored
      This corresponds to NEO/py commit 27e3f620 (New --new-nid storage option
      for fast cloning).
      98869a9b
    • Kirill Smelkov's avatar
      Merge tag v1.12 · c85756a0
      Kirill Smelkov authored
      NEO 1.12
      
      * tag 'v1.12': (28 commits)
        Release version 1.12
        master: reject drop/tweak ctl commands that could lead to unwanted status
        qa: extend test reproducing the migration of a big ZODB to NEO
        neoctl: better display of full partition tables
        Bump protocol version
        tweak: add option to simulate
        tweak: do not crash when trying to remove all nodes
        tweak: do not touch cells of nodes that are intended to be dropped
        Better error reporting from the master to neoctl for denied requests
        Make 'neoctl print pt' report the number of replicas
        Make the number of replicas modifiable when the cluster is running
        New --new-nid storage option for fast cloning
        qa: fix 2 tests with ZODB5
        qa: new tools/stress options to evaluate MySQL engines
        qa: provide a way to let tests start 1 mysqld per storage node
        mysql: make 'user' actually optional in the DB connection string
        mysql: specify column families for RocksDB
        qa: add testIncremental (testImporter) test
        importer: fix hidden "maximum recursion depth exceeded" at startup
        importer: fix closure of ZODB, and also do it when the import is finished
        sqlite: fix resumption of migration to NEO with Importer
        qa: fix a random failure in threaded tests
        importer: speed up startup when the import is already finished
        importer: fix replication (as source) once import is finished
        storage: fix DatabaseManager.getLastTID with max_tid
        qa: remove 2 useless unit tests
        storage: allow the master to change our node id
        Rename --uuid command-line options into --nid
        importer: fix possible data loss on writeback
      c85756a0
    • Kirill Smelkov's avatar
      Sync with NEO/py v1.12 · f4b87d77
      Kirill Smelkov authored
      f4b87d77
    • Kirill Smelkov's avatar
      go/neo/proto: Version 5 -> 6 · 837f49d1
      Kirill Smelkov authored
      This protocol version corresponds to protocol version used by NEO/py
      v1.12 and was set in NEO/py commit c6453626 (Bump protocol version).
      
      The protocol definition was updated to match that NEO/py release in
      the previous patches.
      837f49d1
    • Kirill Smelkov's avatar
      go/neo/proto: tweak += dry-run · 20cacd14
      Kirill Smelkov authored
      This corresponds to NEO/py commit 2a27239d (tweak: add option to
      simulate).
      20cacd14