1. 14 Jul, 2020 11 commits
    • Kirill Smelkov's avatar
      go/zodb/zeo: Introduce notion of encoding · 0696cd64
      Kirill Smelkov authored
      Keep information about which message encoding is used on the wire in
      encoding type. Make pktDecode/pktEncode and data type conversion
      utilities be methods of this type. For now there is only 'pickles'
      encoding, but we'll soon introduce 'msgpack'.
      
      Currently not everything related to pickles is localized in encoding -
      we'll be moving more bits to encoding in the followup patches.
      0696cd64
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
      go/zodb/zeo: Start moving things related to protocol and messages encoding into proto.go · 362e555a
      Kirill Smelkov authored
      First step:
      
      - move msg and msgFlags
      - move pktDecode
      - move functions to pack/unpack tid and oid
      
      more to come.
      362e555a
    • Kirill Smelkov's avatar
      go/zodb/zeo: Shutdown zlink on first error · e7ae218b
      Kirill Smelkov authored
      The same behaviour as ZEO/py does.
      e7ae218b
    • Kirill Smelkov's avatar
      go/zodb/zeo: Use zlink as term for link in between peers · b82154d5
      Kirill Smelkov authored
      Just renaming.
      b82154d5
    • Kirill Smelkov's avatar
      go/zodb/zeo: Don't avoid defer · 5a48e702
      Kirill Smelkov authored
      Starting from Go1.14 defer is no longer slow:
      https://golang.org/doc/go1.14#runtime
      5a48e702
    • Kirill Smelkov's avatar
      8889743e
    • Kirill Smelkov's avatar
      go/zodb/zeo: Initial tests · 589faf07
      Kirill Smelkov authored
      As promised in b65f6d0f (go/zodb: Teach ZODB/go to access ZEO (draft))
      let's start to add tests for ZEO/go client.
      
      The client is tested against ZEO/py server.
      Cover handshake and Load.
      
      Load currently fails beause in ZODB/py POSKeyError does not allow to
      distinguish whether it is no object at all or object exists and its data
      was not found for tid_before.
      
          --- FAIL: TestLoad/py (0.22s)
              zeo_test.go:217: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected:
                  have: /tmp/zeo273044293/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: no such object
                  want: /tmp/zeo273044293/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
              zeo_test.go:217: load 0285cbad858bf2e6:0000000000000006: returned err unexpected:
                  have: /tmp/zeo273044293/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: no such object
                  want: /tmp/zeo273044293/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
              zeo_test.go:217: load 7fffffffffffffff:0000000000000006: returned err unexpected:
                  have: /tmp/zeo273044293/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: no such object
                  want: /tmp/zeo273044293/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
              zeo_test.go:217: load 7fffffffffffffff:0000000000000007: returned err unexpected:
                  have: /tmp/zeo273044293/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: no such object
                  want: /tmp/zeo273044293/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
      
      Problem reported upstream:
      
      -> https://github.com/zopefoundation/ZODB/issues/318
      589faf07
    • Kirill Smelkov's avatar
      go/zodb/zeo: Fix thinko in openByURL error context · 24c9f49f
      Kirill Smelkov authored
      No need to use trailing : suffix as xerr.Contextf automatically adds one.
      24c9f49f
    • Kirill Smelkov's avatar
      go/internal/xtesting: LoadDBHistory · e7a65f87
      Kirill Smelkov authored
      This is utility to load whole content of a ZODB database.
      
      It will be useful to load data to expect from e.g. dataok.fs and feed it
      as txnvOk to DrvTestLoad when verifying another storage driver.
      e7a65f87
    • Kirill Smelkov's avatar
      go/internal/xtesting: FatalIf · 41240d2e
      Kirill Smelkov authored
      This is utility to create X function, similar to exc.Raisif, but that
      would call t.Fatal instead.
      41240d2e
  2. 10 Jul, 2020 3 commits
    • Kirill Smelkov's avatar
      go/zodb/fs1: tests: Factor-out infrastructure to verify Watch into common place · 13c77191
      Kirill Smelkov authored
      We will soon need this infrastructure to test other storage drivers.
      13c77191
    • Kirill Smelkov's avatar
      go/internal/xtesting: DrvTestLoad: More easier to read error messages · 88d0165f
      Kirill Smelkov authored
      E.g. before:
      
          --- FAIL: TestLoad/py (0.22s)
              zeo_test.go:217: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected: /tmp/zeo484021674/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: no such object  ; want: /tmp/zeo484021674/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
              zeo_test.go:217: load 0285cbad858bf2e6:0000000000000006: returned err unexpected: /tmp/zeo484021674/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: no such object  ; want: /tmp/zeo484021674/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
              zeo_test.go:217: load 7fffffffffffffff:0000000000000006: returned err unexpected: /tmp/zeo484021674/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: no such object  ; want: /tmp/zeo484021674/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
              zeo_test.go:217: load 7fffffffffffffff:0000000000000007: returned err unexpected: /tmp/zeo484021674/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: no such object  ; want: /tmp/zeo484021674/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
      
      After:
      
          --- FAIL: TestLoad/py (0.22s)
              zeo_test.go:217: load 0285cbacc06d3a4c:0000000000000007: returned err unexpected:
                  have: /tmp/zeo087978981/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: no such object
                  want: /tmp/zeo087978981/1.fs.zeosock: load 0285cbacc06d3a4c:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
              zeo_test.go:217: load 0285cbad858bf2e6:0000000000000006: returned err unexpected:
                  have: /tmp/zeo087978981/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: no such object
                  want: /tmp/zeo087978981/1.fs.zeosock: load 0285cbad858bf2e6:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
              zeo_test.go:217: load 7fffffffffffffff:0000000000000007: returned err unexpected:
                  have: /tmp/zeo087978981/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: no such object
                  want: /tmp/zeo087978981/1.fs.zeosock: load 7fffffffffffffff:0000000000000007: 0000000000000007: object was deleted @0285cbacc06d3a4c
              zeo_test.go:217: load 7fffffffffffffff:0000000000000006: returned err unexpected:
                  have: /tmp/zeo087978981/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: no such object
                  want: /tmp/zeo087978981/1.fs.zeosock: load 7fffffffffffffff:0000000000000006: 0000000000000006: object was deleted @0285cbad858bf2e6
      88d0165f
    • Kirill Smelkov's avatar
      go/zodb/fs1: tests: Factor-out infrastructure to verify Load into common place · 9b584e08
      Kirill Smelkov authored
      We will soon need this infrastructure to test other storage drivers.
      9b584e08
  3. 07 Jul, 2020 7 commits
  4. 09 Jun, 2020 1 commit
  5. 05 Apr, 2020 2 commits
  6. 14 Feb, 2020 1 commit
  7. 10 Feb, 2020 4 commits
  8. 16 Jan, 2020 1 commit
  9. 12 Apr, 2019 1 commit
  10. 12 Mar, 2019 7 commits
    • Kirill Smelkov's avatar
      go/zodb: Cosmetics · adb2ada3
      Kirill Smelkov authored
      More comments, typos, ...
      adb2ada3
    • Kirill Smelkov's avatar
      go/zodb: LiveCache: Reclassify all objects on cache control change · 1e6aeeee
      Kirill Smelkov authored
      If cache control changes, all objects needs to go through new cache
      control as what was e.g. pinned before could be in other class
      from the point of view of new cache control.
      
      Tests pending...
      1e6aeeee
    • Kirill Smelkov's avatar
      go/zodb: LiveCache: Allow objects to be pinned / omitted from the cache · 4357db3f
      Kirill Smelkov authored
      We already have LiveCacheControl and policy to keep object state in live
      cache. However that state is currently kept only until the object is
      present in the cache, and the object is there currently only until there
      are live pointers to the object from anywhere. If all such pointers go
      away, LiveCache currently discards the object.
      
      Add a way (PCachePinObject) to tell live cache that it must retain an
      object even if there are no other pointers to it. This is needed for
      Wendelin.core which semantically relies on some objects to be present in
      live cache, even if in ghost state, for propagating ZODB invalidations
      into OS pagecache:
      
      https://lab.nexedi.com/kirr/wendelin.core/blob/000bf16359/wcfs/wcfs.go#L245
      
      For symmetry add a way (PCacheDropObject) to tell live cache that it
      must not retain an object. This makes sense for objects that are read in
      large only once to avoid needlessly evicting all other objects while
      making the room for what won't be soon used again.
      
      For completeness add also a way (PCacheDropState) to tell live cache
      that the state of this object should not be cached. This can make sense
      even for objects that are pinned: for example Wendelin.core pins ZBlk
      objects to rely on volatile metadata it attaches to them, but have to
      drop ZBlk state that was loaded from database: that state is propagated
      to be cached in OS pagecache, and keeping it also in ZODB live cache
      would just waste RAM.
      
      Finally add test to cover LiveCache/LiveCacheControl functionality in
      detail.
      4357db3f
    • Kirill Smelkov's avatar
      go/zodb: Prepare to rework/fix LiveCache control · cb6fd5a8
      Kirill Smelkov authored
      We are going to rework LiveCache to support both retaining objects in
      cache even though if all pointers to them went away (currently such
      objects disappear from cache), and to tell the cache that such and such
      object should not be cached at all, e.g. not to evict all other objects
      on one-time large reads.
      
      This will be done by LiveCacheControl giving cache policy classification
      for an object. Prepare for that and change
      
      	WantEvict(obj) -> bool
      
      to
      
      	PCacheClassify(obj) PCachePolicy
      
      which can give more information to LiveCache about how to handle an
      object.
      
      This is only a preparatory patch as there is neither no new
      functionality nor fixes here.
      cb6fd5a8
    • Kirill Smelkov's avatar
      go/zodb: persistent tests: Factor test database setup out of TestPersistentDB · 1a2772e3
      Kirill Smelkov authored
      We already have "testing connection" functionality in common place (was
      named tPersistentDB -> now tConnection). However the functionality to
      "setup and use testing database" was inside TestPersistentDB only.
      
      Since we are going to add more tests, factor that functionality out of
      TestPersistentDB into shared infrastructure that could be reused in
      between tests (-> tDB).
      1a2772e3
    • Kirill Smelkov's avatar
      go/zodb: persistent test: Fix thinko in tPersistentDB.checkObj · e1d57c16
      Kirill Smelkov authored
      It was printing oid instead of obj.
      e1d57c16
    • Kirill Smelkov's avatar
      go/zodb: Don't expose LiveCache internal organization onto its users · 2e8448ad
      Kirill Smelkov authored
      Provide functions to:
      
      - set cache element
      - iterate through all elements
      
      LiveCache organization will be changed in the following patch, and it is
      better to keep the knowledge about its internal structure localized.
      2e8448ad
  11. 10 Mar, 2019 2 commits
    • Kirill Smelkov's avatar
      go/zodb: OpenStorage -> Open · 6badef15
      Kirill Smelkov authored
      For ZODB we always open database at storage level. Thus `zodb.Open(zurl)`
      is relevant. At application level we open database creating DB from
      IStorage and using DB.Open() .
      6badef15
    • Kirill Smelkov's avatar
      go/zodb: LastTid -> Sync + Head · 151d8b79
      Kirill Smelkov authored
      Today LastTid is ambiguous: does it return locally cached last transaction ID,
      or it performs round-trip to server to get more uptodate view of what has been
      last committed? As the result of this ambiguity some drivers (e.g. FileStorage,
      ZEO) return cached view, while other drivers (upcoming NEO, living on t branch)
      was doing the full round-trip.
      
      There are also situations where whether or not LastTid performs syncing to
      server affects semantics: for example if there are two ERP5 nodes and one
      node commits something into ZODB and conveys the fact that something has been
      committed via another channel, the second ERP5 node should have a reliable way
      to observe committed data. But currently there is no such way: on next DB.Open,
      even with at=0 meaning "use latest transaction" and with opt.NoSync=false,
      DB.Open uses storage.LastTid() as a way to sync and oops - for some drivers it
      won't result in real synchronization and thus opened connection could
      potentially have older view with not yet latest data.
      
      To fix this ambiguity require drivers to provide Sync functionality only. That
      should return ID of last transaction committed to storage data as observed from
      some time _afterwards_ Sync call was made. In other words for particular
      client-server case, Sync cannot return cached view of storage and has to
      perform round-trip to the server.
      
      At IStorage level we provide two entry points: Sync, to perform syncing, and
      Head that returns database head (last transaction ID) as viewed by local cache
      without any synchronization.
      
      Please see changes to IStorageDriver and IStorage interfaces for details.
      
      The rest of the changes are:
      
      - teach zodb.storage (the IStorage implementer) to actually implement Sync/Head.
      - adapt zodb.DB to perform Sync for !opt.NoSync case.
      - teach FileStorage to do proper syncing instead of returning local cache.
      - teach ZEO to perform "lastTransaction" RPC call for syncing instead of
        returning local cache.
      - adapt the rest of the tree to interfaces change.
      151d8b79