An error occurred fetching the project authors.
  1. 24 Sep, 2024 2 commits
    • Kirill Smelkov's avatar
      go/zodb/fs1: Fix index save wrt py3 · ef615203
      Kirill Smelkov authored
      We were saving index data with bytestrings inside index records. This
      works for py2 but on py3 it causes failure on unpickling because, by default,
      *STRING opcodes are unpickled into unicode on py3:
      
          === RUN   TestIndexSaveToPy/py2_pickle1/py3
          UnicodeDecodeError: 'ascii' codec can't decode byte 0x8d in position 25: ordinal not in range(128)
      
          The above exception was the direct cause of the following exception:
      
          Traceback (most recent call last):
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/./py/indexcmp", line 43, in <module>
              main()
            File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/./py/indexcmp", line 30, in main
              d1 = fsIndex.load(path1)
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/fsIndex.py", line 138, in load
              v = unpickler.load()
          SystemError: <built-in method read of _io.BufferedReader object at 0x7fb631a86670> returned a result with an error set
              index_test.go:229: zodb/py read/compare index: exit status 1
      
      -> Fix it by explicitly emitting index entries in binary form.
      
      To be able to compare "index from py2 -> loaded into go -> saved by go ->
      compared by py3" implement a bit of backward compatibility for  loading py2
      index files on py3. Do this compatibility only for known-good py2 files, not
      index produced by go code which is loaded by py3 without any compatibility
      shims.
      ef615203
    • Kirill Smelkov's avatar
      go/zodb/fs1: Fix index load from py3 data · 82098c49
      Kirill Smelkov authored
      Python3 emits index entries using bytes, but index.go was expecting only
      bytestrings. As the result loading index produced by py3 was failing
      e.g. as
      
          index_test.go:201: index load: testdata/py3_pickle3/1.fs.index: pickle @6: invalid oidPrefix: type ogórek.Bytes
      
      -> Fix it by accepting both bytes and bytestrings inside the index.
      82098c49
  2. 20 Aug, 2024 3 commits
    • Kirill Smelkov's avatar
      go/*: Switch to StrictUnicode mode for pickling/unpickling · 078b6262
      Kirill Smelkov authored
      Because without StrictUnicode both bytestrings (str from py2) and
      unicode (unicode from py2 and str from py3) load into the same Go type
      string so it becomes impossible to distinguish them from each other.
      Re-saving data, thus, also generally introduces changes as e.g.
      string loaded via bytestring will be saved as unicode when pickling with
      protocol 3. Or a loaded unicode will be saved as bytestring with
      pickling via protocol=2.
      
      -> Switching to StrictUnicode mode solves all those problems.
      
      Please see updated documentatin for zodbpickle.go and
      https://github.com/kisielk/og-rek/commit/b28613c2 for more details about
      StrictUnicode mode.
      
      For ZODB/go this is change in behaviour exposed to outside. However
      there is currently only one known ZODB/go user - WCFS in Wendelin.core -
      and that user will be updated correspondingly as well.
      078b6262
    • Kirill Smelkov's avatar
      go/*: Move creation of a Pickler and Unpickler to central place · 02076be2
      Kirill Smelkov authored
      We already have 3 places where we create picklers and unpicklers:
      zodb/pydata.go, fs1/index.go and zeo/proto.go and they are already
      diverging a bit: for example pydata was explicitly specifying protocol
      for pickle encoder, while other places were using defaults.
      
      We will soon want to use StrictUnicode option for all picklers and
      unpicklers we create, which means it is better to keep this
      customization only in one place instead of copy-pasting it in between
      callsites increasing the possibility for divergence and errors.
      
      -> Move the code to create pickler and unpickler to internal zodbpickle
      package as a preparatory step for that.
      
      For the reference: this package is merely a wrapper around ogórek, not a
      fork of any code like zodbpickle/py is.
      02076be2
    • Kirill Smelkov's avatar
      go/*: Switch pickletools.Xint64 to ogórek.AsInt64 · e0063782
      Kirill Smelkov authored
      ogórek.AsInt64 was added in https://github.com/kisielk/og-rek/commit/010fbd2e
      with functionality similar to what we had in pickletools.Xint64 before.
      e0063782
  3. 18 Jan, 2019 1 commit
    • Kirill Smelkov's avatar
      go/zodb/fs1: Teach it to support notifications on database change · db852511
      Kirill Smelkov authored
      Following-up on 4d2c8b1d (go/zodb: Require drivers to provide
      notifications for database change events) let's teach FileStorage to
      support watching for database changes:
      
      - add watcher who observes changes to data file via fsnotify.
      
      - when we see that the file was updated - there is a tricky case to distinguish
        in-progress writes from just some garbage added at tail. See comments in
        _watcher for details.
      
      - if the watcher sees there is indeed new transactions, the watcher
        updates index & txnh{Min,Max} and sends corresponding event to watchq.
      
      - since index / txnh{Min,Max} can now be changed (by watcher) -> they
        are now protected by mu.
      
      - consequently operations like LastTid, LastOid, Load, ... are all taught to
        observe index / txnh{Min,Max} state in read-snapshot way.
      
      - add down & friends to indicate that the storage is no longer
        operational, if watcher sees there is a problem with data file.
      
      - Open is reworked to start from loading/rebuilding index first, and then tail
        to watcher to detect whether what's after found topPos is garbage or another
        in-progress transaction. Consequently it is now possible to correctly open
        filestorage that is being currently written to and has in-progress transaction
        at tail.
      
      The patch is a bit big, but the changes here are all tightly
      interrelated.
      db852511
  4. 04 Dec, 2018 1 commit
  5. 28 Sep, 2018 1 commit
    • Kirill Smelkov's avatar
      go/zodb/fs1/index: Don't rely on []byte being pickled as string · c72aaa0d
      Kirill Smelkov authored
      As https://github.com/kisielk/og-rek/pull/57 maybe shows []byte was
      pickling as string only unintentionally and that might change.
      
      We are already explicitly checking for string in corresponding index
      load place:
      
      	https://lab.nexedi.com/kirr/neo/blob/2dba8607/go/zodb/storage/fs1/index.go#L282
      
      so it is better we also explicitly save the bits as string.
      
      If we don't and https://github.com/kisielk/og-rek/pull/57 gets accepted,
      tests will fail:
      
      	--- FAIL: TestIndexSaveLoad (0.00s)
      	    index_test.go:176: index load: /tmp/t-index893650059/458967662/1.fs.index: pickle @6: invalid oidPrefix: type []uint8
      	Traceback (most recent call last):
      	  File "./py/indexcmp", line 41, in <module>
      	    main()
      	  File "./py/indexcmp", line 29, in main
      	    d2 = fsIndex.load(path2)
      	  File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/fsIndex.py", line 138, in load
      	    data[ensure_bytes(k)] = fsBucket().fromString(ensure_bytes(v))
      	  File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/fsIndex.py", line 71, in ensure_bytes
      	    return s.encode('ascii') if not isinstance(s, bytes) else s
      	AttributeError: 'bytearray' object has no attribute 'encode'
      	--- FAIL: TestIndexSaveToPy (0.04s)
      	    index_test.go:218: zodb/py read/compare index: exit status 1
      c72aaa0d
  6. 08 Jul, 2018 1 commit
  7. 15 Jan, 2018 2 commits