An error occurred fetching the project authors.
- 24 Sep, 2024 2 commits
-
-
Kirill Smelkov authored
We were saving index data with bytestrings inside index records. This works for py2 but on py3 it causes failure on unpickling because, by default, *STRING opcodes are unpickled into unicode on py3: === RUN TestIndexSaveToPy/py2_pickle1/py3 UnicodeDecodeError: 'ascii' codec can't decode byte 0x8d in position 25: ordinal not in range(128) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/./py/indexcmp", line 43, in <module> main() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/./py/indexcmp", line 30, in main d1 = fsIndex.load(path1) File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/fsIndex.py", line 138, in load v = unpickler.load() SystemError: <built-in method read of _io.BufferedReader object at 0x7fb631a86670> returned a result with an error set index_test.go:229: zodb/py read/compare index: exit status 1 -> Fix it by explicitly emitting index entries in binary form. To be able to compare "index from py2 -> loaded into go -> saved by go -> compared by py3" implement a bit of backward compatibility for loading py2 index files on py3. Do this compatibility only for known-good py2 files, not index produced by go code which is loaded by py3 without any compatibility shims.
-
Kirill Smelkov authored
Python3 emits index entries using bytes, but index.go was expecting only bytestrings. As the result loading index produced by py3 was failing e.g. as index_test.go:201: index load: testdata/py3_pickle3/1.fs.index: pickle @6: invalid oidPrefix: type ogórek.Bytes -> Fix it by accepting both bytes and bytestrings inside the index.
-
- 20 Aug, 2024 3 commits
-
-
Kirill Smelkov authored
Because without StrictUnicode both bytestrings (str from py2) and unicode (unicode from py2 and str from py3) load into the same Go type string so it becomes impossible to distinguish them from each other. Re-saving data, thus, also generally introduces changes as e.g. string loaded via bytestring will be saved as unicode when pickling with protocol 3. Or a loaded unicode will be saved as bytestring with pickling via protocol=2. -> Switching to StrictUnicode mode solves all those problems. Please see updated documentatin for zodbpickle.go and https://github.com/kisielk/og-rek/commit/b28613c2 for more details about StrictUnicode mode. For ZODB/go this is change in behaviour exposed to outside. However there is currently only one known ZODB/go user - WCFS in Wendelin.core - and that user will be updated correspondingly as well.
-
Kirill Smelkov authored
We already have 3 places where we create picklers and unpicklers: zodb/pydata.go, fs1/index.go and zeo/proto.go and they are already diverging a bit: for example pydata was explicitly specifying protocol for pickle encoder, while other places were using defaults. We will soon want to use StrictUnicode option for all picklers and unpicklers we create, which means it is better to keep this customization only in one place instead of copy-pasting it in between callsites increasing the possibility for divergence and errors. -> Move the code to create pickler and unpickler to internal zodbpickle package as a preparatory step for that. For the reference: this package is merely a wrapper around ogórek, not a fork of any code like zodbpickle/py is.
-
Kirill Smelkov authored
ogórek.AsInt64 was added in https://github.com/kisielk/og-rek/commit/010fbd2e with functionality similar to what we had in pickletools.Xint64 before.
-
- 18 Jan, 2019 1 commit
-
-
Kirill Smelkov authored
Following-up on 4d2c8b1d (go/zodb: Require drivers to provide notifications for database change events) let's teach FileStorage to support watching for database changes: - add watcher who observes changes to data file via fsnotify. - when we see that the file was updated - there is a tricky case to distinguish in-progress writes from just some garbage added at tail. See comments in _watcher for details. - if the watcher sees there is indeed new transactions, the watcher updates index & txnh{Min,Max} and sends corresponding event to watchq. - since index / txnh{Min,Max} can now be changed (by watcher) -> they are now protected by mu. - consequently operations like LastTid, LastOid, Load, ... are all taught to observe index / txnh{Min,Max} state in read-snapshot way. - add down & friends to indicate that the storage is no longer operational, if watcher sees there is a problem with data file. - Open is reworked to start from loading/rebuilding index first, and then tail to watcher to detect whether what's after found topPos is garbage or another in-progress transaction. Consequently it is now possible to correctly open filestorage that is being currently written to and has in-progress transaction at tail. The patch is a bit big, but the changes here are all tightly interrelated.
-
- 04 Dec, 2018 1 commit
-
-
Kirill Smelkov authored
- Put dot after the subject, - Indent lists, - ...
-
- 28 Sep, 2018 1 commit
-
-
Kirill Smelkov authored
As https://github.com/kisielk/og-rek/pull/57 maybe shows []byte was pickling as string only unintentionally and that might change. We are already explicitly checking for string in corresponding index load place: https://lab.nexedi.com/kirr/neo/blob/2dba8607/go/zodb/storage/fs1/index.go#L282 so it is better we also explicitly save the bits as string. If we don't and https://github.com/kisielk/og-rek/pull/57 gets accepted, tests will fail: --- FAIL: TestIndexSaveLoad (0.00s) index_test.go:176: index load: /tmp/t-index893650059/458967662/1.fs.index: pickle @6: invalid oidPrefix: type []uint8 Traceback (most recent call last): File "./py/indexcmp", line 41, in <module> main() File "./py/indexcmp", line 29, in main d2 = fsIndex.load(path2) File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/fsIndex.py", line 138, in load data[ensure_bytes(k)] = fsBucket().fromString(ensure_bytes(v)) File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/fsIndex.py", line 71, in ensure_bytes return s.encode('ascii') if not isinstance(s, bytes) else s AttributeError: 'bytearray' object has no attribute 'encode' --- FAIL: TestIndexSaveToPy (0.04s) index_test.go:218: zodb/py read/compare index: exit status 1
-
- 08 Jul, 2018 1 commit
-
-
Kirill Smelkov authored
As the first step factor-out int64 Xint64 checker from zodb/storagefs1/index.go into there. We'll need the checker in the next patch.
-
- 15 Jan, 2018 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Build index type on top of fsb.Tree introduced in the previous patch and add routines to save and load it to/from disk. We ensure ZODB/py compatibility via generating test FileStorage database + its index and checking we can load index from it and also that if we save an index ZODB/py can load it back. FileStorage index is hard to get bit-to-bit identical since this index uses python pickles which can encode the same objects in several different ways.
-