- 19 Nov, 2021 1 commit
-
-
Kirill Smelkov authored
* t: (129 commits) X go.mod: v↑ * go/zodb/btree: Change V<op> family to also provide visited node key coverage on visit callback go/zodb/btree: Add KeyRange type go/zodb/btree: Introduce constants for min/max key value go/zodb/btree: tests: Don't forget to close storage go/zodb/btree: Cosmetics . . . . . . . X Start reworking BTree to provide keycov on visit callback X go/neo: Fix credentials parsing with go1.17 fixup! Y client: Fix URI scheme to move credentials out of query go/internal/xtesting: Add missing X go/zodb/{fs1,zeo}: ~staticcheck go/zodb/btree: Fix missing return on data-consistency error go/zodb, go/zodb/btree: Fix go generate after rename on zodbtools side ...
-
- 12 Nov, 2021 1 commit
-
-
Kirill Smelkov authored
go get: upgraded github.com/fsnotify/fsnotify v1.4.10-0.20200417215612-7f4cf4dd2b52 => v1.5.1 go get: upgraded github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b => v1.0.0 go get: upgraded github.com/gwenn/gosqlite v0.0.0-20201008200117-82e079acf5b6 => v0.0.0-20211101095637-b18efb2e44c8 go get: upgraded github.com/gwenn/yacr v0.0.0-20200112083327-bbe82c1f4d60 => v0.0.0-20211101095056-492fb0c571bc go get: upgraded github.com/kisielk/og-rek v1.1.0 => v1.2.0 go get: upgraded github.com/soheilhy/cmux v0.1.5-0.20210205191134-5ec6847320e5 => v0.1.5 go get: upgraded github.com/tinylib/msgp v1.1.5 => v1.1.6 go get: upgraded golang.org/x/net v0.0.0-20210226172049-e18ecbb05110 => v0.0.0-20211111160137-58aab5ef257a go get: upgraded golang.org/x/sys v0.0.0-20210301091718-77cc2087c03b => v0.0.0-20211111213525-f221eed1c01e go get: upgraded golang.org/x/text v0.3.5 => v0.3.7 go get: upgraded lab.nexedi.com/kirr/go123 v0.0.0-20210128150852-c20e95f0f789 => v0.0.0-20210906140734-c9eb28d9e408 Highlights: - og-rek brings support for pickle protocol 5 - go123 brings in support for Go1.17
-
- 04 Oct, 2021 13 commits
-
-
Kirill Smelkov authored
* master: go/zodb/btree: Change V<op> family to also provide visited node key coverage on visit callback go/zodb/btree: Add KeyRange type go/zodb/btree: Introduce constants for min/max key value go/zodb/btree: tests: Don't forget to close storage go/zodb/btree: Cosmetics
-
Kirill Smelkov authored
WCFS needs to know key coverage for every visited node. WARNING: this is API change.
-
Kirill Smelkov authored
KeyRange represents [lo,hi) key range. It simplifies working with ranges of keys. We will use it in the next commit. KeyRange originated in WCFS and was copied from there: https://lab.nexedi.com/kirr/wendelin.core/blob/57be0126/wcfs/internal/xbtree/blib/keyrange.go
-
Kirill Smelkov authored
We were already using math.Min<Key> in one place, but the number of such places is going to increase. -> Keep min/max definition in only one place.
-
Kirill Smelkov authored
NOTE: db was already being closed in the test's code.
-
Kirill Smelkov authored
-
Kirill Smelkov authored
* t+btree: . . . . . . . X Start reworking BTree to provide keycov on visit callback
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 01 Oct, 2021 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
- 08 Sep, 2021 8 commits
-
-
Kirill Smelkov authored
Due to security concerns, go1.17, even though very reluctantly, similarly to Python broke backward compatibility and stopped to treat ';' as separator when parsing URL queries. We still want to accept ';' as separator in credential part. -> Work it around by replacing ';' -> to '&'. See https://golang.org/doc/go1.17#semicolons and https://github.com/golang/go/issues/25192 for context. See also: cf685fb5 (fixup! Y client: Fix URI scheme to move credentials out of query)
-
Kirill Smelkov authored
parse_qsl no longer treats ';' as valid query separator for security reason because most proxies did not do so and it was possible to poison proxy cache due to difference in query separator handling (see bugs.python.org/issue42967). To handle credentials we don't have any proxy here, and it is still perfectly valid to use ';' as credentials separator. -> Fix it with ';' -> '&' replace workaround, before feeding credentials string to parse_qsl. Amends: b9a42957.
-
Kirill Smelkov authored
* master: go/internal/xtesting: Add missing X go/zodb/{fs1,zeo}: ~staticcheck go/zodb/btree: Fix missing return on data-consistency error go/zodb, go/zodb/btree: Fix go generate after rename on zodbtools side
-
Kirill Smelkov authored
staticcheck says: xtesting.go:386:2: this value of err is never used (SA4006)
-
Kirill Smelkov authored
fs1/format.go:204:2: only the first constant in this group has an explicit type (SA9004) zeo/proto.go:56:2: only the first constant in this group has an explicit type (SA9004)
-
Kirill Smelkov authored
staticcheck reports ziobtree.go:606:4: Errorf is a pure function but its return value is ignored (SA4017) ziobtree.go:626:4: Errorf is a pure function but its return value is ignored (SA4017) zlobtree.go:606:4: Errorf is a pure function but its return value is ignored (SA4017) zlobtree.go:626:4: Errorf is a pure function but its return value is ignored (SA4017)
-
Kirill Smelkov authored
run_with_zodb3py2_compat was renamed to run_with_zodb4py2_compat in nexedi/zodbtools@c59a54ca . Without the fix go genrate was failing as (neo) (z-dev) (g.env) kirr@deca:~/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/btree$ go generate Traceback (most recent call last): File "./testdata/gen-testdata", line 26, in <module> from zodbtools.test.gen_testdata import run_with_zodb3py2_compat ImportError: cannot import name run_with_zodb3py2_compat and (neo) (z-dev) (g.env) kirr@deca:~/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb$ go generate Traceback (most recent call last): File "./py/pydata-gen-testdata", line 24, in <module> from zodbtools.test.gen_testdata import run_with_zodb3py2_compat ImportError: cannot import name run_with_zodb3py2_compat This amends commit fc69e00d (go/zodb/fs1: Fix Python database generator to work with recent zodbtools).
-
Kirill Smelkov authored
-
- 20 Jul, 2021 2 commits
-
-
Kirill Smelkov authored
* master: go/zodb: Fix ΔTail.LastRevOf based on WCFS experience
-
Kirill Smelkov authored
- the only valid range for at is [tail, head]. Don't try to return anything meaningful for queries outside of this range and just panic instead. This is consistent with SliceByRev, which also panics on invalid query, and it is also consistent with semantic model that ΔTail is a vector with data keyed by tid in range (tail, head]: if key is out of vector range, access to the vector should panic, isn't it? - instead of returning revision of minimum entry on exact=n, always return (tail, exact=n) in that case. The change in behaviour is consistent with ΔFtail and ΔBtail from WCFS and is needed for ΔFtail to function correctly: https://lab.nexedi.com/kirr/wendelin.core/blob/22f5f096/wcfs/internal/xbtree/δbtail.go https://lab.nexedi.com/kirr/wendelin.core/blob/22f5f096/wcfs/internal/zdata/δftail.go
-
- 11 Jul, 2021 1 commit
-
-
Kirill Smelkov authored
-
- 02 Jul, 2021 1 commit
-
-
Kirill Smelkov authored
-
- 24 May, 2021 11 commits
-
-
Kirill Smelkov authored
* master: go/zodb: Prevent newly-created not-yet committed objects to go to ghost on deactivate go/zodb: Fix PActivate not to panic after an error go/zodb/fs1tools: Verify go/zodb/fs1tools: Dumper += DumpEndOK go/zodb: tests: Add a couple of missed t.Helper() go/zodb: tests: Teach tDB.Commit to return committed TID
-
Kirill Smelkov authored
-
Kirill Smelkov authored
When object is just created, it is not yet assigned an OID, but can be reachable from other objects. The code that processes transaction can reach to that new object and try to PActivate/PDeactivate it. And currently PDeactivate will drop the object state completely. Another example of object without an OID is Bucket embedded into a Tree object. There, the code that scans the tree can reach to that bucket and try to activate/deactivate it, leading, again, to dropping state of that bucket. -> Fix it.
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Persistent.PActivate used to panic when called the second time, if the first time it hit an error. WCFS hit this in practice via object, that was previously accessed and pinned into the cache, but later deleted in the storage. -> Fix PActivate to reset .loading on an error, so that next time PActivate is called, it tries to trigger load again instead of panicking. Change doload criteria from state==GHOST && refcnt==1 to state==GHOST && loading==nil because now, after failed PActivate, refcnt can be != 0, if there are several other PActivate calls that were waiting for the failed PActivate but did not yet woke up. Here is how added test fails without the fix: --- FAIL: TestActivateAfterDelete (1.65s) panic: t.zodb.MyObject(0000000000000065): activate: need to load, but .loading != nil [recovered] panic: t.zodb.MyObject(0000000000000065): activate: need to load, but .loading != nil goroutine 10085 [running]: testing.tRunner.func1.2(0x649020, 0xc000520660) /home/kirr/src/tools/go/go/src/testing/testing.go:1143 +0x332 testing.tRunner.func1(0xc0001cb080) /home/kirr/src/tools/go/go/src/testing/testing.go:1146 +0x4b6 panic(0x649020, 0xc000520660) /home/kirr/src/tools/go/go/src/runtime/panic.go:965 +0x1b9 lab.nexedi.com/kirr/neo/go/zodb.(*Persistent).PActivate(0xc0001184d0, 0x6e8360, 0xc00019ac90, 0x0, 0x0) /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/persistent.go:191 +0xce5 lab.nexedi.com/kirr/neo/go/zodb.TestActivateAfterDelete(0xc0001cb080) /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/persistent_test.go:786 +0x72c
-
Kirill Smelkov authored
Add utility to verify FileStorage data for consistency. To verify we just need to iterate through all records, because FileStorage driver performs all consistency checks by itself. Mimic normal output to be the same as in fstest from ZODB/py. Example runs of fstest.py and `fs1 verify` on a broken file: $ python ~/src/wendelin/z/ZODB/src/ZODB/scripts/fstest.py -v 1.fs 4: transaction tid 0x03e044f6448c8022 #0 213: transaction tid 0x03e044f646e044bb #1 1.fs has data records that extend beyond the transaction record; end at 466 $ fs1 verify -v 1.fs 4: transaction tid 0x03e044f6448c8022 #0 213: transaction tid 0x03e044f646e044bb #1 2021/05/24 12:43:37 fsverify: 1.fs: 1.fs: transaction record @355: -> (iter data): 1.fs: data record @416: check: data record [..., 466) overlaps txn boundary [..., 458) As can be seen, in fs1 case, the error contains more details: [start, end) of both mismatching transaction and data records. In addition to fstest-like verbosity, add progress-mode, where % of total completion is printed in a style similar to one used by `fs1 verify-index`. The Go-based implementation is also faster even when data is on HDD. For example on a 73GB database provided by @jerome[1] fsrefs.py takes ~15 minutes to run and occupy ~70-100% of CPU. On the other hand `fs1 verify` takes ~7 minutes to run and occupy ~ 20-30% of CPU. Tests pending. [1] nexedi/zodbtools!19 (comment 129480)
-
Kirill Smelkov authored
Some dumpers might want to print something at the end of their dump. We will need this functionality for Verify (see next patch).
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-