1. 19 Nov, 2021 1 commit
    • Kirill Smelkov's avatar
      Merge branch 't' into t2 · 95fbc76f
      Kirill Smelkov authored
      * t: (129 commits)
        X go.mod: v↑ *
        go/zodb/btree: Change V<op> family to also provide visited node key coverage on visit callback
        go/zodb/btree: Add KeyRange type
        go/zodb/btree: Introduce constants for min/max key value
        go/zodb/btree: tests: Don't forget to close storage
        go/zodb/btree: Cosmetics
        .
        .
        .
        .
        .
        .
        .
        X Start reworking BTree to provide keycov on visit callback
        X go/neo: Fix credentials parsing with go1.17
        fixup! Y client: Fix URI scheme to move credentials out of query
        go/internal/xtesting: Add missing X
        go/zodb/{fs1,zeo}: ~staticcheck
        go/zodb/btree: Fix missing return on data-consistency error
        go/zodb, go/zodb/btree: Fix go generate after rename on zodbtools side
        ...
      95fbc76f
  2. 12 Nov, 2021 1 commit
    • Kirill Smelkov's avatar
      X go.mod: v↑ * · ea538368
      Kirill Smelkov authored
      go get: upgraded github.com/fsnotify/fsnotify v1.4.10-0.20200417215612-7f4cf4dd2b52 => v1.5.1
      go get: upgraded github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b => v1.0.0
      go get: upgraded github.com/gwenn/gosqlite v0.0.0-20201008200117-82e079acf5b6 => v0.0.0-20211101095637-b18efb2e44c8
      go get: upgraded github.com/gwenn/yacr v0.0.0-20200112083327-bbe82c1f4d60 => v0.0.0-20211101095056-492fb0c571bc
      go get: upgraded github.com/kisielk/og-rek v1.1.0 => v1.2.0
      go get: upgraded github.com/soheilhy/cmux v0.1.5-0.20210205191134-5ec6847320e5 => v0.1.5
      go get: upgraded github.com/tinylib/msgp v1.1.5 => v1.1.6
      go get: upgraded golang.org/x/net v0.0.0-20210226172049-e18ecbb05110 => v0.0.0-20211111160137-58aab5ef257a
      go get: upgraded golang.org/x/sys v0.0.0-20210301091718-77cc2087c03b => v0.0.0-20211111213525-f221eed1c01e
      go get: upgraded golang.org/x/text v0.3.5 => v0.3.7
      go get: upgraded lab.nexedi.com/kirr/go123 v0.0.0-20210128150852-c20e95f0f789 => v0.0.0-20210906140734-c9eb28d9e408
      
      Highlights:
      
      - og-rek brings support for pickle protocol 5
      - go123  brings in support for Go1.17
      ea538368
  3. 04 Oct, 2021 13 commits
  4. 01 Oct, 2021 2 commits
  5. 08 Sep, 2021 8 commits
    • Kirill Smelkov's avatar
      X go/neo: Fix credentials parsing with go1.17 · 87199da2
      Kirill Smelkov authored
      Due to security concerns, go1.17, even though very reluctantly,
      similarly to Python broke backward compatibility and stopped to treat ';'
      as separator when parsing URL queries.
      
      We still want to accept ';' as separator in credential part.
      -> Work it around by replacing ';' -> to '&'.
      
      See https://golang.org/doc/go1.17#semicolons and https://github.com/golang/go/issues/25192 for context.
      
      See also: cf685fb5 (fixup! Y client: Fix URI scheme to move credentials out of query)
      87199da2
    • Kirill Smelkov's avatar
      fixup! Y client: Fix URI scheme to move credentials out of query · cf685fb5
      Kirill Smelkov authored
      parse_qsl no longer treats ';' as valid query separator for security
      reason because most proxies did not do so and it was possible to poison
      proxy cache due to difference in query separator handling (see bugs.python.org/issue42967).
      
      To handle credentials we don't have any proxy here, and it is still
      perfectly valid to use ';' as credentials separator.
      
      -> Fix it with ';' -> '&' replace workaround, before feeding credentials
      string to parse_qsl.
      
      Amends: b9a42957.
      cf685fb5
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · b7a42db0
      Kirill Smelkov authored
      * master:
        go/internal/xtesting: Add missing X
        go/zodb/{fs1,zeo}: ~staticcheck
        go/zodb/btree: Fix missing return on data-consistency error
        go/zodb, go/zodb/btree: Fix go generate after rename on zodbtools side
      b7a42db0
    • Kirill Smelkov's avatar
      go/internal/xtesting: Add missing X · e7bea476
      Kirill Smelkov authored
      staticcheck says:
      
      	xtesting.go:386:2: this value of err is never used (SA4006)
      e7bea476
    • Kirill Smelkov's avatar
      go/zodb/{fs1,zeo}: ~staticcheck · 3ff4eb14
      Kirill Smelkov authored
      fs1/format.go:204:2: only the first constant in this group has an explicit type (SA9004)
      zeo/proto.go:56:2: only the first constant in this group has an explicit type (SA9004)
      3ff4eb14
    • Kirill Smelkov's avatar
      go/zodb/btree: Fix missing return on data-consistency error · ca630163
      Kirill Smelkov authored
      staticcheck reports
      
          ziobtree.go:606:4: Errorf is a pure function but its return value is ignored (SA4017)
          ziobtree.go:626:4: Errorf is a pure function but its return value is ignored (SA4017)
          zlobtree.go:606:4: Errorf is a pure function but its return value is ignored (SA4017)
          zlobtree.go:626:4: Errorf is a pure function but its return value is ignored (SA4017)
      ca630163
    • Kirill Smelkov's avatar
      go/zodb, go/zodb/btree: Fix go generate after rename on zodbtools side · 3d27ed5d
      Kirill Smelkov authored
      run_with_zodb3py2_compat was renamed to run_with_zodb4py2_compat in
      nexedi/zodbtools@c59a54ca .
      
      Without the fix go genrate was failing as
      
          (neo) (z-dev) (g.env) kirr@deca:~/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/btree$ go generate
          Traceback (most recent call last):
            File "./testdata/gen-testdata", line 26, in <module>
              from zodbtools.test.gen_testdata import run_with_zodb3py2_compat
          ImportError: cannot import name run_with_zodb3py2_compat
      
      and
      
          (neo) (z-dev) (g.env) kirr@deca:~/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb$ go generate
          Traceback (most recent call last):
            File "./py/pydata-gen-testdata", line 24, in <module>
              from zodbtools.test.gen_testdata import run_with_zodb3py2_compat
          ImportError: cannot import name run_with_zodb3py2_compat
      
      This amends commit fc69e00d (go/zodb/fs1: Fix Python database generator
      to work with recent zodbtools).
      3d27ed5d
    • Kirill Smelkov's avatar
      ~staticcheck · 34f4f310
      Kirill Smelkov authored
      34f4f310
  6. 20 Jul, 2021 2 commits
  7. 11 Jul, 2021 1 commit
  8. 02 Jul, 2021 1 commit
  9. 24 May, 2021 11 commits
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · d02d6555
      Kirill Smelkov authored
      * master:
        go/zodb: Prevent newly-created not-yet committed objects to go to ghost on deactivate
        go/zodb: Fix PActivate not to panic after an error
        go/zodb/fs1tools: Verify
        go/zodb/fs1tools: Dumper += DumpEndOK
        go/zodb: tests: Add a couple of missed t.Helper()
        go/zodb: tests: Teach tDB.Commit to return committed TID
      d02d6555
    • Kirill Smelkov's avatar
      . · 85475968
      Kirill Smelkov authored
      85475968
    • Kirill Smelkov's avatar
      go/zodb: Prevent newly-created not-yet committed objects to go to ghost on deactivate · bb618ce1
      Kirill Smelkov authored
      When object is just created, it is not yet assigned an OID, but can be
      reachable from other objects. The code that processes transaction can
      reach to that new object and try to PActivate/PDeactivate it. And
      currently PDeactivate will drop the object state completely.
      
      Another example of object without an OID is Bucket embedded into a Tree
      object. There, the code that scans the tree can reach to that bucket and
      try to activate/deactivate it, leading, again, to dropping state of that bucket.
      
      -> Fix it.
      bb618ce1
    • Kirill Smelkov's avatar
      . · 15f12104
      Kirill Smelkov authored
      15f12104
    • Kirill Smelkov's avatar
      . · 9de09953
      Kirill Smelkov authored
      9de09953
    • Kirill Smelkov's avatar
      go/zodb: Fix PActivate not to panic after an error · 67f0d2cd
      Kirill Smelkov authored
      Persistent.PActivate used to panic when called the second time, if the
      first time it hit an error. WCFS hit this in practice via object, that was
      previously accessed and pinned into the cache, but later deleted in the
      storage.
      
      -> Fix PActivate to reset .loading on an error, so that next time PActivate is
      called, it tries to trigger load again instead of panicking. Change doload
      criteria from
      
      	state==GHOST && refcnt==1
      
      to
      
      	state==GHOST && loading==nil
      
      because now, after failed PActivate, refcnt can be != 0, if there are several
      other PActivate calls that were waiting for the failed PActivate but did not
      yet woke up.
      
      Here is how added test fails without the fix:
      
          --- FAIL: TestActivateAfterDelete (1.65s)
          panic: t.zodb.MyObject(0000000000000065): activate: need to load, but .loading != nil [recovered]
                  panic: t.zodb.MyObject(0000000000000065): activate: need to load, but .loading != nil
      
          goroutine 10085 [running]:
          testing.tRunner.func1.2(0x649020, 0xc000520660)
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1143 +0x332
          testing.tRunner.func1(0xc0001cb080)
                  /home/kirr/src/tools/go/go/src/testing/testing.go:1146 +0x4b6
          panic(0x649020, 0xc000520660)
                  /home/kirr/src/tools/go/go/src/runtime/panic.go:965 +0x1b9
          lab.nexedi.com/kirr/neo/go/zodb.(*Persistent).PActivate(0xc0001184d0, 0x6e8360, 0xc00019ac90, 0x0, 0x0)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/persistent.go:191 +0xce5
          lab.nexedi.com/kirr/neo/go/zodb.TestActivateAfterDelete(0xc0001cb080)
                  /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/persistent_test.go:786 +0x72c
      67f0d2cd
    • Kirill Smelkov's avatar
      go/zodb/fs1tools: Verify · 5a26fb31
      Kirill Smelkov authored
      Add utility to verify FileStorage data for consistency.
      To verify we just need to iterate through all records, because
      FileStorage driver performs all consistency checks by itself.
      
      Mimic normal output to be the same as in fstest from ZODB/py.
      Example runs of fstest.py and `fs1 verify` on a broken file:
      
          $ python ~/src/wendelin/z/ZODB/src/ZODB/scripts/fstest.py -v 1.fs
                   4: transaction tid 0x03e044f6448c8022 #0
                 213: transaction tid 0x03e044f646e044bb #1
          1.fs has data records that extend beyond the transaction record; end at 466
      
          $ fs1 verify -v 1.fs
                   4: transaction tid 0x03e044f6448c8022 #0
                 213: transaction tid 0x03e044f646e044bb #1
          2021/05/24 12:43:37 fsverify: 1.fs: 1.fs: transaction record @355: -> (iter data): 1.fs: data record @416: check: data record [..., 466) overlaps txn boundary [..., 458)
      
      As can be seen, in fs1 case, the error contains more details: [start, end) of
      both mismatching transaction and data records.
      
      In addition to fstest-like verbosity, add progress-mode, where % of total
      completion is printed in a style similar to one used by `fs1 verify-index`.
      
      The Go-based implementation is also faster even when data is on HDD. For
      example on a 73GB database provided by @jerome[1] fsrefs.py takes ~15 minutes
      to run and occupy ~70-100% of CPU. On the other hand `fs1 verify` takes ~7
      minutes to run and occupy ~ 20-30% of CPU.
      
      Tests pending.
      
      [1] nexedi/zodbtools!19 (comment 129480)
      5a26fb31
    • Kirill Smelkov's avatar
      go/zodb/fs1tools: Dumper += DumpEndOK · c66dc12d
      Kirill Smelkov authored
      Some dumpers might want to print something at the end of their dump.
      We will need this functionality for Verify (see next patch).
      c66dc12d
    • Kirill Smelkov's avatar
      . · 86db839d
      Kirill Smelkov authored
      86db839d
    • Kirill Smelkov's avatar
      . · bf50c7a7
      Kirill Smelkov authored
      bf50c7a7
    • Kirill Smelkov's avatar
      . · b80071ce
      Kirill Smelkov authored
      b80071ce