- gen-δtail is not used - we are building ΔFtail from scratch.
- gogcccrash did its work and the problem is now understood.

* t:

Now both normal and random tests for all Update and rebuild could be
exercised as part of regular test runs.

* t2:
  X rebuild: tests: Random testing
  X rebuild: tests: Don't exercise keys from keys2 that already became tracked after Track(keys1) + Update
  X rebuild: tests: Inline _assertTrack
  X rebuild: tests: Don't compute keyCover in trackSet
  X rebuild: tests: Don't recompute trackSet(keys1R2) several times
  X rebuild: tests: Move ΔBtail.Clone test out of hot inner loop into separate test
  X tests: Factor-out tree-test-env into tTreeEnv
  X xbtree: Less copy/garbage in RangedKeySet ops
  X rebuild: tests: Precompute kadj10·kadj21
  X rebuild: tests: Don't access ZODB in xtrackKeys
  X rebuild: tests: Don't access ZODB in XGetδKV
  X rebuild: tests: Don't reflect.DeepEqual in inner loop
  .
  .
  .
  .
  .
  .

Rework py allstructs to emit just set of random trees for one kv.
Rework TestΔBTailAllStructs to iterate through such obtained sets and
generate all pairs and triplets. Previously it was only pairs which is
not enough to verify rebuild properly.

X rebuild: tests: Don't exercise keys from keys2 that already became tracked after Track(keys1) + Update

58s -> 14s  (in -short mode)

t.Helper() is relatively expensive

61s -> 58s

63s -> 61s

68s -> 63s

73s -> 68s

We will need this not only in test update/rebuild.

rebuild tests ~75 -> ~73s.

82s -> 75s.

Use preloaded data, similarly to previous commit.

95s -> 82s.

Preload data when tTreeCommit is created and use that data in
the inner loops instead of accessing ZODB from those inner loops.

130s -> 95s

154s -> 130s

* t2: (50 commits)
  .
  X rebuild: Serial tests now started to PASS draftly
  .
  X rebuild: Don't return nil for empty ΔPPTreeSubSet - that leads to SIGSEGV
  X treediff: Fix BUG while computing AB coverage
  X ΔBtail.Clone had bug that it was aliasing klon and orig data
  X Fix rebuild with ø @at2
  X Fix bug in PPTreeSubSet.Difference  - it was always leaving root node alive
  .
  .
  X found why TestΔBTailAllStructs was not effective to find δtkeycov bugs
  .
  .
  X wcfs: assert that keycov only grow
  .
  X another bug in δtkeyconv computation
  .
  .
  .
  .
  ...

    $ go test -v -failfast -run 'TestΔBTail$/rebuild'
    ...
    PASS
    ok      lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree        176.661s

--- FAIL: TestΔBTail (0.74s)
    --- FAIL: TestΔBTail/rebuild/T/T1/T-T2/B0:e-B1:f-B2:g→T1/T-T/B0:g-T2/B1:h-B2:g (0.00s)
        --- FAIL: TestΔBTail/rebuild/T/T1/T-T2/B0:e-B1:f-B2:g→T1/T-T/B0:g-T2/B1:h-B2:g/_T{};R (0.00s)
            --- FAIL: TestΔBTail/rebuild/T/T1/T-T2/B0:e-B1:f-B2:g→T1/T-T/B0:g-T2/B1:h-B2:g/_T{};R/_→T1/T-T/B0:g-T/B1:e (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x632cc1]

goroutine 238 [running]:
testing.tRunner.func1.2(0x696aa0, 0x893f70)
        /home/kirr/src/tools/go/go/src/testing/testing.go:1143 +0x332
testing.tRunner.func1(0xc000001b00)
        /home/kirr/src/tools/go/go/src/testing/testing.go:1146 +0x4b6
panic(0x696aa0, 0x893f70)
        /home/kirr/src/tools/go/go/src/runtime/panic.go:965 +0x1b9
lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.PPTreeSubSet.ApplyΔ(0xc00040f9b0, 0x0)
        /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/pptreesubset.go:515 +0x41
lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.(*ΔTtail).rebuild(0xc00040f830, 0xc, 0xc00040f500, 0xc000142b40, 0x0, 0x0, 0x0)
        /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/δbtail.go:371 +0x145
lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.(*ΔBtail).rebuildAll(0xc00015dc00, 0x0, 0x0)
        /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/δbtail.go:319 +0x18c
lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.xverifyΔBTail_rebuild_TR(0xc000001b00, 0xc000142b40, 0xc00015dc00, 0xc00006a960, 0xc, 0xc0004863c0, 0xc00040f3b0, 0xc00040f5c0, 0xc00040f650, 0xc00040f740, ...)
        /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/δbtail_test.go:1127 +0x1b6
lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.xverifyΔBTail_rebuild.func1.1.1(0xc000001b00)
        /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/δbtail_test.go:1074 +0xafa
testing.tRunner(0xc000001b00, 0xc000555740)
        /home/kirr/src/tools/go/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
        /home/kirr/src/tools/go/go/src/testing/testing.go:1238 +0x2b3
exit status 2

Access to node entries is valid only after the node was activated.
And it fails otherwise.

--- FAIL: TestΔBTail (0.63s)
    --- FAIL: TestΔBTail/rebuild/T/T1/B0:a-B1:b→T/T1/T-T/B0:c-B1:d (0.00s)
        --- FAIL: TestΔBTail/rebuild/T/T1/B0:a-B1:b→T/T1/T-T/B0:c-B1:d/_T{};R (0.00s)
            --- FAIL: TestΔBTail/rebuild/T/T1/B0:a-B1:b→T/T1/T-T/B0:c-B1:d/_T{};R/_→T/T/B0:a (0.00s)
panic: BUG: T000000000000003a points to T000000000000000c as parent in trackSet, but not found in T000000000000000c children [recovered]
        panic: BUG: T000000000000003a points to T000000000000000c as parent in trackSet, but not found in T000000000000000c children

Rebuild tests were "surprisingly" failing when run as part of whole
testsuite, but succeeding when run separately.

So that even A \ A was not ø.

It was due to thinko in iterating through keys set - we were using key
ordinals instead of keys themselvs. Now it find such bugs:

=== RUN   TestΔBTailAllStructs/Update/T1,3/B0:d-B1:a-B3:d,4:g→T1,4/B0:e-B1:a-B4:c
    δbtail_test.go:839: 2: trackSet:
                have: map[000000000000000c:{pffffffffffffffff c2} 00000000000005c2:{p000000000000000c c0} 0000000000000618:{p000000000000000c c0}]
                want: map[000000000000000c:{pffffffffffffffff c3} 00000000000005c2:{p000000000000000c c0} 0000000000000618:{p000000000000000c c0} 0000000000000620:{p000000000000000c c0}]
    δbtail_test.go:708: T1,3/B0:d-B1:a-B3:d,4:g -> T1,4/B0:e-B1:a-B4:c  ; tracked={0 1} :

        d12:  map[0:{d e} 3:{d ø} 4:{g c}]
        δTok: map[0:{d e} 3:{d ø} 4:{g c}]
        δT:   map[0:{d e}]

        δZ:               {000000000000000c 0000000000000618 0000000000000620}
        Tracked^δZ:       {0 1}
        kadj[Tracked^δZ]: {0 1 3 4 9223372036854775807}
        kadj: map[0:{0} 1:{1 3 4 9223372036854775807} 3:{1 3 4 9223372036854775807} 4:{1 3 4 9223372036854775807} 9223372036854775807:{1 3 4 9223372036854775807}]

        δtkeycov wrong:
        have: {}
        want: {[3,∞)}
        δT    ∌  δTok[3]
        δT    ∌  δTok[4]

Currently fails with e.g.:

(neo) (z-dev) (g.env) kirr@deca:~/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree$ go test -short -v -failfast -run 'TestΔBTail$/Update/T3/B1:a-B8:c→T7/B1:a-B8:c'
=== RUN   TestΔBTail
2021/06/23 14:08:45 zodb: FIXME: open /tmp/δBTail926507346/1.fs: raw cache is not ready for invalidations -> NoCache forced
=== RUN   TestΔBTail/Update/T3/B1:a-B8:c→T7/B1:a-B8:c
    δbtail_test.go:812: BUG: tkeyCov1 ⊄ tkeyCov2:
                tkeyCov1: {[3,∞)}
                tkeyCov2: {[7,∞)}
                tkeyCov1 \ tkeyCov2: {[3,7)}
    δbtail_test.go:708: T3/B1:a-B8:c -> T7/B1:a-B8:c  ; tracked={8} :

        d12:  map[]
        δTok: map[]
        δT:   map[]

        δZ:               {000000000000000c}
        Tracked^δZ:       {}
        kadj[Tracked^δZ]: {}
        kadj: map[1:{1 8 9223372036854775807} 8:{1 8 9223372036854775807} 9223372036854775807:{1 8 9223372036854775807}]

--- FAIL: TestΔBTail (0.84s)
    --- FAIL: TestΔBTail/Update/T3/B1:a-B8:c→T7/B1:a-B8:c (0.00s)