1. 01 Jul, 2021 5 commits
    • Kirill Smelkov's avatar
      Merge branch 't2' into t3 · 8b31bce0
      Kirill Smelkov authored
      * t2:
        X No longer rely on ZODB cache invariant for invalidations
        X ΔFtail: Keep reference to ZBigFile via Oid, not via *ZBigFile
      8b31bce0
    • Kirill Smelkov's avatar
      . · 7540653c
      Kirill Smelkov authored
      7540653c
    • Kirill Smelkov's avatar
      X No longer rely on ZODB cache invariant for invalidations · bf9a7405
      Kirill Smelkov authored
      Move all aspects of what is tracked into ΔFtail.
      It is more robust if ΔFtail does not rely on properties of outside
      zconn.Cache in order to work correctly.
      It also potentially allows to move ΔFtail out of zdata package.
      
      And it is even more memeory efficient, as before it was
      
      	Connection.cache[zoid] -> ZBlk(ghost)->inΔFtail
      
      and now it is
      
      	ΔFtail.trackSetZBlk[zoid] -> zblkTrack
      
      becuase Persistent embedded in ZBlk even in ghost state occupies alone
      occupies ~ 16 words.
      
      Now we keep only what is needed to be kept for tracking.
      bf9a7405
    • Kirill Smelkov's avatar
      X ΔFtail: Keep reference to ZBigFile via Oid, not via *ZBigFile · ef74aebc
      Kirill Smelkov authored
      *ZBigFile is live object associated with particular ZODB Connection.
      ef74aebc
    • Kirill Smelkov's avatar
      . · 70a8eb7d
      Kirill Smelkov authored
      70a8eb7d
  2. 30 Jun, 2021 6 commits
  3. 29 Jun, 2021 4 commits
    • Kirill Smelkov's avatar
      X δbtail += ForgetPast · d0fe680a
      Kirill Smelkov authored
      d0fe680a
    • Kirill Smelkov's avatar
      Merge branch 't' into t2 · a8177067
      Kirill Smelkov authored
      * t:
      a8177067
    • Kirill Smelkov's avatar
      X Get rebuild tests to run in a sane time; Add proper random-based testing for rebuild · c9f13fc7
      Kirill Smelkov authored
      Now both normal and random tests for all Update and rebuild could be
      exercised as part of regular test runs.
      
      * t2:
        X rebuild: tests: Random testing
        X rebuild: tests: Don't exercise keys from keys2 that already became tracked after Track(keys1) + Update
        X rebuild: tests: Inline _assertTrack
        X rebuild: tests: Don't compute keyCover in trackSet
        X rebuild: tests: Don't recompute trackSet(keys1R2) several times
        X rebuild: tests: Move ΔBtail.Clone test out of hot inner loop into separate test
        X tests: Factor-out tree-test-env into tTreeEnv
        X xbtree: Less copy/garbage in RangedKeySet ops
        X rebuild: tests: Precompute kadj10·kadj21
        X rebuild: tests: Don't access ZODB in xtrackKeys
        X rebuild: tests: Don't access ZODB in XGetδKV
        X rebuild: tests: Don't reflect.DeepEqual in inner loop
        .
        .
        .
        .
        .
        .
      c9f13fc7
    • Kirill Smelkov's avatar
      X rebuild: tests: Random testing · e9c4b619
      Kirill Smelkov authored
      Rework py allstructs to emit just set of random trees for one kv.
      Rework TestΔBTailAllStructs to iterate through such obtained sets and
      generate all pairs and triplets. Previously it was only pairs which is
      not enough to verify rebuild properly.
      e9c4b619
  4. 28 Jun, 2021 7 commits
  5. 25 Jun, 2021 3 commits
  6. 24 Jun, 2021 13 commits
    • Kirill Smelkov's avatar
      X rebuild: tests: Don't reflect.DeepEqual in inner loop · 324241eb
      Kirill Smelkov authored
      154s -> 130s
      324241eb
    • Kirill Smelkov's avatar
      . · 086ae670
      Kirill Smelkov authored
      086ae670
    • Kirill Smelkov's avatar
      . · e4e35723
      Kirill Smelkov authored
      e4e35723
    • Kirill Smelkov's avatar
      . · 05644ccc
      Kirill Smelkov authored
      05644ccc
    • Kirill Smelkov's avatar
      . · 986e4a38
      Kirill Smelkov authored
      986e4a38
    • Kirill Smelkov's avatar
      . · ee9d003b
      Kirill Smelkov authored
      ee9d003b
    • Kirill Smelkov's avatar
      . · 45ce369a
      Kirill Smelkov authored
      45ce369a
    • Kirill Smelkov's avatar
      X ΔBtail.rebuild started to work draftly · 52c72dbb
      Kirill Smelkov authored
      * t2: (50 commits)
        .
        X rebuild: Serial tests now started to PASS draftly
        .
        X rebuild: Don't return nil for empty ΔPPTreeSubSet - that leads to SIGSEGV
        X treediff: Fix BUG while computing AB coverage
        X ΔBtail.Clone had bug that it was aliasing klon and orig data
        X Fix rebuild with ø @at2
        X Fix bug in PPTreeSubSet.Difference  - it was always leaving root node alive
        .
        .
        X found why TestΔBTailAllStructs was not effective to find δtkeycov bugs
        .
        .
        X wcfs: assert that keycov only grow
        .
        X another bug in δtkeyconv computation
        .
        .
        .
        .
        ...
      52c72dbb
    • Kirill Smelkov's avatar
      . · 8402f4c3
      Kirill Smelkov authored
      8402f4c3
    • Kirill Smelkov's avatar
      X rebuild: Serial tests now started to PASS draftly · c34907cf
      Kirill Smelkov authored
          $ go test -v -failfast -run 'TestΔBTail$/rebuild'
          ...
          PASS
          ok      lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree        176.661s
      c34907cf
    • Kirill Smelkov's avatar
      . · 22d712a5
      Kirill Smelkov authored
      22d712a5
    • Kirill Smelkov's avatar
      X rebuild: Don't return nil for empty ΔPPTreeSubSet - that leads to SIGSEGV · ddb28043
      Kirill Smelkov authored
      --- FAIL: TestΔBTail (0.74s)
          --- FAIL: TestΔBTail/rebuild/T/T1/T-T2/B0:e-B1:f-B2:g→T1/T-T/B0:g-T2/B1:h-B2:g (0.00s)
              --- FAIL: TestΔBTail/rebuild/T/T1/T-T2/B0:e-B1:f-B2:g→T1/T-T/B0:g-T2/B1:h-B2:g/_T{};R (0.00s)
                  --- FAIL: TestΔBTail/rebuild/T/T1/T-T2/B0:e-B1:f-B2:g→T1/T-T/B0:g-T2/B1:h-B2:g/_T{};R/_→T1/T-T/B0:g-T/B1:e (0.00s)
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
              panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x632cc1]
      
      goroutine 238 [running]:
      testing.tRunner.func1.2(0x696aa0, 0x893f70)
              /home/kirr/src/tools/go/go/src/testing/testing.go:1143 +0x332
      testing.tRunner.func1(0xc000001b00)
              /home/kirr/src/tools/go/go/src/testing/testing.go:1146 +0x4b6
      panic(0x696aa0, 0x893f70)
              /home/kirr/src/tools/go/go/src/runtime/panic.go:965 +0x1b9
      lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.PPTreeSubSet.ApplyΔ(0xc00040f9b0, 0x0)
              /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/pptreesubset.go:515 +0x41
      lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.(*ΔTtail).rebuild(0xc00040f830, 0xc, 0xc00040f500, 0xc000142b40, 0x0, 0x0, 0x0)
              /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/δbtail.go:371 +0x145
      lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.(*ΔBtail).rebuildAll(0xc00015dc00, 0x0, 0x0)
              /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/δbtail.go:319 +0x18c
      lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.xverifyΔBTail_rebuild_TR(0xc000001b00, 0xc000142b40, 0xc00015dc00, 0xc00006a960, 0xc, 0xc0004863c0, 0xc00040f3b0, 0xc00040f5c0, 0xc00040f650, 0xc00040f740, ...)
              /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/δbtail_test.go:1127 +0x1b6
      lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree.xverifyΔBTail_rebuild.func1.1.1(0xc000001b00)
              /home/kirr/src/neo/src/lab.nexedi.com/nexedi/wendelin.core/wcfs/internal/xbtree/δbtail_test.go:1074 +0xafa
      testing.tRunner(0xc000001b00, 0xc000555740)
              /home/kirr/src/tools/go/go/src/testing/testing.go:1193 +0xef
      created by testing.(*T).Run
              /home/kirr/src/tools/go/go/src/testing/testing.go:1238 +0x2b3
      exit status 2
      ddb28043
    • Kirill Smelkov's avatar
      X treediff: Fix BUG while computing AB coverage · 9d20f8e8
      Kirill Smelkov authored
      Access to node entries is valid only after the node was activated.
      And it fails otherwise.
      
      --- FAIL: TestΔBTail (0.63s)
          --- FAIL: TestΔBTail/rebuild/T/T1/B0:a-B1:b→T/T1/T-T/B0:c-B1:d (0.00s)
              --- FAIL: TestΔBTail/rebuild/T/T1/B0:a-B1:b→T/T1/T-T/B0:c-B1:d/_T{};R (0.00s)
                  --- FAIL: TestΔBTail/rebuild/T/T1/B0:a-B1:b→T/T1/T-T/B0:c-B1:d/_T{};R/_→T/T/B0:a (0.00s)
      panic: BUG: T000000000000003a points to T000000000000000c as parent in trackSet, but not found in T000000000000000c children [recovered]
              panic: BUG: T000000000000003a points to T000000000000000c as parent in trackSet, but not found in T000000000000000c children
      9d20f8e8
  7. 23 Jun, 2021 2 commits