1. 09 Aug, 2021 3 commits
    • Kirill Smelkov's avatar
      X ΔBtail: Tweak tests to run faster in normal mode · 2f5582e6
      Kirill Smelkov authored
      t(TestΔBTailRandom) ~320s -> ~100s.
      2f5582e6
    • Kirill Smelkov's avatar
      Merge remote-tracking branch 'kirr/t' into t · e5a2ecf3
      Kirill Smelkov authored
      * kirr/t:
        X Fix mlock2 build on Debian 8
      e5a2ecf3
    • Kirill Smelkov's avatar
      X ΔFtail fixes · 124688f9
      Kirill Smelkov authored
      * t2:
        X ΔFtail.SliceByFileRev: Fix untracked entries to be present uniformly in result
        .
        .
        .
        .
        .
        X test that shows problem of SliceByRootRev where untracked blocks are not added uniformly into whole history
        .
        .
        .
        .
        .
        .
        .
        .
        X Size no longer tracks [0,∞) since we start tracking when zfile is non-empty
        X ΔFtail: `go test -failfast -short -v -run Random -randseed=1626793016249041295` discovered problems
      124688f9
  2. 06 Aug, 2021 6 commits
  3. 05 Aug, 2021 4 commits
    • Kirill Smelkov's avatar
      X test that shows problem of SliceByRootRev where untracked blocks are not... · 4f707117
      Kirill Smelkov authored
      X test that shows problem of SliceByRootRev where untracked blocks are not added uniformly into whole history
      
      === RUN   TestΔFtailSliceXXX
      2021/08/05 18:07:35 zodb: FIXME: open /tmp/TestΔFtailSliceXXX2265944622/001/1.fs: raw cache is not ready for invalidations -> NoCache forced
          δftail_test.go:689: slice (@at0,@at2]:
              have: [@at1·{0 1}S @at2·{1}S]
              want: [@at1·{0 1}S @at2·{0 1}S]
      4f707117
    • Kirill Smelkov's avatar
      . · 8b190e24
      Kirill Smelkov authored
      8b190e24
    • Kirill Smelkov's avatar
      . · e7bc6180
      Kirill Smelkov authored
      e7bc6180
    • Kirill Smelkov's avatar
      . · ec505b13
      Kirill Smelkov authored
      ec505b13
  4. 04 Aug, 2021 5 commits
  5. 03 Aug, 2021 1 commit
  6. 30 Jul, 2021 1 commit
    • Kirill Smelkov's avatar
      X Fix mlock2 build on Debian 8 · c2423296
      Kirill Smelkov authored
      @rporchetto reports build failure on Debian 8 / Linux 3.16
      
          [2021-07-30 15:40:35,677] INFO     gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/srv/slapgrid/slappart19/srv/runner/shared/python2.7/3b7a074d1ded44046871b13502341482/include/python2.7 -c wcfs/internal/mm.c -o build/temp.linux-x86_64-2.7/wcfs/internal/mm.o
          [2021-07-30 15:40:35,677] INFO     wcfs/internal/mm.c: In function 'mlock2':
          [2021-07-30 15:40:35,677] INFO     wcfs/internal/mm.c:618:28: error: 'SYS_mlock2' undeclared (first use in this function); did you mean 'SYS_mlock'?
          [2021-07-30 15:40:35,677] INFO              long err = syscall(SYS_mlock2, addr, len, flags);
          [2021-07-30 15:40:35,677] INFO                                 ^~~~~~~~~~
          [2021-07-30 15:40:35,677] INFO                                 SYS_mlock
      
      Fix the build.
      
      NOTE mlock2 was added in Linux 4.3.
      
      Similarly MCL_ONFAULT is not provided on that old glibc 2.19:
      
          [2021-07-30 15:40:35,677] INFO     wcfs/internal/mm.c:986:55: error: 'MCL_ONFAULT' undeclared here (not in a function); did you mean 'MLOCK_ONFAULT'?
          [2021-07-30 15:40:35,677] INFO        __pyx_e_8wendelin_4wcfs_8internal_2mm_MCL_ONFAULT = MCL_ONFAULT,
          [2021-07-30 15:40:35,678] INFO                                                            ^~~~~~~~~~~
          [2021-07-30 15:40:35,678] INFO                                                            MLOCK_ONFAULT
      
      -> Comment MCL_ONFAULT for now since we do not actually use it anywhere yet.
      c2423296
  7. 23 Jul, 2021 1 commit
    • Kirill Smelkov's avatar
      X ΔFtail: `go test -failfast -short -v -run Random... · 46624787
      Kirill Smelkov authored
      X ΔFtail: `go test -failfast -short -v -run Random -randseed=1626793016249041295` discovered problems
      
      === RUN   TestΔFtailRandom
          δftail_test.go:141: # n=10 seed=1626793016249041295
      2021/07/23 12:26:01 zodb: FIXME: open /tmp/TestΔFtailRandom1363232041/001/1.fs: raw cache is not ready for invalidations -> NoCache forced
          δftail_test.go:191: #   @at0 (03e19cb6064c58dd)
          δftail_test.go:203: # → @at1 (03e19cb6064ddd99)  t0:a Da:a,b:b,c:c,d:d,e:e,f:f,g:g,h:h,i:i,j:j      ; not-yet-tracked
          δftail_test.go:375: # → @at2 (03e19cb6064fc922)  δT2:i,3:c,5:d,9:c  δD{a b c d e f g h i}   ; t0:a,2:i,3:c,5:d,9:c Da:a2,b:b2,c:c2,d:d2,e:e2,f:f2,g:g2,h:h2,i:i2,j:j        δ{0 2 3 5 9}
          δftail_test.go:472: δf:
              have: &{03e19cb6064fc922 false {2 3 5 9} true}
              want: &{03e19cb6064fc922 false {0 2 3 5 9} true}
          δftail_test.go:499: .trackSetZBlk:
              ~have: map[c:{3 9} d:{5} i:{2}]
               want: map[a:{0} c:{3 9} d:{5} i:{2}]
      
          ...
      46624787
  8. 20 Jul, 2021 9 commits
    • Kirill Smelkov's avatar
      X ΔFtail + tests · 0853cc9f
      Kirill Smelkov authored
      - Reimplement ΔFtail queries via gluing ΔBtail and ΔZtail data on the fly.
        This helps to avoid implementing complex rebuild logic in ΔFtail.
        The only place that needs to have that complexity is now ΔBtail, and there it
        already works draftly.
      
      - Add ΔFtail tests.
      
      - Add notion of epochs to ΔFtail. Epochs correspond to ZBigFile objects changes
        (creation and deletion). Unfortunately handling ZBigFile object changes
        turned out to be necessary to keep wcfs tests in passing state.
      
      - Move common testing infrastructure - that is used by both ΔBtail and ΔFtail - to xbtreetest package.
      
      - Add tests for ΔBtail.SliceByRootRev aliasing
      
      - Lazy rebuild is now on
      
      - ΔBtail.GetAt reworked
      
      ...
      
      * t2: (112 commits)
        X wcfs: v↑ NEO/go (checkpoint)
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        X ΔFtail: Rebuild vδE after first track
        .
        .
        .
        .
        .
        .
        .
        .
        ...
      0853cc9f
    • Kirill Smelkov's avatar
      X wcfs: v↑ NEO/go (checkpoint) · d13f11ca
      Kirill Smelkov authored
      To pick up neo@bc3b5ec3
      d13f11ca
    • Kirill Smelkov's avatar
      . · d2be40dd
      Kirill Smelkov authored
      d2be40dd
    • Kirill Smelkov's avatar
      X Rework ΔFtail so that BlkRevAt works with ZBigFile checkout from any at ∈ (tail, head] · 22f5f096
      Kirill Smelkov authored
      This is necessarry because previously both ΔFtail.BlkRevAt and
      ΔBtail.GetAt required zfile/root checkouts from @head. However during
      update process in wcfs it is ΔFtail that is updated first, and only
      after some invalidations zconn is resynced to new @head. And if we
      move zconn.resync to happen before ΔFtail.Update, that would break
      the invariant that data in OS cache correspond to zconn.At.
      
      The rework triggered and clarified logic of what ΔBtail.GetAt
      responsibility and functionality should be. Now it computes query result
      only based on ΔBtail data, and returns valueExact=false or
      revExact=false, if those data is insufficient. Previously it was further
      trying to load data from the database which is conflicting with
      
      	// ΔBtail semantically consists of
      	//
      	//	[]δB			; rev ∈ (tail, head]
      
      Now that database access moved to GetAt user - to ΔFtail.BlkRevAt.
      
      WCFS tests - that were broken after ΔFtail rework - now finally pass again.
      
      * t2+qoldhead:
        .
        .
        .
        .
        .
        .
        .
        .
      22f5f096
    • Kirill Smelkov's avatar
      . · 95c1dac0
      Kirill Smelkov authored
      95c1dac0
    • Kirill Smelkov's avatar
      . · 4f04dc6b
      Kirill Smelkov authored
      4f04dc6b
    • Kirill Smelkov's avatar
      . · 5af13240
      Kirill Smelkov authored
      5af13240
    • Kirill Smelkov's avatar
      . · 024dde83
      Kirill Smelkov authored
      024dde83
    • Kirill Smelkov's avatar
      . · 190617f5
      Kirill Smelkov authored
      190617f5
  9. 19 Jul, 2021 6 commits
    • Kirill Smelkov's avatar
      . · 03de8208
      Kirill Smelkov authored
      03de8208
    • Kirill Smelkov's avatar
      . · e072a39f
      Kirill Smelkov authored
      e072a39f
    • Kirill Smelkov's avatar
      . · e2eb170c
      Kirill Smelkov authored
      e2eb170c
    • Kirill Smelkov's avatar
      . · cd97de63
      Kirill Smelkov authored
      cd97de63
    • Kirill Smelkov's avatar
      ΔFtail += .Epoch in δf · 30f5ddc7
      Kirill Smelkov authored
      Unfortunately I could not avoid computing diff for ZBigFile objects
      themselves, because after recent ΔFtail and ΔZtail fixes, wcfs tests
      started to fail becase in the following scenario
      
          δFtail is created                        @at0
          file is created                          @at1
          file starts to be tracked when δFtail is @at2
          δFtail query is made about file block -> boom reports @at0, because the file does not exist there at all
      
      To fix this ΔFtail now detects when ZBigFile objects are changed
      themselves and indicate such a change with specifal δf with
      δf.Epoch=true .
      
      * t2+δfepoch: (38 commits)
        X ΔFtail: Rebuild vδE after first track
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        .
        ...
      30f5ddc7
    • Kirill Smelkov's avatar
      X ΔFtail: Rebuild vδE after first track · bf3ace66
      Kirill Smelkov authored
      This is needed to detect where the file was created and indicate with
      δF.Epoch=true correspondingly.
      bf3ace66
  10. 16 Jul, 2021 4 commits