-
Kirill Smelkov authored
Add utility to verify FileStorage data for consistency. To verify we just need to iterate through all records, because FileStorage driver performs all consistency checks by itself. Mimic normal output to be the same as in fstest from ZODB/py. Example runs of fstest.py and `fs1 verify` on a broken file: $ python ~/src/wendelin/z/ZODB/src/ZODB/scripts/fstest.py -v 1.fs 4: transaction tid 0x03e044f6448c8022 #0 213: transaction tid 0x03e044f646e044bb #1 1.fs has data records that extend beyond the transaction record; end at 466 $ fs1 verify -v 1.fs 4: transaction tid 0x03e044f6448c8022 #0 213: transaction tid 0x03e044f646e044bb #1 2021/05/24 12:43:37 fsverify: 1.fs: 1.fs: transaction record @355: -> (iter data): 1.fs: data record @416: check: data record [..., 466) overlaps txn boundary [..., 458) As can be seen, in fs1 case, the error contains more details: [start, end) of both mismatching transaction and data records. In addition to fstest-like verbosity, add progress-mode, where % of total completion is printed in a style similar to one used by `fs1 verify-index`. The Go-based implementation is also faster even when data is on HDD. For example on a 73GB database provided by @jerome[1] fsrefs.py takes ~15 minutes to run and occupy ~70-100% of CPU. On the other hand `fs1 verify` takes ~7 minutes to run and occupy ~ 20-30% of CPU. Tests pending. [1] nexedi/zodbtools!19 (comment 129480)
5a26fb31