1. 07 Mar, 2019 1 commit
  2. 31 Jan, 2019 6 commits
  3. 30 Jan, 2019 3 commits
  4. 11 Jan, 2019 2 commits
  5. 10 Jan, 2019 4 commits
    • Kirill Smelkov's avatar
      Test coverage for ZODB{3,4,5} · 8ff7020c
      Kirill Smelkov authored
      Use tox to test with all kinds of ZODB.
      With preceding 3 patches tests pass with all versions of upstream ZODB.
      
      TODO: test coverage for both py2 and py3.
      8ff7020c
    • Kirill Smelkov's avatar
      zodbcommit: Fix it for ZODB3 and ZODB4 · 7a94e312
      Kirill Smelkov authored
      maxtid is in ZODB.utils starting only from ZODB5.
      
      ZODB{3,4} want txn._extension, while ZODB5 deprecate it in favour of
      txn.extension.
      7a94e312
    • Kirill Smelkov's avatar
      zodbdump: Fix it for ZODB3 and ZODB4 · 0e5d2f81
      Kirill Smelkov authored
      IStorageTransactionMetaData is ZODB5-only interface. Bug introduced in
      dd959b28 (zodbdump += DumpReader - to read/parse zodbdump stream).
      0e5d2f81
    • Kirill Smelkov's avatar
      dump, commit: Test for both non-empty and empty transaction extensions · 425e6656
      Kirill Smelkov authored
      Currently we exercise zodbdump and zodbcommit+zodbdump with non-empty
      extensions, which works if ZODB is patched for txn.extension_bytes
      support, but fails on pristine ZODB.
      
      Support for txn.extension_bytes cannot get into upstream ZODB for
      more than a year:
      
      https://github.com/zopefoundation/ZODB/pull/183
      https://github.com/zopefoundation/ZODB/pull/207
      
      and  even if it somehow will make it, it will likely be only in ZODB5,
      while we still care to support ZODB4 and ZODB3.
      
      Skipping zodbdump / zodbcommit tests, if a ZODB does not have
      txn.extension_bytes support, would result in significant reduction of
      zodbtools test coverage, because practically that is the current
      situation with all upstream ZODB{3,4,5}. Dropping test coverage for
      non-empty extensions is neither a good option.
      
      For those reason, let's rework the tests and test both zodbdump and
      zodbcommit with two scenarios:
      
      1. on a test database where transactions extensions are always empty.
         This should work on all ZODB irregardless of whether
         txn.extension_bytes patch is there or not.
      
      2. on a test database where transactions extensions are present.
         This should work if ZODB has txn.extension_bytes support, but if not,
         we can mark this case as xfail, since the failure is expected.
      
      This way we make the testsuite pass irregardless of whether
      txn.extension_bytes support is there, and we don't abandon dump/commit
      testing coverage.
      
      /helped-by Jérome Perrin <jerome@nexedi.com>
      425e6656
  6. 09 Jan, 2019 2 commits
    • Jérome Perrin's avatar
      Fix zodb analyze with empty reports · 8c76eae2
      Jérome Perrin authored
      Fix for this kind of errors:
      
      ```
      (env)$ zodb analyze demo.fs ffffffffffffffff..
      # ø
      Processed 0 records in 0 transactions
      Traceback (most recent call last):
        File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/env/bin/zodb", line 11, in <module>
          load_entry_point('zodbtools', 'console_scripts', 'zodb')()
        File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodb.py", line 130, in main
          return command_module.main(argv)
        File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 305, in main
          report(analyze(path, use_dbm, delta_fs, tidmin, tidmax), csv)
        File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 102, in report
          print "Average record size is %7.2f bytes" % (rep.DBYTES * 1.0 / rep.OIDS)
      ZeroDivisionError: float division by zero
      ```
      
      and also small fixes for python3 compatibility
      
      /reviewed-on nexedi/zodbtools!9
      8c76eae2
    • Jérome Perrin's avatar
      analyze: always display processed tid range (even ø) · d37746c6
      Jérome Perrin authored
      To keep a consistent output.
      d37746c6
  7. 08 Jan, 2019 6 commits
  8. 07 Jan, 2019 1 commit
  9. 30 Dec, 2018 4 commits
  10. 17 Dec, 2018 1 commit
    • Kirill Smelkov's avatar
      *: Don't forget to close opened ZODB storage · 9dbe70f3
      Kirill Smelkov authored
      Storages need to be closed to indicate a clean access end. If a storage is
      not closed cleanly it might require to spend time and resources on next
      open. For example FileStorage might need to recompute the index.
      
      /reviewed-on !6
      9dbe70f3
  11. 13 Dec, 2018 5 commits
  12. 12 Dec, 2018 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Always end a transaction with LF · 624aeb09
      Kirill Smelkov authored
      Before now we were emitting extra LF only in between transactions as a
      separator. However the dump format states LF always goes after
      transaction, and there is a reason for it:
      
      	without LF in the end, it becomes ambiguous at EOF - whether it
      	is a proper transaction end, or the transaction was cut.
      
      So avoid the ambiguity by always emitting trailing LF after transaction
      record.
      624aeb09
  13. 02 Jul, 2018 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Switch to using qq from pygolang · b1163449
      Kirill Smelkov authored
      I originally added escapeqq as part of 75c03368 (zodbdump: Start to
      stabilize output format) with the task for this utility to quote string
      into valid "..." string always quoted with ".
      
      This utility was later copied to pygolang:
      
      	kirr/pygolang@afa46cf5
      
      and then further improved there to work under both Python2 and Python3
      and to not escape printable UTF-8 characters:
      
      	kirr/pygolang@02dddb97
      
      So stop the duplication and simply switch to the better version.
      b1163449
  14. 11 May, 2018 3 commits
    • Kirill Smelkov's avatar
      zodbdump: Start to stabilize format + add test · 2801fae9
      Kirill Smelkov authored
      We start to stabilize output format of `zodb dump`. It is actually now robust and the only thing I would contemplate to potentially change is to also cover transaction metadata by hash checksum. So please take a look at updated format (details in patch 1) to provide feedback because it is likely close to what it  will be in its final form.
      
      We also add a program to generate test database which uses various fancy ZODB features and check `zodb dump` output on it to golden one (patch 3).
      
      To be able to dump transaction metadata in raw form ZODB is patched a bit:
      
      https://github.com/zopefoundation/ZODB/pull/183
      
      and we try to detect whether appropriate support is there at runtime and if yes use it to streamline obtaining transaction extension as raw (patch 2).
      Pleae see patch 1 (second half of `zodbdump.py` about what has to be taken on without such support and that it still can't work fully reliably).
      
      /cc @nexedi
      /reviewed-on !3
      2801fae9
    • Kirill Smelkov's avatar
      zodbdump: Add golden test · 7f0bbf7e
      Kirill Smelkov authored
      We add a program to generate a test database with all fancy features and
      then check `zodb dump` output on it to golden on.
      
      The test database itself is commited to git because we want to make sure
      zodbdump works ok for particular exact data and because transaction
      extension dict is potentially saved differently on various runs.
      
      Quoting
      
        https://docs.python.org/2.7/library/stdtypes.html#dict.items    and
        https://docs.python.org/3.7/library/stdtypes.html#dictionary-view-objects
      
      """ CPython implementation detail: Keys and values are listed in an arbitrary
          order which is non-random, varies across Python implementations, and depends
          on the dictionary’s history of insertions and deletions. """
      
      This way on test/gen_testdata.py changes it has to be run manually, and
      then the output result of the run committed back together with
      gen_testdata.py changes.
      7f0bbf7e
    • Kirill Smelkov's avatar
      zodbdump: Retrieve transaction metadata as raw from ZODB if storage support is available · 33230940
      Kirill Smelkov authored
      This patch test at runtime whether used storage can provide transaction
      metadata in raw form and if so used thi form directly without going
      through long fragile way of stable pickling extension back to bytes.
      
      Corresponding ZODB support is at
      
      https://github.com/zopefoundation/ZODB/pull/183
      33230940