1. 24 May, 2019 4 commits
    • Kirill Smelkov's avatar
      *: s.decode('hex') -> fromhex(s) · b508f108
      Kirill Smelkov authored
      Because on Py3:
              def test_dumpreader():
                  in_ = b"""\
              txn 0123456789abcdef " "
              user "my name"
              description "o la-la..."
              extension "zzz123 def"
              obj 0000000000000001 delete
              obj 0000000000000002 from 0123456789abcdee
              obj 0000000000000003 54 adler32:01234567 -
              obj 0000000000000004 4 sha1:9865d483bc5a94f2e30056fc256ed3066af54d04
              obj 0000000000000005 9 crc32:52fdeac5
              txn 0123456789abcdf0 " "
              user "author2"
              description "zzz"
              extension "qqq"
                  r = DumpReader(BytesIO(in_))
                  t1 = r.readtxn()
                  assert isinstance(t1, Transaction)
          >       assert t1.tid == '0123456789abcdef'.decode('hex')
          E       AttributeError: 'str' object has no attribute 'decode'
          test/test_dump.py:77: AttributeError
      Based on patch by Jérome Perrin.
    • Kirill Smelkov's avatar
      *: Pass bytes - not unicode - literals to sha1() · a7eee284
      Kirill Smelkov authored
      	data = 'data1'
      	    def sha1(data):
      	        m = hashlib.sha1()
      	>       m.update(data)
      	E       TypeError: Unicode-objects must be encoded before hashing
      	zodbtools/util.py:38: TypeError
      Based on patch by Jérome Perrin.
    • Kirill Smelkov's avatar
      *: cStringIO.StringIO -> io.BytesIO · 62b21d01
      Kirill Smelkov authored
      There is no cStringIO on Python3:
      	test_dump.py:26: in <module>
      	    from cStringIO import StringIO
      	E   ModuleNotFoundError: No module named 'cStringIO'
      Based on patch by Jérome Perrin.
    • Jérome Perrin's avatar
      test: add a test for zodb commmad and help driver · 2d94ae9d
      Jérome Perrin authored
      kirr: factor running `zodb ...` into zodbrun + add test for `zodb -h`.
      Added test currently passes on py2, but fails on py3:
      	out = <_io.TextIOWrapper encoding='UTF-8'>
      	    def usage(out):
      	    Zodb is a tool for managing ZODB databases.
      	        zodb command [arguments]
      	    The commands are:
      	    """, file=out)
      	        cmdv = command_dict.keys()
      	>       cmdv.sort()
      	E       AttributeError: 'dict_keys' object has no attribute 'sort'
      	zodbtools/zodb.py:55: AttributeError
      It will be fixed in the next patch.
  2. 30 Jan, 2019 2 commits
  3. 10 Jan, 2019 2 commits
    • Kirill Smelkov's avatar
      zodbcommit: Fix it for ZODB3 and ZODB4 · 7a94e312
      Kirill Smelkov authored
      maxtid is in ZODB.utils starting only from ZODB5.
      ZODB{3,4} want txn._extension, while ZODB5 deprecate it in favour of
    • Kirill Smelkov's avatar
      dump, commit: Test for both non-empty and empty transaction extensions · 425e6656
      Kirill Smelkov authored
      Currently we exercise zodbdump and zodbcommit+zodbdump with non-empty
      extensions, which works if ZODB is patched for txn.extension_bytes
      support, but fails on pristine ZODB.
      Support for txn.extension_bytes cannot get into upstream ZODB for
      more than a year:
      and  even if it somehow will make it, it will likely be only in ZODB5,
      while we still care to support ZODB4 and ZODB3.
      Skipping zodbdump / zodbcommit tests, if a ZODB does not have
      txn.extension_bytes support, would result in significant reduction of
      zodbtools test coverage, because practically that is the current
      situation with all upstream ZODB{3,4,5}. Dropping test coverage for
      non-empty extensions is neither a good option.
      For those reason, let's rework the tests and test both zodbdump and
      zodbcommit with two scenarios:
      1. on a test database where transactions extensions are always empty.
         This should work on all ZODB irregardless of whether
         txn.extension_bytes patch is there or not.
      2. on a test database where transactions extensions are present.
         This should work if ZODB has txn.extension_bytes support, but if not,
         we can mark this case as xfail, since the failure is expected.
      This way we make the testsuite pass irregardless of whether
      txn.extension_bytes support is there, and we don't abandon dump/commit
      testing coverage.
      /helped-by Jérome Perrin <jerome@nexedi.com>
  4. 09 Jan, 2019 1 commit
  5. 08 Jan, 2019 2 commits
  6. 13 Dec, 2018 2 commits
    • Kirill Smelkov's avatar
      zodbcommit - Tool to commit new transaction into ZODB · 960c5e17
      Kirill Smelkov authored
      Zodbcommit reads transaction description from stdin and commits read data into
      ZODB. The transaction to be committed is read in zodbdump format, but without
      first 'txn' header line. For example:
          user "author"
          description "change 123"
          extension ""
          obj 0000000000000001 4 null:00
      This tool could be useful for testing and for low-level database
      maintenance. Please see zodbcommit.py docstring for more details.
    • Kirill Smelkov's avatar
      zodbdump += DumpReader - to read/parse zodbdump stream · dd959b28
      Kirill Smelkov authored
      We will likely need this reader for `zodb restore` in the future.
      We will also use this reader for `zodb commit` in the next patch.
      pygolang dependency v↑ becuase we use recently introduced
      golang.strconv to unquote user/desc/extension strings.
      Python2 works. Python3 support is only minimal and incomplete.
  7. 12 Dec, 2018 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Always end a transaction with LF · 624aeb09
      Kirill Smelkov authored
      Before now we were emitting extra LF only in between transactions as a
      separator. However the dump format states LF always goes after
      transaction, and there is a reason for it:
      	without LF in the end, it becomes ambiguous at EOF - whether it
      	is a proper transaction end, or the transaction was cut.
      So avoid the ambiguity by always emitting trailing LF after transaction
  8. 02 Jul, 2018 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Switch to using qq from pygolang · b1163449
      Kirill Smelkov authored
      I originally added escapeqq as part of 75c03368 (zodbdump: Start to
      stabilize output format) with the task for this utility to quote string
      into valid "..." string always quoted with ".
      This utility was later copied to pygolang:
      and then further improved there to work under both Python2 and Python3
      and to not escape printable UTF-8 characters:
      So stop the duplication and simply switch to the better version.
  9. 11 May, 2018 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Add golden test · 7f0bbf7e
      Kirill Smelkov authored
      We add a program to generate a test database with all fancy features and
      then check `zodb dump` output on it to golden on.
      The test database itself is commited to git because we want to make sure
      zodbdump works ok for particular exact data and because transaction
      extension dict is potentially saved differently on various runs.
        https://docs.python.org/2.7/library/stdtypes.html#dict.items    and
      """ CPython implementation detail: Keys and values are listed in an arbitrary
          order which is non-random, varies across Python implementations, and depends
          on the dictionary’s history of insertions and deletions. """
      This way on test/gen_testdata.py changes it has to be run manually, and
      then the output result of the run committed back together with
      gen_testdata.py changes.
  10. 02 Nov, 2017 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Start to stabilize output format · 75c03368
      Kirill Smelkov authored
      Since zodbdump start (c0a6299f "zodbdump - Tool to dump content of a
      ZODB database   (draft)") and up till now zodbdump output format was not
      good. For example user and description transaction properties were
      output without proper quoting, which in situation when there would be
      fancy characters in there would break the output.
      So start the format stabilization:
      - user and description are output as quoted, so now they are guaranteed
        to be on one line. The quoting character is always " (instead of e.g.
        smartly quoting either by ' or " as python does) for easier
        compatibility with ZODB implementations in other languages.
      - transaction extension is now printed as raw bytes, not as dict.
        The idea here is that `zodb dump`
        * should perform dump of raw data as stored inside ZODB so that later
          `zodb restore` could restore the database identically to the same state.
        * we should dump raw data instead of unpickled ones because generally
          on-disk extension's format can be any raw bytes and this information
          should be preserved.
      - transaction status is now also output as quoted to preserve line
        breakage on fancy status codes.
      - it is documented that sha1 is not the only allowed hash function that
        might be used.
      - in hashonly mode we add trailing " -" to obj string so that it is
        possible to distinguish outputs of `zodb dump` and `zodb dump -hashonly`
        without knowing a-priory the way it was produced.
        The reason to do so is that it would be not good to e.g. by accident
        feed hashonly output to (future) `zodb restore`, which, without having
        a way to see it should not consume object data would read following
        transaction information as raw object data with confusing later
        errors (and a small chance to restore completely different database
        without reporting error at all).
      Because ZODB iteration API gives us already unpickled extension and only
      that, for now to dump it as raw we get a long road to pickle it back
      also caring to try to pickle in stable order.
      Hopefully this will be only a fallback because of
      and next zodbtools patch.
      For testing purposes (currently only quoting function unit test) py.test
      usage is introduced.
      The code is also generally polished here and there.