1. 24 May, 2019 8 commits
    • Kirill Smelkov's avatar
      zodbdump: Default out to stdout in binary mode · c5f20201
      Kirill Smelkov authored
      Zodbdump format is mixed text+binary so dumping to unicode stdout won't
      work.
      
      Based on patch by Jérome Perrin.
      c5f20201
    • Kirill Smelkov's avatar
      *: s.decode('hex') -> fromhex(s) · b508f108
      Kirill Smelkov authored
      Because on Py3:
      
              def test_dumpreader():
                  in_ = b"""\
              txn 0123456789abcdef " "
              user "my name"
              description "o la-la..."
              extension "zzz123 def"
              obj 0000000000000001 delete
              obj 0000000000000002 from 0123456789abcdee
              obj 0000000000000003 54 adler32:01234567 -
              obj 0000000000000004 4 sha1:9865d483bc5a94f2e30056fc256ed3066af54d04
              ZZZZ
              obj 0000000000000005 9 crc32:52fdeac5
              ABC
      
              DEF!
      
              txn 0123456789abcdf0 " "
              user "author2"
              description "zzz"
              extension "qqq"
      
              """
      
                  r = DumpReader(BytesIO(in_))
                  t1 = r.readtxn()
                  assert isinstance(t1, Transaction)
          >       assert t1.tid == '0123456789abcdef'.decode('hex')
          E       AttributeError: 'str' object has no attribute 'decode'
      
          test/test_dump.py:77: AttributeError
      
      Based on patch by Jérome Perrin.
      b508f108
    • Kirill Smelkov's avatar
      utils: Initialize hashers with bytes · 1418c86f
      Kirill Smelkov authored
      	self = <zodbtools.util.CRC32Hasher object at 0x7f887ae465f8>
      
      	    def __init__(self):
      	>       self._h = crc32('')
      	E       TypeError: a bytes-like object is required, not 'str'
      
      	util.py:208: TypeError
      
      Based on patch by Jérome Perrin.
      1418c86f
    • Kirill Smelkov's avatar
      *: Pass bytes - not unicode - literals to sha1() · a7eee284
      Kirill Smelkov authored
      	data = 'data1'
      
      	    def sha1(data):
      	        m = hashlib.sha1()
      	>       m.update(data)
      	E       TypeError: Unicode-objects must be encoded before hashing
      
      	zodbtools/util.py:38: TypeError
      
      Based on patch by Jérome Perrin.
      a7eee284
    • Kirill Smelkov's avatar
      util: Fix ashex for Python3 · 7a7370e6
      Kirill Smelkov authored
      	s = b'\x03\xc4\x85v\x00\x00\x00\x00'
      
      	    def ashex(s):
      	>       return s.encode('hex')
      	E       AttributeError: 'bytes' object has no attribute 'encode'
      
      	zodbtools/util.py:29: AttributeError
      
      s.encode('hex') used to work on Py2 but fails on Py3:
      
      	In [1]: s = "abc"
      
      	In [2]: b = b"def"
      
      	In [3]: s.encode('hex')
      	---------------------------------------------------------------------------
      	LookupError                               Traceback (most recent call last)
      	<ipython-input-3-75ae843597fe> in <module>()
      	----> 1 s.encode('hex')
      
      	LookupError: 'hex' is not a text encoding; use codecs.encode() to handle arbitrary codecs
      
      	In [4]: b.encode('hex')
      	---------------------------------------------------------------------------
      	AttributeError                            Traceback (most recent call last)
      	<ipython-input-4-ec2fccff20bc> in <module>()
      	----> 1 b.encode('hex')
      
      	AttributeError: 'bytes' object has no attribute 'encode'
      
      	In [5]: import codecs
      
      	In [6]: codecs.encode(b, 'hex')
      	Out[6]: b'646566'
      
      	In [7]: codecs.encode(s, 'hex')
      	---------------------------------------------------------------------------
      	TypeError                                 Traceback (most recent call last)
      	/usr/lib/python3.7/encodings/hex_codec.py in hex_encode(input, errors)
      	     14     assert errors == 'strict'
      	---> 15     return (binascii.b2a_hex(input), len(input))
      	     16
      
      	TypeError: a bytes-like object is required, not 'str'
      
      	The above exception was the direct cause of the following exception:
      
      	TypeError                                 Traceback (most recent call last)
      	<ipython-input-7-7fcb16cead4f> in <module>()
      	----> 1 codecs.encode(s, 'hex')
      
      	TypeError: encoding with 'hex' codec failed (TypeError: a bytes-like object is required, not 'str')
      
      After the patch it works with bytes and raises for str.
      Fromhex does not need to be changed - it already uses codecs.decode way as
      originally added in dd959b28 (zodbdump += DumpReader - to read/parse zodbdump
      stream).
      
      Based on patch by Jérome Perrin.
      7a7370e6
    • Kirill Smelkov's avatar
      *: cStringIO.StringIO -> io.BytesIO · 62b21d01
      Kirill Smelkov authored
      There is no cStringIO on Python3:
      
      	test_dump.py:26: in <module>
      	    from cStringIO import StringIO
      	E   ModuleNotFoundError: No module named 'cStringIO'
      
      Based on patch by Jérome Perrin.
      62b21d01
    • Jérome Perrin's avatar
      zodb: rework command driver for python3 compatibility · 00a534ef
      Jérome Perrin authored
      This makes zodb command driver tests added in the previous patch to pass
      on both python2 and python3.
      00a534ef
    • Jérome Perrin's avatar
      test: add a test for zodb commmad and help driver · 2d94ae9d
      Jérome Perrin authored
      ----
      
      kirr: factor running `zodb ...` into zodbrun + add test for `zodb -h`.
      
      Added test currently passes on py2, but fails on py3:
      
      	out = <_io.TextIOWrapper encoding='UTF-8'>
      
      	    def usage(out):
      	        print("""\
      	    Zodb is a tool for managing ZODB databases.
      
      	    Usage:
      
      	        zodb command [arguments]
      
      	    The commands are:
      	    """, file=out)
      
      	        cmdv = command_dict.keys()
      	>       cmdv.sort()
      	E       AttributeError: 'dict_keys' object has no attribute 'sort'
      
      	zodbtools/zodb.py:55: AttributeError
      
      It will be fixed in the next patch.
      2d94ae9d
  2. 07 Mar, 2019 1 commit
  3. 31 Jan, 2019 6 commits
  4. 30 Jan, 2019 3 commits
  5. 11 Jan, 2019 2 commits
  6. 10 Jan, 2019 4 commits
    • Kirill Smelkov's avatar
      Test coverage for ZODB{3,4,5} · 8ff7020c
      Kirill Smelkov authored
      Use tox to test with all kinds of ZODB.
      With preceding 3 patches tests pass with all versions of upstream ZODB.
      
      TODO: test coverage for both py2 and py3.
      8ff7020c
    • Kirill Smelkov's avatar
      zodbcommit: Fix it for ZODB3 and ZODB4 · 7a94e312
      Kirill Smelkov authored
      maxtid is in ZODB.utils starting only from ZODB5.
      
      ZODB{3,4} want txn._extension, while ZODB5 deprecate it in favour of
      txn.extension.
      7a94e312
    • Kirill Smelkov's avatar
      zodbdump: Fix it for ZODB3 and ZODB4 · 0e5d2f81
      Kirill Smelkov authored
      IStorageTransactionMetaData is ZODB5-only interface. Bug introduced in
      dd959b28 (zodbdump += DumpReader - to read/parse zodbdump stream).
      0e5d2f81
    • Kirill Smelkov's avatar
      dump, commit: Test for both non-empty and empty transaction extensions · 425e6656
      Kirill Smelkov authored
      Currently we exercise zodbdump and zodbcommit+zodbdump with non-empty
      extensions, which works if ZODB is patched for txn.extension_bytes
      support, but fails on pristine ZODB.
      
      Support for txn.extension_bytes cannot get into upstream ZODB for
      more than a year:
      
      https://github.com/zopefoundation/ZODB/pull/183
      https://github.com/zopefoundation/ZODB/pull/207
      
      and  even if it somehow will make it, it will likely be only in ZODB5,
      while we still care to support ZODB4 and ZODB3.
      
      Skipping zodbdump / zodbcommit tests, if a ZODB does not have
      txn.extension_bytes support, would result in significant reduction of
      zodbtools test coverage, because practically that is the current
      situation with all upstream ZODB{3,4,5}. Dropping test coverage for
      non-empty extensions is neither a good option.
      
      For those reason, let's rework the tests and test both zodbdump and
      zodbcommit with two scenarios:
      
      1. on a test database where transactions extensions are always empty.
         This should work on all ZODB irregardless of whether
         txn.extension_bytes patch is there or not.
      
      2. on a test database where transactions extensions are present.
         This should work if ZODB has txn.extension_bytes support, but if not,
         we can mark this case as xfail, since the failure is expected.
      
      This way we make the testsuite pass irregardless of whether
      txn.extension_bytes support is there, and we don't abandon dump/commit
      testing coverage.
      
      /helped-by Jérome Perrin <jerome@nexedi.com>
      425e6656
  7. 09 Jan, 2019 2 commits
    • Jérome Perrin's avatar
      Fix zodb analyze with empty reports · 8c76eae2
      Jérome Perrin authored
      Fix for this kind of errors:
      
      ```
      (env)$ zodb analyze demo.fs ffffffffffffffff..
      # ø
      Processed 0 records in 0 transactions
      Traceback (most recent call last):
        File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/env/bin/zodb", line 11, in <module>
          load_entry_point('zodbtools', 'console_scripts', 'zodb')()
        File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodb.py", line 130, in main
          return command_module.main(argv)
        File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 305, in main
          report(analyze(path, use_dbm, delta_fs, tidmin, tidmax), csv)
        File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 102, in report
          print "Average record size is %7.2f bytes" % (rep.DBYTES * 1.0 / rep.OIDS)
      ZeroDivisionError: float division by zero
      ```
      
      and also small fixes for python3 compatibility
      
      /reviewed-on !9
      8c76eae2
    • Jérome Perrin's avatar
      analyze: always display processed tid range (even ø) · d37746c6
      Jérome Perrin authored
      To keep a consistent output.
      d37746c6
  8. 08 Jan, 2019 6 commits
  9. 07 Jan, 2019 1 commit
  10. 30 Dec, 2018 4 commits
  11. 17 Dec, 2018 1 commit
  12. 13 Dec, 2018 2 commits