1. 29 Apr, 2020 6 commits
    • Kirill Smelkov's avatar
      tidrange: test: Fix for py3 · 2236aaaf
      Kirill Smelkov authored
      ashex gives bytes, whereas reference_tid was str.
      2236aaaf
    • Kirill Smelkov's avatar
      *: dict.keys() returns sequence, not [] on py3 · 7851a964
      Kirill Smelkov authored
      The sequence cannot be randomly accessed, e.g.
      
          In [5]: d = {1:2}
      
          In [6]: kv = d.keys()
      
          In [7]: kv
          Out[7]: dict_keys([1])
      
          In [8]: kv[0]
          ---------------------------------------------------------------------------
          TypeError                                 Traceback (most recent call last)
          <ipython-input-8-643f90e1910b> in <module>()
          ----> 1 kv[0]
      
          TypeError: 'dict_keys' object is not subscriptable
      
      -> Use list(dict.keys()) in places where we need random access.
      7851a964
    • Kirill Smelkov's avatar
      *: Pass bytes literal into BytesIO · 2f9e0623
      Kirill Smelkov authored
      Otherwise it breaks with str on py3:
      
      	In [1]: from io import BytesIO
      
      	In [2]: BytesIO("abc")
      	---------------------------------------------------------------------------
      	TypeError                                 Traceback (most recent call last)
      	<ipython-input-2-52a130edd46d> in <module>()
      	----> 1 BytesIO("abc")
      
      	TypeError: a bytes-like object is required, not 'str'
      2f9e0623
    • Kirill Smelkov's avatar
      zodbdump: Use bytes to emit its output · d3152c78
      Kirill Smelkov authored
      Zodbdump format is text-binary and is saved into files opened in binary
      mode. -> We have to emit bytes - not strings - into it, since otherwise
      on Python3 it would break.
      
      This needs qq support from pygolang[1] to be able to use qq with both
      string and bytestring format, e.g. for
      
      	 "hello %s" % qq(name),	and
      	b"hello %s" % qq(name)
      
      to give the same output irregardless of whether name is str or bytes.
      
      [1] nexedi/pygolang!1
      d3152c78
    • Kirill Smelkov's avatar
      *: Zodbdump format is semi text-binary: Mark it as such + handle zdump output as binary · ddd5fd03
      Kirill Smelkov authored
      Zodbdump format is already described as semi text-binary in top-level
      zodbdump.py documentation. However zdump() docstring was referring to it
      as "text". Fix it and use binary to handle places where zdump is
      loaded/saved.
      ddd5fd03
    • Kirill Smelkov's avatar
      *: Don't use %r to print/report lines/bytes to outside · bc608aea
      Kirill Smelkov authored
      %r has different output for strings and bytes on python3:
      
      	In [1]: a = 'hello'
      	In [2]: b = b'hello'
      
      	In [3]: repr(a)
      	Out[3]: "'hello'"
      
      	In [4]: repr(b)
      	Out[4]: "b'hello'"
      
      -> Use qq whose output is stable irregardless of whether input is string or bytes.
      bc608aea
  2. 13 Mar, 2020 1 commit
  3. 14 Feb, 2020 1 commit
  4. 24 May, 2019 8 commits
    • Kirill Smelkov's avatar
      zodbdump: Default out to stdout in binary mode · c5f20201
      Kirill Smelkov authored
      Zodbdump format is mixed text+binary so dumping to unicode stdout won't
      work.
      
      Based on patch by Jérome Perrin.
      c5f20201
    • Kirill Smelkov's avatar
      *: s.decode('hex') -> fromhex(s) · b508f108
      Kirill Smelkov authored
      Because on Py3:
      
              def test_dumpreader():
                  in_ = b"""\
              txn 0123456789abcdef " "
              user "my name"
              description "o la-la..."
              extension "zzz123 def"
              obj 0000000000000001 delete
              obj 0000000000000002 from 0123456789abcdee
              obj 0000000000000003 54 adler32:01234567 -
              obj 0000000000000004 4 sha1:9865d483bc5a94f2e30056fc256ed3066af54d04
              ZZZZ
              obj 0000000000000005 9 crc32:52fdeac5
              ABC
      
              DEF!
      
              txn 0123456789abcdf0 " "
              user "author2"
              description "zzz"
              extension "qqq"
      
              """
      
                  r = DumpReader(BytesIO(in_))
                  t1 = r.readtxn()
                  assert isinstance(t1, Transaction)
          >       assert t1.tid == '0123456789abcdef'.decode('hex')
          E       AttributeError: 'str' object has no attribute 'decode'
      
          test/test_dump.py:77: AttributeError
      
      Based on patch by Jérome Perrin.
      b508f108
    • Kirill Smelkov's avatar
      utils: Initialize hashers with bytes · 1418c86f
      Kirill Smelkov authored
      	self = <zodbtools.util.CRC32Hasher object at 0x7f887ae465f8>
      
      	    def __init__(self):
      	>       self._h = crc32('')
      	E       TypeError: a bytes-like object is required, not 'str'
      
      	util.py:208: TypeError
      
      Based on patch by Jérome Perrin.
      1418c86f
    • Kirill Smelkov's avatar
      *: Pass bytes - not unicode - literals to sha1() · a7eee284
      Kirill Smelkov authored
      	data = 'data1'
      
      	    def sha1(data):
      	        m = hashlib.sha1()
      	>       m.update(data)
      	E       TypeError: Unicode-objects must be encoded before hashing
      
      	zodbtools/util.py:38: TypeError
      
      Based on patch by Jérome Perrin.
      a7eee284
    • Kirill Smelkov's avatar
      util: Fix ashex for Python3 · 7a7370e6
      Kirill Smelkov authored
      	s = b'\x03\xc4\x85v\x00\x00\x00\x00'
      
      	    def ashex(s):
      	>       return s.encode('hex')
      	E       AttributeError: 'bytes' object has no attribute 'encode'
      
      	zodbtools/util.py:29: AttributeError
      
      s.encode('hex') used to work on Py2 but fails on Py3:
      
      	In [1]: s = "abc"
      
      	In [2]: b = b"def"
      
      	In [3]: s.encode('hex')
      	---------------------------------------------------------------------------
      	LookupError                               Traceback (most recent call last)
      	<ipython-input-3-75ae843597fe> in <module>()
      	----> 1 s.encode('hex')
      
      	LookupError: 'hex' is not a text encoding; use codecs.encode() to handle arbitrary codecs
      
      	In [4]: b.encode('hex')
      	---------------------------------------------------------------------------
      	AttributeError                            Traceback (most recent call last)
      	<ipython-input-4-ec2fccff20bc> in <module>()
      	----> 1 b.encode('hex')
      
      	AttributeError: 'bytes' object has no attribute 'encode'
      
      	In [5]: import codecs
      
      	In [6]: codecs.encode(b, 'hex')
      	Out[6]: b'646566'
      
      	In [7]: codecs.encode(s, 'hex')
      	---------------------------------------------------------------------------
      	TypeError                                 Traceback (most recent call last)
      	/usr/lib/python3.7/encodings/hex_codec.py in hex_encode(input, errors)
      	     14     assert errors == 'strict'
      	---> 15     return (binascii.b2a_hex(input), len(input))
      	     16
      
      	TypeError: a bytes-like object is required, not 'str'
      
      	The above exception was the direct cause of the following exception:
      
      	TypeError                                 Traceback (most recent call last)
      	<ipython-input-7-7fcb16cead4f> in <module>()
      	----> 1 codecs.encode(s, 'hex')
      
      	TypeError: encoding with 'hex' codec failed (TypeError: a bytes-like object is required, not 'str')
      
      After the patch it works with bytes and raises for str.
      Fromhex does not need to be changed - it already uses codecs.decode way as
      originally added in dd959b28 (zodbdump += DumpReader - to read/parse zodbdump
      stream).
      
      Based on patch by Jérome Perrin.
      7a7370e6
    • Kirill Smelkov's avatar
      *: cStringIO.StringIO -> io.BytesIO · 62b21d01
      Kirill Smelkov authored
      There is no cStringIO on Python3:
      
      	test_dump.py:26: in <module>
      	    from cStringIO import StringIO
      	E   ModuleNotFoundError: No module named 'cStringIO'
      
      Based on patch by Jérome Perrin.
      62b21d01
    • Jérome Perrin's avatar
      zodb: rework command driver for python3 compatibility · 00a534ef
      Jérome Perrin authored
      This makes zodb command driver tests added in the previous patch to pass
      on both python2 and python3.
      00a534ef
    • Jérome Perrin's avatar
      test: add a test for zodb commmad and help driver · 2d94ae9d
      Jérome Perrin authored
      ----
      
      kirr: factor running `zodb ...` into zodbrun + add test for `zodb -h`.
      
      Added test currently passes on py2, but fails on py3:
      
      	out = <_io.TextIOWrapper encoding='UTF-8'>
      
      	    def usage(out):
      	        print("""\
      	    Zodb is a tool for managing ZODB databases.
      
      	    Usage:
      
      	        zodb command [arguments]
      
      	    The commands are:
      	    """, file=out)
      
      	        cmdv = command_dict.keys()
      	>       cmdv.sort()
      	E       AttributeError: 'dict_keys' object has no attribute 'sort'
      
      	zodbtools/zodb.py:55: AttributeError
      
      It will be fixed in the next patch.
      2d94ae9d
  5. 31 Jan, 2019 3 commits
  6. 30 Jan, 2019 2 commits
  7. 10 Jan, 2019 3 commits
    • Kirill Smelkov's avatar
      zodbcommit: Fix it for ZODB3 and ZODB4 · 7a94e312
      Kirill Smelkov authored
      maxtid is in ZODB.utils starting only from ZODB5.
      
      ZODB{3,4} want txn._extension, while ZODB5 deprecate it in favour of
      txn.extension.
      7a94e312
    • Kirill Smelkov's avatar
      zodbdump: Fix it for ZODB3 and ZODB4 · 0e5d2f81
      Kirill Smelkov authored
      IStorageTransactionMetaData is ZODB5-only interface. Bug introduced in
      dd959b28 (zodbdump += DumpReader - to read/parse zodbdump stream).
      0e5d2f81
    • Kirill Smelkov's avatar
      dump, commit: Test for both non-empty and empty transaction extensions · 425e6656
      Kirill Smelkov authored
      Currently we exercise zodbdump and zodbcommit+zodbdump with non-empty
      extensions, which works if ZODB is patched for txn.extension_bytes
      support, but fails on pristine ZODB.
      
      Support for txn.extension_bytes cannot get into upstream ZODB for
      more than a year:
      
      https://github.com/zopefoundation/ZODB/pull/183
      https://github.com/zopefoundation/ZODB/pull/207
      
      and  even if it somehow will make it, it will likely be only in ZODB5,
      while we still care to support ZODB4 and ZODB3.
      
      Skipping zodbdump / zodbcommit tests, if a ZODB does not have
      txn.extension_bytes support, would result in significant reduction of
      zodbtools test coverage, because practically that is the current
      situation with all upstream ZODB{3,4,5}. Dropping test coverage for
      non-empty extensions is neither a good option.
      
      For those reason, let's rework the tests and test both zodbdump and
      zodbcommit with two scenarios:
      
      1. on a test database where transactions extensions are always empty.
         This should work on all ZODB irregardless of whether
         txn.extension_bytes patch is there or not.
      
      2. on a test database where transactions extensions are present.
         This should work if ZODB has txn.extension_bytes support, but if not,
         we can mark this case as xfail, since the failure is expected.
      
      This way we make the testsuite pass irregardless of whether
      txn.extension_bytes support is there, and we don't abandon dump/commit
      testing coverage.
      
      /helped-by Jérome Perrin <jerome@nexedi.com>
      425e6656
  8. 09 Jan, 2019 1 commit
  9. 08 Jan, 2019 6 commits
  10. 30 Dec, 2018 3 commits
  11. 17 Dec, 2018 1 commit
  12. 13 Dec, 2018 3 commits
    • Kirill Smelkov's avatar
      zodbcommit - Tool to commit new transaction into ZODB · 960c5e17
      Kirill Smelkov authored
      Zodbcommit reads transaction description from stdin and commits read data into
      ZODB. The transaction to be committed is read in zodbdump format, but without
      first 'txn' header line. For example:
      
          user "author"
          description "change 123"
          extension ""
          obj 0000000000000001 4 null:00
          ZZZZ
      
      This tool could be useful for testing and for low-level database
      maintenance. Please see zodbcommit.py docstring for more details.
      960c5e17
    • Kirill Smelkov's avatar
      zodbdump += DumpReader - to read/parse zodbdump stream · dd959b28
      Kirill Smelkov authored
      We will likely need this reader for `zodb restore` in the future.
      We will also use this reader for `zodb commit` in the next patch.
      
      pygolang dependency v↑ becuase we use recently introduced
      golang.strconv to unquote user/desc/extension strings.
      
      Python2 works. Python3 support is only minimal and incomplete.
      dd959b28
    • Kirill Smelkov's avatar
      util += hashRegistry, null, adler32, crc32 hashers · e973d519
      Kirill Smelkov authored
      hashRegistry will be needed for zodbdump reader to create a hasher by
      its name. Since std hashlib does not have adler32/crc32 nor null - let's
      add our own implementation for those hashers too.
      
      The code is based on
      
      	neo@a60c472c, and
      	neo@3f578560
      e973d519
  13. 12 Dec, 2018 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Always end a transaction with LF · 624aeb09
      Kirill Smelkov authored
      Before now we were emitting extra LF only in between transactions as a
      separator. However the dump format states LF always goes after
      transaction, and there is a reason for it:
      
      	without LF in the end, it becomes ambiguous at EOF - whether it
      	is a proper transaction end, or the transaction was cut.
      
      So avoid the ambiguity by always emitting trailing LF after transaction
      record.
      624aeb09
  14. 02 Jul, 2018 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Switch to using qq from pygolang · b1163449
      Kirill Smelkov authored
      I originally added escapeqq as part of 75c03368 (zodbdump: Start to
      stabilize output format) with the task for this utility to quote string
      into valid "..." string always quoted with ".
      
      This utility was later copied to pygolang:
      
      	pygolang@afa46cf5
      
      and then further improved there to work under both Python2 and Python3
      and to not escape printable UTF-8 characters:
      
      	pygolang@02dddb97
      
      So stop the duplication and simply switch to the better version.
      b1163449