1. 16 Mar, 2021 2 commits
    • Kirill Smelkov's avatar
      zodbcommit: Provide full context when reporting errors · 129afa67
      Kirill Smelkov authored
      In the previous patch we taught object copy handler to report more
      details, but it was still incomplete - the error was missing details
      about which operation was run - commit, or restore of particular
      transaction.
      
      Noting that it can be also noted that other errors reported from that
      function lack such context.
      
      -> So fix it universally, at least for zodbcommit for now: set top-level
      runctx to topic of what we are doing, and use that runctx when
      generating errors. Runctx describes what we are running, and could be
      also later used for logging and tracing. That's why it is called runctx
      instead of just errctx for "error context".
      
      TODO currently it is only exceptions that we explicitly raise which get
      the context. If an exception is raised by something that we call - the
      context won't be added. It would be good to later rework error handling
      and append such context for any raised error. Defer and
      https://lab.nexedi.com/kirr/go123/blob/863c4602/xerr/__init__.py has
      something preliminary for this.
      
      The particular error when restoring a missing object copy becomes
      
          ValueError: /tmp/demo002868462/δ0285cbac75555580/δ.fs: restore 0285cbacb70a3db3 @0285cbacb258bf66: object 0000000000000003: copy from @0285cbac70a3d733: no data
      
      instead of older
      
          ValueError: /tmp/demo358030847/δ0285cbac75555580/δ.fs: object 0000000000000003: copy from @0285cbac70a3d733: no data
      
      /reviewed-by @jerome
      /reviewed-on !20
      129afa67
    • Kirill Smelkov's avatar
      zodbcommit: Robustify copy handling · fa00c283
      Kirill Smelkov authored
      When zodbdump input says to copy an object, we first load that object.
      However if object does not exist loadBefore raises POSKeyError, and when
      object at copied-from revision was deleted loadBefore returns None.
      
      -> Handle that explicitly to provide failure details to the user, so
      that instead of cryptic
      
          === RUN   TestLoad/δstart=0285cbac75555580
          Traceback (most recent call last):
            File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
              "__main__", fname, loader, pkg_name)
            File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
              exec code in run_globals
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodb.py", line 133, in <module>
              main()
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodb.py", line 129, in main
              return command_module.main(argv)
            File "<decorator-gen-6>", line 2, in main
            File "/home/kirr/src/tools/go/pygolang/golang/__init__.py", line 103, in _
              return f(*argv, **kw)
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbrestore.py", line 94, in main
              zodbrestore(stor, asbinstream(sys.stdin), _)
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbrestore.py", line 43, in zodbrestore
              zodbcommit(stor, at, txn)
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbcommit.py", line 122, in zodbcommit
              _()
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbcommit.py", line 91, in _
              data, _, _ = stor.loadBefore(obj.oid, p64(u64(obj.copy_from)+1))
          TypeError: 'NoneType' object is not iterable
              xtesting.go:483: /tmp/demo009767458/δ0285cbac75555580/δ.fs: zpyrestore: exit status 1
      
      it fails with something more understandable:
      
          === RUN   TestLoad/δstart=0285cbac75555580
          Traceback (most recent call last):
            File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
              "__main__", fname, loader, pkg_name)
            File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
              exec code in run_globals
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodb.py", line 133, in <module>
              main()
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodb.py", line 129, in main
              return command_module.main(argv)
            File "<decorator-gen-6>", line 2, in main
            File "/home/kirr/src/tools/go/pygolang/golang/__init__.py", line 103, in _
              return f(*argv, **kw)
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbrestore.py", line 94, in main
              zodbrestore(stor, asbinstream(sys.stdin), _)
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbrestore.py", line 43, in zodbrestore
              zodbcommit(stor, at, txn)
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbcommit.py", line 129, in zodbcommit
              _()
            File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbcommit.py", line 97, in _
              (stor.getName(), ashex(obj.oid), ashex(obj.copy_from)))
          ValueError: /tmp/demo358030847/δ0285cbac75555580/δ.fs: object 0000000000000003: copy from @0285cbac70a3d733: no data
              xtesting.go:483: /tmp/demo358030847/δ0285cbac75555580/δ.fs: zpyrestore: exit status 1
      
      For the implementation it would be easier to use loadAt
      (https://github.com/zopefoundation/ZODB/pull/323), but we don't have
      that yet.
      
      /reviewed-by @jerome
      /reviewed-on !20
      fa00c283
  2. 15 Mar, 2021 4 commits
  3. 10 Mar, 2021 2 commits
    • Kirill Smelkov's avatar
      Drop support for ZODB3 · c59a54ca
      Kirill Smelkov authored
      Nexedi stack is dropping support for that old ZODB version - see e.g.
      
      - slapos@70d05199
      - neoppod@3a8f6f03
      - wendelin.core@0802da2b
      
      Regarding test/gen_testdata.py: even though ZODB4 uses zodbpickle, and
      so should be able to load pickles encoded with protocol 3 even on
      python2, in practice it does not work so well: ZODB4 tests fail if I set
      
          --- a/src/ZODB/_compat.py
          +++ b/src/ZODB/_compat.py
          @@ -34,7 +34,7 @@
               HIGHEST_PROTOCOL = cPickle.HIGHEST_PROTOCOL
               IMPORT_MAPPING = {}
               NAME_MAPPING = {}
          -    _protocol = 1
          +    _protocol = 3
               FILESTORAGE_MAGIC = b"FS21"
           else:
               # Python 3.x: can't use stdlib's pickle because
      
      -> so continue to preserve protocol < 3 when generating the test
      database for compatibility - now with ZODB4/py2.
      
      /reviewed-by @jerome
      /reviewed-on !18
      c59a54ca
    • Kirill Smelkov's avatar
      tox: Don't run tests agains ZODB+PR183 anymore · 986baf02
      Kirill Smelkov authored
      The patch that provides raw-extension functionality was merged into ZODB 5.6:
      
      https://github.com/zopefoundation/ZODB/commit/2f8cc67a3ba3
      
      So when testing with ZODB5 >= 5.6 the tests will excercise code path
      that uses txn.extension_bytes, and when testing with ZODB4 the tests
      will excercise code path that work-arounds lack of txn.extension_bytes.
      
      /reviewed-by @jerome
      /reviewed-on !18
      986baf02
  4. 02 Nov, 2020 1 commit
  5. 30 Apr, 2020 1 commit
  6. 29 Apr, 2020 6 commits
    • Kirill Smelkov's avatar
      tidrange: test: Fix for py3 · 2236aaaf
      Kirill Smelkov authored
      ashex gives bytes, whereas reference_tid was str.
      2236aaaf
    • Kirill Smelkov's avatar
      *: dict.keys() returns sequence, not [] on py3 · 7851a964
      Kirill Smelkov authored
      The sequence cannot be randomly accessed, e.g.
      
          In [5]: d = {1:2}
      
          In [6]: kv = d.keys()
      
          In [7]: kv
          Out[7]: dict_keys([1])
      
          In [8]: kv[0]
          ---------------------------------------------------------------------------
          TypeError                                 Traceback (most recent call last)
          <ipython-input-8-643f90e1910b> in <module>()
          ----> 1 kv[0]
      
          TypeError: 'dict_keys' object is not subscriptable
      
      -> Use list(dict.keys()) in places where we need random access.
      7851a964
    • Kirill Smelkov's avatar
      *: Pass bytes literal into BytesIO · 2f9e0623
      Kirill Smelkov authored
      Otherwise it breaks with str on py3:
      
      	In [1]: from io import BytesIO
      
      	In [2]: BytesIO("abc")
      	---------------------------------------------------------------------------
      	TypeError                                 Traceback (most recent call last)
      	<ipython-input-2-52a130edd46d> in <module>()
      	----> 1 BytesIO("abc")
      
      	TypeError: a bytes-like object is required, not 'str'
      2f9e0623
    • Kirill Smelkov's avatar
      zodbdump: Use bytes to emit its output · d3152c78
      Kirill Smelkov authored
      Zodbdump format is text-binary and is saved into files opened in binary
      mode. -> We have to emit bytes - not strings - into it, since otherwise
      on Python3 it would break.
      
      This needs qq support from pygolang[1] to be able to use qq with both
      string and bytestring format, e.g. for
      
      	 "hello %s" % qq(name),	and
      	b"hello %s" % qq(name)
      
      to give the same output irregardless of whether name is str or bytes.
      
      [1] pygolang!1
      d3152c78
    • Kirill Smelkov's avatar
      *: Zodbdump format is semi text-binary: Mark it as such + handle zdump output as binary · ddd5fd03
      Kirill Smelkov authored
      Zodbdump format is already described as semi text-binary in top-level
      zodbdump.py documentation. However zdump() docstring was referring to it
      as "text". Fix it and use binary to handle places where zdump is
      loaded/saved.
      ddd5fd03
    • Kirill Smelkov's avatar
      *: Don't use %r to print/report lines/bytes to outside · bc608aea
      Kirill Smelkov authored
      %r has different output for strings and bytes on python3:
      
      	In [1]: a = 'hello'
      	In [2]: b = b'hello'
      
      	In [3]: repr(a)
      	Out[3]: "'hello'"
      
      	In [4]: repr(b)
      	Out[4]: "b'hello'"
      
      -> Use qq whose output is stable irregardless of whether input is string or bytes.
      bc608aea
  7. 13 Mar, 2020 1 commit
  8. 14 Feb, 2020 1 commit
  9. 09 Jul, 2019 1 commit
  10. 03 Jun, 2019 1 commit
    • Kirill Smelkov's avatar
      More python3 compatibility · b44f9c0d
      Kirill Smelkov authored
      @jerome, I was trying to make zodbtools work with Python3 and along that road picked some bits of your work from !12. At present the migration to Python3 is not complete, and even though now I have the answer to how handle strings in both python2/3 in compatible and reasonable way (I can share details if you are interested), I have to put that work on hold for some time and use https://pypi.org/project/pep3134 directly in wcfs tests, since getting all string details right, even after figuring on how to do it, will take time. Anyway the bits presented here should be ready for master and could be merged now. Could you please have a look?
      
      Thanks beforehand,  
      Kirill
      
      /reviewed-on !13
      b44f9c0d
  11. 24 May, 2019 8 commits
    • Kirill Smelkov's avatar
      zodbdump: Default out to stdout in binary mode · c5f20201
      Kirill Smelkov authored
      Zodbdump format is mixed text+binary so dumping to unicode stdout won't
      work.
      
      Based on patch by Jérome Perrin.
      c5f20201
    • Kirill Smelkov's avatar
      *: s.decode('hex') -> fromhex(s) · b508f108
      Kirill Smelkov authored
      Because on Py3:
      
              def test_dumpreader():
                  in_ = b"""\
              txn 0123456789abcdef " "
              user "my name"
              description "o la-la..."
              extension "zzz123 def"
              obj 0000000000000001 delete
              obj 0000000000000002 from 0123456789abcdee
              obj 0000000000000003 54 adler32:01234567 -
              obj 0000000000000004 4 sha1:9865d483bc5a94f2e30056fc256ed3066af54d04
              ZZZZ
              obj 0000000000000005 9 crc32:52fdeac5
              ABC
      
              DEF!
      
              txn 0123456789abcdf0 " "
              user "author2"
              description "zzz"
              extension "qqq"
      
              """
      
                  r = DumpReader(BytesIO(in_))
                  t1 = r.readtxn()
                  assert isinstance(t1, Transaction)
          >       assert t1.tid == '0123456789abcdef'.decode('hex')
          E       AttributeError: 'str' object has no attribute 'decode'
      
          test/test_dump.py:77: AttributeError
      
      Based on patch by Jérome Perrin.
      b508f108
    • Kirill Smelkov's avatar
      utils: Initialize hashers with bytes · 1418c86f
      Kirill Smelkov authored
      	self = <zodbtools.util.CRC32Hasher object at 0x7f887ae465f8>
      
      	    def __init__(self):
      	>       self._h = crc32('')
      	E       TypeError: a bytes-like object is required, not 'str'
      
      	util.py:208: TypeError
      
      Based on patch by Jérome Perrin.
      1418c86f
    • Kirill Smelkov's avatar
      *: Pass bytes - not unicode - literals to sha1() · a7eee284
      Kirill Smelkov authored
      	data = 'data1'
      
      	    def sha1(data):
      	        m = hashlib.sha1()
      	>       m.update(data)
      	E       TypeError: Unicode-objects must be encoded before hashing
      
      	zodbtools/util.py:38: TypeError
      
      Based on patch by Jérome Perrin.
      a7eee284
    • Kirill Smelkov's avatar
      util: Fix ashex for Python3 · 7a7370e6
      Kirill Smelkov authored
      	s = b'\x03\xc4\x85v\x00\x00\x00\x00'
      
      	    def ashex(s):
      	>       return s.encode('hex')
      	E       AttributeError: 'bytes' object has no attribute 'encode'
      
      	zodbtools/util.py:29: AttributeError
      
      s.encode('hex') used to work on Py2 but fails on Py3:
      
      	In [1]: s = "abc"
      
      	In [2]: b = b"def"
      
      	In [3]: s.encode('hex')
      	---------------------------------------------------------------------------
      	LookupError                               Traceback (most recent call last)
      	<ipython-input-3-75ae843597fe> in <module>()
      	----> 1 s.encode('hex')
      
      	LookupError: 'hex' is not a text encoding; use codecs.encode() to handle arbitrary codecs
      
      	In [4]: b.encode('hex')
      	---------------------------------------------------------------------------
      	AttributeError                            Traceback (most recent call last)
      	<ipython-input-4-ec2fccff20bc> in <module>()
      	----> 1 b.encode('hex')
      
      	AttributeError: 'bytes' object has no attribute 'encode'
      
      	In [5]: import codecs
      
      	In [6]: codecs.encode(b, 'hex')
      	Out[6]: b'646566'
      
      	In [7]: codecs.encode(s, 'hex')
      	---------------------------------------------------------------------------
      	TypeError                                 Traceback (most recent call last)
      	/usr/lib/python3.7/encodings/hex_codec.py in hex_encode(input, errors)
      	     14     assert errors == 'strict'
      	---> 15     return (binascii.b2a_hex(input), len(input))
      	     16
      
      	TypeError: a bytes-like object is required, not 'str'
      
      	The above exception was the direct cause of the following exception:
      
      	TypeError                                 Traceback (most recent call last)
      	<ipython-input-7-7fcb16cead4f> in <module>()
      	----> 1 codecs.encode(s, 'hex')
      
      	TypeError: encoding with 'hex' codec failed (TypeError: a bytes-like object is required, not 'str')
      
      After the patch it works with bytes and raises for str.
      Fromhex does not need to be changed - it already uses codecs.decode way as
      originally added in dd959b28 (zodbdump += DumpReader - to read/parse zodbdump
      stream).
      
      Based on patch by Jérome Perrin.
      7a7370e6
    • Kirill Smelkov's avatar
      *: cStringIO.StringIO -> io.BytesIO · 62b21d01
      Kirill Smelkov authored
      There is no cStringIO on Python3:
      
      	test_dump.py:26: in <module>
      	    from cStringIO import StringIO
      	E   ModuleNotFoundError: No module named 'cStringIO'
      
      Based on patch by Jérome Perrin.
      62b21d01
    • Jérome Perrin's avatar
      zodb: rework command driver for python3 compatibility · 00a534ef
      Jérome Perrin authored
      This makes zodb command driver tests added in the previous patch to pass
      on both python2 and python3.
      00a534ef
    • Jérome Perrin's avatar
      test: add a test for zodb commmad and help driver · 2d94ae9d
      Jérome Perrin authored
      ----
      
      kirr: factor running `zodb ...` into zodbrun + add test for `zodb -h`.
      
      Added test currently passes on py2, but fails on py3:
      
      	out = <_io.TextIOWrapper encoding='UTF-8'>
      
      	    def usage(out):
      	        print("""\
      	    Zodb is a tool for managing ZODB databases.
      
      	    Usage:
      
      	        zodb command [arguments]
      
      	    The commands are:
      	    """, file=out)
      
      	        cmdv = command_dict.keys()
      	>       cmdv.sort()
      	E       AttributeError: 'dict_keys' object has no attribute 'sort'
      
      	zodbtools/zodb.py:55: AttributeError
      
      It will be fixed in the next patch.
      2d94ae9d
  12. 07 Mar, 2019 1 commit
  13. 31 Jan, 2019 6 commits
  14. 30 Jan, 2019 3 commits
  15. 11 Jan, 2019 2 commits