- 24 May, 2019 5 commits
-
-
Kirill Smelkov authored
data = 'data1' def sha1(data): m = hashlib.sha1() > m.update(data) E TypeError: Unicode-objects must be encoded before hashing zodbtools/util.py:38: TypeError Based on patch by Jérome Perrin.
-
Kirill Smelkov authored
s = b'\x03\xc4\x85v\x00\x00\x00\x00' def ashex(s): > return s.encode('hex') E AttributeError: 'bytes' object has no attribute 'encode' zodbtools/util.py:29: AttributeError s.encode('hex') used to work on Py2 but fails on Py3: In [1]: s = "abc" In [2]: b = b"def" In [3]: s.encode('hex') --------------------------------------------------------------------------- LookupError Traceback (most recent call last) <ipython-input-3-75ae843597fe> in <module>() ----> 1 s.encode('hex') LookupError: 'hex' is not a text encoding; use codecs.encode() to handle arbitrary codecs In [4]: b.encode('hex') --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-4-ec2fccff20bc> in <module>() ----> 1 b.encode('hex') AttributeError: 'bytes' object has no attribute 'encode' In [5]: import codecs In [6]: codecs.encode(b, 'hex') Out[6]: b'646566' In [7]: codecs.encode(s, 'hex') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /usr/lib/python3.7/encodings/hex_codec.py in hex_encode(input, errors) 14 assert errors == 'strict' ---> 15 return (binascii.b2a_hex(input), len(input)) 16 TypeError: a bytes-like object is required, not 'str' The above exception was the direct cause of the following exception: TypeError Traceback (most recent call last) <ipython-input-7-7fcb16cead4f> in <module>() ----> 1 codecs.encode(s, 'hex') TypeError: encoding with 'hex' codec failed (TypeError: a bytes-like object is required, not 'str') After the patch it works with bytes and raises for str. Fromhex does not need to be changed - it already uses codecs.decode way as originally added in dd959b28 (zodbdump += DumpReader - to read/parse zodbdump stream). Based on patch by Jérome Perrin.
-
Kirill Smelkov authored
There is no cStringIO on Python3: test_dump.py:26: in <module> from cStringIO import StringIO E ModuleNotFoundError: No module named 'cStringIO' Based on patch by Jérome Perrin.
-
Jérome Perrin authored
This makes zodb command driver tests added in the previous patch to pass on both python2 and python3.
-
Jérome Perrin authored
---- kirr: factor running `zodb ...` into zodbrun + add test for `zodb -h`. Added test currently passes on py2, but fails on py3: out = <_io.TextIOWrapper encoding='UTF-8'> def usage(out): print("""\ Zodb is a tool for managing ZODB databases. Usage: zodb command [arguments] The commands are: """, file=out) cmdv = command_dict.keys() > cmdv.sort() E AttributeError: 'dict_keys' object has no attribute 'sort' zodbtools/zodb.py:55: AttributeError It will be fixed in the next patch.
-
- 07 Mar, 2019 1 commit
-
-
Jérome Perrin authored
-
- 31 Jan, 2019 6 commits
-
-
Jérome Perrin authored
-
Jérome Perrin authored
---- kirr: use loggging as log and keep emitting warnings on one line.
-
Jérome Perrin authored
this silents a warning about \w being unknown escape sequence ---- kirr: preserved _obj_re definition to be on 1 line.
-
Jérome Perrin authored
-
Jérome Perrin authored
until https://github.com/zopefoundation/ZODB/pull/183 gets merged, let's run also the tests for this, since we have support for this extension.
-
Jérome Perrin authored
also simplify a bit definition as ZODB is common in all versions ---- kirr: - cover only last 2 py3 releases: 3.6 and 3.7 for now (3.8 is not yet released) - separate ZODB3 as it supports only python2. Py3 tests are failing for now and we'll be getting them to pass incrementally - step by step.
-
- 30 Jan, 2019 3 commits
-
-
Jérome Perrin authored
Using dateparser to support absolute and relative dates in natural language. /reviewed-by @kirr /reviewed-on !8
-
Jérome Perrin authored
-
Jérome Perrin authored
-
- 11 Jan, 2019 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Fixes to get all tests passing with all ZODB versions we care about. Organize testing coverage for all cases via tox, similarly to wendelin.core . /reviewed-on nexedi/zodbtools!11
-
- 10 Jan, 2019 4 commits
-
-
Kirill Smelkov authored
Use tox to test with all kinds of ZODB. With preceding 3 patches tests pass with all versions of upstream ZODB. TODO: test coverage for both py2 and py3.
-
Kirill Smelkov authored
maxtid is in ZODB.utils starting only from ZODB5. ZODB{3,4} want txn._extension, while ZODB5 deprecate it in favour of txn.extension.
-
Kirill Smelkov authored
IStorageTransactionMetaData is ZODB5-only interface. Bug introduced in dd959b28 (zodbdump += DumpReader - to read/parse zodbdump stream).
-
Kirill Smelkov authored
Currently we exercise zodbdump and zodbcommit+zodbdump with non-empty extensions, which works if ZODB is patched for txn.extension_bytes support, but fails on pristine ZODB. Support for txn.extension_bytes cannot get into upstream ZODB for more than a year: https://github.com/zopefoundation/ZODB/pull/183 https://github.com/zopefoundation/ZODB/pull/207 and even if it somehow will make it, it will likely be only in ZODB5, while we still care to support ZODB4 and ZODB3. Skipping zodbdump / zodbcommit tests, if a ZODB does not have txn.extension_bytes support, would result in significant reduction of zodbtools test coverage, because practically that is the current situation with all upstream ZODB{3,4,5}. Dropping test coverage for non-empty extensions is neither a good option. For those reason, let's rework the tests and test both zodbdump and zodbcommit with two scenarios: 1. on a test database where transactions extensions are always empty. This should work on all ZODB irregardless of whether txn.extension_bytes patch is there or not. 2. on a test database where transactions extensions are present. This should work if ZODB has txn.extension_bytes support, but if not, we can mark this case as xfail, since the failure is expected. This way we make the testsuite pass irregardless of whether txn.extension_bytes support is there, and we don't abandon dump/commit testing coverage. /helped-by Jérome Perrin <jerome@nexedi.com>
-
- 09 Jan, 2019 2 commits
-
-
Jérome Perrin authored
Fix for this kind of errors: ``` (env)$ zodb analyze demo.fs ffffffffffffffff.. # ø Processed 0 records in 0 transactions Traceback (most recent call last): File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/env/bin/zodb", line 11, in <module> load_entry_point('zodbtools', 'console_scripts', 'zodb')() File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodb.py", line 130, in main return command_module.main(argv) File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 305, in main report(analyze(path, use_dbm, delta_fs, tidmin, tidmax), csv) File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 102, in report print "Average record size is %7.2f bytes" % (rep.DBYTES * 1.0 / rep.OIDS) ZeroDivisionError: float division by zero ``` and also small fixes for python3 compatibility /reviewed-on !9
-
Jérome Perrin authored
To keep a consistent output.
-
- 08 Jan, 2019 6 commits
-
-
Jérome Perrin authored
and use six.moves for python3 compatibility. Previously we were using "anydbm" which selects dbhash, gdbm or dbm, but opening the db with the f flag that's only valid for gdm, so de-facto we were supporting only gdbm.
-
Jérome Perrin authored
this also solves the following error on python3: AttributeError: 'dict_keys' object has no attribute 'sort'
-
Jérome Perrin authored
-
Jérome Perrin authored
-
Jérome Perrin authored
-
Jérome Perrin authored
-
- 07 Jan, 2019 1 commit
-
-
Jérome Perrin authored
/reviewed-on nexedi/zodbtools!10
-
- 30 Dec, 2018 4 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
- add help tidrange topic. - change all commands to refer to it. - add TODO to parse tid from absolute and relative dates (e.g. 1.month.ago, similarly to how git can do). Dateparser https://dateparser.readthedocs.io/ will probably be of help here. /reviewed-on !7
-
Kirill Smelkov authored
Currently zodbanalyze analyzes whole storage. However it becomes non practical to make a full zodbanalyze run on whole storage because usually there are many transactions and objects and the time to run full zodbanalyze is huge. However, similarly to zodbdump, we can teach zodbanalyze to analyze a particular range of transactions. This should help to analyze a range of changes for e.g. yesterday, or for last week or similar. /reviewed-on nexedi/zodbtools!7
-
Kirill Smelkov authored
Analyze uses regular ZODB storage API: .iterator() & friends. This way it should be possible apply it not only to FileStorage, but to other type of storages as well - for example to NEO and ZEO. Use zodbtools.util.storageFromURL to open a storage by knowing its URL. Preserve support to directly apply zodbanalyze to FileStorage deltas. /reviewed-on nexedi/zodbtools!7
-
- 17 Dec, 2018 1 commit
-
-
Kirill Smelkov authored
Storages need to be closed to indicate a clean access end. If a storage is not closed cleanly it might require to spend time and resources on next open. For example FileStorage might need to recompute the index. /reviewed-on nexedi/zodbtools!6
-
- 13 Dec, 2018 5 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
This should be in 7f0bbf7e (zodbdump: Add golden test).
-
Kirill Smelkov authored
Zodbcommit reads transaction description from stdin and commits read data into ZODB. The transaction to be committed is read in zodbdump format, but without first 'txn' header line. For example: user "author" description "change 123" extension "" obj 0000000000000001 4 null:00 ZZZZ This tool could be useful for testing and for low-level database maintenance. Please see zodbcommit.py docstring for more details.
-
Kirill Smelkov authored
We will likely need this reader for `zodb restore` in the future. We will also use this reader for `zodb commit` in the next patch. pygolang dependency v↑ becuase we use recently introduced golang.strconv to unquote user/desc/extension strings. Python2 works. Python3 support is only minimal and incomplete.
-
Kirill Smelkov authored
hashRegistry will be needed for zodbdump reader to create a hasher by its name. Since std hashlib does not have adler32/crc32 nor null - let's add our own implementation for those hashers too. The code is based on kirr/neo@a60c472c, and kirr/neo@3f578560
-