zodbtools:d6bde57c96b162094fa9e36a7a9c127899b35c7f commitshttps://lab.nexedi.com/kirr/zodbtools/-/commits/d6bde57c96b162094fa9e36a7a9c127899b35c7f2019-01-31T14:02:18+03:00https://lab.nexedi.com/kirr/zodbtools/-/commit/d6bde57c96b162094fa9e36a7a9c127899b35c7fzodbdump: use r-strings for regexp2019-01-31T14:02:18+03:00Jérome Perrinjerome@nexedi.com
this silents a warning about \w being unknown escape sequence
----
kirr: preserved _obj_re definition to be on 1 line.https://lab.nexedi.com/kirr/zodbtools/-/commit/9697b0afd4fca9b2086059aec4ed71b66dcf7c5etox: fix outdated comment2019-01-31T14:00:44+03:00Jérome Perrinjerome@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/c50bfb00883a0beba7c6b522c539552d034cdc06tox: run tests against ZODB #1832019-01-31T14:00:20+03:00Jérome Perrinjerome@nexedi.com
until <a href="https://github.com/zopefoundation/ZODB/pull/183" rel="nofollow noreferrer noopener" target="_blank">https://github.com/zopefoundation/ZODB/pull/183</a> gets merged, let's
run also the tests for this, since we have support for this extension.https://lab.nexedi.com/kirr/zodbtools/-/commit/eaa3aec7a16b47b22770a28bcc3fda8cfa275250tox: run tests for python 32019-01-31T13:57:01+03:00Jérome Perrinjerome@nexedi.com
also simplify a bit definition as ZODB is common in all versions
----
kirr:
- cover only last 2 py3 releases: 3.6 and 3.7 for now (3.8 is not yet
released)
- separate ZODB3 as it supports only python2.
Py3 tests are failing for now and we'll be getting them to pass
incrementally - step by step.https://lab.nexedi.com/kirr/zodbtools/-/commit/4037002c2c13f384c23f7310ded34d26bf18dd36Support absolute and relative time in tidrange2019-01-30T10:16:52+03:00Jérome Perrinjerome@nexedi.com
Using dateparser to support absolute and relative dates in natural language.
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a>
/reviewed-on <a href="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/8" data-original="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/8" data-link="false" data-link-reference="true" data-project="462" data-merge-request="2664" data-project-path="nexedi/zodbtools" data-iid="8" data-mr-title="Support absolute and relative time in tidrange" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/zodbtools!8</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/81b566c44a03b25d760f0ae69514bd2c8e836c86tidrange: supports giving range as time2019-01-30T02:39:44+01:00Jérome Perrinjerome@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/3e35453c948d5d57a4a637001788bbea977ccceatest: test for tidrange with tids2019-01-30T02:39:14+01:00Jérome Perrinjerome@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/f0399d50cfed5d2dfc17154e115f7d6b06620a83zodbtools v0.0.0.dev72019-01-11T17:54:36+03:00Kirill Smelkovkirr@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/25aeda62d854d326585e2c1a49fc82ca20753eabTest coverage for ZODB{3,4,5}2019-01-11T09:12:57+01:00Kirill Smelkovkirr@nexedi.com
Fixes to get all tests passing with all ZODB versions we care about.
Organize testing coverage for all cases via tox, similarly to wendelin.core .
/reviewed-on <a href="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/11" data-original="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/11" data-link="false" data-link-reference="true" data-project="462" data-merge-request="2667" data-project-path="nexedi/zodbtools" data-iid="11" data-mr-title="Test coverage for ZODB{3,4,5}" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/zodbtools!11</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/8ff7020c5523ecda69a15e2f40c80ea28735559eTest coverage for ZODB{3,4,5}2019-01-10T14:44:24+03:00Kirill Smelkovkirr@nexedi.com
Use tox to test with all kinds of ZODB.
With preceding 3 patches tests pass with all versions of upstream ZODB.
TODO: test coverage for both py2 and py3.https://lab.nexedi.com/kirr/zodbtools/-/commit/7a94e312bc08439ae6bc2bfba693b9046db1ef5dzodbcommit: Fix it for ZODB3 and ZODB42019-01-10T14:44:24+03:00Kirill Smelkovkirr@nexedi.com
maxtid is in ZODB.utils starting only from ZODB5.
ZODB{3,4} want txn._extension, while ZODB5 deprecate it in favour of
txn.extension.https://lab.nexedi.com/kirr/zodbtools/-/commit/0e5d2f81e7dafcb6bcbe79d39d3ce51db1d5acd8zodbdump: Fix it for ZODB3 and ZODB42019-01-10T14:44:24+03:00Kirill Smelkovkirr@nexedi.com
IStorageTransactionMetaData is ZODB5-only interface. Bug introduced in
<a href="/kirr/zodbtools/-/commit/dd959b28c4835e77cecb7549e41a6f21a20f4e30" data-original="dd959b28" data-link="false" data-link-reference="false" data-project="461" data-commit="dd959b28c4835e77cecb7549e41a6f21a20f4e30" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="zodbdump += DumpReader - to read/parse zodbdump stream" class="gfm gfm-commit has-tooltip">dd959b28</a> (zodbdump += DumpReader - to read/parse zodbdump stream).https://lab.nexedi.com/kirr/zodbtools/-/commit/425e66565b23883d054dece4a86219802c2c3158dump, commit: Test for both non-empty and empty transaction extensions2019-01-10T14:44:04+03:00Kirill Smelkovkirr@nexedi.com
Currently we exercise zodbdump and zodbcommit+zodbdump with non-empty
extensions, which works if ZODB is patched for txn.extension_bytes
support, but fails on pristine ZODB.
Support for txn.extension_bytes cannot get into upstream ZODB for
more than a year:
<a href="https://github.com/zopefoundation/ZODB/pull/183" rel="nofollow noreferrer noopener" target="_blank">https://github.com/zopefoundation/ZODB/pull/183</a>
<a href="https://github.com/zopefoundation/ZODB/pull/207" rel="nofollow noreferrer noopener" target="_blank">https://github.com/zopefoundation/ZODB/pull/207</a>
and even if it somehow will make it, it will likely be only in ZODB5,
while we still care to support ZODB4 and ZODB3.
Skipping zodbdump / zodbcommit tests, if a ZODB does not have
txn.extension_bytes support, would result in significant reduction of
zodbtools test coverage, because practically that is the current
situation with all upstream ZODB{3,4,5}. Dropping test coverage for
non-empty extensions is neither a good option.
For those reason, let's rework the tests and test both zodbdump and
zodbcommit with two scenarios:
1. on a test database where transactions extensions are always empty.
This should work on all ZODB irregardless of whether
txn.extension_bytes patch is there or not.
2. on a test database where transactions extensions are present.
This should work if ZODB has txn.extension_bytes support, but if not,
we can mark this case as xfail, since the failure is expected.
This way we make the testsuite pass irregardless of whether
txn.extension_bytes support is there, and we don't abandon dump/commit
testing coverage.
/helped-by Jérome Perrin <jerome@nexedi.com>https://lab.nexedi.com/kirr/zodbtools/-/commit/8c76eae251cc6c4c705c95abb1e13ec4262c90a3Fix zodb analyze with empty reports2019-01-09T03:41:47+01:00Jérome Perrinjerome@nexedi.com
Fix for this kind of errors:
```
(env)$ zodb analyze demo.fs ffffffffffffffff..
# ø
Processed 0 records in 0 transactions
Traceback (most recent call last):
File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/env/bin/zodb", line 11, in <module>
load_entry_point('zodbtools', 'console_scripts', 'zodb')()
File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodb.py", line 130, in main
return command_module.main(argv)
File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 305, in main
report(analyze(path, use_dbm, delta_fs, tidmin, tidmax), csv)
File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 102, in report
print "Average record size is %7.2f bytes" % (rep.DBYTES * 1.0 / rep.OIDS)
ZeroDivisionError: float division by zero
```
and also small fixes for python3 compatibility
/reviewed-on <a href="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/9" data-original="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/9" data-link="false" data-link-reference="true" data-project="462" data-merge-request="2665" data-project-path="nexedi/zodbtools" data-iid="9" data-mr-title="Fix zodb analyze with empty reports" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/zodbtools!9</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/d37746c6c321c9dc03eb67a30e25b1ad36c23a14analyze: always display processed tid range (even ø)2019-01-09T03:30:30+01:00Jérome Perrinjerome@nexedi.com
To keep a consistent output.https://lab.nexedi.com/kirr/zodbtools/-/commit/5e2ed5e7d5ebf5c53e42af7b89fd548becc1f116analyze: explicitly use gdbm as dbm2019-01-08T01:33:04+01:00Jérome Perrinjerome@nexedi.com
and use six.moves for python3 compatibility.
Previously we were using "anydbm" which selects dbhash, gdbm or dbm, but
opening the db with the f flag that's only valid for gdm, so de-facto we
were supporting only gdbm.https://lab.nexedi.com/kirr/zodbtools/-/commit/79aa0c4599001611bc18305c3d4aeddbb826d195analyze: don't introduce useless variable2019-01-08T01:32:11+01:00Jérome Perrinjerome@nexedi.com
this also solves the following error on python3:
AttributeError: 'dict_keys' object has no attribute 'sort'https://lab.nexedi.com/kirr/zodbtools/-/commit/5585361543271d955c18331b27de1b41cb04412eanalyze: minor syntax tweaks for python3 compatibility2019-01-08T01:31:36+01:00Jérome Perrinjerome@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/7d24147b670b4eb8ce7ab262b61e38d1f3952a41analyze: use print function for python3 compatibility2019-01-08T01:31:36+01:00Jérome Perrinjerome@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/b4824ad5cb756745bc74c2047dba17463013adaeanalyze: fix ZeroDivisionErrors when report is empty2019-01-08T01:31:36+01:00Jérome Perrinjerome@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/474a05591b756b08d55d39e83c1a37beaf711893test: minimal test for zodb analyze command2019-01-08T01:19:53+01:00Jérome Perrinjerome@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/7c5bb0b525edf2c911fde4a8a96d5272f975628csetup: don't declare python3 compatbility yet2019-01-07T17:10:22+03:00Jérome Perrinjerome@nexedi.com
/reviewed-on <a href="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/10" data-original="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/10" data-link="false" data-link-reference="true" data-project="462" data-merge-request="2666" data-project-path="nexedi/zodbtools" data-iid="10" data-mr-title="setup: don't declare python3 compatibility yet" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/zodbtools!10</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/98d84b910a4b25138f4ab955e99a803203b503a8zodbtools v0.0.0.dev62018-12-30T13:31:59+03:00Kirill Smelkovkirr@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/f7eff5fec20171ec5378cef80c16903181393126*: Refer to `zodb help tidrange` about how history range should be specified2018-12-30T10:15:46+01:00Kirill Smelkovkirr@nexedi.com
- add help tidrange topic.
- change all commands to refer to it.
- add TODO to parse tid from absolute and relative dates (e.g.
1.month.ago, similarly to how git can do). Dateparser
<a href="https://dateparser.readthedocs.io/" rel="nofollow noreferrer noopener" target="_blank">https://dateparser.readthedocs.io/</a>
will probably be of help here.
/reviewed-on <a href="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/7" data-original="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/7" data-link="false" data-link-reference="true" data-project="462" data-merge-request="2656" data-project-path="nexedi/zodbtools" data-iid="7" data-mr-title="zodbanalyze: Teach it to work with any ZODB storage, and analyze a particular range of history" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/zodbtools!7</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/7ad9e1dfce3e944635436c163c1089d482a18a1czodbanalyze: Add support to analyze a particular range of transaction2018-12-30T10:15:46+01:00Kirill Smelkovkirr@nexedi.com
Currently zodbanalyze analyzes whole storage. However it becomes
non practical to make a full zodbanalyze run on whole storage because
usually there are many transactions and objects and the time to run full
zodbanalyze is huge.
However, similarly to zodbdump, we can teach zodbanalyze to analyze a
particular range of transactions. This should help to analyze a range of
changes for e.g. yesterday, or for last week or similar.
/reviewed-on <a href="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/7" data-original="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/7" data-link="false" data-link-reference="true" data-project="462" data-merge-request="2656" data-project-path="nexedi/zodbtools" data-iid="7" data-mr-title="zodbanalyze: Teach it to work with any ZODB storage, and analyze a particular range of history" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/zodbtools!7</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/3ce22f28a69f82f45f9704adf83454a2438d7109zodbanalyze: Teach it to work with any ZODB storage2018-12-30T10:15:46+01:00Kirill Smelkovkirr@nexedi.com
Analyze uses regular ZODB storage API: .iterator() & friends. This way
it should be possible apply it not only to FileStorage, but to other
type of storages as well - for example to NEO and ZEO.
Use zodbtools.util.storageFromURL to open a storage by knowing its URL.
Preserve support to directly apply zodbanalyze to FileStorage deltas.
/reviewed-on <a href="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/7" data-original="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/7" data-link="false" data-link-reference="true" data-project="462" data-merge-request="2656" data-project-path="nexedi/zodbtools" data-iid="7" data-mr-title="zodbanalyze: Teach it to work with any ZODB storage, and analyze a particular range of history" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/zodbtools!7</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/9dbe70f32f2975873c943a0cef5f9e215e2e493f*: Don't forget to close opened ZODB storage2018-12-17T08:50:45+01:00Kirill Smelkovkirr@nexedi.com
Storages need to be closed to indicate a clean access end. If a storage is
not closed cleanly it might require to spend time and resources on next
open. For example FileStorage might need to recompute the index.
/reviewed-on <a href="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/6" data-original="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/6" data-link="false" data-link-reference="true" data-project="462" data-merge-request="2613" data-project-path="nexedi/zodbtools" data-iid="6" data-mr-title="*: Don't forget to close opened ZODB storage" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/zodbtools!6</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/220d1121ed253d5608f35cefceb098707f4e54bczodbtools v0.0.0.dev52018-12-13T18:13:43+03:00Kirill Smelkovkirr@nexedi.comhttps://lab.nexedi.com/kirr/zodbtools/-/commit/dc60b22b2ff55368a31b5a9a85579f619f30aa94Don't forget to include testdata files into dist tarball2018-12-13T18:13:32+03:00Kirill Smelkovkirr@nexedi.com
This should be in <a href="/nexedi/zodbtools/-/commit/7f0bbf7e6dadbea4806d8940f5d62a480204c41f" data-original="7f0bbf7e" data-link="false" data-link-reference="false" data-project="462" data-commit="7f0bbf7e6dadbea4806d8940f5d62a480204c41f" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="zodbdump: Add golden test" class="gfm gfm-commit has-tooltip">7f0bbf7e</a> (zodbdump: Add golden test).https://lab.nexedi.com/kirr/zodbtools/-/commit/960c5e1778dbfbd3f52360c929df2fd5abeb1972zodbcommit - Tool to commit new transaction into ZODB2018-12-13T18:00:13+03:00Kirill Smelkovkirr@nexedi.com
Zodbcommit reads transaction description from stdin and commits read data into
ZODB. The transaction to be committed is read in zodbdump format, but without
first 'txn' header line. For example:
user "author"
description "change 123"
extension ""
obj 0000000000000001 4 null:00
ZZZZ
This tool could be useful for testing and for low-level database
maintenance. Please see zodbcommit.py docstring for more details.https://lab.nexedi.com/kirr/zodbtools/-/commit/dd959b28c4835e77cecb7549e41a6f21a20f4e30zodbdump += DumpReader - to read/parse zodbdump stream2018-12-13T17:54:37+03:00Kirill Smelkovkirr@nexedi.com
We will likely need this reader for `zodb restore` in the future.
We will also use this reader for `zodb commit` in the next patch.
pygolang dependency v↑ becuase we use recently introduced
golang.strconv to unquote user/desc/extension strings.
Python2 works. Python3 support is only minimal and incomplete.https://lab.nexedi.com/kirr/zodbtools/-/commit/e973d519a9847f87e673671e62477bea2b52d2ffutil += hashRegistry, null, adler32, crc32 hashers2018-12-13T17:41:02+03:00Kirill Smelkovkirr@nexedi.com
hashRegistry will be needed for zodbdump reader to create a hasher by
its name. Since std hashlib does not have adler32/crc32 nor null - let's
add our own implementation for those hashers too.
The code is based on
<a href="https://lab.nexedi.com/kirr/neo/commit/a60c472c#d1c7bd5aa21a79d572554358240bc7ab061b6bf5_0_31" data-original="https://lab.nexedi.com/kirr/neo/commit/a60c472c#d1c7bd5aa21a79d572554358240bc7ab061b6bf5_0_31" data-link="false" data-link-reference="true" data-project="73" data-commit="a60c472c76da315f918afeb06f994c8610cf7c84" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="go/neo/t/neotest: CPU information & benchmarks" class="gfm gfm-commit has-tooltip">kirr/neo@a60c472c</a>, and
<a href="https://lab.nexedi.com/kirr/neo/commit/3f578560#4ecae6b34b98e72445132117149bbbc811693ff0_0_36" data-original="https://lab.nexedi.com/kirr/neo/commit/3f578560#4ecae6b34b98e72445132117149bbbc811693ff0_0_36" data-link="false" data-link-reference="true" data-project="73" data-commit="3f578560c49222cf4b0f8ba2b408d3db9c94444d" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="go/neo/t/neotest: ZODB benchmarks" class="gfm gfm-commit has-tooltip">kirr/neo@3f578560</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/624aeb096d49ea09e5c48e397ef68b37cb97e62azodbdump: Always end a transaction with LF2018-12-12T18:40:34+03:00Kirill Smelkovkirr@nexedi.com
Before now we were emitting extra LF only in between transactions as a
separator. However the dump format states LF always goes after
transaction, and there is a reason for it:
without LF in the end, it becomes ambiguous at EOF - whether it
is a proper transaction end, or the transaction was cut.
So avoid the ambiguity by always emitting trailing LF after transaction
record.https://lab.nexedi.com/kirr/zodbtools/-/commit/b11634494ac93e856ac9a0b4601fbc3c21f6aaefzodbdump: Switch to using qq from pygolang2018-07-02T20:09:46+03:00Kirill Smelkovkirr@nexedi.com
I originally added escapeqq as part of <a href="/kirr/zodbtools/-/commit/75c03368907c20f90491d7ffa7a00f221e970ce8" data-original="75c03368" data-link="false" data-link-reference="false" data-project="461" data-commit="75c03368907c20f90491d7ffa7a00f221e970ce8" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="zodbdump: Start to stabilize output format" class="gfm gfm-commit has-tooltip">75c03368</a> (zodbdump: Start to
stabilize output format) with the task for this utility to quote string
into valid "..." string always quoted with ".
This utility was later copied to pygolang:
<a href="https://lab.nexedi.com/kirr/pygolang/commit/afa46cf5#aafe23bd3b5a060d8afd59367774e219700c7064_0_22" data-original="https://lab.nexedi.com/kirr/pygolang/commit/afa46cf5#aafe23bd3b5a060d8afd59367774e219700c7064_0_22" data-link="false" data-link-reference="true" data-project="1158" data-commit="afa46cf539037ff343ad0c12ce6549420f1d1cb0" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Turn pygopath into full pygolang" class="gfm gfm-commit has-tooltip">pygolang@afa46cf5</a>
and then further improved there to work under both Python2 and Python3
and to not escape printable UTF-8 characters:
<a href="https://lab.nexedi.com/kirr/pygolang/commit/02dddb97" data-original="https://lab.nexedi.com/kirr/pygolang/commit/02dddb97" data-link="false" data-link-reference="true" data-project="1158" data-commit="02dddb97eea28e8192da6de0ee3d56b00da9542f" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="gcompat: Teach qq to accept both str and unicode + emit printable UTF-8 as is" class="gfm gfm-commit has-tooltip">pygolang@02dddb97</a>
So stop the duplication and simply switch to the better version.https://lab.nexedi.com/kirr/zodbtools/-/commit/2801fae9afbf57a6a83e41289a5c66fd8feb5a2fzodbdump: Start to stabilize format + add test2018-05-11T20:02:05+02:00Kirill Smelkovkirr@nexedi.com
We start to stabilize output format of `zodb dump`. It is actually now robust and the only thing I would contemplate to potentially change is to also cover transaction metadata by hash checksum. So please take a look at updated format (details in patch 1) to provide feedback because it is likely close to what it will be in its final form.
We also add a program to generate test database which uses various fancy ZODB features and check `zodb dump` output on it to golden one (patch 3).
To be able to dump transaction metadata in raw form ZODB is patched a bit:
<a href="https://github.com/zopefoundation/ZODB/pull/183" rel="nofollow noreferrer noopener" target="_blank">https://github.com/zopefoundation/ZODB/pull/183</a>
and we try to detect whether appropriate support is there at runtime and if yes use it to streamline obtaining transaction extension as raw (patch 2).
Pleae see patch 1 (second half of `zodbdump.py` about what has to be taken on without such support and that it still can't work fully reliably).
/cc <a href="/nexedi" data-group="7" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="nexedi">@nexedi</a>
/reviewed-on <a href="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/3" data-original="https://lab.nexedi.com/nexedi/zodbtools/merge_requests/3" data-link="false" data-link-reference="true" data-project="462" data-merge-request="1555" data-project-path="nexedi/zodbtools" data-iid="3" data-mr-title="zodbdump: Start to stabilize format + add test" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/zodbtools!3</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/7f0bbf7e6dadbea4806d8940f5d62a480204c41fzodbdump: Add golden test2018-05-11T21:01:13+03:00Kirill Smelkovkirr@nexedi.com
We add a program to generate a test database with all fancy features and
then check `zodb dump` output on it to golden on.
The test database itself is commited to git because we want to make sure
zodbdump works ok for particular exact data and because transaction
extension dict is potentially saved differently on various runs.
Quoting
<a href="https://docs.python.org/2.7/library/stdtypes.html#dict.items" rel="nofollow noreferrer noopener" target="_blank">https://docs.python.org/2.7/library/stdtypes.html#dict.items</a> and
<a href="https://docs.python.org/3.7/library/stdtypes.html#dictionary-view-objects" rel="nofollow noreferrer noopener" target="_blank">https://docs.python.org/3.7/library/stdtypes.html#dictionary-view-objects</a>
""" CPython implementation detail: Keys and values are listed in an arbitrary
order which is non-random, varies across Python implementations, and depends
on the dictionary’s history of insertions and deletions. """
This way on test/gen_testdata.py changes it has to be run manually, and
then the output result of the run committed back together with
gen_testdata.py changes.https://lab.nexedi.com/kirr/zodbtools/-/commit/33230940bc493ba5e716106a77d0fd70c1a0d303zodbdump: Retrieve transaction metadata as raw from ZODB if storage support i...2018-05-11T21:00:42+03:00Kirill Smelkovkirr@nexedi.com
This patch test at runtime whether used storage can provide transaction
metadata in raw form and if so used thi form directly without going
through long fragile way of stable pickling extension back to bytes.
Corresponding ZODB support is at
<a href="https://github.com/zopefoundation/ZODB/pull/183" rel="nofollow noreferrer noopener" target="_blank">https://github.com/zopefoundation/ZODB/pull/183</a>https://lab.nexedi.com/kirr/zodbtools/-/commit/75c03368907c20f90491d7ffa7a00f221e970ce8zodbdump: Start to stabilize output format2017-11-02T19:44:02+03:00Kirill Smelkovkirr@nexedi.com
Since zodbdump start (<a href="/nexedi/zodbtools/-/commit/c0a6299f55dd2f9e742a96beba1bff425693b8ce" data-original="c0a6299f" data-link="false" data-link-reference="false" data-project="462" data-commit="c0a6299f55dd2f9e742a96beba1bff425693b8ce" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="zodbdump - Tool to dump content of a ZODB database (draft)" class="gfm gfm-commit has-tooltip">c0a6299f</a> "zodbdump - Tool to dump content of a
ZODB database (draft)") and up till now zodbdump output format was not
good. For example user and description transaction properties were
output without proper quoting, which in situation when there would be
fancy characters in there would break the output.
So start the format stabilization:
- user and description are output as quoted, so now they are guaranteed
to be on one line. The quoting character is always " (instead of e.g.
smartly quoting either by ' or " as python does) for easier
compatibility with ZODB implementations in other languages.
- transaction extension is now printed as raw bytes, not as dict.
The idea here is that `zodb dump`
* should perform dump of raw data as stored inside ZODB so that later
`zodb restore` could restore the database identically to the same state.
* we should dump raw data instead of unpickled ones because generally
on-disk extension's format can be any raw bytes and this information
should be preserved.
- transaction status is now also output as quoted to preserve line
breakage on fancy status codes.
- it is documented that sha1 is not the only allowed hash function that
might be used.
- in hashonly mode we add trailing " -" to obj string so that it is
possible to distinguish outputs of `zodb dump` and `zodb dump -hashonly`
without knowing a-priory the way it was produced.
The reason to do so is that it would be not good to e.g. by accident
feed hashonly output to (future) `zodb restore`, which, without having
a way to see it should not consume object data would read following
transaction information as raw object data with confusing later
errors (and a small chance to restore completely different database
without reporting error at all).
Because ZODB iteration API gives us already unpickled extension and only
that, for now to dump it as raw we get a long road to pickle it back
also caring to try to pickle in stable order.
Hopefully this will be only a fallback because of
<a href="https://github.com/zopefoundation/ZODB/pull/183" rel="nofollow noreferrer noopener" target="_blank">https://github.com/zopefoundation/ZODB/pull/183</a>
and next zodbtools patch.
~~~~
For testing purposes (currently only quoting function unit test) py.test
usage is introduced.
The code is also generally polished here and there.https://lab.nexedi.com/kirr/zodbtools/-/commit/79cf177aa0b20c078182a5efaeba3ac4cea9be1bRelicense to GPLv3+ with wide exception for all Free Software / Open Source...2017-10-24T21:08:42+03:00Kirill Smelkovkirr@nexedi.comRelicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options.
Nexedi stack is licensed under Free Software licenses with various exceptions
that cover three business cases:
- Free Software
- Proprietary Software
- Rebranding
As long as one intends to develop Free Software based on Nexedi stack, no
license cost is involved. Developing proprietary software based on Nexedi stack
may require a proprietary exception license. Rebranding Nexedi stack is
prohibited unless rebranding license is acquired.
Through this licensing approach, Nexedi expects to encourage Free Software
development without restrictions and at the same time create a framework for
proprietary software to contribute to the long term sustainability of the
Nexedi stack.
Please see <a href="https://www.nexedi.com/licensing" rel="nofollow noreferrer noopener" target="_blank">https://www.nexedi.com/licensing</a> for details, rationale and options.
https://lab.nexedi.com/kirr/zodbtools/-/commit/f28417ff32acf8b10b64a20c7055630f82b0f6400.0.0.dev42017-04-05T20:49:50+03:00Kirill Smelkovkirr@nexedi.com