- 17 Dec, 2018 1 commit
-
-
Kirill Smelkov authored
Storages need to be closed to indicate a clean access end. If a storage is not closed cleanly it might require to spend time and resources on next open. For example FileStorage might need to recompute the index. /reviewed-on nexedi/zodbtools!6
-
- 13 Dec, 2018 5 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
This should be in 7f0bbf7e (zodbdump: Add golden test).
-
Kirill Smelkov authored
Zodbcommit reads transaction description from stdin and commits read data into ZODB. The transaction to be committed is read in zodbdump format, but without first 'txn' header line. For example: user "author" description "change 123" extension "" obj 0000000000000001 4 null:00 ZZZZ This tool could be useful for testing and for low-level database maintenance. Please see zodbcommit.py docstring for more details.
-
Kirill Smelkov authored
We will likely need this reader for `zodb restore` in the future. We will also use this reader for `zodb commit` in the next patch. pygolang dependency v↑ becuase we use recently introduced golang.strconv to unquote user/desc/extension strings. Python2 works. Python3 support is only minimal and incomplete.
-
Kirill Smelkov authored
hashRegistry will be needed for zodbdump reader to create a hasher by its name. Since std hashlib does not have adler32/crc32 nor null - let's add our own implementation for those hashers too. The code is based on kirr/neo@a60c472c, and kirr/neo@3f578560
-
- 12 Dec, 2018 1 commit
-
-
Kirill Smelkov authored
Before now we were emitting extra LF only in between transactions as a separator. However the dump format states LF always goes after transaction, and there is a reason for it: without LF in the end, it becomes ambiguous at EOF - whether it is a proper transaction end, or the transaction was cut. So avoid the ambiguity by always emitting trailing LF after transaction record.
-
- 02 Jul, 2018 1 commit
-
-
Kirill Smelkov authored
I originally added escapeqq as part of 75c03368 (zodbdump: Start to stabilize output format) with the task for this utility to quote string into valid "..." string always quoted with ". This utility was later copied to pygolang: kirr/pygolang@afa46cf5 and then further improved there to work under both Python2 and Python3 and to not escape printable UTF-8 characters: kirr/pygolang@02dddb97 So stop the duplication and simply switch to the better version.
-
- 11 May, 2018 3 commits
-
-
Kirill Smelkov authored
We start to stabilize output format of `zodb dump`. It is actually now robust and the only thing I would contemplate to potentially change is to also cover transaction metadata by hash checksum. So please take a look at updated format (details in patch 1) to provide feedback because it is likely close to what it will be in its final form. We also add a program to generate test database which uses various fancy ZODB features and check `zodb dump` output on it to golden one (patch 3). To be able to dump transaction metadata in raw form ZODB is patched a bit: https://github.com/zopefoundation/ZODB/pull/183 and we try to detect whether appropriate support is there at runtime and if yes use it to streamline obtaining transaction extension as raw (patch 2). Pleae see patch 1 (second half of `zodbdump.py` about what has to be taken on without such support and that it still can't work fully reliably). /cc @nexedi /reviewed-on !3
-
Kirill Smelkov authored
We add a program to generate a test database with all fancy features and then check `zodb dump` output on it to golden on. The test database itself is commited to git because we want to make sure zodbdump works ok for particular exact data and because transaction extension dict is potentially saved differently on various runs. Quoting https://docs.python.org/2.7/library/stdtypes.html#dict.items and https://docs.python.org/3.7/library/stdtypes.html#dictionary-view-objects """ CPython implementation detail: Keys and values are listed in an arbitrary order which is non-random, varies across Python implementations, and depends on the dictionary’s history of insertions and deletions. """ This way on test/gen_testdata.py changes it has to be run manually, and then the output result of the run committed back together with gen_testdata.py changes.
-
Kirill Smelkov authored
This patch test at runtime whether used storage can provide transaction metadata in raw form and if so used thi form directly without going through long fragile way of stable pickling extension back to bytes. Corresponding ZODB support is at https://github.com/zopefoundation/ZODB/pull/183
-
- 02 Nov, 2017 1 commit
-
-
Kirill Smelkov authored
Since zodbdump start (c0a6299f "zodbdump - Tool to dump content of a ZODB database (draft)") and up till now zodbdump output format was not good. For example user and description transaction properties were output without proper quoting, which in situation when there would be fancy characters in there would break the output. So start the format stabilization: - user and description are output as quoted, so now they are guaranteed to be on one line. The quoting character is always " (instead of e.g. smartly quoting either by ' or " as python does) for easier compatibility with ZODB implementations in other languages. - transaction extension is now printed as raw bytes, not as dict. The idea here is that `zodb dump` * should perform dump of raw data as stored inside ZODB so that later `zodb restore` could restore the database identically to the same state. * we should dump raw data instead of unpickled ones because generally on-disk extension's format can be any raw bytes and this information should be preserved. - transaction status is now also output as quoted to preserve line breakage on fancy status codes. - it is documented that sha1 is not the only allowed hash function that might be used. - in hashonly mode we add trailing " -" to obj string so that it is possible to distinguish outputs of `zodb dump` and `zodb dump -hashonly` without knowing a-priory the way it was produced. The reason to do so is that it would be not good to e.g. by accident feed hashonly output to (future) `zodb restore`, which, without having a way to see it should not consume object data would read following transaction information as raw object data with confusing later errors (and a small chance to restore completely different database without reporting error at all). Because ZODB iteration API gives us already unpickled extension and only that, for now to dump it as raw we get a long road to pickle it back also caring to try to pickle in stable order. Hopefully this will be only a fallback because of https://github.com/zopefoundation/ZODB/pull/183 and next zodbtools patch. ~~~~ For testing purposes (currently only quoting function unit test) py.test usage is introduced. The code is also generally polished here and there.
-
- 24 Oct, 2017 1 commit
-
-
Kirill Smelkov authored
Relicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options. Nexedi stack is licensed under Free Software licenses with various exceptions that cover three business cases: - Free Software - Proprietary Software - Rebranding As long as one intends to develop Free Software based on Nexedi stack, no license cost is involved. Developing proprietary software based on Nexedi stack may require a proprietary exception license. Rebranding Nexedi stack is prohibited unless rebranding license is acquired. Through this licensing approach, Nexedi expects to encourage Free Software development without restrictions and at the same time create a framework for proprietary software to contribute to the long term sustainability of the Nexedi stack. Please see https://www.nexedi.com/licensing for details, rationale and options.
-
- 05 Apr, 2017 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Hello up there. Recently with Ivan we needed a way to obtain last transaction ID of a ZODB storage for resiliency checking. For this `zodb info` utility to print general information about a ZODB database is introduced. Along the way, as it was already planned, all zodbtools utilities are now covered by only one `zodb` command which can invoke other subcommands and show help topics. I also used the occassion to switch how storages are specified from being ZConfig-file-always to be specified by URL e.g. neo://neo1@127.0.0.1:24573 zeo://... file://... without loosing generality because zconfig:// scheme is also supported. Please find more details about all changes in individal commit messages. Thanks for feedback, Kirill /cc @kazuhiko, @jm, @vpelletier, @jerome, @Tyagov, @klaus, @alain.takoudjou, @rafael /reviewed-on nexedi/zodbtools!2
-
- 04 Apr, 2017 4 commits
-
-
Kirill Smelkov authored
Either all general parameters at once: $ zodb info neo://neo1@127.0.0.1:24573 name=NEOStorage(neo1) size=0 last_tid=03be7484ddc7f6ee or one particular parameter: $ zodb info neo://neo1@127.0.0.1:24573 last_tid 03be7484ddc7f6ee
-
Kirill Smelkov authored
Most of our tools need only read access for working. However e.g. FileStorage, when opened in read-write mode, automatically creates database file and index. This way if database is opened in read-write mode a simple typo in path, e.g. to `zodb dump path` would lead to: - new database at path will be created - the dump will print nothing (empty database) - exit status will be 0 (ok) and no error will be reported. For this reason it is better tools declare access level they need so for read-only access request we can catch it with an error from storage. This, however, requires quite recent ZODB to work: https://github.com/zopefoundation/ZODB/pull/153 P.S. We don't want to force users to always specify read-only in URLs or zconf files because: - this is error prone - URL or zconf can be though as of file - when a program opens a file the program, not file, declares which type of access it wants. That's why access mode declaration has to be internal.
-
Kirill Smelkov authored
Previously for specifying a storage we were requiring users to create zconfig file and put information about storage there. This is not always convenient e.g. for quickly inspecting a file storage or a NEO instance here and there for which address and name is already at hand. So thanks to zodburi we can switch to specifying storages by URL without loosing generality as there is still zconfig:// schema which allows to configure storages zconfig way. P.S. for convenience we allow paths without schema to be treated as FileStorage (i.e. file:// schema is implicitly used). P.P.S. zodbanalyze is not affected (for now ?) as it currently works with FileStorage only. [1] http://docs.pylonsproject.org/projects/zodburi/
-
Kirill Smelkov authored
We already have 3 commands in zodbtools suite (zodbanalyze, zodbdump & zodbcmp) and this is going to grow. And it was already noted some time ago with TODO (in 66946b8d) that we need only one command driver to invoke everything. So do it: introduce `zodb` command which can invoke other subcommands and show general help or help for subcommand or a topic. The structure is modelled after `git` and `go` commands. Help topics are for now empty but we'll add one help topic in the next patch.
-
- 30 Mar, 2017 1 commit
-
-
Kirill Smelkov authored
- Nexedi adheres to GPLv3+ with wide exception for Free/Open-source software. Use this for our original work. - Zodbanalyze was initially developed by Zope corp & co. See d86d04dc (initial copy of ZODB3-3.9.7's ZODB/scripts/analyze.py.) ab17cf2d (Hook in Nexedi's zodbanalyze) So follow original Zope's licensing terms for zodbanalyze.
-
- 14 Mar, 2017 1 commit
-
-
Kirill Smelkov authored
The package was named zodbtools from start (see fd6ad1b9 "Start of zodbtools.git"). However in 66a03ae5 I've made a thinko naming some parts as zodbtool. Fix the confusion and name things uniformly as zodbtools. /reviewed-by TrustMe
-
- 17 Nov, 2016 16 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
zodbanalyze was added in 1e506a81 and ab17cf2d.
-
Kirill Smelkov authored
-
Kazuhiko Shiozaki authored
/reviewed-on nexedi/zodbtools!1 /see-also nexedi/slapos!116
-
Kirill Smelkov authored
This is enhanced version of ZODB original scripts/analyze.py . Nexedi version was originally developed as part of ERP5 here: https://lab.nexedi.com/nexedi/erp5/commits/master/erp5/util/zodbanalyze It was agreed to move zodbanalyze to zodbutils /see nexedi/slapos!116
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
* it is faster. * it does not require Products.
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
Kirill Smelkov authored
(where `python setup.py sdist` puts result tarballs)
-
-
- 16 Nov, 2016 3 commits
-
-
Kirill Smelkov authored
WARNING output format is not stable yet.
-
Kirill Smelkov authored
-
Kirill Smelkov authored
This is a tool to compare two ZODB databases in between tidmin..tidmax transaction range with default range being -∞..+∞ - (whole database). For comparision both databases are scanned at storage layer and every transaction content is compared bit-to-bit between the two. The program stops either at first difference found, or when whole requested transaction range is scanned with no difference detected. Database storages are specified in files with ZConfig-based storage definition, e.g. %import neo.client <NEOStorage> master_nodes ... name ... </NEOStorage> Please see nexedi/neoppod!4 for one of possible contexts. The tool is generic though and is not NEO-specific. It should be able to even check two different storages like ZEO & NEO, or FileStorage and NEO etc and thus can be handy. TODO tests
-