Commit b9070f8f authored by Kirill Smelkov's avatar Kirill Smelkov

Merge branch 'y/restore' into x/catobj

* y/restore: (56 commits)
  zodbrestore - Tool to restore content of a ZODB database from zodbdump output
  zodbcommit: Prepare to compute current serial of an oid lazily
  zodbcommit: Don't forget to call tpc_abort on an error
  Drop support for ZODB3
  tox: Don't run tests agains ZODB+PR183 anymore
  Add way to run tests via nxdtest
  tidrange: test: Fix for py3
  *: dict.keys() returns sequence, not [] on py3
  *: Pass bytes literal into BytesIO
  zodbdump: Use bytes to emit its output
  *: Zodbdump format is semi text-binary: Mark it as such + handle zdump output as binary
  *: Don't use %r to print/report lines/bytes to outside
  zodbinfo: Provide "head" as command to query DB head; Turn "last_tid" into deprecated alias for head
  test/gen_testdata: Fix for ZODB5 > 5.5.1 + preserve database compatibility with ZODB3/py2
  tox: Don't duplicate setup.py on which for-tests dependencies we need
  zodbdump: Default out to stdout in binary mode
  *: s.decode('hex') -> fromhex(s)
  utils: Initialize hashers with bytes
  *: Pass bytes - not unicode - literals to sha1()
  util: Fix ashex for Python3
  ...
parents 6bac5920 198b8df4
/dist
/zodbtools.egg-info
/.tox/
# setup to run tests on Nexedi testing infrastructure.
# https://stack.nexedi.com/test_status
TestCase('pytest', ['python', '-m', 'pytest'], summaryf=PyTest.summary)
Zodbtools change history
========================
0.0.0.dev8 (2019-03-07)
-----------------------
- Support using absolute and relative time in tidrange. One example usage is:
``zodb analyze data.fs 2018-01-01T10:30:00Z..yesterday`` (commit__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/4037002c
- Python3 support progressed (`commit 1`__, 2__, 3__), but zodbtools does not
support python3 yet. The test suite was extended to run on python3 (commit__)
and also was extended to also run on ZODB with raw extensions from ongoing
pull request `#183`__ (commit__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/d6bde57c
__ https://lab.nexedi.com/nexedi/zodbtools/commit/f16ccfd4
__ https://lab.nexedi.com/nexedi/zodbtools/commit/b338d004
__ https://lab.nexedi.com/nexedi/zodbtools/commit/eaa3aec7
__ https://github.com/zopefoundation/ZODB/pull/183
__ https://lab.nexedi.com/nexedi/zodbtools/commit/c50bfb00
0.0.0.dev7 (2019-01-11)
-----------------------
- Fix zodbtools to work with all ZODB3, ZODB4 and ZODB5 (`commit 1`__, 2__,
3__, 4__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/425e6656
__ https://lab.nexedi.com/nexedi/zodbtools/commit/0e5d2f81
__ https://lab.nexedi.com/nexedi/zodbtools/commit/7a94e312
__ https://lab.nexedi.com/nexedi/zodbtools/commit/8ff7020c
- Fix `zodb analyze` for the case when history range is empty (`commit 1`__,
2__, 3__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/b4824ad5
__ https://lab.nexedi.com/nexedi/zodbtools/commit/d37746c6
__ https://lab.nexedi.com/nexedi/zodbtools/commit/474a0559
- Zodbtools is not yet Python3-ready (commit__), but we started to fix it
step-by-step (`commit 1`__, 2__, 3__, 4__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/7c5bb0b5
__ https://lab.nexedi.com/nexedi/zodbtools/commit/7d24147b
__ https://lab.nexedi.com/nexedi/zodbtools/commit/55853615
__ https://lab.nexedi.com/nexedi/zodbtools/commit/79aa0c45
__ https://lab.nexedi.com/nexedi/zodbtools/commit/5e2ed5e7
0.0.0.dev6 (2018-12-30)
-----------------------
- `zodb analyze` can now work with any ZODB storage and supports analyzing a
particular range of history (`commit 1`__, 2__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/3ce22f28
__ https://lab.nexedi.com/nexedi/zodbtools/commit/7ad9e1df
- Add help for specifying TID ranges (commit__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/f7eff5fe
- Always close opened storages (commit__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/9dbe70f3
0.0.0.dev5 (2018-12-13)
-----------------------
- Start to stabilize `zodb dump` format. The format is close to be stable now
and will likely be changed, if at all, only in minor ways (`commit 1`__, 2__,
3__, 4__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/75c03368
__ https://lab.nexedi.com/nexedi/zodbtools/commit/33230940
__ https://lab.nexedi.com/nexedi/zodbtools/commit/7f0bbf7e
__ https://lab.nexedi.com/nexedi/zodbtools/commit/624aeb09
- Add `DumpReader` - class to read/parse input in `zodbdump` format (commit__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/dd959b28
- Add `zodb commit` subcommand to commit new transaction into ZODB (commit__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/960c5e17
0.0.0.dev4 (2017-04-05)
-----------------------
- Clarify licensing (`commit 1`__, 2__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/9e4305b8
__ https://lab.nexedi.com/nexedi/zodbtools/commit/79cf177a
- Add `zodb` tool to drive all subcommands (commit__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/984cfe22
- Add `zodb info` subcommand to print general information about a ZODB database
(commit__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/37b9fbde
- Switch to open ZODB storages by URL, not only via ZConfig files. URL support
comes from `zodburi` (`commit 1`__, 2__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/82b06413
__ https://lab.nexedi.com/nexedi/zodbtools/commit/bfeb1690
0.0.0.dev3 (2016-11-17)
-----------------------
- Move Nexedi version of `zodbanalyze` from ERP5 into zodbtools.
Compared to original `zodbanalyze` Nexedi version is faster, prints not only
total, but also current sizes, and supports running on bigger databases where
keeping all working set to analyze in RAM is not feasible. It also supports
analyzing a Repozo deltafs file directly.
(`commit 1`__, 2__, 3__, 4__, 5__, 6__, 7__, 8__, 9__)
__ https://lab.nexedi.com/nexedi/zodbtools/commit/ab17cf2d
__ https://lab.nexedi.com/nexedi/zodbtools/commit/1e506a81
__ https://lab.nexedi.com/nexedi/zodbtools/commit/d86d04dc
__ https://lab.nexedi.com/nexedi/zodbtools/commit/5fd2c0eb
__ https://lab.nexedi.com/nexedi/zodbtools/commit/a9346784
__ https://lab.nexedi.com/nexedi/zodbtools/commit/1a489502
__ https://lab.nexedi.com/nexedi/zodbtools/commit/8dc37247
__ https://lab.nexedi.com/nexedi/zodbtools/commit/e4d4762a
__ https://lab.nexedi.com/nexedi/zodbtools/commit/2e834aaf
0.0.0.dev2 (2016-11-17)
-----------------------
- Add initial draft of `zodbdump` - tool to dump content of a ZODB database
(`commit 1`__, 2__).
__ https://lab.nexedi.com/nexedi/zodbtools/commit/c0a6299f
__ https://lab.nexedi.com/nexedi/zodbtools/commit/d955f79a
0.0.0.dev1 (2016-11-16)
-----------------------
- Initial release of zodbtools with `zodbcmp` (`commit 1`__, 2__, 3__).
We originally tried to put `zodbcmp` into ZODB itself, but Jim Fulton asked__
not to load ZODB with scripts anymore. This way zodbtools was created.
__ https://lab.nexedi.com/nexedi/zodbtools/commit/fd6ad1b9
__ https://lab.nexedi.com/nexedi/zodbtools/commit/66a03ae5
__ https://lab.nexedi.com/nexedi/zodbtools/commit/66946b8d
__ https://github.com/zopefoundation/ZODB/pull/128#issuecomment-260970932
include COPYING LICENSE-ZPL.txt README.rst CHANGELOG.rst tox.ini
recursive-include zodbtools/test/testdata *.fs *.index *.ok *.txt
......@@ -8,7 +8,9 @@ scripts anymore. So we are here:
__ https://github.com/zopefoundation/ZODB/pull/128#issuecomment-260970932
- `zodb analyze` - analyze FileStorage or repozo deltafs usage.
- `zodb analyze` - analyze ZODB database or repozo deltafs usage.
- `zodb cmp` - compare content of two ZODB databases bit-to-bit.
- `zodb commit` - commit new transaction into a ZODB database.
- `zodb dump` - dump content of a ZODB database.
- `zodb restore` - restore content of a ZODB database.
- `zodb info` - print general information about a ZODB database.
......@@ -8,9 +8,10 @@ def readfile(path):
setup(
name = 'zodbtools',
version = '0.0.0.dev4',
version = '0.0.0.dev8',
description = 'ZODB-related utilities',
long_description = readfile('README.rst'),
long_description = '%s\n----\n\n%s' % (
readfile('README.rst'), readfile('CHANGELOG.rst')),
url = 'https://lab.nexedi.com/nexedi/zodbtools',
license = 'GPLv3+ with wide exception for Open-Source; ZPL 2.1',
author = 'Nexedi + Zope Foundation + Community',
......@@ -19,23 +20,21 @@ setup(
keywords = 'zodb utility tool',
packages = find_packages(),
install_requires = ['ZODB', 'zodburi', 'pygolang >= 0.0.0.dev3', 'six'],
install_requires = ['ZODB', 'zodburi', 'zope.interface', 'pygolang >= 0.0.0.dev6', 'six', 'dateparser'],
extras_require = {
'test': ['pytest'],
'test': ['pytest', 'freezegun', 'pytz', 'mock;python_version<="2.7"'],
},
entry_points= {'console_scripts': ['zodb = zodbtools.zodb:main']},
# FIXME restore py3 support
classifiers = [_.strip() for _ in """\
Development Status :: 3 - Alpha
Intended Audience :: Developers
Operating System :: POSIX :: Linux
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
Topic :: Database
Topic :: Utilities
Framework :: ZODB\
......
# zodbtools | tox setup
[tox]
envlist = py{27,36,37}-ZODB{4,5}
[testenv]
deps =
.[test]
# latest current ZODB 4
ZODB4: ZODB >=4.0, <5.0dev
ZODB4: ZEO >=4.0, <5.0dev
# ZEO4 depends on transaction <2
ZODB4: transaction <2.0dev
# latest current ZODB 5
ZODB5: ZODB >=5.6, <6.0dev
ZODB5: ZEO >=5.0, <6.0dev
commands= {envpython} -m pytest
# -*- coding: utf-8 -*-
# zodbtools - help topics
# Copyright (C) 2017 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
# Copyright (C) 2017-2018 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
......@@ -51,4 +52,39 @@ Please see zodburi documentation for full details:
http://docs.pylonsproject.org/projects/zodburi/
"""
topic_dict['zurl'] = "specifying database URL", help_zurl
help_tidrange = """\
Many zodb commands can be invoked on specific range of database history and
accept <tidrange> parameter for that. The syntax for <tidrange> is
tidmin..tidmax
where tidmin and tidmax specify [tidmin, tidmax] range of transactions, ends
inclusive. Both tidmin and tidmax are optional and default to
tidmin: 0 (start of database history)
tidmax: +∞ (end of database history)
If a tid (tidmin or tidmax) is given, it has to be specified as follows:
- a 16-digit hex number specifying transaction ID, e.g. 0285cbac258bf266
- absolute timestamp, in RFC3339 or RFC822 formats
- relative timestamp, e.g. yesterday, 1 week ago
Example tid ranges:
.. whole database history
000000000000aaaa.. transactions starting from 000000000000aaaa till latest
..000000000000bbbb transactions starting from database beginning till 000000000000bbbb
000000000000aaaa..000000000000bbbb transactions starting from 000000000000aaaa till 000000000000bbbb
1985-04-12T23:20:50.52Z..2018-01-01T10:30:00Z
transactions starting from 1985-04-12 at 23 hours
20 minutes 50 seconds and 520000000 nano seconds
in UTC till 2018-01-01 at 10 hours 30 minutes in UTC
1_week_ago..yesterday transactions from one week ago until yesterday.
In commands <tidrange> is optional - if it is not given at all, it defaults to
0..+∞, i.e. to whole database history.
"""
topic_dict['zurl'] = "specifying database URL", help_zurl
topic_dict['tidrange'] = "specifying history range", help_tidrange
# Copyright (C) 2019 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
# option) any later version, as published by the Free Software Foundation.
#
# You can also Link and Combine this program with other software covered by
# the terms of any of the Free Software licenses or any of the Open Source
# Initiative approved licenses and Convey the resulting work. Corresponding
# source of such a combination shall include the source code for all other
# software used.
#
# This program is distributed WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
import pytest
from zodbtools.test.testutil import zext_supported
# zext is a test fixture function object that allows to exercise 2 cases:
#
# - when ZODB does not have txn.extension_bytes support
# - when ZODB might have txn.extension_bytes support
#
# in a test, zext should be used as as follows:
#
# def test_something(zext):
# # bytes for an extension dict
# raw_ext = dumps({...})
#
# # will be either same as raw_ext, or b'' if ZODB lacks txn.extension_bytes support
# raw_ext = zext(raw_ext)
#
# # zext.disabled indicates whether testing for non-empty extension was disabled.
# if zext.disabled:
# ...
@pytest.fixture(params=['!zext', 'zext'])
def zext(request):
if request.param == '!zext':
# txn.extension_bytes is not working - always test with empty extension
def _(ext):
return b''
_.disabled = True
return _
else:
# txn.extension_bytes might be working - test with given extension and
# xfail if ZODB does not have necessary support.
def _(ext):
return ext
_.disabled = False
if not zext_supported():
request.applymarker(pytest.mark.xfail(reason='ZODB does not have txn.extension_bytes support'))
return _
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Copyright (C) 2017 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
# Copyright (C) 2017-2021 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
......@@ -39,6 +39,7 @@
from ZODB.FileStorage import FileStorage
from ZODB import DB
from ZODB.Connection import TransactionMetaData
from ZODB.POSException import UndoError
from persistent import Persistent
import transaction
......@@ -60,7 +61,12 @@ def hex64(packed):
return '0x%016x' % unpack64(packed)
# make time.time() predictable
_xtime = time.mktime(time.strptime("04 Jan 1979", "%d %b %Y"))
_xtime0 = time.mktime(time.strptime("04 Jan 1979", "%d %b %Y"))
def xtime_reset():
global _xtime
_xtime = _xtime0
xtime_reset()
def xtime():
global _xtime
_xtime += 1.1
......@@ -94,7 +100,7 @@ class Object(Persistent):
# prepare extension dictionary for subject
alnum = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
def ext(subj):
def ext4subj(subj):
d = {"x-generator": "zodb/py%s (%s)" % (sys.version_info.major, subj)}
# also add some random 'x-cookie'
......@@ -107,7 +113,7 @@ def ext(subj):
d[xcookie] = cookie
# shufle extension dict randomly - to likely trigger different ordering on save
keyv = d.keys()
keyv = list(d.keys())
random.shuffle(keyv)
ext = {}
for key in keyv:
......@@ -115,8 +121,66 @@ def ext(subj):
return ext
# gen_testdb generates test FileStorage database @ outfs_path
def gen_testdb(outfs_path):
# run_with_zodb4py2_compat(f) runs f preserving database compatibility with
# ZODB4/py2, which generates pickles encoded with protocol < 3.
#
# ZODB5 started to use protocol 3 and binary for oids starting from ZODB 5.4.0:
# https://github.com/zopefoundation/ZODB/commit/12ee41c4
# Undo it, while we generate test database.
def run_with_zodb4py2_compat(f):
import ZODB.ConflictResolution
import ZODB.Connection
import ZODB.ExportImport
import ZODB.FileStorage.FileStorage
import ZODB._compat
import ZODB.broken
import ZODB.fsIndex
import ZODB.serialize
binary = getattr(ZODB.serialize, 'binary', None)
_protocol = getattr(ZODB.serialize, '_protocol', None)
Pz4 = 2
try:
ZODB.serialize.binary = bytes
# XXX cannot change just ZODB._compat._protocol, because many modules
# do `from ZODB._compat import _protocol` and just `import ZODB`
# imports many ZODB.X modules. In other words we cannot change
# _protocol just in one place.
ZODB.ConflictResolution._protocol = Pz4
ZODB.Connection._protocol = Pz4
ZODB.ExportImport._protocol = Pz4
ZODB.FileStorage.FileStorage._protocol = Pz4
ZODB._compat._protocol = Pz4
ZODB.broken._protocol = Pz4
ZODB.fsIndex._protocol = Pz4
ZODB.serialize._protocol = Pz4
f()
finally:
ZODB.serialize.binary = binary
ZODB.ConflictResolution._protocol = _protocol
ZODB.Connection._protocol = _protocol
ZODB.ExportImport._protocol = _protocol
ZODB.FileStorage.FileStorage._protocol = _protocol
ZODB._compat._protocol = _protocol
ZODB.broken._protocol = _protocol
ZODB.fsIndex._protocol = _protocol
ZODB.serialize._protocol = _protocol
# gen_testdb generates test FileStorage database @ outfs_path.
#
# zext indicates whether or not to include non-empty extension into transactions.
def gen_testdb(outfs_path, zext=True):
def _():
_gen_testdb(outfs_path, zext)
run_with_zodb4py2_compat(_)
def _gen_testdb(outfs_path, zext):
xtime_reset()
ext = ext4subj
if not zext:
def ext(subj): return {}
logging.basicConfig()
# generate random changes to objects hooked to top-level root by a/b/c/... key
......@@ -163,7 +227,7 @@ def gen_testdb(outfs_path):
break
# delete an object
name = random.choice(root.keys())
name = random.choice(list(root.keys()))
obj = root[name]
root[name] = Object("%s%i*" % (name, i))
# NOTE user/ext are kept empty on purpose - to also test this case
......@@ -178,14 +242,16 @@ def gen_testdb(outfs_path):
''.join(chr(_) for _ in range(32)), # <- NOTE all control characters
u"delete %i\nalpha beta gamma'delta\"lambda\n\nqqq ..." % i,
ext("delete %s" % unpack64(obj._p_oid)))
stor.tpc_begin(txn)
stor.deleteObject(obj._p_oid, obj_tid_lastchange, txn)
stor.tpc_vote(txn)
# at low level stor requires ZODB.IStorageTransactionMetaData not txn (ITransaction)
txn_stormeta = TransactionMetaData(txn.user, txn.description, txn.extension)
stor.tpc_begin(txn_stormeta)
stor.deleteObject(obj._p_oid, obj_tid_lastchange, txn_stormeta)
stor.tpc_vote(txn_stormeta)
# TODO different txn status vvv
# XXX vvv it does the thing, but py fs iterator treats this txn as EOF
#if i != Niter-1:
# stor.tpc_finish(txn)
stor.tpc_finish(txn)
# stor.tpc_finish(txn_stormeta)
stor.tpc_finish(txn_stormeta)
# close db & rest not to get conflict errors after we touched stor
# directly a bit. everything will be reopened on next iteration.
......@@ -196,13 +262,22 @@ def gen_testdb(outfs_path):
# ----------------------------------------
from zodbtools.zodbdump import zodbdump
from zodbtools.test.testutil import zext_supported
def main():
# check that ZODB supports txn.extension_bytes; refuse to work if not.
if not zext_supported():
raise RuntimeError("gen_testdata must be used with ZODB that supports txn.extension_bytes")
out = "testdata/1"
gen_testdb("%s.fs" % out)
stor = FileStorage("%s.fs" % out, read_only=True)
with open("%s.zdump.ok" % out, "w") as f:
zodbdump(stor, None, None, out=f)
for zext in [True, False]:
dbname = out
if not zext:
dbname += "_!zext"
gen_testdb("%s.fs" % dbname, zext=zext)
stor = FileStorage("%s.fs" % dbname, read_only=True)
with open("%s.zdump.ok" % dbname, "wb") as f:
zodbdump(stor, None, None, out=f)
if __name__ == '__main__':
main()
# -*- coding: utf-8 -*-
# Copyright (C) 2019 Nexedi SA and Contributors.
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
# option) any later version, as published by the Free Software Foundation.
#
# You can also Link and Combine this program with other software covered by
# the terms of any of the Free Software licenses or any of the Open Source
# Initiative approved licenses and Convey the resulting work. Corresponding
# source of such a combination shall include the source code for all other
# software used.
#
# This program is distributed WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
from zodbtools.zodbanalyze import analyze, report
import os.path
def test_zodbanalyze(capsys):
for use_dbm in (False, True):
report(
analyze(
os.path.join(os.path.dirname(__file__), "testdata", "1.fs"),
use_dbm=use_dbm,
delta_fs=False,
tidmin=None,
tidmax=None,
),
csv=False,
)
captured = capsys.readouterr()
assert "Processed 68 records in 59 transactions" in captured.out
assert captured.err == ""
# csv output
report(
analyze(
os.path.join(os.path.dirname(__file__), "testdata", "1.fs"),
use_dbm=False,
delta_fs=False,
tidmin=None,
tidmax=None,
),
csv=True,
)
captured = capsys.readouterr()
assert (
"""Class Name,T.Count,T.Bytes,Pct,AvgSize,C.Count,C.Bytes,O.Count,O.Bytes
persistent.mapping.PersistentMapping,10,1578,45.633314%,157.800000,1,213,9,1365
__main__.Object,56,1880,54.366686%,33.571429,9,303,47,1577
"""
== captured.out
)
assert captured.err == ""
# empty range
report(
analyze(
os.path.join(os.path.dirname(__file__), "testdata", "1.fs"),
use_dbm=False,
delta_fs=False,
tidmin="ffffffffffffffff",
tidmax=None,
),
csv=False,
)
captured = capsys.readouterr()
assert "# ø\nNo transactions processed\n" == captured.out.encode('utf-8')
assert captured.err == ""
# -*- coding: utf-8 -*-
# Copyright (C) 2018-2020 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
# Jérome Perrin <jerome@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
# option) any later version, as published by the Free Software Foundation.
#
# You can also Link and Combine this program with other software covered by
# the terms of any of the Free Software licenses or any of the Open Source
# Initiative approved licenses and Convey the resulting work. Corresponding
# source of such a combination shall include the source code for all other
# software used.
#
# This program is distributed WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
from zodbtools.zodbcommit import zodbcommit
from zodbtools.zodbdump import zodbdump, Transaction, ObjectData, ObjectDelete, ObjectCopy
from zodbtools.util import storageFromURL, sha1
from ZODB.utils import p64, u64, z64
from ZODB._compat import BytesIO, dumps, _protocol # XXX can't yet commit with arbitrary ext.bytes
from tempfile import mkdtemp
from shutil import rmtree
from golang import func, defer
# verify zodbcommit.
@func
def test_zodbcommit(zext):
tmpd = mkdtemp('', 'zodbcommit.')
defer(lambda: rmtree(tmpd))
stor = storageFromURL('%s/2.fs' % tmpd)
defer(stor.close)
head = stor.lastTransaction()
# commit some transactions via zodbcommit and verify if storage dump gives
# what is expected.
t1 = Transaction(z64, ' ', b'user name', b'description ...', zext(dumps({'a': 'b'}, _protocol)), [
ObjectData(p64(1), b'data1', 'sha1', sha1(b'data1')),
ObjectData(p64(2), b'data2', 'sha1', sha1(b'data2'))])
t1.tid = zodbcommit(stor, head, t1)
t2 = Transaction(z64, ' ', b'user2', b'desc2', b'', [
ObjectDelete(p64(2))])
t2.tid = zodbcommit(stor, t1.tid, t2)
buf = BytesIO()
zodbdump(stor, p64(u64(head)+1), None, out=buf)
dumped = buf.getvalue()
assert dumped == b''.join([_.zdump() for _ in (t1, t2)])
# ObjectCopy. XXX zodbcommit handled ObjectCopy by actually copying data,
# not referencing previous transaction via backpointer.
t3 = Transaction(z64, ' ', b'user3', b'desc3', b'', [
ObjectCopy(p64(1), t1.tid)])
t3.tid = zodbcommit(stor, t2.tid, t3)
data1_1, _, _ = stor.loadBefore(p64(1), p64(u64(t1.tid)+1))
data1_3, _, _ = stor.loadBefore(p64(1), p64(u64(t3.tid)+1))
assert data1_1 == data1_3
assert data1_1 == b'data1' # just in case
# Copyright (C) 2017 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
# -*- coding: utf-8 -*-
# Copyright (C) 2017-2020 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
# Jérome Perrin <jerome@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
......@@ -17,21 +19,129 @@
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
from zodbtools.zodbdump import zodbdump
from zodbtools.zodbdump import (
zodbdump, DumpReader, Transaction, ObjectDelete, ObjectCopy,
ObjectData, HashOnly
)
from zodbtools.util import fromhex
from ZODB.FileStorage import FileStorage
from cStringIO import StringIO
from ZODB.utils import p64
from io import BytesIO
from os.path import dirname
from zodbtools.test.testutil import zext_supported
from pytest import raises, xfail
# verify zodbdump output against golden
def test_zodbdump():
tdir = dirname(__file__)
stor = FileStorage('%s/testdata/1.fs' % tdir, read_only=True)
def test_zodbdump(zext):
tdir = dirname(__file__)
zkind = '_!zext' if zext.disabled else ''
stor = FileStorage('%s/testdata/1%s.fs' % (tdir, zkind), read_only=True)
with open('%s/testdata/1.zdump.ok' % tdir) as f:
with open('%s/testdata/1%s.zdump.ok' % (tdir, zkind), 'rb') as f:
dumpok = f.read()
out = StringIO()
out = BytesIO()
zodbdump(stor, None, None, out=out)
assert out.getvalue() == dumpok
# verify zodbdump.DumpReader
def test_dumpreader():
in_ = b"""\
txn 0123456789abcdef " "
user "my name"
description "o la-la..."
extension "zzz123 def"
obj 0000000000000001 delete
obj 0000000000000002 from 0123456789abcdee
obj 0000000000000003 54 adler32:01234567 -
obj 0000000000000004 4 sha1:9865d483bc5a94f2e30056fc256ed3066af54d04
ZZZZ
obj 0000000000000005 9 crc32:52fdeac5
ABC
DEF!
txn 0123456789abcdf0 " "
user "author2"
description "zzz"