Starting with upcoming ZODB 5.5.2 ZODB tries to preserve extension_bytes transaction metadata property in the raw form as it was stored on disk in the database:

https://github.com/zopefoundation/ZODB/commit/2f8cc67a

However now when running test/gen_testdata.py with ZODB with that patch (and gen_testdata.py refuses to work if it detects that ZODB does not properly supports .extension_bytes property because we want it to be present in the generated test database [1,2]) it now breaks:

$ ./gen_testdata.py
Traceback (most recent call last):
  File "./gen_testdata.py", line 230, in <module>
    main()
  File "./gen_testdata.py", line 224, in main
    gen_testdb("%s.fs" % dbname, zext=zext)
  File "./gen_testdata.py", line 194, in gen_testdb
    stor.tpc_begin(txn)
  File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/BaseStorage.py", line 193, in tpc_begin
    ext = transaction.extension_bytes
AttributeError: 'Transaction' object has no attribute 'extension_bytes'

The breakage is because, as specified in ZODB interfaces[3,4], storage requires ZODB.IStorageTransactionMetaData, not transaction.ITransaction instance gen_testdata.py was using. The script used to work before just by luck.

The fix is to convert transaction instance into storage transaction metadata object for the place where we talk to storage at raw level.

HOWEVER, when checking regenerated database and its dump I noticed:

ZODB >= 5.4.0 uses pickle protocol 3 on both python2 and python3

https://github.com/zopefoundation/ZODB/commit/12ee41c4

In other words it saves e.g. OID of an object as pickle binary, which decodes as bytes on py3 and zodbpickle.binary on py2 when decoding via zodbpickle. However it will result in DecodeError when decoding on py2 with standard pickle module. The latter means that ZODB3 will fail to load data from test database, because ZODB3 - contrary to ZODB4 and ZODB5 - uses std pickle module, not zodbpickle.

Until now we used to care about ZODB3 and it is included into zodbtools test matrix:

https://lab.nexedi.com/nexedi/zodbtools/blob/7bc0385e/tox.ini#L9-14

Thus, if we continue to preserve ZODB3 support we cannot go on with this patch.

@klaus, could you please provide feedback, whether you still need ZODB3 support, or you plan to drop it in some future? If yes, we can go further here, just by dropping ZODB3 support o zodbtools side too. If no - and it is ok - I will try to think what to do and overcome the obstacle.

Thanks beforehand,
Kirill

/cc @jerome, @jm, @romain

[1] https://lab.nexedi.com/nexedi/zodbtools/blob/7bc0385e/zodbtools/test/gen_testdata.py#L215-217
[2] https://lab.nexedi.com/nexedi/zodbtools/blob/7bc0385e/zodbtools/test/testutil.py#L31-63
[3] https://github.com/zopefoundation/ZODB/blob/5.5.1-35-gb5895a5c2/src/ZODB/interfaces.py#L815-L818
[4] https://github.com/zopefoundation/ZODB/blob/5.5.1-35-gb5895a5c2/src/ZODB/interfaces.py#L538-L575