• Kirill Smelkov's avatar
    tests: Adjust testdata FileStorage for current Python on the fly · e825f80f
    Kirill Smelkov authored
    FileStorage/py2 uses `FS21` magic in file header, whereas
    FileStorage/py3 uses `FS30` magic:
    
        https://github.com/zopefoundation/ZODB/blob/0e72b8b13657/src/ZODB/_compat.py#L39
        https://github.com/zopefoundation/ZODB/blob/0e72b8b13657/src/ZODB/_compat.py#L74
    
    And if, upon opening the database, file magic does not match to what ZODB
    expects, open is rejected:
    
        https://github.com/zopefoundation/ZODB/blob/0e72b8b13657/src/ZODB/FileStorage/FileStorage.py#L88
        https://github.com/zopefoundation/ZODB/blob/0e72b8b13657/src/ZODB/FileStorage/FileStorage.py#L1625-L1630
    
    This is done with the idea for a database, that was written from
    Python2, to be rejected to be opened from Python3 and vice-versa because
    strings/bytes semantics changed in between py23.
    
    As the result, many zodbtools tests currently fail on py3 when they try
    to access prepared FileStorage database in testdata, because that
    database was originally prepared on py2. Here is, for example, how
    test_zodbdump fails:
    
        ___________________________ test_zodbdump[zext-raw] ____________________________
    
        zext = <function zext.<locals>._ at 0x7f28530bf9d0>, pretty = 'raw'
    
            @mark.parametrize('pretty', ('raw', 'zpickledis'))
            def test_zodbdump(zext, pretty):
                tdir  = dirname(__file__)
                zkind = '_!zext' if zext.disabled else ''
        >       stor  = FileStorage('%s/testdata/1%s.fs' % (tdir, zkind), read_only=True)
    
        zodbtools/test/test_dump.py:41:
        _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
        ../ZODB/src/ZODB/FileStorage/FileStorage.py:315: in __init__
            self._pos, self._oid, tid = read_index(
        _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
        file = <_io.BufferedReader name='/home/kirr/src/wendelin/z/zodbtools/zodbtools/test/testdata/1.fs'>
        name = '/home/kirr/src/wendelin/z/zodbtools/zodbtools/test/testdata/1.fs'
        index = <ZODB.fsIndex.fsIndex object at 0x7f2852fee2b0>, tindex = {}
        stop = b'\xff\xff\xff\xff\xff\xff\xff\xff'
        ltid = b'\x00\x00\x00\x00\x00\x00\x00\x00', start = 4
        maxoid = b'\x00\x00\x00\x00\x00\x00\x00\x00', recover = 0, read_only = True
    
            def read_index(file, name, index, tindex, stop=b'\377'*8,
                           ltid=z64, start=4, maxoid=z64, recover=0, read_only=0):
                """Scan the file storage and update the index."""
                ...
                if file_size:
                    if file_size < start:
                        raise FileStorageFormatError(file.name)
                    seek(0)
                    if read(4) != packed_version:
        >               raise FileStorageFormatError(name)
        E               ZODB.FileStorage.FileStorage.FileStorageFormatError: /home/kirr/src/wendelin/z/zodbtools/zodbtools/test/testdata/1.fs
    
        ../ZODB/src/ZODB/FileStorage/FileStorage.py:1630: FileStorageFormatError
    
    Since zodbtools primarily work on raw data without decoding stored
    pickles, unlike Zope or ERP5, it should not be a problem for zodbtools
    to work on py3 with the database that was prepared on py2.
    
    -> Adjust all tests to use FileStorage data generated on the fly based
    on original files in testdata/ but with FileStorage header being
    rewritten to match current python.
    e825f80f
test_dump.py 4.6 KB