Commit 2ca0f076 by Kirill Smelkov

ZBigArray: Compatibility fix to read arrays from DB that were previously saved without order info

Commit ab9ca2df (bigarray: Add support for FORTRAN ordering) added
ability to define array order, but there I made a mistake of not caring
about how previously-saved to DB arrays would be read back.

The thing is BigArray gained new data member ._order which is
automatically saved to DB thanks to ZBigArray inheriting from
Persistent; on load-from-db path we just read object state from DB,
which for ZBigArray is dict, and restore object attributes from it.

But for previously-saved data, obviously, there is no 'order' entry and thus
this way restored objects are restored not in full to current code expectations
and it can boom e.g. this way:

  Module wendelin.bigarray, line 190, in resize
    self._init0(new_shape, self.dtype, order=self._order)
  AttributeError: 'ZBigArray' object has no attribute '_order'

Solution to fix is: on restore-from-DB path, see if a data member is not
present on restored object, and if it has default value in BigArray set it to

( code to get function defaults is from )

/cc @Tyagov, @klaus
1 parent 18b40b18
......@@ -64,7 +64,11 @@ class BigArray(object):
# TODO doc -> see ndarray
# NOTE does not accept strides
# NOTE please be cooperative to ZBigArray and name helper data members starting with _v_
# NOTE please be cooperative to ZBigArray and
# - name helper data members starting with _v_.
# - do not rename data members which go to DB (not starting with _v_).
# - if new argument is added - it has to have default which is
# semantically equivalent to how the code was working before.
def __init__(self, shape, dtype_, bigfileh, order='C'):
self._init0(shape, dtype_, order)
self._v_fileh = bigfileh
......@@ -24,10 +24,16 @@ TODO text
from wendelin.bigarray import BigArray
from wendelin.bigfile.file_zodb import ZBigFile, LivePersistent
import inspect
# TODO document that first access data must be either before commit or Connection.add
# {} BigArray arg -> default value
_ = inspect.getargspec(BigArray.__init__)
BigArray_defaults = dict(zip(reversed(_.args), reversed(_.defaults)))
class ZBigArray(BigArray,
# Live: don't allow us to go to ghost
# (not to loose ._v_fileh which works as DataManager)
......@@ -52,6 +58,14 @@ class ZBigArray(BigArray,
def __setstate__(self, state):
super(ZBigArray, self).__setstate__(state)
# for backward compatibility: if a member is missing in state - set it
# to BigArray default. Ex. `order` was not set in early versions of
# ZBigArray and when loading such objects from DB, without adjustment,
# they won't work properly.
for k, v in BigArray_defaults.items():
if not hasattr(self, '_'+k):
setattr(self, '_'+k, v)
# NOTE __setstate__() is done after either
# - 1st time loading from DB, or
# - loading from DB after invalidation.
......@@ -189,6 +189,53 @@ def test_zbigarray():
# test array ordering is saved properly into DB and is picked up in
# backward-compatible manner - for data saved before order parameter was
# introduced.
# (actual ordering indexing test is in BigArray tests, not here)
def test_zbigarray_order():
# make sure order is properly saved/restored to/from DB
root = testdb.dbopen()
root['carray'] = ZBigArray((16*1024*1024,), uint8)
root['farray'] = ZBigArray((16*1024*1024,), uint8, order='F')
del root
root = testdb.dbopen()
C = root['carray']
F = root['farray']
assert isinstance(C, ZBigArray)
assert C.shape == (16*1024*1024,)
assert C.dtype == dtype(uint8)
assert C._order == 'C'
assert isinstance(F, ZBigArray)
assert F.shape == (16*1024*1024,)
assert F.dtype == dtype(uint8)
assert F._order == 'F'
# make sure we can read previously saved data which had no order set
root['coldarray'] = Cold = ZBigArray((16*1024*1024,), uint8)
del Cold._order # simulate that it is without
assert '_order' not in Cold.__getstate__()
del root, Cold
root = testdb.dbopen()
Cold = root['coldarray']
assert Cold._order == 'C'
del root
# the same as test_bigfile_filezodb_vs_conn_migration but explicitly for ZBigArray
# ( NOTE this test is almost dup of test_zbigarray_vs_conn_migration() )
def test_zbigarray_vs_conn_migration():
Styling with Markdown is supported
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!