• Kirill Smelkov's avatar
    bigfile/zodb: Resync _ZBigFileH to Connection.transaction_manager on every connection reopen · d9d6adec
    Kirill Smelkov authored
    This continues c7c01ce4 (bigfile/zodb: ZODB.Connection can migrate
    between threads on close/open and we have to care): Until now we were
    retrieving zconn.transaction_manager on _ZBigFileH init, and further using
    that transaction manager for every connection reopen. However that is
    not correct because on every reopen connection can be given new
    transaction manager.
    We were not practically hitting the bug because until recently ZODB was,
    by default, using the same ThreadTransactionManager manager instance as
    Connection.transaction_manager for all connections, and not doing all
    steps needed to keep _ZBigFileH.transaction_manager in sync to
    Connection was forgiven - a particular transaction manager that was used
    was TransactionManager instance implicitly associated with current
    thread by global threading.Local transaction.manager . However starting
    from ZODB 5.5.0 Connection code was changed to remember as
    .transaction_manager the particular TransactionManager instance without
    any threading.Local games:
    Given that we were not syncing properly that broke wendelin.core tests, for
        bigfile/tests/test_filezodb.py::test_bigfile_filezodb_vs_conn_migration Exception in thread Thread-1:
        Traceback (most recent call last):
          File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
          File "/usr/lib/python2.7/threading.py", line 754, in run
            self.__target(*self.__args, **self.__kwargs)
          File "/home/kirr/src/wendelin/wendelin.core/bigfile/tests/test_filezodb.py", line 401, in T11
            transaction.commit()    # should be nothing
          File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/_manager.py", line 252, in commit
            return self.manager.commit()
          File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/_manager.py", line 131, in commit
            return self.get().commit()
          File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/_transaction.py", line 298, in commit
            self._synchronizers.map(lambda s: s.beforeCompletion(self))
          File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/weakset.py", line 61, in map
          File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/_transaction.py", line 298, in <lambda>
            self._synchronizers.map(lambda s: s.beforeCompletion(self))
          File "/home/kirr/src/wendelin/wendelin.core/bigfile/file_zodb.py", line 671, in beforeCompletion
            assert txn is zconn.transaction_manager.get()
    What is happening here is that one thread used the connection and
    ZBigFile/_ZBigFileH associated with it, then the connection was closed
    and released to DB pool. Then the connection was reopened but by another
    thread and thus with different TransactionManager instance and oops -
    _ZBigFileH.transaction_manager is different because it is
    TransactionManager instance that was used by the first thread.
    Fix it by resyncing _ZBigFileH.transaction_manager on every
    connection reopen. No new test as existing tests already cover the
    problem when run with ZODB >= 5.5.0 .
file_zodb.py 28.4 KB