1. 04 May, 2016 1 commit
    • Jim Fulton's avatar
      Simplify MVCC by determining transaction start time using lastTransaction. · 227953b9
      Jim Fulton authored
      This implements: https://github.com/zopefoundation/ZODB/issues/50
      
      Rather than watching invalidations, simply use 1 + the storages
      lastTransaction, which is equivalent to but much simpler than waiting
      for the first invalidation after a transaction starts.
      
      More importantly, it means we can always use loadBefore and get away
      from load.  We no longer have to worry about ordering of invalidations
      and load() results.
      
      Much thanks to NEO for pointing the way toward this simplification!
      
      Implementing this initially caused a deadlock, because DB.open()
      called Connection.open() while holding a database lock and
      Connection.open() now calls IStotage.lastTransaction(), which acquires a
      storage lock. (It's not clear that lastTransaction() really needs a
      storage lock.)  Meanwhile, IStotage.tpc_finish() calls a DB function
      that requires the DB lock while holding the storage lock.  Fixing this
      required moving the call to Connection.open() outside the region where
      the DB lock was held.
      
      To debug the problem above, I greatly improved lock-debugging
      support. Now all of the ZODB code imports Lock, RLock and Condition
      from ZODB.utils. If the DEBUG_LOCKING is set to a non-empty value,
      then these are wrapped in such a way that debugging information is
      printed as they are used. This made spotting the nature of the
      deadlock easier.
      
      Of course, a change this basic broke lots of tests. Most of the
      breakage arises from the fact that connections now call
      lastTransaction on storages at transaction boundaries.  Lots of tests
      didn't clean up databases and connections properly.  I fixed many
      tests, but ultimately gave up and added some extra cleanup code that
      violated transaction-manager underware (and the underware's privates)
      to clear transaction synchonizers in test setup and tear-down.  I plan
      to add a transaction manager API for this and to use it in a
      subsequent PR.
      
      This tests makes database and connection hygiene a bit more important,
      especially for tests, because a connection will continue to interact
      with storages if it isn't properly closed, which can lead to errors if
      the storage is closed.  I chose not to swallow these errors in
      Connection, choosing rather to clean up tests.
      
      The thread debugging and test changes make this PR larger than I would
      have liked. Apologies in advance to the reviewers.
      227953b9
  2. 29 Apr, 2016 2 commits
  3. 28 Apr, 2016 2 commits
  4. 27 Apr, 2016 4 commits
    • Julien Muchembled's avatar
      Changelog for PR #52 · 06df0eba
      Julien Muchembled authored
      06df0eba
    • Julien Muchembled's avatar
      Fix possible data corruption after FileStorage is truncated to roll back a transaction · 028b1922
      Julien Muchembled authored
      Multi-threaded IO support, which is new to ZODB 3.10, allows clients to read
      data (load & loadBefore) even after tpc_vote has started to write a new
      transaction to disk. This is done by using different 'file' objects.
      
      Issues start when a transaction is rolled back after data has been appended
      (using the writing file object). Truncating is not enough because the FilePool
      may have been used concurrently to read the end of the last transaction:
      file objects have their own read buffers which, in this case, may also contain
      the beginning of the aborted transaction.
      
      So a solution is to invalidate read buffers whenever they may contain wrong
      data. This patch does it on truncation, which happens rarely enough to not
      affect performance.
      
      We discovered this bug in the following conditions:
      - ZODB splitted in several FileStorage
      - many conflicts in the first committed DB, but always resolved
      - unresolved conflict in another DB
      If the transaction is replayed with success (no more conflict in the other DB),
      a subsequent load of the object that could be resolved in the first DB may, for
      example, return a wrong serial (tid of the aborted transaction) if the layout
      of the committed transaction matches that of the aborted one.
      
      The bug usually manifests with POSKeyError & CorruptedDataError exceptions in
      ZEO logs, for example while trying to resolve a conflict (and restarting the
      transaction does not help, causing Site Errors in Zope). But theorically,
      this could also cause silent corruption or unpickling errors at client side.
      028b1922
    • Julien Muchembled's avatar
      Changelog for PR #54 · b83ac1c2
      Julien Muchembled authored
      b83ac1c2
    • Julien Muchembled's avatar
      Merge pull request #54 from zopefoundation/deps-cleanup · eebae5a2
      Julien Muchembled authored
      Cleanup dependencies, remove ZEO leftovers
      eebae5a2
  5. 25 Apr, 2016 2 commits
  6. 04 Apr, 2016 1 commit
    • Jason Madden's avatar
      patch time.time in historical_connections to avoid windows failures; fix race... · 6b707535
      Jason Madden authored
      patch time.time in historical_connections to avoid windows failures; fix race condition in pool.availableGC
      
      * patch time.time in historical_connections to avoid windows failures.
      
      * Also use a mock version of datetime.datetime.utcnow.
      
      We can't patch the method in-place because it's an extension type.
      
      * synchronize sleep with time
      
      * increment by less than the pool timeout to avoid a race condition.
      
      * Avoid mutating the dict as we iter over it.
      
      Seen in Python 3.4 tests:
      
      File "/home/travis/build/zopefoundation/ZODB/src/ZODB/historical_connections.txt", line 263, in historical_connections.txt
      Failed example:
          conn = db.open()
      Exception raised:
          Traceback (most recent call last):
            File "/opt/python/3.4.2/lib/python3.4/doctest.py", line 1324, in __run
              compileflags, 1), test.globs)
            File "<doctest historical_connections.txt[0]>", line 1, in <module>
              conn = db.open()
            File "/home/travis/build/zopefoundation/ZODB/src/ZODB/DB.py", line 761, in open
              self.historical_pool.availableGC()
            File "/home/travis/build/zopefoundation/ZODB/src/ZODB/DB.py", line 277, in availableGC
              for key, pool in self.pools.items():
          RuntimeError: dictionary changed size during iteration
      6b707535
  7. 18 Mar, 2016 1 commit
  8. 17 Mar, 2016 6 commits
  9. 16 Nov, 2015 1 commit
  10. 13 Nov, 2015 1 commit
  11. 07 Nov, 2015 4 commits
  12. 03 Nov, 2015 1 commit
  13. 14 Jul, 2015 1 commit
    • Marius Gedminas's avatar
      Oh come on! · e76c5b02
      Marius Gedminas authored
      (IIRC a bug in python-modernize causes duplicate future imports)
      e76c5b02
  14. 09 Jun, 2015 1 commit
  15. 02 Jun, 2015 4 commits
  16. 25 May, 2015 4 commits
  17. 22 May, 2015 4 commits