1. 18 Nov, 2021 3 commits
    • Kirill Smelkov's avatar
      fixup! DemoStorage: Add support for deleteObject (IExternalGC) · 5a1df8c0
      Kirill Smelkov authored
      Make flake8 happy
      
      src/ZODB/DemoStorage.py:388:80: E501 line too long (89 > 79 characters)
      src/ZODB/DemoStorage.py:391:80: E501 line too long (87 > 79 characters)
      src/ZODB/DemoStorage.py:393:80: E501 line too long (83 > 79 characters)
      src/ZODB/DemoStorage.py:395:80: E501 line too long (87 > 79 characters)
      src/ZODB/DemoStorage.py:397:80: E501 line too long (100 > 79 characters)
      src/ZODB/DemoStorage.py:398:80: E501 line too long (86 > 79 characters)
      src/ZODB/DemoStorage.py:408:5: E303 too many blank lines (2)
      src/ZODB/tests/testDemoStorage.py:397:9: F401 'ZODB.FileStorage.FileStorage' imported but unused
      src/ZODB/tests/testDemoStorage.py:399:80: E501 line too long (101 > 79 characters)
      5a1df8c0
    • Kirill Smelkov's avatar
      c3820ff4
    • Kirill Smelkov's avatar
      Merge branch 'y/loadAt.8' into y/demo-delete · fb91aa62
      Kirill Smelkov authored
      * y/loadAt.8: (35 commits)
        Make lint happy
        Let the year float.
        Configuring for pure-python
        Specify a PyPy2 version.
        Lint the code.
        Configuring for pure-python
        Fix TypeError for fsoids (#351)
        Fix deprecation warnings occurring on Python 3.10.
        fix more PY3 incompatibilities in `fsstats`
        fix Python 3 incompatibility for `fsstats`
        add `fsdump/fsstats` test
        fsdump/fsstats improvements
        Undeprecate loadBefore
        fixup! changes: Add draft entry for loadAt/loadBeforeEx/DemoStorage fix
        changes: Add draft entry for loadAt/loadBeforeEx/DemoStorage fix
        fixup! Handle NotImplementedError raised by loadBefore/loadBeforeEx as "interface not provided"
        *: Don't emit warnings on loadBefore
        Handle NotImplementedError raised by loadBefore/loadBeforeEx as "interface not provided"
        loadAt -> loadBeforeEx
        - add coverage combine step
        ...
      fb91aa62
  2. 11 Nov, 2021 2 commits
  3. 09 Nov, 2021 2 commits
  4. 05 Nov, 2021 2 commits
  5. 29 Oct, 2021 3 commits
    • Kirill Smelkov's avatar
      Merge remote-tracking branch 'origin/master' into y/loadAt.7 · 133ebeb3
      Kirill Smelkov authored
      to resolve trivial conflict on CHANGES.rst
      
      * origin/master: (22 commits)
        Fix TypeError for fsoids (#351)
        Fix deprecation warnings occurring on Python 3.10.
        fix more PY3 incompatibilities in `fsstats`
        fix Python 3 incompatibility for `fsstats`
        add `fsdump/fsstats` test
        fsdump/fsstats improvements
        - add coverage combine step
        - first cut moving tests from Travis CI to GitHub Actions
        - ignore virtualenv artifacts [ci skip]
        tests: Run race-related tests with high frequency of switches between threads
        tests: Add test for load vs external invalidation race
        tests: Add test for open vs invalidation race
        fixup! doc/requirements: Require pygments < 2.6 on py2
        doc/requirements: Require pygments < 2.6 on py2
        fixup! buildout: Fix Sphinx install on Python2
        buildout: Fix Sphinx install on Python2
        Update README.rst
        Security fix documentation dependencies (#342)
        changes: Correct link to UnboundLocalError fsoids.py fix
        fsrefs: Optimize IO  (take 2) (#340)
        ...
      133ebeb3
    • Michael Howitz's avatar
      Lint the code. · 6e5baffd
      Michael Howitz authored
      Add support for Python 3.9 and 3.10.
      6e5baffd
    • Michael Howitz's avatar
      Configuring for pure-python · 1f3a0d62
      Michael Howitz authored
      1f3a0d62
  6. 28 Oct, 2021 1 commit
  7. 27 Oct, 2021 1 commit
  8. 06 Oct, 2021 1 commit
  9. 05 Oct, 2021 3 commits
  10. 03 Oct, 2021 1 commit
  11. 06 Jun, 2021 1 commit
  12. 11 May, 2021 1 commit
  13. 06 May, 2021 2 commits
  14. 04 May, 2021 3 commits
  15. 03 May, 2021 1 commit
  16. 23 Apr, 2021 4 commits
  17. 21 Apr, 2021 3 commits
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
      tests: Add test for load vs external invalidation race · e923c9a8
      Kirill Smelkov authored
      For ZEO this data corruption bug was reported at
      https://github.com/zopefoundation/ZEO/issues/155 and fixed at
      https://github.com/zopefoundation/ZEO/pull/169.
      
      Without that fix the failure shows e.g. as follows when running ZEO test
      suite:
      
          Failure in test check_race_load_vs_external_invalidate (ZEO.tests.testZEO.BlobAdaptedFileStorageTests)
          Traceback (most recent call last):
            File "/usr/lib/python2.7/unittest/case.py", line 329, in run
              testMethod()
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/BasicStorage.py", line 621, in check_race_load_vs_external_invalidate
              self.fail([_ for _ in failure if _])
            File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
              raise self.failureException(msg)
          AssertionError: ['T1: obj1.value (7)  !=  obj2.value (8)']
      
      Even if added test is somewhat similar to
      check_race_loadopen_vs_local_invalidate, it is added anew without trying
      to unify code. The reason here is that the probability to catch load vs
      external invalidation race is significantly reduced when there are only
      1 modify and 1 verify workers. The unification with preserving both
      tests semantic would make test for "load vs local invalidate" harder to
      follow. Sometimes a little copying is better than trying to unify too
      much.
      
      For the test to work, test infrastructure is amended with
      ._new_storage_client() method that complements ._storage attribute:
      client-server storages like ZEO, NEO and RelStorage allow several
      storage clients to be connected to single storage server. For
      client-server storages test subclasses should implement
      _new_storage_client to return new storage client that is connected to
      the same storage server self._storage is connected to.
      
      For ZEO ._new_storage_client() is added by https://github.com/zopefoundation/ZEO/pull/170
      
      Other client-server storages can follow to implement ._new_storage_client()
      and this way automatically activate this "load vs external invalidation"
      test when their testsuite is run.
      
      Contrary to test for "load vs local invalidate" N is set to lower value (100),
      because with 8 workers the bug is usually reproduced at not-so-high iteration
      number (5-10-20).
      
      /cc @d-maurer, @jamadden, @jmuchemb
      /reviewed-on https://github.com/zopefoundation/ZODB/pull/345
      e923c9a8
    • Kirill Smelkov's avatar
      tests: Add test for open vs invalidation race · 5b4dd5f7
      Kirill Smelkov authored
      Add test that exercises open vs invalidation race condition that, if
      happen, leads to data corruption. We are seeing such race happening on
      storage level in ZEO (https://github.com/zopefoundation/ZEO/issues/166),
      and previously we've seen it also to happen on Connection level
      (https://github.com/zopefoundation/ZODB/issues/290). By adding this test
      to be exercised wrt all storages we make sure that all storages stay
      free from this race.
      
      And it payed out. Besides catching original problems from
      https://github.com/zopefoundation/ZODB/issues/290 and
      https://github.com/zopefoundation/ZEO/issues/166 , this test also
      discovered a concurrency bug in MVCCMappingStorage:
      
          Failure in test check_race_open_vs_invalidate (ZODB.tests.testMVCCMappingStorage.MVCCMappingStorageTests)
          Traceback (most recent call last):
            File "/usr/lib/python2.7/unittest/case.py", line 329, in run
              testMethod()
            File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/tests/BasicStorage.py", line 492, in check_race_open_vs_invalidate
              self.fail(failure[0])
            File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
              raise self.failureException(msg)
          AssertionError: T1: obj1.value (24)  !=  obj2.value (23)
      
      The problem with MVCCMappingStorage was that instance.poll_invalidations
      was correctly taking main_lock with intention to make sure main data is
      not mutated during analysis, but instance.tpc_finish and
      instance.tpc_abort did _not_ taken main lock, which was leading to
      committed data to be propagating into main storage in non-atomic way.
      
      This bug was also observable if both obj1 and obj2 in the added test
      were always loaded from the storage (added obj2._p_invalidate after
      obj1._p_invalidate).
      
      -> Fix MVCCMappingStorage by correctly locking main MVCCMappingStorage
      instance when processing transaction completion.
      
      /cc @d-maurer, @jamadden, @jmuchemb
      /reviewed-on https://github.com/zopefoundation/ZODB/pull/345
      5b4dd5f7
  18. 20 Apr, 2021 5 commits
  19. 01 Apr, 2021 1 commit
    • Claudius Ellsel's avatar
      Update README.rst · dad77801
      Claudius Ellsel authored
      Fix tiny issue with a remaining colon that was probably not deleted during updates of the README.
      dad77801