1. 03 Apr, 2020 1 commit
  2. 29 Sep, 2019 2 commits
    • Kirill Smelkov's avatar
      . · cb6b2abd
      Kirill Smelkov authored
      cb6b2abd
    • Kirill Smelkov's avatar
      X wcfs: Don't forbid simultaneous watch requests · 43915fe9
      Kirill Smelkov authored
      The limitation that only one simultaneous watch request over one wlink
      is possible was added in April (b4857f66) when both Watch and WatchLink
      locking was not there and marked as XXX. That locking was added in July
      (85d86a32) though, so there should be real need to simultaneous watch
      requests over one wlink.
      
      Also the limit implementation had a bug: it was setting handlingWatch
      back to 0, but _after_ sending reply to client. This way a situation was
      possible when client is woken up first, sends another watch request,
      wcfs was not yet scheduled, handlingWatch is still _1_, and watch
      request is rejected. This bug is very likely to happen when running wcfs
      tests with 2 CPU machine or with just GOMAXPROCS=2:
      
          C: setup watch f<0000000000000043> @at1 (03d2c23f46d04dcc)
          #  pinok: {2: @at1 (03d2c23f46d04dcc), 3: @at0 (03d2c23f46c44300), 5: @at0 (03d2c23f46c44300)}
          S: wlink 6: rx: "1 watch 0000000000000043 @03d2c23f46d04dcc\n"
          S: wlink 6: tx: "2 pin 0000000000000043 #3 @03d2c23f46c44300\n"
          C: watch  : rx: '2 pin 0000000000000043 #3 @03d2c23f46c44300\n'
          S: wlink 6: tx: "4 pin 0000000000000043 #2 @03d2c23f46d04dcc\n"
          S: wlink 6: tx: "6 pin 0000000000000043 #5 @03d2c23f46c44300\n"
          C: watch  : rx: '4 pin 0000000000000043 #2 @03d2c23f46d04dcc\n'
          C: watch  : rx: '6 pin 0000000000000043 #5 @03d2c23f46c44300\n'
          S: wlink 6: rx: "2 ack\n"
          S: wlink 6: rx: "4 ack\n"
          S: wlink 6: rx: "6 ack\n"
          S: wlink 6: tx: "1 ok\n"
          C: watch  : rx: '1 ok\n'
      
          C: setup watch f<0000000000000043> (@at1 (03d2c23f46d04dcc) ->) @at2 (03d2c23f46e91daa)
          # pin@old: {2: @at1 (03d2c23f46d04dcc), 3: @at0 (03d2c23f46c44300), 5: @at0 (03d2c23f46c44300)}
          # pin@new: {2: @at2 (03d2c23f46e91daa), 3: @at2 (03d2c23f46e91daa), 5: @at2 (03d2c23f46e91daa)}
          #  pinok: {2: @at2 (03d2c23f46e91daa), 3: @at2 (03d2c23f46e91daa), 5: @at2 (03d2c23f46e91daa)}
          S: wlink 6: rx: "3 watch 0000000000000043 @03d2c23f46e91daa\n"
          S: wlink 6: tx: "0 error: 3: another watch request is already in progress\n"
          C: watch  : rx: '0 error: 3: another watch request is already in progress\n'
          C: watch  : rx fatal: 'error: 3: another watch request is already in progress'
          C: watch  : rx: ''
      
      If we would need to maintain the limit, we should move setting
      handlingWatch=0 just before sending final reply to client, but since the
      need for the limit is not there anymore, let's fix it by removing the
      limit altogether.
      43915fe9
  3. 08 Jul, 2019 1 commit
  4. 18 Jun, 2019 3 commits
    • Kirill Smelkov's avatar
      tox -= Python3.5 · 76e8dc34
      Kirill Smelkov authored
      Let's keep on test coverage for last 2 stable Python releases.
      76e8dc34
    • Kirill Smelkov's avatar
      Fix build for Python 3.7 · bca5f79e
      Kirill Smelkov authored
      Starting from Python 3.7 the place to keep exception state was changed:
      https://github.com/python/cpython/commit/ae3087c638
      
      NOTE ZEO4 does not wok with Python3.7, because ZEO4 uses "async" for a
      variable and that breaks because starting from Python3.7 "async" became
      a keyword.
      
      After the fix wendelin.core tests pass under all python2.7, python3.6
      and python3.7.
      bca5f79e
    • Kirill Smelkov's avatar
      tox: ZODB5.test.util needs mock · 6b5384ae
      Kirill Smelkov authored
      Dependency added here:
      
      	https://github.com/zopefoundation/ZODB/commit/e0bc8bd567
      
      If we don't provide mock, e.g. py27-ZODB5-*-zeo-* breaks:
      
      	    def setup_module():
      	        global testdb
      	>       testdb = getTestDB()
      
      	bigarray/tests/test_arrayzodb.py:38:
      	_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      	.tox/py27-ZODB5-zblk1-zeo-numpy115/lib/python2.7/site-packages/wendelin/lib/testing.py:342: in getTestDB
      	    testdb = testdb_factory(testdb_uri)
      	.tox/py27-ZODB5-zblk1-zeo-numpy115/lib/python2.7/site-packages/wendelin/lib/testing.py:245: in __init__
      	    from ZEO.tests import forker
      	.tox/py27-ZODB5-zblk1-zeo-numpy115/lib/python2.7/site-packages/ZEO/tests/forker.py:29: in <module>
      	    import ZODB.tests.util
      	_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      
      	    """
      	    from ZODB.MappingStorage import DB
      
      	    import atexit
      	    import os
      	    import persistent
      	    import re
      	    import tempfile
      	    import time
      	    import transaction
      	    import unittest
      	    import warnings
      	    import ZODB.utils
      	    from ZODB.Connection import TransactionMetaData
      	    import zope.testing.setupstack
      	    from zope.testing import renormalizing
      
      	    try:
      	        from unittest import mock
      	    except ImportError:
      	>       import mock
      	E       ImportError: No module named mock
      
      	.tox/py27-ZODB5-zblk1-zeo-numpy115/lib/python2.7/site-packages/ZODB/tests/util.py:35: ImportError
      6b5384ae
  5. 23 May, 2019 4 commits
  6. 04 Mar, 2019 2 commits
    • Kirill Smelkov's avatar
      . · e1ad2ae3
      Kirill Smelkov authored
      e1ad2ae3
    • Kirill Smelkov's avatar
      X wcfs: Care to disable OS polling on us · 59552328
      Kirill Smelkov authored
      If we don't early-disable it, we can see a situation where when
      handling invalidations wcfs calls open("@revX/bigfile/...") to upload
      cache data there, go runtime tries to use epoll on that fd, and it gets
      stuck as described in the commit referenced in comments.
      
      In particular the deadlock was easy to trigger with nproc=1 environment
      (either a VM with 1 cpu, or e.g. under `taskset -c 0`)
      
      Bug reported by @romain.
      59552328
  7. 22 Feb, 2019 1 commit
  8. 16 Apr, 2018 2 commits
    • Kirill Smelkov's avatar
      tox: v↑ everything · 0dbf3c44
      Kirill Smelkov authored
      - start testing NEO with ZODB4 and ZODB5. ZODB4-*-NEO works while
        ZODB5-*-NEO currently fails [1].
      - update NumPy to latest releases.
      - update Python 3 to latest releases.
      0dbf3c44
    • Kirill Smelkov's avatar
      tox/ZODB3: Limit ZConfig < 3.2.0 · c96daa2f
      Kirill Smelkov authored
      With ZConfig==3.2.0 (released 2017-06-22) ZEO started to get file names
      as unicode, which eventually breaks on assert in ZODB3.FileStorage code:
      
          Traceback (most recent call last):
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/tests/zeoserver.py", line 215, in <module>
              main()
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/tests/zeoserver.py", line 188, in main
              server = ZEO.runzeo.create_server({"1": storage}, zo)
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/runzeo.py", line 366, in create_server
              auth_realm = options.auth_realm,
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/StorageServer.py", line 890, in __init__
              self._setup_invq(name, storage)
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZEO/StorageServer.py", line 928, in _setup_invq
              self.invq[name] = list(lastInvalidations(self.invq_bound))
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZODB/FileStorage/FileStorage.py", line 1238, in lastInvalidations
              for trans in FileIterator(self._file_name, pos=pos)]
            File "/home/kirr/src/wendelin/r/work/venv/local/lib/python2.7/site-packages/ZODB/FileStorage/FileStorage.py", line 1640, in __init__
              assert isinstance(filename, str), `filename`
          AssertionError: u'/home/kirr/src/wendelin/r/work/wendelin.core/Data.fs'
      
      -> So for ZODB3 pin ZConfig to ZConfig <3.2.0 which is known to work with ZODB3.
      c96daa2f
  9. 17 Jan, 2017 1 commit
  10. 06 Jul, 2016 1 commit
    • Kirill Smelkov's avatar
      Fix build for Python 3.5 · e6beab19
      Kirill Smelkov authored
      @kazuhiko reports that wendelin.core build is currently broken on Python 3.5.
      Indeed it was:
      
          In file included from bigfile/_bigfile.c:37:0:
          ./include/wendelin/compat_py2.h: In function ‘_PyThreadState_UncheckedGetx’:
          ./include/wendelin/compat_py2.h:66:28: warning: implicit declaration of function ‘_Py_atomic_load_relaxed’ [-Wimplicit-function-declaration]
               return (PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current);
                                      ^
          ./include/wendelin/compat_py2.h:66:53: error: ‘_PyThreadState_Current’ undeclared (first use in this function)
               return (PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current);
                                                               ^
          ./include/wendelin/compat_py2.h:66:53: note: each undeclared identifier is reported only once for each function it appears in
          ./include/wendelin/compat_py2.h:67:1: warning: control reaches end of non-void function [-Wreturn-type]
           }
           ^
      
      The story here is that in 3.5 they decided to remove direct access to
      _PyThreadState_Current and atomic implementations - because that might
      semantically conflict with other headers implementing atomics - and
      provide only access by function.
      
      Starting from Python 3.5.2rc1 the function to get current thread state
      without asserting it is !NULL - _PyThreadState_UncheckedGet() - was added:
      
          https://github.com/python/cpython/commit/df858591
      
      so for those python versions we can directly use it.
      
      After the fix wendelin.core tox tests pass under all python2.7, python3.4 and python3.5.
      
      More context here:
      
          https://bugs.python.org/issue26154
          https://bugs.python.org/issue25150
      
      Fixes: nexedi/wendelin.core#1
      e6beab19
  11. 01 Jul, 2016 1 commit
  12. 13 Jun, 2016 2 commits
  13. 15 Dec, 2015 1 commit
  14. 24 Sep, 2015 2 commits
    • Kirill Smelkov's avatar
      bigfile/zodb: Format #1 which is optimized for small changes · 13c0c17c
      Kirill Smelkov authored
      Our current approach is that each file block is represented by 1 zodb
      object, with block size being 2M. Even with trailing \0 trimming, which
      halves the overhead on average, DB size grows very fast if we do a lot
      of small appends or changes. So another format needs to be introduced
      which has lower overhead for storing small changes:
      
      In general, to represent BigFile as ZODB objects, each file block could
      be represented separately either as
      
          1) one ZODB object, or          (ZBlk0 - this what we have already)
          2) group of ZODB objects        (ZBlk1 - this is what we introduce)
      
      with top-level BTree directory #blk -> objects representing block.
      
      For "1" we have
      
          - low-overhead access time (only 1 object loaded from DB), but
          - high-overhead in terms of ZODB size (with FileStorage / ZEO, every change
            to a block causes it to be written into DB in full again)
      
      For "2" we have
      
          - low-overhead in terms of ZODB size (only part of a block is overwritten
            in DB on single change), but
          - high-overhead in terms of access time
            (several objects need to be loaded for 1 block)
      
      In general it is not possible to have low-overhead for both i) access-time, and
      ii) DB size, with approach where we do block objects representation /
      management on *client* side.
      
      On the other hand, if object management is moved to DB *server* side, it is
      possible to deduplicate them there and this way have low-overhead for both
      access-time and DB size with just client storing 1 object per file block. This
      will be our future approach after we teach NEO about object deduplication.
      
      ~~~~
      
      As shown above in the last paragraph it is not possible to perform
      optimally on client side. Thus ZBlk1 should be only an intermediate
      solution until we move data management to DB server side, with main
      criteria for ZBlk1 to keep it simple.
      
      In this patch a simple scheme is used, where every block is divided into
      chunks organized via BTree. When a block part changes, only corresponding
      chunk is updated. Chunk size is chosen to be 4K which creates ~ 512
      fanout for 2M block.
      
      DB size after tests is changed as follows:
      
              bigfile     bigarray
      
      ZBlk0     24K       6200K
      ZBlk1     36K         36K
      
      ( slight size increase for bigfile tests is because of btree structures
        overhead )
      
      Time to run tests stays approximately the same.
      
      /cc @Tyagov, @klaus
      13c0c17c
    • Kirill Smelkov's avatar
      bigfile/zodb: Prepare to have several ZBlk formats · 70ea8573
      Kirill Smelkov authored
      - current ZBlk becomes format 0
      - write format can be selected via WENDELIN_CORE_ZBLK_FMT env var
      - upon writing a block we always make sure we write it in current write
        format - so if a block was previously written in one format, it could
        be changed on the next write.
      - tox is prepared to test all write formats (so far only ZBlk0 there).
      
      The reason is - in the next patch we'll introduce another format for
      blocks which is optimized for small changes.
      70ea8573
  15. 06 Aug, 2015 1 commit
  16. 26 Jun, 2015 1 commit
  17. 28 May, 2015 1 commit
  18. 03 Apr, 2015 1 commit