1. 17 Nov, 2020 2 commits
    • Kirill Smelkov's avatar
      bigfile/py: Garbage-collect BigFile <=> BigFileH cycles · a6a8f5ba
      Kirill Smelkov authored
      Since ZBigFile keeps references to fileh objects that are created
      through it it forms a file <=> fileh cycle that is not collected without
      cyclic GC:
      
      https://lab.nexedi.com/nexedi/wendelin.core/blob/v0.13-52-ga702d41/bigfile/file_zodb.py#L497
      https://lab.nexedi.com/nexedi/wendelin.core/blob/v0.13-52-ga702d41/bigfile/file_zodb.py#L566-571
      
      We did not noticed this leak until now because it is small, but with
      upcoming wendelin.core 2 it is important to release a fileh, because
      there is WCFS connection associated with fileh, and if fileh is not
      released, that connection also stays alive, keeping on-WCFS resources
      still being used, and preventing WCFS from being unmounted cleanly.
      
      -> Add cyclic GC support to PyBigFile / PyBigFileH
      
      NOTE: we still don't allow PyVMA <=> PyBigFileH cycles to be collected,
      because fileh_close called from fileh.__del__ asserts that there are no
      live mappings left. See added comments for details. There is no
      known practical need to use such cycles, so this should be ok.
      
      See also other patches on cyclic GC topic:
      
      - 450ad804 (bigarray: ArrayRef support for BigArray)  // adds cyclic GC support for PyVMA
      - d97641d2 (bigfile/py: Properly untrack PyVMA from GC before dealloc)
      
      /proposed-for-review-on !12
      a6a8f5ba
    • Kirill Smelkov's avatar
      bigfile/py: Move PyVMA's support for cyclic GC close to pyvma_dealloc · 7cc35422
      Kirill Smelkov authored
      The logic in pyvma_traverse and pyvma_clear needs to be synchronized
      with PyVMA deallocation. In the next patche we'll be amending this
      logic, and it will help a reader to keep all those functions together.
      
      For the reference: PyVMA support for cyclic GC was introduced in
      450ad804 (bigarray: ArrayRef support for BigArray). See also d97641d2
      (bigfile/py: Properly untrack PyVMA from GC before dealloc).
      
      /proposed-for-review-on !12
      7cc35422
  2. 18 Dec, 2019 3 commits
    • Kirill Smelkov's avatar
      bigfile/py: Move data structures to public .h file · 907bd9d4
      Kirill Smelkov authored
      This is needed so that e.g. a Python class implemented in C or Cython
      (cdef class) could inherit from PyBigFile.
      
      Don't put _bigfile.h into separate include/ directory, and keep it along
      main .c file, similarly to how pygolang is organized.
      907bd9d4
    • Kirill Smelkov's avatar
      bigfile/py: Provide package-level documentation · 3684d164
      Kirill Smelkov authored
      Provide package-level documentation that gives brief overview of what
      this package does. Split internal notes into separate comment.
      3684d164
    • Kirill Smelkov's avatar
      bigfile/py: Stop using Plan9 C extensions · bf44905b
      Kirill Smelkov authored
      Starting from 5755a6b3 (Basic setup.py / Makefile to build/install/sdist
      stuff + bigfile.so skeleton) and 35eb95c2 (bigfile: Python wrapper
      around virtual memory subsystem) we were using Plan9 C extensions[1] for
      simple inheritance. Those extensions are supported by GCC with
      -fplan9-extensions option. However that option is supported only for C,
      while for C++ it does not work at all with error produced by the compiler
      on Plan9 syntax.
      
      Soon we'll need to add another extension - written in C++ - to
      wendelin.core . This extension will be providing client side of WCFS and
      integrating that with virtmem. In that extension we'll need to use
      _bigfile data structures - in particular we'll need to use PyBigFile and
      extend it with another `cdef class` children written in Cython/C++.
      
      This patch prepares for that: first stop using Plan9 C extensions in
      _bigfile py module data structures and adapt the code correspondingly.
      In the next patch we'll move those data structures into an .h file.
      
      We don't drop -fplan9-extensions from setup.py, because Plan9-style
      inheritance continues to be used internally by virtmem - e.g. in
      ram_shmfs.c and friends.
      
      A bit pity to drop that good stuff, but given that we'll need to use C++
      for WCFS client for other good stuff provided by pygolang[2], it is a
      reasonable compromise.
      
      [1] http://9p.io/sys/doc/comp.html  "Extensions" section
      [2] https://pypi.org/project/pygolang
      bf44905b
  3. 15 Jul, 2019 1 commit
    • Kirill Smelkov's avatar
      bigfile/py: It is ok to have .fileh_open() as BigFile method · 7ee02038
      Kirill Smelkov authored
      There was an XXX of whether fileh_open should be a BigFile method or
      global function. However if it would be a global function it will need
      to anyway accept file parameter to indicate which file is opened, and
      that in turn suggests that it should be a file method. Remove XXX.
      7ee02038
  4. 11 Jul, 2019 1 commit
  5. 18 Jun, 2019 1 commit
    • Kirill Smelkov's avatar
      Fix build for Python 3.7 · bca5f79e
      Kirill Smelkov authored
      Starting from Python 3.7 the place to keep exception state was changed:
      https://github.com/python/cpython/commit/ae3087c638
      
      NOTE ZEO4 does not wok with Python3.7, because ZEO4 uses "async" for a
      variable and that breaks because starting from Python3.7 "async" became
      a keyword.
      
      After the fix wendelin.core tests pass under all python2.7, python3.6
      and python3.7.
      bca5f79e
  6. 11 Jan, 2019 1 commit
    • Kirill Smelkov's avatar
      bigfile/py: Properly untrack PyVMA from GC before dealloc · d97641d2
      Kirill Smelkov authored
      On a testing instance we started to see segfaults in pyvma_dealloc()
      with inside calls to vma_unmap but with NULL pyvma->fileh. That was
      strange, becuse before calling vma_unmap(), the code explicitly checks
      whether pyvma->fileh is !NULL.
      
      That was, as it turned out, due to pyvma_dealloc being called twice at the
      same time from two python threads. Here is how that was possible:
      
      T1 decrefs pyvma and finds its reference count drops to zero. It calls
      pyvma_dealloc. From there vma_unmap() is called, which calls virt_lock()
      and that releases GIL first. Another thread T2 was waiting for GIL, it
      acquires it, does some work at python level and somehow triggers GC.
      Since PyVMA supports cyclic GC, it was on GC list and thus GC calls
      dealloc for the same vma again. Here is how it looks in the backtraces:
      
      T1:
      
      	#0  0x00007f6aefc57827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x1e011d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      	#1  do_futex_wait (sem=sem@entry=0x1e011d0, abstime=0x0) at sem_waitcommon.c:111
      	#2  0x00007f6aefc578d4 in __new_sem_wait_slow (sem=0x1e011d0, abstime=0x0) at sem_waitcommon.c:181
      	#3  0x00007f6aefc5797a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
      	#4  0x00000000004ffbc4 in PyThread_acquire_lock ()
      	#5  0x00000000004dbe8a in PyEval_RestoreThread ()
      	#6  0x00007f6ac6d3b8fc in py_gil_retake_if_waslocked (arg=0x4f18f00) at bigfile/_bigfile.c:1048
      	#7  0x00007f6ac6d3dcfc in virt_gil_retake_if_waslocked (gilstate=0x4f18f00) at bigfile/virtmem.c:78
      	#8  0x00007f6ac6d3dd30 in virt_lock () at bigfile/virtmem.c:92
      	#9  0x00007f6ac6d3e724 in vma_unmap (vma=0x7f6a7e0c4100) at bigfile/virtmem.c:271
      	#10 0x00007f6ac6d3a0bc in pyvma_dealloc (pyvma0=0x7f6a7e0c40e0) at bigfile/_bigfile.c:284
      	...
      	#13 0x00000000004d76b0 in PyEval_EvalFrameEx ()
      
      T2:
      
      	#5  0x00007f6ac6d3a081 in pyvma_dealloc (pyvma0=0x7f6a7e0c40e0) at bigfile/_bigfile.c:276
      	#6  0x0000000000500450 in ?? ()
      	#7  0x00000000004ffd82 in _PyObject_GC_New ()
      	#8  0x0000000000485392 in PyList_New ()
      	#9  0x00000000004d3bff in PyEval_EvalFrameEx ()
      
      T2 does the work of vma_unmap and clears C-level vma. Then, when T1 wakes up and
      returns to vma_unmap, it sees vma->file and all other fields cleared -> oops
      segfault.
      
      Fix it by removing pyvma from GC list before going to do actual destruction.
      This way if a concurrent GC triggers, it won't see the vma object on its list,
      and thus won't have a chance to invoke its destructor the second time.
      
      The bug was introduced in 450ad804 (bigarray: ArrayRef support for BigArray)
      when PyVMA was changed to be cyclic-GC aware. However at that time, even Python
      documentation itself was not saying PyObject_GC_UnTrack is needed, as it was
      added only in 2.7.15 after finding that many types in CPython itself are
      vulnerable to similar segfaults:
      
      https://github.com/python/cpython/commit/4cde4bdcc86
      https://bugs.python.org/issue31095
      
      It is pity, that CPython took the approach to force all type authors to
      care to invoke PyObject_GC_UnTrack explicitly, instead of doing that
      automatically in Python runtime before calling tp_dealloc.
      
      /cc @Tyagov, @klaus
      /reviewed-on !11
      d97641d2
  7. 02 Apr, 2018 1 commit
    • Kirill Smelkov's avatar
      bigarray: ArrayRef support for BigArray · 450ad804
      Kirill Smelkov authored
      Rationale
      ---------
      
      Array reference could be useful in situations where one needs to pass arrays
      between processes and instead of copying array data, leverage the fact that
      top-level array, for example ZBigArray, is already persisted separately, and
      only send small amount of information referencing data in question.
      
      Implementation
      --------------
      
      BigArray is not regular NumPy array and so needs explicit support in
      ArrayRef code to find root object and indices. This patch adds such
      support via the following way:
      
      - when BigArray.__getitem__ creates VMA, it remembers in the VMA
        the top-level BigArray object under which this VMA was created.
      
      - when ArrayRef is finding root, it can detect such VMAs, because it will
        be pointed to by the most top regular ndarray's .base, and in turn gets
        top-level BigArray object from the VMA.
      
      - further all indices computations are performed, similarly to complete regular
        ndarrays case, on ndarrays root and a. But in the end .lo and .hi are
        adjusted for the corresponding offset of where root is inside whole
        BigArray.
      
      - there is no need to adjust .deref() at all.
      
      For remembering information into a VMA and also to be able to get
      (readonly) its mapping addresses _bigfile.c extension has to be extended
      a bit. Since we are now storing arbitrary python object attached to
      PyVMA - it can create cycles - and so PyVMA accordingly adjusted to
      support cyclic garbage collector.
      
      Please see the patch itself for more details and comments.
      450ad804
  8. 24 Oct, 2017 1 commit
    • Kirill Smelkov's avatar
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source... · f11386a4
      Kirill Smelkov authored
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options.
      
      Nexedi stack is licensed under Free Software licenses with various exceptions
      that cover three business cases:
      
      - Free Software
      - Proprietary Software
      - Rebranding
      
      As long as one intends to develop Free Software based on Nexedi stack, no
      license cost is involved. Developing proprietary software based on Nexedi stack
      may require a proprietary exception license. Rebranding Nexedi stack is
      prohibited unless rebranding license is acquired.
      
      Through this licensing approach, Nexedi expects to encourage Free Software
      development without restrictions and at the same time create a framework for
      proprietary software to contribute to the long term sustainability of the
      Nexedi stack.
      
      Please see https://www.nexedi.com/licensing for details, rationale and options.
      f11386a4
  9. 21 Aug, 2017 1 commit
    • Kirill Smelkov's avatar
      bigfile/py: Don't forget to clear exception state after retrieving pybuf referrers · 4228d8b6
      Kirill Smelkov authored
      A buffer object (pybuf) is passed by C-level loadblk to python loadblk
      implementation. Since pybuf points to memory that will go away after
      loadblk call returns to virtmem, PyBigFile tries hard to make sure
      nothing stays referencing pybuf so it can be released.
      
      It tries to:
      
      1. automatically GC cycles referencing pybuf (9aa6a5d7 "bigfile/py: Teach
         loadblk() to automatically break reference cycles to pybuf")
      2. replace pybuf with stub object if a calling frame referencing it still
         stays alive (61b18a40 "bigfile/py/loadblk: Replace pybuf with a stub
         object in calling frame in case it stays alive")
      3. and as a last resort unpins pybuf from original buffer memeory to
         point it to NULL (024c246c "bigfile/py/loadblk: Resort to pybuf
         unpinning, if nothing helps")
      
      Step #1 invokes GC. Step #2 calls gc.get_referrers(pybuf) and looks for
      frames in there.
      
      The gc.get_referrers() call happens at python level with allocating some
      objects, e.g. tuple to pass arguments, resulting list etc. And we all
      know that any object allocation might cause automatic garbage
      collection, and GC'ing can in turn ran arbitrary code due to __del__ in
      release objects and weakrefs callbacks.
      
      At a first glance the scenario that GC will be triggered at step #2
      looks unrealistic because the GC was just run at step #1 and it is only
      a few objects being allocated for the call at step #2. However if
      arbitrary code runs from under GC it can create new garbage and thus
      upon returning from gc.collect() the garbage list is not empty as the
      following program demonstrates:
      
          ---- 8< ----
          import gc
      
          # just an object we can set attributes on
          class X:
              pass
      
          # call f on __del__
          class DelCall:
              def __init__(self, f):
                  self.f = f
      
              def __del__(self):
                  self.f()
      
          # _mkgarbage creates n objects of garbage kept referenced from an object cycle
          # so that only cyclic GC can free them.
          def _mkgarbage(n):
              # cycle
              a, b = X(), X()
              a.b, b.a = b, a
      
              # cycle references [n] garbage
              a.objv = [X() for _ in range(n)]
              return a
      
          # mkgarbage creates cycled garbage and arranges for twice more garbage to be
          # created when original garbage is collected
          def mkgarbage(n):
              a = _mkgarbage(n)
              a.ondel = DelCall(lambda : _mkgarbage(2*n))
      
          def main():
              for i in xrange(10):
                  mkgarbage(1000)
                  print '> %s' % (gc.get_count(),)
                  n = gc.collect()
                  print '< %s' % (gc.get_count(),)
      
          main()
          ---- 8< ----
      
          kirr@deco:~/tmp/trashme/t$ ./gcmoregarbage.py
          > (482, 11, 0)
          < (1581, 0, 0)
          > (531, 3, 0)
          < (2070, 0, 0)
          > (531, 3, 0)
          < (2070, 0, 0)
          > (531, 3, 0)
          < (2070, 0, 0)
          > (531, 3, 0)
          < (2070, 0, 0)
          > (531, 3, 0)
          < (2070, 0, 0)
          > (531, 3, 0)
          < (2070, 0, 0)
          > (531, 3, 0)
          < (2070, 0, 0)
          > (531, 3, 0)
          < (2070, 0, 0)
          > (531, 3, 0)
          < (2070, 0, 0)
      
      here lines starting with "<" show amount of live garbage objects after
      gc.collect() call has been finished.
      
      This way on a busy server there could be arrangements when GC is
      explicitly ran at step #1 and then automatically run at step #2 (because of
      gc.get_referrers() python-level call) and from under GC #2 arbitrary code runs
      thus potentially mutating exception state which shows in logs as
      
          bigfile/_bigfile.c:685: pybigfile_loadblk: Assertion `!(ts->exc_type || ts->exc_value || ts->exc_traceback)' failed.
      
      ----
      
      So don't assume we end with clean exception state after collecting pybuf
      referrers and just clear exception state once again as we do after explicit GC.
      Don't make a similar assumption for buffer unpinning as an object is
      decrefed there and in theory this can run some code.
      
      A test is added to automatically exercise exception state clearing for
      get_referrers code path via approach similar to demonstrated in above -
      - we generate more garbage from under garbage and also arrange for
      finalizers, which mutate exceptions state, to be run at GC #2.
      
      The test without the fix applied fails like this:
      
          bigfile/_bigfile.c:710 pybigfile_loadblk WARN: python thread-state found with handled but not cleared exception state
          bigfile/_bigfile.c:711 pybigfile_loadblk WARN: I will dump it and then crash
          ts->exc_type:   None
          ts->exc_value:  <nil>
          ts->exc_traceback:      <nil>
          Segmentation fault (core dumped)
      
      The None in ts->exc_type and nil value and traceback are probably coming from
      here in cpython runtime:
      
          https://github.com/python/cpython/blob/883520a8/Python/ceval.c#L3717
      
      Since this took some time to find, more diagnostics is also added before
      BUG_ONs corresponding to finding unclean exception state.
      4228d8b6
  10. 17 Aug, 2017 1 commit
    • Kirill Smelkov's avatar
      bigfile/py: Stop caring about sys.exc_{type,value,traceback} variables · 3804cc39
      Kirill Smelkov authored
      Before py3k python stores exception information not only in thread-local state
      but also globally in sys.exc_* variables (wrt sys.exc_info()) for
      "backward compatibility". However using them is not thread-safe as the
      following example demonstrates:
      
      ---- 8< ----
      from threading import Thread
      import sys
      
      def T1():
          print 'T1'
          while 1:
              exc_type = sys.exc_type
              if exc_type is not None:
                  print 'AAA: %r' % exc_type
      
      def f(): g()
      def g(): h()
      def h(): 1/0
      
      def T2():
          print 'T2'
          while 1:
              try:
                  f()
              except:
                  pass
      
      t1, t2 = Thread(target=T1), Thread(target=T2)
      t1.start(); t2.start()
      ---- 8< ----
      
      ---- 8< ----
      kirr@deco:~/tmp/trashme/t$ ./excthreads.py
      T1
      T2
      
      AAA: <type 'exceptions.ZeroDivisionError'>
      AAA: <type 'exceptions.ZeroDivisionError'>
      AAA: <type 'exceptions.ZeroDivisionError'>
      AAA: <type 'exceptions.ZeroDivisionError'>
      AAA: <type 'exceptions.ZeroDivisionError'>
      AAA: <type 'exceptions.ZeroDivisionError'>
      ^\Выход
      ---- 8< ----
      
      Because of the above nothing modern (I've explicitly checked at least CPython
      itself and Zope) uses this variables - wherever needed per-thread exception
      state is retrieved with sys.exc_info().
      
      So on wendelin.core side it is thus thankless job to try to preserve sys.exc_*
      vars state because on a busy server they are literally changing all the -
      arbitrary from the point of view of particular thread - time while its python
      code runs.
      3804cc39
  11. 16 Jan, 2017 3 commits
    • Kirill Smelkov's avatar
      bigfile/py/loadblk: Resort to pybuf unpinninf, if nothing helps · 024c246c
      Kirill Smelkov authored
      There are situations possible when both exc_traceback and frame objects are
      garbage-collected, but frame's f_locals remains not collected because e.g. it
      was explicitly added to somewhere. We cannot detect such cases (dicts are not
      listed in referrers).
      
      So if nothing helped, as a last resort, unpin pybuf from its original
      memory and make it point to zero-sized NULL.
      
      In general this is not strictly correct to do as other buffers &
      memoryview objects created from pybuf, copy its pointer on
      initialization and thus pybuf unpinning won't adjust them.
      
      However we require BigFile implementations to make sure not to use
      such-created objects, if any, after return from loadblk().
      
      Finally fixes #7
      024c246c
    • Kirill Smelkov's avatar
      bigfile/py: Factor out code to "unpin" a buffer to separate functions · 3a8e1beb
      Kirill Smelkov authored
      This code was added in 6da5172e (bigfile/py: Teach storeblk() how to
      correctly propagate traceback on error) to unpin a storeblk pybuf to not
      care whether its refcount == 1 - this way to be able to propagate python
      error upper not caring whether pybuf is still referenced or not.
      
      9aa6a5d7 (bigfile/py: Teach loadblk() to automatically break reference
      cycles to pybuf) adds a note that such unpinning is not strictly
      correct: becuase of other buffer objects were created from pybuf - they
      are copying pointers on initialization and unpinning pybuf won't adjust
      them.
      
      However for loadblk codepath it turned out (see next patch) it is not
      completely possible to unreference pybuf in all cases. For this reason
      loadblk will be falling back to unpinning too.
      
      As a preparatory step move common code to shared functions.
      3a8e1beb
    • Kirill Smelkov's avatar
      bigfile/py/loadblk: Replace pybuf with a stub object in calling frame in case it stays alive · 61b18a40
      Kirill Smelkov authored
      It turns out some code wants to store tracebacks e.g. for further
      logging/whatever. This way GC won't help to free up references to pybuf.
      However if pybuf remain referenced only from calling frames, we can
      change there reference to pybuf to a stub object "<pybuf>" and this way
      remove the reference.
      
      With added test but without loadblk changes the failure would be as:
      
          pybigfile_loadblk WARN: pybuf->ob_refcnt != 1 even after GC:
          pybuf (ob_refcnt=2):    <read-write buffer ptr 0x7fae4911f000, size 2097152 at 0x7fae4998cef0>
          pybuf referrers:        [<frame object at 0x556daff41aa0>]		<-- NOTE
          bigfile/_bigfile.c:613 pybigfile_loadblk        BUG!
      61b18a40
  12. 12 Jan, 2017 2 commits
    • Kirill Smelkov's avatar
      bigfile/py: Factor outcode to get list of objects that refer to obj to XPyObject_GetReferrers() · ca7c1b6d
      Kirill Smelkov authored
      We'll need it in next patch to get and analyze this list.
      ca7c1b6d
    • Kirill Smelkov's avatar
      bigfile/py: Dump pybuf referrers if pybuf->ob_refcnt != 1 before dying in loadblk epilogue · 20b41a5a
      Kirill Smelkov authored
      Instead of only printing "BUG" let's print information about objects
      which still refer to pybuf - to help debugging.
      
      For example with the following artificial pybuf leak
      
      ```
      diff --git a/bigfile/tests/test_basic.py b/bigfile/tests/test_basic.py
      index c737621..f5e057a 100644
      --- a/bigfile/tests/test_basic.py
      +++ b/bigfile/tests/test_basic.py
      @@ -126,6 +126,7 @@ def test_basic():
      
       # test that python exception state is preserved across pagefaulting
       def test_pagefault_savestate():
      +    zzz = []
           class BadFile(BigFile):
               def loadblk(self, blk, buf):
                   # simulate some errors in-between to overwrite thread exception
      @@ -154,6 +155,7 @@ def loadblk(self, blk, buf):
                   # which result in holding additional ref to buf, but loadblk caller
                   # will detect and handle this situation via garbage-collecting
                   # above cycle.
      +            zzz.append(buf)
      
                   self.loadblk_run = 1
      ```
      
      it dies this way:
      
          bigfile/_bigfile.c:567 pybigfile_loadblk WARN: pybuf->ob_refcnt != 1 even after GC:
          pybuf (ob_refcnt=2):    <read-write buffer ptr 0x7f08d3e88000, size 2097152 at 0x7f08d48b7070>
          pybuf referrers:        [[<read-write buffer ptr 0x7f08d3e88000, size 2097152 at 0x7f08d48b7070>]]
          bigfile/_bigfile.c:573 pybigfile_loadblk        BUG!
      20b41a5a
  13. 11 Jan, 2017 2 commits
    • Kirill Smelkov's avatar
      bigfile/py: Teach loadblk() to automatically break reference cycles to pybuf · 9aa6a5d7
      Kirill Smelkov authored
      Because otherwise we bug on pybuf->ob_refcnt != 1.
      
      Such cycles might happen if inside loadblk implementation an exception
      is internally raised and then caught even in deeply internal function
      which does not receive pybuf as argument or by some other way:
      
      After
      
      	_, _, exc_traceback = sys.exc_info()
      
      there is a reference loop created:
      
      	exc_traceback
      	  |        ^
      	  |        |
      	  v     .f_localsplus
      	 frame
      
      and since exc_traceback object holds reference to deepest frame, which via f_back
      will be holding reference to frames up to frame with pybuf argument, it
      will result in additional reference to pybuf being held until the above
      cycle is garbage collected.
      
      So to solve the problem while leaving loadblk, if pybuf->ob_refcnt !=
      let's first do garbage-collection, and only then recheck left
      references. After GC reference-loops created by exceptions should go
      away.
      
      NOTE PyGC_Collect() (C way to call gc.collect()) always performs
          GC - it is not affected by gc.disable() which disables only
          _automatic_ garbage collection.
      
      NOTE it turned out out storeblk logic to unpin pybuf (see
          6da5172e "bigfile/py: Teach storeblk() how to correctly propagate
          traceback on error") is flawed, because when e.g. creating memoryview
          from pybuf internal pointer is copied and then clearing original buf
          does not result in clearing the copy.
      
      NOTE it is ok to do gc.collect() from under sighandler - at least we are
          already doing it for a long time via running non-trivial python code
          which for sure triggers automatic GC from time to time (see also
          786d418d "bigfile: Simple test that we can handle GC from-under
          sighandler" for the reference)
      
      Fixes: #7
      9aa6a5d7
    • Kirill Smelkov's avatar
      bigfile/py: Factor exception state clearing code to XPyErr_FullClear() · b4c269eb
      Kirill Smelkov authored
      In the next patch we will need to use this from several places.
      b4c269eb
  14. 10 Jan, 2017 1 commit
    • Kirill Smelkov's avatar
      bigfile/virtmem: Maintain dirty pages list for a fileh · 8bb7f2f2
      Kirill Smelkov authored
      This allows writeout code not to scan whole pagemap to find dirty pages
      to write out, which should be faster.
      
      But more importantly iterating whole pagemap on writeout would become
      unsafe, when in upcoming patch storeblk() will be called with virt_lock
      released: because there pagemap could be modified e.g. due to processing
      other read accesses.
      
      So maintain fileh->dirty_pages list and use it when we need to go
      through dirtied pages.
      
      Updates: #6
      8bb7f2f2
  15. 01 Jul, 2016 1 commit
    • Kirill Smelkov's avatar
      bigfile/virtmem: Move _PyThreadState_Current_GET to compat_py2.h · 20115391
      Kirill Smelkov authored
      _PyThreadState_Current_GET() is a function to get current python thread
      state without asserting it is !NULL. It was added as part of d53271b9
      (bigfile/virtmem: Big Virtmem lock)
      
      We are going to adapt it to Python 3.5 (see next patch), so before doing
      so move it to our compatibility place.
      
      In the new place the name is _PyThreadState_UncheckedGet -- like such
      function is named in Python 3.5 (again, see next patch).
      
      Updates: #1
      20115391
  16. 17 Aug, 2015 1 commit
    • Kirill Smelkov's avatar
      bigfile: ZODB -> BigFileH invalidate propagation · 92bfd03e
      Kirill Smelkov authored
      Continuing theme from the previous patch, here is propagation of
      invalidation messages from ZODB to BigFileH memory.
      
      The use-case here is that e.g. one fileh mapping was created in one
      connection, another in another, and after doing changes in second
      connection and committing there, the first fileh has to invalidate
      appropriate already-loaded pages, so its next transaction won't work
      with stale data.
      
      To do it, we hook into ZBlk._p_invalidate() and propagate the
      invalidation message to ZBigFile which then notifies all
      opened-through-it ZBigFileH to invalidate a page.
      
      ZBlk -> ZBigFile lookup is done without storing backpointer in ZODB -
      instead, every time ZBigFile touches ZBlk object (and thus potentially
      does GHOST -> Live transition to it), we (re-)bind it back to ZBigFile.
      Since ZBigFile is the only class that works with ZBlk objects it is safe
      to do so.
      
      For ZBigFile to notify "all-opened-through-it" ZBigFileH, a weakset is
      introduced to track them.
      
      Otherwise the real page invalidation work is done by virtmem (see
      previous patch).
      92bfd03e
  17. 09 Aug, 2015 1 commit
    • Kirill Smelkov's avatar
      bigfile/py: Teach storeblk() how to correctly propagate traceback on error · 6da5172e
      Kirill Smelkov authored
      Previously we were limited to printing traceback starting down from just
      storeblk() via explicit PyErr_PrintEx() - because pybuf was attached to
      memory which could go away right after return from C function - so we
      had to destroy that object for sure, not letting any traceback to hold a
      reference to it.
      
      This turned out to be too limiting and not showing full context where
      errors happen.
      
      So do the following trick: before returning, reattach pybuf to empty
      region at NULL, and this way we don't need to worry about pybuf pointing
      to memory which can go away -> thus instead of printing exception locally
      - just return it the usual way it is done with C api in Python.
      
      NOTE In contrast to PyMemoryViewObject, PyBufferObject definition is not
      public, so to support Python2 - had to copy its definition to PY2 compat
      header.
      
      NOTE2 loadblk() is not touched - the loading is done from sighandler
      context, which simulates as if it work in separate python thread, so it
      is leaved as is for now.
      6da5172e
  18. 06 Aug, 2015 1 commit
    • Kirill Smelkov's avatar
      bigfile/virtmem: Big Virtmem lock · d53271b9
      Kirill Smelkov authored
      At present several threads running can corrupt internal virtmem
      datastructures (e.g. ram->lru_list, fileh->pagemap, etc).
      
      This can happen even if we have zope instances only with 1 worker thread
      - because there are other "system" thread, and python garbage collection
      can trigger at any thread, so if a virtmem object, e.g. VMA or FileH was
      there sitting at GC queue to be collected, their collection, and thus
      e.g. vma_unmap() and fileh_close() will be called from
      different-from-worker thread.
      
      Because of that virtmem just has to be aware of threads not to allow
      internal datastructure corruption.
      
      On the other hand, the idea of introducing userspace virtual memory
      manager turned out to be not so good from performance and complexity
      point of view, and thus the plan is to try to move it back into the
      kernel. This way it does not make sense to do a well-optimised locking
      implementation for userspace version.
      
      So we do just a simple single "protect-all" big lock for virtmem.
      
      Of a particular note is interaction with Python's GIL - any long-lived
      lock has to be taken with GIL released, because else it can deadlock:
      
          t1  t2
      
          G
          V   G
         !G   V
          G
      
      so we introduce helpers to make sure the GIL is not taken, and to retake
      it back if we were holding it initially.
      
      Those helpers (py_gil_ensure_unlocked / py_gil_retake_if_waslocked) are
      symmetrical opposites to what Python provides to make sure the GIL is
      locked (via PyGILState_Ensure / PyGILState_Release).
      
      Otherwise, the patch is more-or-less straightforward application for
      one-big-lock to protect everything idea.
      d53271b9
  19. 02 Jun, 2015 3 commits
  20. 03 Apr, 2015 2 commits
    • Kirill Smelkov's avatar
      bigfile: Python wrapper around virtual memory subsystem · 35eb95c2
      Kirill Smelkov authored
      Exposes BigFile - this way users can define BigFile backend in Python.
      
      Also exposed are BigFile handles, and VMA objects which are results of
      mmaping.
      35eb95c2
    • Kirill Smelkov's avatar
      Basic setup.py / Makefile to build/install/sdist stuff + bigfile.so skeleton · 5755a6b3
      Kirill Smelkov authored
      It is an early project decision to use gnu99 & Plan9 C extensions, to
      simplify C code.
      
      So far we only build stub wendelin/bigfile/_bigfile.so .
      
      Makefile is introduced because there will be targets which are easier to
      handle at make level. For end users no make knowledge is required -
      usual `python setup.py build|install|...` work, redirecting to make
      where necessary.
      
      As was promised (e870781d "Top-level in-tree import redirector")
      setup.py contains install-time hooks to handle in-tree wendelin.py and
      install it as a module namespace.
      
      For sdist, we just use `git ls-files` info if we are in a checkout.
      5755a6b3