1. 04 Feb, 2020 7 commits
    • Kirill Smelkov's avatar
      golang: qq: Use u for UTF-8 decoding · 3073ac98
      Kirill Smelkov authored
      U is preffered way to make sure an object is unicode string.
      3073ac98
    • Kirill Smelkov's avatar
      gcompat: Move qq into golang · 8c459a99
      Kirill Smelkov authored
      This will allow to integrate qq with u in the next patch.
      
      Moving to compiled code for string processing functions is also
      generally better for performance.
      8c459a99
    • Kirill Smelkov's avatar
      golang: Provide b, u for strings · bcb95cd5
      Kirill Smelkov authored
      With Python3 I've got tired to constantly use .encode() and .decode();
      getting exception if original argument was unicode on e.g. b.decode();
      getting exception on raw bytes that are invalid UTF-8, not being able to
      use bytes literal with non-ASCII characters, etc.
      
      So instead of this pain provide two functions that make sure an object
      is either bytes or unicode:
      
      - b converts str/unicode/bytes s to UTF-8 encoded bytestring.
      
      	Bytes input is preserved as-is:
      
      	   b(bytes_input) == bytes_input
      
      	Unicode input is UTF-8 encoded. The encoding always succeeds.
      	b is reverse operation to u - the following invariant is always true:
      
      	   b(u(bytes_input)) == bytes_input
      
      - u converts str/unicode/bytes s to unicode string.
      
      	Unicode input is preserved as-is:
      
      	   u(unicode_input) == unicode_input
      
      	Bytes input is UTF-8 decoded. The decoding always succeeds and input
      	information is not lost: non-valid UTF-8 bytes are decoded into
      	surrogate codes ranging from U+DC80 to U+DCFF.
      	u is reverse operation to b - the following invariant is always true:
      
      	   u(b(unicode_input)) == unicode_input
      
      NOTE: encoding _and_ decoding *never* fail nor loose information. This
      is achieved by using 'surrogateescape' error handler on Python3, and
      providing manual fallback that behaves the same way on Python2.
      
      The naming is chosen with the idea so that b(something) resembles
      b"something", and u(something) resembles u"something".
      
      This, even being only a part of strings solution discussed in [1],
      should help handle byte- and unicode- strings in more robust and
      distraction free way.
      
      Top-level documentation is TODO.
      
      [1] nexedi/zodbtools!13
      bcb95cd5
    • Kirill Smelkov's avatar
      libgolang: Provide Nil as alias for std::nullptr_t · 230c81c4
      Kirill Smelkov authored
      This continues 60f6db6f (libgolang: Provide nil as alias for nullptr and
      NULL): I've tried to compile pygolang with Clang on my Debian 10
      workstation and got:
      
          $ CC=clang CXX=clang++ python setup.py build_dso -i
      
          In file included from ./golang/fmt.h:32:
          ./golang/libgolang.h:381:11: error: unknown type name 'nullptr_t'; did you mean 'std::nullptr_t'?
          constexpr nullptr_t nil = nullptr;
                    ^~~~~~~~~
                    std::nullptr_t
          /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/c++config.h:242:29: note: 'std::nullptr_t' declared here
            typedef decltype(nullptr)     nullptr_t;
                                          ^
          :
          In file included from ./golang/context.h
          In file included from golang/runtime/libgolang.cpp:30:
          ./golang/libgolang.h:381:11: error: unknown type name 'nullptr_t'; did you mean 'std::nullptr_t'?
          constexpr nullptr_t nil = nullptr;
                    ^~~~~~~~~
                    std::nullptr_t
          /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/c++config.h:242:29: note: 'std::nullptr_t' declared here
            typedef decltype(nullptr)     nullptr_t;
                                          ^
          :39:
          ./golang/libgolang.h:381:11: error: unknown type In file included from golang/fmt.cpp:25:
          In file included from ./golang/fmt.h:32:
          ./golang/libgolang.h:421:17: error: unknown type name 'nullptr_t'; did you mean 'std::nullptr_t'?
              inline chan(nullptr_t) { _ch = nil; }
                          ^~~~~~~~~
                          std::nullptr_t
      
          ...
      
      It seems with GCC and Clang under macOS nullptr_t is automatically provided in
      builtin namespace, while with older Clang on Linux (clang version 7.0.1-8) only
      in std:: namespace - rightfully as nullptr_t is described to be present there:
      
      https://en.cppreference.com/w/cpp/types/nullptr_t
      
      This way we either have to correct all occurrences of nullptr_t to
      std::nullptr_t, or do something similar with providing nil under golang:: .
      
      To reduce noise I prefer the later and let it be named as Nil.
      230c81c4
    • Kirill Smelkov's avatar
      golang: tests: pypanicWhenBlocked: Fix thinko in __exit__ · b5a2f9dc
      Kirill Smelkov authored
      The code was assigning nil to local, _not_ global _tblockforever. As a
      result _tblockforever was left set with a test hook even after leaving
      test context. Fix it.
      
      The bug was there starting from 3b241983 (Port/move channels to
      C/C++/Pyx).
      
      Had to change `= nil` to `= NULL` because with nil Cython complains as
      
              def __exit__(pypanicWhenBlocked t, typ, val, tb):
                  global _tblockforever
                  _tblockforever = nil
                                  ^
          ------------------------------------------------------------
      
          golang/_golang_test.pyx:86:25: Cannot assign type 'nullptr_t' to 'void (*)(void) nogil'
      
      This is https://github.com/cython/cython/issues/3314.
      b5a2f9dc
    • Kirill Smelkov's avatar
      time: Kill unused nullptr_t cimport · 9b63ec01
      Kirill Smelkov authored
      It's a leftover originating from b073f6df (time: Move/Port timers to
      C++/Pyx nogil).
      9b63ec01
    • Kirill Smelkov's avatar
      context: Fix thinko/typo in PyContext.__dealloc__ · 371d50b5
      Kirill Smelkov authored
      Instead of `pyctx.ctx = nil` it was just `ctx = nil` - i.e. assign nil
      to local variable instead of changing pyctx instance data. We were not
      observing this bug because Cython, for C++ fields of cdef classes,
      automatically emits in-place destructor calls in generated __dealloc__
      
          https://github.com/cython/cython/blob/0.29.14-11-g8c620c388/Cython/Compiler/ModuleNode.py#L1477-L1478
      
      and so this way there was no leak. However we want to be explicit and the
      code was not correct. Fix it.
      
      The bug was there from 2a359791 (context: Move/Port context package to
      C++/Pyx nogil).
      371d50b5
  2. 17 Jan, 2020 3 commits
    • Kirill Smelkov's avatar
      *: NULL -> nil (Pyx and only where we can) · 01ade7ac
      Kirill Smelkov authored
      Convert Pyx part of the project to use nil instead of NULL.
      
      Not every usage of NULL was converted and some places were left to use
      NULL where changing it to nil currently hits Cython compilation error:
      
      https://github.com/cython/cython/issues/3314
      01ade7ac
    • Kirill Smelkov's avatar
      *: NULL/nullptr -> nil (C++ only) · fc1c3e24
      Kirill Smelkov authored
      Convert C++ part of the project to use nil instead of NULL/nullptr.
      
      We do not convert pyx part yet, because Cython currently does not
      understand that nullptr_t has properties of NULL and with e.g. the
      following change
      
          --- a/golang/_context.pyx
          +++ b/golang/_context.pyx
          @@ -116,7 +116,7 @@ cdef cppclass _PyValue (_interface, gobject) nogil:
               __dealloc__():
                   with gil:
                       obj = <object>this.pyobj
          -            this.pyobj = NULL
          +            this.pyobj = nil
                       Py_DECREF(obj)
      
      errors as
      
          Error compiling Cython file:
          ------------------------------------------------------------
          ...
                  if __decref():
                      del self
              __dealloc__():
                  with gil:
                      obj = <object>this.pyobj
                      this.pyobj = nil
                                  ^
          ------------------------------------------------------------
      
          golang/_context.pyx:119:25: Cannot assign type 'nullptr_t' to 'PyObject *'
      
      https://github.com/cython/cython/issues/3314
      fc1c3e24
    • Kirill Smelkov's avatar
      libgolang: Provide nil as alias for nullptr and NULL · 60f6db6f
      Kirill Smelkov authored
      Nil is more native to Go.
      60f6db6f
  3. 13 Jan, 2020 1 commit
  4. 08 Jan, 2020 1 commit
  5. 06 Dec, 2019 1 commit
  6. 27 Nov, 2019 9 commits
    • Kirill Smelkov's avatar
      pygolang v0.0.5 · c5c3071b
      Kirill Smelkov authored
      This release is driven by wendelin.core v2 needs with one of the changes
      being that now most of the library was moved into nogil code and can be
      used fully from inside nogil world(*). Python modules are now just wrappers
      of their nogil counterparts. The way for Python and nogil worlds to
      communicate is also provided.
      
      The move to nogil required many other enhancements along the way. Please
      see CHANGELOG for overview.
      
      The move to nogil brought some speedup automatically.
      Below are benchmark results of this release compared to pygolang v0.0.4
      (1573d101) for python-level benchmarks (we have only those at present):
      
      	(on i7@2.6GHz)
      
      thread runtime:
      
          name             old time/op  new time/op  delta
          go               18.3µs ± 0%  18.3µs ± 1%     ~     (p=1.000 n=10+10)
          chan             2.91µs ± 3%  2.99µs ± 5%   +2.73%  (p=0.022 n=10+10)
          select           3.57µs ± 3%  3.57µs ± 4%     ~     (p=0.720 n=9+10)
          def              55.0ns ± 0%  54.0ns ± 0%   -1.82%  (p=0.002 n=8+10)
          func_def         43.8µs ± 2%  44.1µs ± 1%   +0.64%  (p=0.035 n=10+9)
          call             64.0ns ± 0%  66.3ns ± 1%   +3.59%  (p=0.000 n=10+10)
          func_call        1.05µs ± 1%  1.24µs ± 0%  +17.80%  (p=0.000 n=10+7)
          try_finally       138ns ± 0%   137ns ± 1%   -0.51%  (p=0.003 n=10+10)
          defer            2.32µs ± 1%  2.63µs ± 1%  +13.52%  (p=0.000 n=10+10)
          workgroup_empty  38.0µs ± 1%  24.1µs ± 1%  -36.43%  (p=0.000 n=10+10)
          workgroup_raise  47.7µs ± 1%  28.2µs ± 0%  -40.76%  (p=0.000 n=10+10)
      
      gevent runtime:
      
          name             old time/op  new time/op  delta
          go               16.9µs ± 1%  17.2µs ± 2%   +1.94%  (p=0.000 n=10+10)
          chan             7.43µs ± 0%  7.82µs ± 0%   +5.34%  (p=0.000 n=10+7)
          select           10.5µs ± 0%  11.2µs ± 0%   +6.74%  (p=0.000 n=10+10)
          def              63.0ns ± 0%  57.6ns ± 1%   -8.57%  (p=0.000 n=9+10)
          func_def         44.0µs ± 1%  44.2µs ± 1%     ~     (p=0.063 n=10+10)
          call             67.0ns ± 0%  64.0ns ± 0%   -4.48%  (p=0.002 n=8+10)
          func_call        1.06µs ± 1%  1.23µs ± 1%  +16.50%  (p=0.000 n=10+10)
          try_finally       144ns ± 0%   136ns ± 0%   -5.90%  (p=0.000 n=10+10)
          defer            2.37µs ± 1%  2.61µs ± 1%  +10.07%  (p=0.000 n=10+10)
          workgroup_empty  57.0µs ± 0%  55.0µs ± 2%   -3.53%  (p=0.000 n=10+9)
          workgroup_raise  72.4µs ± 0%  69.6µs ± 6%   -3.95%  (p=0.035 n=9+10)
      
      workgroup_* changes for thread runtime is the speedup I am talking about.
      defer/func_call slowdown is due to added exception chaining. We did not
      optimize Python-level defer yet, and if/when that would be needed, it should
      be possible to optimize by moving pydefer implementation into Cython.
      
      (*) go and channels were moved into nogil world in Pygolang v0.0.3 +
      v0.0.4 . Now it is the rest of the library that was moved with packages
      like context, time, sync etc.
      
      wendelin.core v2 needs nogil to run pinner thread on client side to
      support isolation property in cooperation with wcfs: since there is a
      `client -> wcfs -> pinner` loop:
      
            - - - - - -
           |           |
              pinner <------.
           |           |   wcfs
              client -------^
           |           |
            - - - - - -
           client process
      
      the pinner thread would deadlock if it tries to take the GIL because
      client thread can be holding GIL already while accessing wcfs-mmaped
      memory (think doing e.g. `x = A[i]` in Python).
      c5c3071b
    • Kirill Smelkov's avatar
      readme: Turn package references into links · 37ae291f
      Kirill Smelkov authored
      For base functionality we have overview in the readme itself, but for
      packages we have only their listing with brief overview and no
      documentation for in-package functionality.
      
      Let's have at least links to .h/.pxd/.py where package functionality is
      documented.
      37ae291f
    • Kirill Smelkov's avatar
      libgolang: Provide top-level overview for interfaces · 45c8cddd
      Kirill Smelkov authored
      See commit 5a99b769 (libgolang: Start providing interfaces) for context.
      45c8cddd
    • Kirill Smelkov's avatar
      libgolang: Provide top-level overview for automatic memory management · 7f0672aa
      Kirill Smelkov authored
      Provide top-level documentation for memory management facilities that
      was marked as TODO in refptr & co matches. See e.g. the following
      commits for context:
      
      - e82b4fab (libgolang: Objects refcounting (initial draft))
      - b2253abf (libgolang: Rename refobj -> object)
      - fd2a6fab (libgolang: Fix globals atexit race condition of ~refptr vs
        access from another thread)
      7f0672aa
    • Kirill Smelkov's avatar
      libgolang: Provide top-level overview for defer · 39f40159
      Kirill Smelkov authored
      defer is now part of libgolang API (see 1d153a45 "libgolang: Expose
      defer as public C++ API"). It should be explained in top-level overview.
      39f40159
    • Kirill Smelkov's avatar
      fixup! time: Move/Port timers to C++/Pyx nogil · c7a1bce7
      Kirill Smelkov authored
      This amends commit b073f6df: clang was complaining that _Ticker/_Timer
      were forward-declared as class, but later declared as struct:
      
          ./golang/time.h:107:1: warning: '_Ticker' defined as a struct here but previously declared as a class [-Wmismatched-tags]
          struct _Ticker : object {
          ^
          ./golang/time.h:77:1: note: did you mean struct here?
          class _Ticker;
          ^~~~~
          struct
          ./golang/time.h:140:1: warning: '_Timer' defined as a struct here but previously declared as a class [-Wmismatched-tags]
          struct _Timer : object {
          ^
          ./golang/time.h:78:1: note: did you mean struct here?
          class _Timer;
          ^~~~~
          struct
      
      -> Fix it by using struct. Also used inline style inside refptr.
      c7a1bce7
    • Kirill Smelkov's avatar
      libgolang/thread: Upstream PyThread_release_lock bug should have been fixed · d51bcd7e
      Kirill Smelkov authored
      Race condition inside PyThread_release_lock should have been fixed on
      both CPython/darwin and PyPy/darwin:
      
          https://github.com/python/cpython/commit/c5abd63e94fc
          https://bitbucket.org/pypy/pypy/commits/6cd7a0d1a940
          https://bitbucket.org/pypy/pypy/commits/a9d36d6af872
      
      The bug is explained here:
      
          https://bugs.python.org/issue38106
          https://bitbucket.org/pypy/pypy/issues/3072
      
      The following commits added tests for the bug on Pygolang side:
      
          34b7a1f4 (golang: Expose Sema and Mutex as public Python and Cython/nogil API)
          5142460d (libgolang/thread: Add links to upstream PyThread_release_lock bug)
      d51bcd7e
    • Kirill Smelkov's avatar
      strings: New package · 0efd4a9a
      Kirill Smelkov authored
      Provide string utilities to verify whether string has suffix/prefix,
      trim it, and split string by a delimiter. The code originated in wcfs
      codebase in wendelin.core .
      
      Pyx/nogil only.
      0efd4a9a
    • Kirill Smelkov's avatar
      fmt: New package · 309963f8
      Kirill Smelkov authored
      With fmt.sprintf() and fmt.errorf() to format strings and errors. The
      code was extracted from wcfs in wendelin.core .
      
      Pyx/nogil only.
      309963f8
  7. 26 Nov, 2019 1 commit
  8. 25 Nov, 2019 2 commits
    • Kirill Smelkov's avatar
      fixup! sync: Move/Port sync.WorkGroup to C++/Pyx nogil · cf96559c
      Kirill Smelkov authored
      This ammends 33cf3113: Kill now unused imports.
      cf96559c
    • Kirill Smelkov's avatar
      libpyxruntime: It should link to libgolang · 6d94fccf
      Kirill Smelkov authored
      It was all working on Debian 10, but running tests on Ubuntu 16.04
      turned out to fail:
      
          $ python -m pytest
          ================================== test session starts ===================================
          platform linux2 -- Python 2.7.12, pytest-4.6.6, py-1.8.0, pluggy-0.13.0
          rootdir: /home/kirr/src/pygolang
          collected 0 items / 10 errors
      
          ========================================= ERRORS =========================================
          ________________________ ERROR collecting golang/_gopath_test.py _________________________
          ImportError while importing test module '/home/kirr/src/pygolang/golang/_gopath_test.py'.
          Hint: make sure your test modules/packages have valid Python names.
          Traceback:
          golang/__init__.py:42: in <module>
              from golang._golang import _pysys_exc_clear as _sys_exc_clear
          golang/_golang.pyx:35: in init golang._golang
              _init_libpyxruntime()
          golang/_golang.pyx:532: in golang._golang._init_libpyxruntime
              import golang.pyx.runtime
          E   ImportError: /home/kirr/src/pygolang/golang/pyx/../runtime/liblibpyxruntime.so.0.1: undefined symbol: _ZN6golang4sync9WaitGroup3addEi
      
          $ c++filt _ZN6golang4sync9WaitGroup3addEi
          golang::sync::WaitGroup::add(int)
      
      So since libpyxruntime is using sync.WaitGroup (and sync.Mutex etc) it should
      be linking to libgolang where those symbols are provided. Fix it.
      
      Fixes: 4fc6e49c (time: Factor-out PyFunc into shared library libpyxruntime.so)
      6d94fccf
  9. 22 Nov, 2019 5 commits
    • Kirill Smelkov's avatar
      golang: Fix defer exception chaining in the presence of subchains · 1477dd02
      Kirill Smelkov authored
      A deferred function can raise exception and this exception itself can
      have .__context__ - consider for example if B1 is raised and its chain
      is B1->B2->B3. Before calling that deferred function, we save
      then-current exception A1 into GoFrame.exc_ctx and link-to .exc_ctx after
      the call. We were previously trying to link-to .exc_ctx from raised
      exception itself - i.e. B1 in the above example which is not correct:
      B1 was raised while B2 was being raised ... etc and it was B3 who was
      raised after A1. The consequence was that A1 was list, since B1 already
      had non-empty .__context__
      
      -> Fix it by linking-to A1 from B3, not from B1. In other words by
      linking-to .exc_ctx chain from tail of exception chain of raised
      exception.
      
      We can be sure that updated test is correct because it passes under
      Python3 where exception chaining is implemented natively.
      
      Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)
      1477dd02
    • Kirill Smelkov's avatar
      golang: Fix defer exception chaining for regular exceptions raised from under @func · 06cac90b
      Kirill Smelkov authored
      Defer support for exception chaining on Python2 (see bb9a94c3 "golang:
      Teach defer to chain exceptions (PEP 3134) even on Python2") simulates
      PEP 3134 by manually setting and chaining exc .__context__, .__cause__
      and .__traceback__: if exceptions are thrown by several deferred
      functions they are linked into exception chain via .__context__
      attribute. Defer support in @func also makes sure that any exception
      that comes out of a function has all those PEP 3134 attributes and
      presets them to their default values if exception object did not have
      them initially. In particular .__context__ is preset to None for first
      raised exception.
      
      There was a logic error in chaining handling: .__context__ was glued to
      previously raised exception only if current exception object did not
      have .__context__ attribute at all. And this was failing to chain them
      if current exception was raised from under another function wrapped in
      @func, because GoFrame wrapping that function makes sure to set
      curexc.__context__=None and oops - it was not changed later.
      
      -> Fix chaining implementation by gluing .__context__ either if there is
      no such attribute in current exception, or if .__context__ is None.
      
      This is correct to do since .__context__, by its definition, represents
      implicitly / automatically chained exceptions - contrary to .__cause__
      which is explicitly set and would be incorrect to automatically change
      from None to something. We also know that the end result is the same as
      Python3 behaviour since updated tests (see bellow) pass OK also when run
      under Python3 where exceptions chaining is implemented by Python runtime
      natively.
      
      Update test_defer_excchain() in golang_test.py, which verifies how
      raised exceptions are chained.
      
      No need to update testprog/golang_test_defer_excchain.{py,txt} since the
      test there verifies that our traceback dumper is correctly hooked into
      Python interpreter - where we know that exceptions are already chained
      correctly and we verify only that automatic traceback dump takes this
      chaining into account.
      
      Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)
      06cac90b
    • Kirill Smelkov's avatar
      golang: Fix defer traceback tests when run somewhere under symlink · 7faaecbc
      Kirill Smelkov authored
      Since realpath was used to find out pygolang dir, it will give different
      prefix compared to what actual prefix of __file__ is if __file__ was
      imported through some symlinked directory. And it is the actual
      __file__ that is included into traceback - not its realpath'ed version.
      
      This was breaking for example like this on py27-gevent tox tests:
      
          __file__:          '.../pygolang/.tox/py27-gevent/local/lib/python2.7/site-packages/golang/golang_test.py'
          dirname(__file__): '.../pygolang/.tox/py27-gevent/local/lib/python2.7/site-packages/golang'
          PYGOLANG:          '.../pygolang/.tox/py27-gevent/lib/python2.7/site-packages'
      
          E           Failed: not equal:
          E           Differences (unified diff with -expected +actual):
          E               @@ -1,8 +1,8 @@
          E                Traceback (most recent call last):
          E               -  File "PYGOLANG/golang/golang_test.py", line ..., in test_defer_excchain_traceback
          E               +  File ".../pygolang/.tox/py27-gevent/local/lib/python2.7/site-packages/golang/golang_test.py", line 1360, in test_defer_excchain_traceback
          E                    alpha()
          E               -  File "PYGOLANG/golang/golang_test.py", line ..., in alpha
          E               +  File ".../pygolang/.tox/py27-gevent/local/lib/python2.7/site-packages/golang/golang_test.py", line 1357, in alpha
          E                    beta()
          E               -  File "PYGOLANG/golang/golang_test.py", line ..., in beta
          E               +  File ".../pygolang/.tox/py27-gevent/local/lib/python2.7/site-packages/golang/golang_test.py", line 1356, in beta
          E                    raise RuntimeError("gamma")
          E                RuntimeError: gamma
      
      -> Fix it by not doing realpath'ification when detecting PYGOLANG
      prefix.
      
      Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)
      7faaecbc
    • Kirill Smelkov's avatar
      libgolang: Fix globals atexit race condition of ~refptr vs access from another thread · fd2a6fab
      Kirill Smelkov authored
      Until now we were defining errors like this:
      
          const error ErrSomething  = errors::New("abc");
      
      However there is a problem here: the runtime will call ErrSomething
      destructor on program exit, and for refptr<T> the destructor does 2
      things: a) decrefs the object, and b) switches embedded pointer to NULL;
      
      If another thread is still running both "a" and "b" can become race
      conditions if that T2 uses ErrSomething via e.g. just
      
          ...
          if (err == ErrSomething)
              ...
      
      Here is, for example TSAN report about ErrPyStopped usage when accessed by timer thread:
      
          WARNING: ThreadSanitizer: data race (pid=4224)
           Read of size 8 at 0x7f0d6840e150 by thread T92:
             #0 golang::refptr<golang::_error>::refptr(golang::refptr<golang::_error> const&) golang/libgolang.h:525 (liblibpyxruntime.so.0.1+0x4edc)
             #1 golang::pyx::runtime::PyFunc::operator()() const golang/runtime/libpyxruntime.cpp:222 (liblibpyxruntime.so.0.1+0x423d)
             #2 std::_Function_handler<void (), golang::pyx::runtime::PyFunc>::_M_invoke(std::_Any_data const&) /usr/include/c++/8/bits/std_function.h:297 (_time.so+0x1382b)
             #3 std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687 (liblibgolang.so.0.1+0x2ede6)
             #4 golang::time::_Timer::_fire(double, int) golang/time.cpp:197 (liblibgolang.so.0.1+0x48cbe)
             #5 operator() golang/time.cpp:171 (liblibgolang.so.0.1+0x4894c)
             #6 __invoke_impl<void, golang::time::_Timer::reset(double)::<lambda(int)>&, int&> /usr/include/c++/8/bits/invoke.h:60 (liblibgolang.so.0.1+0x4a142)
             #7 __invoke<golang::time::_Timer::reset(double)::<lambda(int)>&, int&> /usr/include/c++/8/bits/invoke.h:95 (liblibgolang.so.0.1+0x4a080)
             #8 __call<void, 0> /usr/include/c++/8/functional:400 (liblibgolang.so.0.1+0x49fcc)
             #9 operator()<> /usr/include/c++/8/functional:484 (liblibgolang.so.0.1+0x49cee)
             #10 _M_invoke /usr/include/c++/8/bits/std_function.h:297 (liblibgolang.so.0.1+0x49858)
             #11 std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687 (liblibgolang.so.0.1+0x2ede6)
             #12 operator() golang/libgolang.h:326 (liblibgolang.so.0.1+0x48fa1)
             #13 _FUN golang/libgolang.h:324 (liblibgolang.so.0.1+0x49004)
             #14 <null> <null> (python2+0x194d13)
      
           Previous write of size 8 at 0x7f0d6840e150 by main thread:
             #0 golang::refptr<golang::_error>::~refptr() golang/libgolang.h:510 (liblibpyxruntime.so.0.1+0x4d39)
             #1 at_exit_wrapper ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:389 (libtsan.so.0+0x28693)
      
           As if synchronized via sleep:
             #0 nanosleep ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:366 (libtsan.so.0+0x49960)
             #1 __pyx_f_6golang_7runtime_15_runtime_thread_nanosleep golang/runtime/_runtime_thread.c:1702 (_runtime_thread.so+0x48f2)
             #2 _tasknanosleep golang/runtime/libgolang.cpp:1237 (liblibgolang.so.0.1+0x2cca7)
             #3 golang::time::sleep(double) golang/runtime/libgolang.cpp:1251 (liblibgolang.so.0.1+0x2ce0b)
             #4 golang::time::_Timer::_fire(double, int) golang/time.cpp:179 (liblibgolang.so.0.1+0x48bd6)
             #5 operator() golang/time.cpp:171 (liblibgolang.so.0.1+0x4894c)
             #6 __invoke_impl<void, golang::time::_Timer::reset(double)::<lambda(int)>&, int&> /usr/include/c++/8/bits/invoke.h:60 (liblibgolang.so.0.1+0x4a142)
             #7 __invoke<golang::time::_Timer::reset(double)::<lambda(int)>&, int&> /usr/include/c++/8/bits/invoke.h:95 (liblibgolang.so.0.1+0x4a080)
             #8 __call<void, 0> /usr/include/c++/8/functional:400 (liblibgolang.so.0.1+0x49fcc)
             #9 operator()<> /usr/include/c++/8/functional:484 (liblibgolang.so.0.1+0x49cee)
             #10 _M_invoke /usr/include/c++/8/bits/std_function.h:297 (liblibgolang.so.0.1+0x49858)
             #11 std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687 (liblibgolang.so.0.1+0x2ede6)
             #12 operator() golang/libgolang.h:326 (liblibgolang.so.0.1+0x48fa1)
             #13 _FUN golang/libgolang.h:324 (liblibgolang.so.0.1+0x49004)
             #14 <null> <null> (python2+0x194d13)
      
           Location is global 'golang::pyx::runtime::ErrPyStopped' of size 8 at 0x7f0d6840e150 (liblibpyxruntime.so.0.1+0x000000009150)
      
           Thread T92 (tid=7834, running) created by thread T15 at:
             #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:915 (libtsan.so.0+0x2be1b)
             #1 PyThread_start_new_thread <null> (python2+0x194ccf)
             #2 _taskgo golang/runtime/libgolang.cpp:124 (liblibgolang.so.0.1+0x288e7)
             #3 go<golang::time::_Timer::reset(double)::<lambda(int)>, int> golang/libgolang.h:324 (liblibgolang.so.0.1+0x490b7)
             #4 golang::time::_Timer::reset(double) golang/time.cpp:170 (liblibgolang.so.0.1+0x48b36)
             #5 __pyx_f_6golang_5_time_timer_reset_pyexc golang/_time.cpp:3271 (_time.so+0xa506)
             #6 __pyx_pf_6golang_5_time_7PyTimer_6reset golang/_time.cpp:2824 (_time.so+0x97d8)
             #7 __pyx_pw_6golang_5_time_7PyTimer_7reset golang/_time.cpp:2789 (_time.so+0x9751)
             #8 PyEval_EvalFrameEx <null> (python2+0xf27f7)
             #9 std::_Function_handler<void (), golang::pyx::runtime::PyFunc>::_M_invoke(std::_Any_data const&) /usr/include/c++/8/bits/std_function.h:297 (_time.so+0x1382b)
             #10 std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687 (liblibgolang.so.0.1+0x2ede6)
             #11 golang::time::_Timer::_fire(double, int) golang/time.cpp:197 (liblibgolang.so.0.1+0x48cbe)
             #12 operator() golang/time.cpp:171 (liblibgolang.so.0.1+0x4894c)
             #13 __invoke_impl<void, golang::time::_Timer::reset(double)::<lambda(int)>&, int&> /usr/include/c++/8/bits/invoke.h:60 (liblibgolang.so.0.1+0x4a142)
             #14 __invoke<golang::time::_Timer::reset(double)::<lambda(int)>&, int&> /usr/include/c++/8/bits/invoke.h:95 (liblibgolang.so.0.1+0x4a080)
             #15 __call<void, 0> /usr/include/c++/8/functional:400 (liblibgolang.so.0.1+0x49fcc)
             #16 operator()<> /usr/include/c++/8/functional:484 (liblibgolang.so.0.1+0x49cee)
             #17 _M_invoke /usr/include/c++/8/bits/std_function.h:297 (liblibgolang.so.0.1+0x49858)
             #18 std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687 (liblibgolang.so.0.1+0x2ede6)
             #19 operator() golang/libgolang.h:326 (liblibgolang.so.0.1+0x48fa1)
             #20 _FUN golang/libgolang.h:324 (liblibgolang.so.0.1+0x49004)
             #21 <null> <null> (python2+0x194d13)
      
          SUMMARY: ThreadSanitizer: data race golang/libgolang.h:525 in golang::refptr<golang::_error>::refptr(golang::refptr<golang::_error> const&)
      
      -> Fix it by not deallocating the object, nor clearing the pointer on smart pointer destruction.
      
      To do so in ergonomic way introduce another template type global<X>,
      which can be used instead of X and change types of all global error
      variables from `const error` to `const global<error>`. Don't change pxd
      to offload pyx users from thinking about global/not-global aspect.
      
      Top-level documentation is TODO.
      fd2a6fab
    • Kirill Smelkov's avatar
      sync: Move/Port sync.WorkGroup to C++/Pyx nogil · 33cf3113
      Kirill Smelkov authored
      Provide sync.WorkGroup that can be used directly from C++ and Pyx/nogil codes.
      Python-level sync.WorkGroup becomes a wrapper around pyx/nogil one.
      
      Like with context (2a359791 "context: Move/Port context package to
      C++/Pyx nogil"), timers (b073f6df "time: Move/Port timers to C++/Pyx
      nogil") and interfaces (5a99b769 "libgolang: Start providing
      interfaces") memory for on-heap allocated WorkGroup is managed
      automatically.
      
      Python-level tests should be enough to cover C++/Pyx functionality at
      zero-level approximation.
      33cf3113
  10. 20 Nov, 2019 3 commits
  11. 19 Nov, 2019 5 commits
    • Kirill Smelkov's avatar
      context: Provide way to create PyContext from raw Context · 4fb9b51c
      Kirill Smelkov authored
      sync.PyWorkGroup will need to wrap raw Context into PyContext for
      spawned workers.
      4fb9b51c
    • Kirill Smelkov's avatar
      libpyxruntime: Teach PyFunc to report Python exception after the call · a36efe6d
      Kirill Smelkov authored
      - Add PyError class that wraps Python exception;
      - change PyFunc::operator() return from void -> to error, which might be
        either PyError or special error indicating that Python interpreter is
        stopped.
      - Provide way to reraise that Python exception.
      
      We will need to see which exception, if any, PyFunc call raised in
      sync.PyWorkGroup, where this python-level exception will need to be
      propagated to PyWorkGroup.wait() and reraised from there.
      a36efe6d
    • Kirill Smelkov's avatar
      libpyxruntime: Fix deadlock in pygil_ensure() · 73182038
      Kirill Smelkov authored
      pygil_ensure() was taking pyexitedMu and then GIL, which can deadlock
      via AB BA scenario if pygil_ensure() is also called in another thread
      with GIL already held.
      
      -> Fix it by always taking GIL with pyexitedMu released.
      
      Without the fix the test added always hangs with thread backend, e.g.
      like this:
      
          golang/pyx/runtime_test.py::test_pyx_pyfunc_vs_gil_deadlock
          STUCK
          terminate called after throwing an instance of 'golang::PanicError'
            what():  STUCK
          Fatal Python error: Aborted
      
      The bug was introduced in b073f6df (time: Move/Port timers to C++/Pyx nogil).
      73182038
    • Kirill Smelkov's avatar
      time: Factor-out PyFunc into shared library libpyxruntime.so · 4fc6e49c
      Kirill Smelkov authored
      Commit b073f6df (time: Move/Port timers to C++/Pyx nogil) added C++
      level PyFunc as a way to call Python function from under nogil code.
      We will soon need this functionality for sync.WorkGroup too, so move it
      into shared place.
      
      We move into regular library - not a pyx module (e.g. golang.pyx) -
      because for pyx modules linking is done at import time by specially
      generated code, while we need the linker support to automatically
      resolve e.g. calls to PyFunc::operator() from inside std::function.
      4fc6e49c
    • Kirill Smelkov's avatar
      libgolang: Provide DSO_EXPORT/DSO_IMPORT macros for other libraries · 5146a416
      Kirill Smelkov authored
      In the next patch we are going to move PyFunc from time.pyx into shared
      library. Before doing that let's provide DSO visibility macros from
      libgolang.h to offload other libraries from duplicating dance on how to
      mark symbols visibility properly depending on compiler/OS.
      5146a416
  12. 18 Nov, 2019 2 commits
    • Kirill Smelkov's avatar
      libgolang/{gevent,thread}: Preserve Python-level exception in runtime calls · 47fac0a9
      Kirill Smelkov authored
      Gevent runtime uses python-level calls internally which might interfere
      with current python state. For example if current python exception is
      set, and e.g. go or makesema runtime call is made, the following
      happens:
      
          golang/golang_test.py::test_pyx_runtime_vs_pyexc RuntimeError: abc
      
          The above exception was the direct cause of the following exception:
      
          SystemError: <class 'gevent.__semaphore.Semaphore'> returned a result with an error set
          Exception ignored in: 'golang.runtime._runtime_gevent._sema_alloc'
          SystemError: <class 'gevent.__semaphore.Semaphore'> returned a result with an error set
          terminate called after throwing an instance of 'golang::PanicError'
            what():  makesema: alloc failed
          Fatal Python error: Aborted
      
      -> Fix all functions in the runtimes that work at Python level to
         save/restore Python-level exception on entry/exit.
      
      This is mostly gevent runtime, but also a couple of non-posix fallbacks in
      thread runtime.
      
      The bug was there from day 1 of runtimes - from ce8152a2 (pyx api:
      Provide sleep), f971a2a8 (pyx api: Provide go) and 69db91bf (libgolang:
      Add internal semaphores).
      47fac0a9
    • Kirill Smelkov's avatar
      libgolang/{gevent,thread}: Restructure to keep functions that access Python localized · 689dc862
      Kirill Smelkov authored
      In the next patch we will fix runtime functions that work at Python
      level to preserve user Python exception state. Before that, as a
      preparatory step, restructure runtimes code, so that all such
      Python-accessing functions are grouped together.
      689dc862