1. 18 Nov, 2019 2 commits
    • Kirill Smelkov's avatar
      libgolang/{gevent,thread}: Preserve Python-level exception in runtime calls · 47fac0a9
      Kirill Smelkov authored
      Gevent runtime uses python-level calls internally which might interfere
      with current python state. For example if current python exception is
      set, and e.g. go or makesema runtime call is made, the following
      happens:
      
          golang/golang_test.py::test_pyx_runtime_vs_pyexc RuntimeError: abc
      
          The above exception was the direct cause of the following exception:
      
          SystemError: <class 'gevent.__semaphore.Semaphore'> returned a result with an error set
          Exception ignored in: 'golang.runtime._runtime_gevent._sema_alloc'
          SystemError: <class 'gevent.__semaphore.Semaphore'> returned a result with an error set
          terminate called after throwing an instance of 'golang::PanicError'
            what():  makesema: alloc failed
          Fatal Python error: Aborted
      
      -> Fix all functions in the runtimes that work at Python level to
         save/restore Python-level exception on entry/exit.
      
      This is mostly gevent runtime, but also a couple of non-posix fallbacks in
      thread runtime.
      
      The bug was there from day 1 of runtimes - from ce8152a2 (pyx api:
      Provide sleep), f971a2a8 (pyx api: Provide go) and 69db91bf (libgolang:
      Add internal semaphores).
      47fac0a9
    • Kirill Smelkov's avatar
      libgolang/{gevent,thread}: Restructure to keep functions that access Python localized · 689dc862
      Kirill Smelkov authored
      In the next patch we will fix runtime functions that work at Python
      level to preserve user Python exception state. Before that, as a
      preparatory step, restructure runtimes code, so that all such
      Python-accessing functions are grouped together.
      689dc862
  2. 15 Nov, 2019 1 commit
    • Kirill Smelkov's avatar
      pyx.build: Allow to use custom build_ext · b4feee6f
      Kirill Smelkov authored
      setuptools_dso hardcodes to use its own build_ext in its setup.
      We amend that build_ext and inject our own version carefully while
      setup() is run - see 7ae8c4f3 (pyx.build: Allow to combine C and C++
      sources in one extension). However, currently, if user uses cmdclass =
      {'build_ext': ...} in his own setup, this will be ignored and
      overwritten by golang.pyx.build.build_ext . One example where this
      breaks things is wendelin.core which hooks into build_ext to first
      configure/build CCAN before any extension:
      
      https://lab.nexedi.com/nexedi/wendelin.core/blob/b26ba558/setup.py#L147-153
      
      -> Fix it by taking into account what user could put into cmdclass.
      b4feee6f
  3. 14 Nov, 2019 17 commits
    • Kirill Smelkov's avatar
      libgolang: Fix double C++ defer · 14a249cb
      Kirill Smelkov authored
      If defer was used multiple times in one function it was failing to
      compile. For example added test is failing like this:
      
          In file included from golang/runtime/libgolang_test.cpp:22:
          golang/runtime/libgolang_test.cpp: In function ‘void __test_defer(int*)’:
          ./golang/libgolang.h:457:59: error: redeclaration of ‘golang::_deferred _defer___COUNTER__’
           #define defer(f) golang::_deferred _defer_ ## __COUNTER__ (f)
                                                                     ^
          golang/runtime/libgolang_test.cpp:473:5: note: in expansion of macro ‘defer’
               defer([&]() {
               ^~~~~
          ./golang/libgolang.h:457:36: note: ‘golang::_deferred _defer___COUNTER__’ previously declared here
           #define defer(f) golang::_deferred _defer_ ## __COUNTER__ (f)
                                              ^~~~~~~
          golang/runtime/libgolang_test.cpp:470:5: note: in expansion of macro ‘defer’
               defer([&]() {
               ^~~~~
      
      It turns out the __COUNTER__ was not expanded and they say you have to
      use 3 level of macro calls. Oh well...
      
      https://stackoverflow.com/a/27611869/9456786
      14a249cb
    • Kirill Smelkov's avatar
      libgolang: Provide func as alias for std::function · de269272
      Kirill Smelkov authored
      std::function is frequently too long to type while func is more native to Go.
      de269272
    • Kirill Smelkov's avatar
      context: Export PyContext for cimport · a6c1c984
      Kirill Smelkov authored
      For example wendelin.core wants to accept Python-level context,
      transform it into C-level context and pass to pyx/nogil code.
      a6c1c984
    • Kirill Smelkov's avatar
      context: Move/Port context package to C++/Pyx nogil · 2a359791
      Kirill Smelkov authored
      Provide context-related functionality that can be used directly from C++
      and Pyx/nogil codes. Python-level classes and functions become small
      wrappers around pyx/nogil ones.
      
      Like with timers (b073f6df "time: Move/Port timers to C++/Pyx nogil")
      and interfaces (5a99b769 "libgolang: Start providing interfaces")
      memory for objects dynamically allocated on heap is managed
      automatically.
      2a359791
    • Kirill Smelkov's avatar
      cxx: New package · 9785f2d3
      Kirill Smelkov authored
      Start new package that provides C++ amendments to be used by libgolang
      and its users. Current functionality: dict and set that wrap
      std::unordered_map and std::unordered_set into ergonomic interface.
      
      The code originates from wendelin.core:
      https://lab.nexedi.com/kirr/wendelin.core/blob/5a045ed1/wcfs/internal/wcfs_misc.h#L214-257
      
      Pyx/nogil only.
      9785f2d3
    • Kirill Smelkov's avatar
      errors: New package · a245ab56
      Kirill Smelkov authored
      With just one function - errors.New() - to create new error with given
      text.
      
      This will be soon used in Pyx/nogil context package to create context.canceled
      and context.deadlineExceeded errors.
      
      Pyx/nogil only for now.
      a245ab56
    • Kirill Smelkov's avatar
      libgolang: Start providing interfaces · 5a99b769
      Kirill Smelkov authored
      - interface is analog of interface{} in Go.
      - error is analog of error in Go.
      
      For now the interfaces are implemented via classic C++ scheme via
      inheritance and additional vtable field in pointed object.
      
      In the future we might want to consider reworking that to Go-style
      interfaces without predeclaring which interfaces a class implements.
      
      Top-level documentation is TODO.
      
      Interfaces will be soon needed to describe Context interface in C
      context package.
      5a99b769
    • Kirill Smelkov's avatar
      golang.pxd: Expose missing memory-management bits · 274afa3f
      Kirill Smelkov authored
      - refptr==refptr  (e.g. to compare whether returned error object is
        something particular)
      - adoptref() and newref()
      - object (exposed as gobject not to be confused with builtin pyx
        "object" referring to python object)
      
      All this will be soon used in C version of context package.
      274afa3f
    • Kirill Smelkov's avatar
      context: Provide package documentation · 9216e2db
      Kirill Smelkov authored
      Briefly describe provided functionality instead of only referring to
      https://golang.org/pkg/context and https://blog.golang.org/context.
      9216e2db
    • Kirill Smelkov's avatar
      libgolang: Rename refobj -> object · b2253abf
      Kirill Smelkov authored
      Thinking a bit more after e82b4fab (libgolang: Objects refcounting
      (initial draft)) and working on interfaces as part of context pyx -> C
      move, I've came to conclusion that the better name for on-heap objects
      managed by libgolang is just "object" without "ref" prefix.
      
      -> Rename refobj -> object and amend its documentation correspondingly.
      b2253abf
    • Kirill Smelkov's avatar
      libgolang: Provide std::hash<refptr> · af4a8d80
      Kirill Smelkov authored
      So that refptr<T> could be used as keys in e.g. dict (unordered_map) or
      set (unordered_set). set<refptr> will soon be used by C version of
      context package.
      af4a8d80
    • Kirill Smelkov's avatar
      libgolang: Provide select(vector<_selcase>) · 66e1e756
      Kirill Smelkov authored
      We already provide select for array and std::initializer_list. However
      when cases are constructed at runtime dynamically both array and
      initializer_list are not convenient to use.
      
      -> Provide select() overload that accepts vector<_selcase>.
      
      This functionality will soon be used in C version of context package.
      66e1e756
    • Kirill Smelkov's avatar
      sync: Fixup package documentation · 85df1e40
      Kirill Smelkov authored
      - indicated that WorkGroup is not provided in Go version (amends
        40f3b90c "sync: Provide package documentation").
      - fix typo in sync.pxd subject (amends 34b7a1f4 "golang: Expose Sema and
        Mutex as public Python and Cython/nogil API")
      85df1e40
    • Kirill Smelkov's avatar
      golang: tests: Silence inspect deprecation warning · 45f4df9e
      Kirill Smelkov authored
      Every time tests are run under Python3 the following warnings are printed:
      
        golang/golang_test.py::test_func
          .../golang/golang_test.py:990: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
            assert inspect.formatargspec(*inspect.getargspec(MyClass.zzz)) == '(self, v, x=2, **kkkkwww)'
      
        golang/golang_test.py::test_func
          .../golang/golang_test.py:990: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
            assert inspect.formatargspec(*inspect.getargspec(MyClass.zzz)) == '(self, v, x=2, **kkkkwww)'
      
      However since we are not going to drop Python2 support soon, and there is no
      reffered "use-instead" functionality on Python2, let's simply silence the
      warning for now.
      45f4df9e
    • Kirill Smelkov's avatar
      context: Switch to comparing value keys by identity instead of equality · 281defb2
      Kirill Smelkov authored
      It will be some extra work to define general equality when moving
      context code into C. However that is unnecessary as usually a key is
      specific to one particular package, and value associated with that key
      is struct, or some other aggregate, that is holding all package-specific
      context values.
      
      Switching to compare keys by identity preserves that use-case while
      making transition to C easier.
      281defb2
    • Kirill Smelkov's avatar
      context: Don't use dict in _ValueCtx · f76c11f3
      Kirill Smelkov authored
      We always store just one key in there.
      This will simplify transition to C.
      f76c11f3
    • Kirill Smelkov's avatar
      context: tests: Don't compare .done() by "is" · 20761c55
      Kirill Smelkov authored
      This continues 2c8063f4 (*: Channels must be compared by ==, not by "is"
      even for nilchan) and changes comparision of .done() for different
      contexts from "is" to ==.
      
      The reason is that soon we are going to move contexts into C, and this
      way in context tree there potentially can be PyCtx -> CCtx -> PyCtx
      chains, and while all those contexts could have the same C-level channel,
      it would be difficult to propagate the knowledge that Python-level
      pychan wrapper should be also exactly the same.
      
      We already treat channel objects as pointers and require to compare them
      by ==, so there should be no loss here. However to keep performance of
      repeated
      
      	select(
      	    ctx.done(),
      	    ...
      	)
      
      we continue to require that .done() for a particular python context
      object returns exactly the same channel object every time.
      20761c55
  4. 13 Nov, 2019 1 commit
  5. 11 Nov, 2019 2 commits
  6. 07 Nov, 2019 10 commits
    • Kirill Smelkov's avatar
      *: Suppress DeprecationWarning on `import imp` · ec2e882b
      Kirill Smelkov authored
      As of 20191107 imp is present in Python 3.8 source tree and is _used_
      there - see e.g.
      
          https://github.com/python/cpython/blob/v3.8.0-66-g7c20888e713/Lib/pkgutil.py#L188-L192
      
      And they say imp will be there until at least Python 4 - see e.g.
      
          https://github.com/python/cpython/commit/e4f41deccf94
      
      This fixes test_defer_excchain_dump failure under python3-dbg, where
      
      E           Failed: not equal:
      E           Differences (unified diff with -expected +actual):
      E               @@ -1,4 +1,6 @@
      E               +PYGOLANG/golang/_gopath.py:37: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
      E               +  import imp
      
      was added into traceback printed on stderr.
      
      Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)
      ec2e882b
    • Kirill Smelkov's avatar
      time: Move/Port timers to C++/Pyx nogil · b073f6df
      Kirill Smelkov authored
      Provide time.Ticker, time.Timer and friends that can be used directly
      from C++ and Pyx/nogil codes. Python-level classes become small wrapper
      around pyx/nogil ones.
      
      This is the first patch that moves to Pyx/nogil classes that are
      dynamically allocated on heap. refptr<T> is used to automatically manage
      lifetime of such objects. At Pyx level exposed API is very similar to
      Python-one, while internally it uses refptr<T> and friends.
      b073f6df
    • Kirill Smelkov's avatar
      libgolang: Switch _chan refcounting to refobj · e614d641
      Kirill Smelkov authored
      Refobj was just added in previous patch as the class that provides base
      functionality for reference-counted objects.
      e614d641
    • Kirill Smelkov's avatar
      libgolang: Objects refcounting (initial draft) · e82b4fab
      Kirill Smelkov authored
      Since we are going to move more pygolang functionality into C++ (timers,
      context, ...), and since C++ does not have garbage collector, we will
      need to find a way to automatically manage memory in leak/error free way.
      
      Let's do this via refptr<T> smart pointer (inspired by WebKit's RefPtr<T>),
      which, similarly to chan<T> automatically manages pointed object's
      lifetime through reference counting.
      
      refptr<T> will be used in follow-up patches.
      
      Top-level documentation is TODO.
      e82b4fab
    • Kirill Smelkov's avatar
      golang: Provide way to wrap pyx/nogil-level chan[T] into pychan · 07f9430d
      Kirill Smelkov authored
      This will be needed in Python-wrappers for C-level code which exposes
      channels. For example in time package whose implementation will soon be
      moved to nogil world fully.
      07f9430d
    • Kirill Smelkov's avatar
      readme: Provide more detailed example for pychan -> chan[T] conversion · 69b80926
      Kirill Smelkov authored
      3121b290 (golang: Teach pychan to work with channels of C types, not
      only PyObjects) added ability to convert pychan to chan[T], but
      described that ability only briefly in README. Provide more details in
      that description before we add description and implementation for
      reflexive pychan <- chan[T] wrapping.
      69b80926
    • Kirill Smelkov's avatar
      libgolang: Split time and sync into their own packages · 1a9dae3b
      Kirill Smelkov authored
      We already have some sync functionality implemented in C++ (e.g.
      sync.Once, sync.WaitGroup) and we are going to add more and also move
      timers implementation to C++. It is getting crowded for that
      functionality to still live in libgolang.{h,cpp}
      
      -> Split sync & time functionality into their own C++ packages (still
      built-into libgolang.so)
      1a9dae3b
    • Kirill Smelkov's avatar
      time: Provide package documentation · 419c8950
      Kirill Smelkov authored
      Briefly describe provided functionality instead of only referring to
      https://golang.org/pkg/time.
      419c8950
    • Kirill Smelkov's avatar
      sync: Provide package documentation · 40f3b90c
      Kirill Smelkov authored
      Briefly describe provided functionality instead of only referring to
      https://golang.org/pkg/sync.
      40f3b90c
    • Kirill Smelkov's avatar
      pyx.build: Extension.depends += sync.pxd, time.pxd · b06ec3d4
      Kirill Smelkov authored
      Make built extensions depend on golang/{sync,time}.pxd and rebuild if
      those files change. A bit overkill to depend globally, but more correct,
      since an extension that is using pygolang can be using those packages.
      
      Before the patch - without those depends - since distutils does not
      implement proper dependency tracking, extensions - even those that
      actually depend on sync/time pxd, were not rebuilt.
      
      sync.pxd was missed to be added in 34b7a1f4 (golang: Expose Sema and
      Mutex as public Python and Cython/nogil API). time.pxd was missed to be
      added in ce8152a2 (pyx api: Provide sleep).
      b06ec3d4
  7. 06 Nov, 2019 3 commits
  8. 04 Nov, 2019 4 commits