1. 27 Feb, 2020 1 commit
  2. 11 Feb, 2020 1 commit
    • Kirill Smelkov's avatar
      golang: Teach pyerror to be a base class · 78d0c76f
      Kirill Smelkov authored
      It is surprising to have an exception class that cannot be derived from.
      
      Besides, in the future we'll use subclassing from golang.error as an
      indicator that an error is a "well-defined" (in simple words - does not
      need traceback to be interpreted).
      78d0c76f
  3. 10 Feb, 2020 1 commit
    • Kirill Smelkov's avatar
      golang: Expose error at Py level · 17798442
      Kirill Smelkov authored
      The first step to expose errors and error chaining to Python:
      
      - Add pyerror that wraps a pyx/nogil C-level error and is exposed as golang.error at py level.
      - py errors must be compared by ==, not by "is"
      - Add (py) errors.New to create a new error from text.
      - a C-level error that has .Unwrap, is exposed with .Unwrap at py level,
        but full py-level chaining will be implemented in a follow-up patch.
      - py error does not support inheritance yet.
      
      Top-level documentation is TODO.
      17798442
  4. 06 Feb, 2020 1 commit
    • Kirill Smelkov's avatar
      golang, errors, fmt: Error chaining (C++/Pyx) · fd95c88a
      Kirill Smelkov authored
      Following errors model in Go, let's add support for errors to wrap other
      errors and to be inspected/unwrapped:
      
      - an error can additionally provide way to unwrap itself, if it
        implements errorWrapper interface;
      - errors.Unwrap(err) tries to extract wrapped error;
      - errors.Is(err) tests whether an item in error's chain matches target;
      - `fmt.errorf("... : %w", ... err)` is similar to `fmt.errorf("... : %s", ... err.c_str())`
        but resulting error, when unwrapped, will return err.
      
      Add C++ implementation for the above + tests.
      Python analogs will follow in the next patches.
      
      Top-level documentation is TODO.
      
      See https://blog.golang.org/go1.13-errors for error chaining overview.
      fd95c88a
  5. 04 Feb, 2020 1 commit
    • Kirill Smelkov's avatar
      libgolang: Provide Nil as alias for std::nullptr_t · 230c81c4
      Kirill Smelkov authored
      This continues 60f6db6f (libgolang: Provide nil as alias for nullptr and
      NULL): I've tried to compile pygolang with Clang on my Debian 10
      workstation and got:
      
          $ CC=clang CXX=clang++ python setup.py build_dso -i
      
          In file included from ./golang/fmt.h:32:
          ./golang/libgolang.h:381:11: error: unknown type name 'nullptr_t'; did you mean 'std::nullptr_t'?
          constexpr nullptr_t nil = nullptr;
                    ^~~~~~~~~
                    std::nullptr_t
          /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/c++config.h:242:29: note: 'std::nullptr_t' declared here
            typedef decltype(nullptr)     nullptr_t;
                                          ^
          :
          In file included from ./golang/context.h
          In file included from golang/runtime/libgolang.cpp:30:
          ./golang/libgolang.h:381:11: error: unknown type name 'nullptr_t'; did you mean 'std::nullptr_t'?
          constexpr nullptr_t nil = nullptr;
                    ^~~~~~~~~
                    std::nullptr_t
          /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/c++config.h:242:29: note: 'std::nullptr_t' declared here
            typedef decltype(nullptr)     nullptr_t;
                                          ^
          :39:
          ./golang/libgolang.h:381:11: error: unknown type In file included from golang/fmt.cpp:25:
          In file included from ./golang/fmt.h:32:
          ./golang/libgolang.h:421:17: error: unknown type name 'nullptr_t'; did you mean 'std::nullptr_t'?
              inline chan(nullptr_t) { _ch = nil; }
                          ^~~~~~~~~
                          std::nullptr_t
      
          ...
      
      It seems with GCC and Clang under macOS nullptr_t is automatically provided in
      builtin namespace, while with older Clang on Linux (clang version 7.0.1-8) only
      in std:: namespace - rightfully as nullptr_t is described to be present there:
      
      https://en.cppreference.com/w/cpp/types/nullptr_t
      
      This way we either have to correct all occurrences of nullptr_t to
      std::nullptr_t, or do something similar with providing nil under golang:: .
      
      To reduce noise I prefer the later and let it be named as Nil.
      230c81c4
  6. 17 Jan, 2020 1 commit
  7. 26 Nov, 2019 1 commit
  8. 14 Nov, 2019 2 commits
    • Kirill Smelkov's avatar
      libgolang: Start providing interfaces · 5a99b769
      Kirill Smelkov authored
      - interface is analog of interface{} in Go.
      - error is analog of error in Go.
      
      For now the interfaces are implemented via classic C++ scheme via
      inheritance and additional vtable field in pointed object.
      
      In the future we might want to consider reworking that to Go-style
      interfaces without predeclaring which interfaces a class implements.
      
      Top-level documentation is TODO.
      
      Interfaces will be soon needed to describe Context interface in C
      context package.
      5a99b769
    • Kirill Smelkov's avatar
      golang.pxd: Expose missing memory-management bits · 274afa3f
      Kirill Smelkov authored
      - refptr==refptr  (e.g. to compare whether returned error object is
        something particular)
      - adoptref() and newref()
      - object (exposed as gobject not to be confused with builtin pyx
        "object" referring to python object)
      
      All this will be soon used in C version of context package.
      274afa3f
  9. 07 Nov, 2019 2 commits
    • Kirill Smelkov's avatar
      libgolang: Objects refcounting (initial draft) · e82b4fab
      Kirill Smelkov authored
      Since we are going to move more pygolang functionality into C++ (timers,
      context, ...), and since C++ does not have garbage collector, we will
      need to find a way to automatically manage memory in leak/error free way.
      
      Let's do this via refptr<T> smart pointer (inspired by WebKit's RefPtr<T>),
      which, similarly to chan<T> automatically manages pointed object's
      lifetime through reference counting.
      
      refptr<T> will be used in follow-up patches.
      
      Top-level documentation is TODO.
      e82b4fab
    • Kirill Smelkov's avatar
      golang: Provide way to wrap pyx/nogil-level chan[T] into pychan · 07f9430d
      Kirill Smelkov authored
      This will be needed in Python-wrappers for C-level code which exposes
      channels. For example in time package whose implementation will soon be
      moved to nogil world fully.
      07f9430d
  10. 14 Oct, 2019 3 commits
    • Kirill Smelkov's avatar
      golang: Teach pychan to work with channels of C types, not only PyObjects · 3121b290
      Kirill Smelkov authored
      Introduce notion of data type (dtype similarly to NumPy) into pychan and
      teach it to accept for send objects only matching that dtype. Likewise
      teach pychan to decode raw bytes received from underlying channel into
      Python object correspodningg to pychan dtype. For C dtypes, e.g.
      'C.int', 'C.double' etc, contrary to chan of python objects, the
      transfer can be done without depending on Python GIL. This way channels
      of such C-level dtypes can be used to implement interaction in between
      Python and nogil worlds.
      3121b290
    • Kirill Smelkov's avatar
      libgolang: Teach select to accept inplace tx data · 47111d3e
      Kirill Smelkov authored
      Currently select, via _selcase, requires users to provide pointers to tx
      and rx buffers. However if element type itself can fit into a pointer
      word, we can put the element directly into _selcase and mark the case
      with a flag, that it contains inplace data instead of referring to
      external storage. This will be helpful in upcoming patch where we'll
      teach pychan to work with several element types, not only pyobject
      elements.
      
      This patch does careful introduction of _selcase.flags - in such a way
      that the size of _selcase stays the same as it was before by using
      bitfields. The .ptxrx pointer is unioned with newly introduced inplace
      uint64 .itxrx data, which is used by select instead of .ptxrx if the
      flag is set. The usage of uint64 should not increase _selcase size on
      64-bit platforms.
      
      Then _selcase.ptx() and .prx() accessors are adapted accordingly and
      the rest of the changes are corresponding test and
      _chanselect2<onstack=false> adaptation.
      
      This functionality is kind of low-level and is not exposed via any
      _selsend() or chan.sends() API changes. Whenever inplace tx should be
      used, the case should be prepared either completely manually, or with
      e.g. first calling _selsend() and then manually changing .flags and
      .itxrx.  Added test serves as the example on how to do it.
      
      Inplace rx is currently forbidden - because supporting that would require
      to drop const from casev select argument. However in the future, for
      symmetry, we might want to support that as well.
      
      P.S.
      
      Since write to selcase.itxrx requires casting pointers e.g. like this:
      
      	*(int *)&sel[0].itxrx = 12345;
      
      it breaks C99 strict aliasing and by default compiler can generate bad
      code on such pattern. To the problem we adapt the build system
      to default compiler to no-strict-aliasing (as many other projects do,
      e.g. Linux kernel) with the idea that in many cases where strict
      aliasing was intended to help it actually does not, because e.g. pointer
      types are the same, while explicitly marking pointers with `restrict`
      keyword does help indeed.
      
      Nothing new for Python2 here, as it is using -fno-strict-aliasing by
      itself. However Python3 is compiling without -fno-strict-aliasing:
      https://python.org/dev/peps/pep-3123 .
      47111d3e
    • Kirill Smelkov's avatar
      libgolang: _selcase: Introduce accessors to get pointers to for-send and for-recv data · 2590e9a7
      Kirill Smelkov authored
      In the next patch we are going to teach _selcase to support both
      external and inplace data. Before that let's do a couple of preparatory
      things.
      
      This patch: introduces .ptx() and .prx() accessors to get to
      corresponding data buffer associated with _selcase. Convert _selcase
      users to use .ptx() and .prx() instead of doing direct .ptxrx access.
      This way when we'll add inplace support to _selcase, we'll need to adapt
      only accessors, not clients.
      
      The only place that is left using .ptxrx directly is one tricky place in
      _chanselect2<onstack=false> where gevent-related code needs to carefully
      deal with proxying tx/rx buffers due to STACK_DEAD_WHILE_PARKED.
      
      NOTE even though new accessors may panic, libgolang.cpp always calls them
      after checking that the conditions for ptx/prx calls are valid.
      2590e9a7
  11. 13 Oct, 2019 2 commits
    • Kirill Smelkov's avatar
      libgolang: _selcase: Rename .data -> .ptxrx · d6c8862d
      Kirill Smelkov authored
      In the next patches we are going to teach _selcase to support both
      external and inplace data. Before that let's do a couple of preparatory
      things.
      
      This patch: rename .data -> .ptxrx . The new name is more clear:
      
      - "p" prefix aligns with other libgolang style, e.g. ptx or prx.
      - "txrx" suffix says that this is used for both send and recv.
      
      Just plain renaming, nothing else in this patch.
      d6c8862d
    • Kirill Smelkov's avatar
      golang: Rework pychan to use C-level channel API · f2847307
      Kirill Smelkov authored
      We will soon rework pychan to be python wrapper not only for
      chan<object>, but also for other channels of various C types - e.g.
      chan<structZ>, chan<int>, etc.
      
      To prepare for this let's first rework pychan from using chan[PyObject*]
      into raw _chan* functions. This will allow us to use the same functions
      over raw channels while dynamically dispatching on channel element type.
      f2847307
  12. 11 Sep, 2019 1 commit
    • Kirill Smelkov's avatar
      golang: Mark chan<T> IO methods as const · 3aca691e
      Kirill Smelkov authored
      chan<T> is a pointer type and e.g. send does not change the pointer - it
      only "modifies" channel buffer. Marking appropriate methods as const is
      needed so that `const chan<T>` could be used to send/receive the same
      way as `chan <T>` is used.
      3aca691e
  13. 29 Aug, 2019 4 commits
    • Kirill Smelkov's avatar
      Port/move channels to C/C++/Pyx · 3b241983
      Kirill Smelkov authored
      - Move channels implementation to be done in C++ inside libgolang. The
        code and logic is based on previous Python-level channels
        implementation, but the new code is just C++ and does not depend on
        Python nor GIL at all, and so works without GIL if libgolang
        runtime works without GIL(*).
      
        (*) for example "thread" runtime works without GIL, while "gevent" runtime
            acquires GIL on every semaphore acquire.
      
        New channels implementation is located in δ(libgolang.cpp).
      
      - Provide low-level C channels API to the implementation. The low-level
        C API was inspired by Libtask[1] and Plan9/Libthread[2].
      
        [1] Libtask: a Coroutine Library for C and Unix. https://swtch.com/libtask.
        [2] http://9p.io/magic/man2html/2/thread.
      
      - Provide high-level C++ channels API that provides type-safety and
        automatic channel lifetime management.
      
        Overview of C and C++ APIs are in δ(libgolang.h).
      
      - Expose C++ channels API at Pyx level as Cython/nogil API so that Cython
        programs could use channels with ease and without need to care about
        lifetime management and low-level details.
      
        Overview of Cython/nogil channels API is in δ(README.rst) and
        δ(_golang.pxd).
      
      - Turn Python channels to be tiny wrapper around chan<PyObject>.
      
      Implementation note:
      
      - gevent case needs special care because greenlet, which gevent uses,
        swaps coroutine stack from C stack to heap on coroutine park, and
        replaces that space on C stack with stack of activated coroutine
        copied back from heap. This way if an object on g's stack is accessed
        while g is parked it would be memory of another g's stack.
      
        The channels implementation explicitly cares about this issue so that
        stack -> * channel send, or * -> stack channel receive work correctly.
      
        It should be noted that greenlet approach, which it inherits from
        stackless, is not only a bit tricky, but also comes with overhead
        (stack <-> heap copy), and prevents a coroutine to migrate from 1 OS
        thread to another OS thread as that would change addresses of on-stack
        things for that coroutine.
      
        As the latter property prevents to use multiple CPUs even if the
        program / runtime are prepared to work without GIL, it would be more
        logical to change gevent/greenlet to use separate stack for each
        coroutine. That would remove stack <-> heap copy and the need for
        special care in channels implementation for stack - stack sends.
        Such approach should be possible to implement with e.g. swapcontext or
        similar mechanism, and a proof of concept of such work wrapped into
        greenlet-compatible API exists[3]. It would be good if at some point
        there would be a chance to explore such approach in Pygolang context.
      
        [3] https://github.com/python-greenlet/greenlet/issues/113#issuecomment-264529838 and below
      
      Just this patch brings in the following speedup at Python level:
      
       (on i7@2.6GHz)
      
      thread runtime:
      
          name             old time/op  new time/op  delta
          go               20.0µs ± 1%  15.6µs ± 1%  -21.84%  (p=0.000 n=10+10)
          chan             9.37µs ± 4%  2.89µs ± 6%  -69.12%  (p=0.000 n=10+10)
          select           20.2µs ± 4%   3.4µs ± 5%  -83.20%  (p=0.000 n=8+10)
          def              58.0ns ± 0%  60.0ns ± 0%   +3.45%  (p=0.000 n=8+10)
          func_def         43.8µs ± 1%  43.9µs ± 1%     ~     (p=0.796 n=10+10)
          call             62.4ns ± 1%  63.5ns ± 1%   +1.76%  (p=0.001 n=10+10)
          func_call        1.06µs ± 1%  1.05µs ± 1%   -0.63%  (p=0.002 n=10+10)
          try_finally       136ns ± 0%   137ns ± 0%   +0.74%  (p=0.000 n=9+10)
          defer            2.28µs ± 1%  2.33µs ± 1%   +2.34%  (p=0.000 n=10+10)
          workgroup_empty  48.2µs ± 1%  34.1µs ± 2%  -29.18%  (p=0.000 n=9+10)
          workgroup_raise  58.9µs ± 1%  45.5µs ± 1%  -22.74%  (p=0.000 n=10+10)
      
      gevent runtime:
      
          name             old time/op  new time/op  delta
          go               24.7µs ± 1%  15.9µs ± 1%  -35.72%  (p=0.000 n=9+9)
          chan             11.6µs ± 1%   7.3µs ± 1%  -36.74%  (p=0.000 n=10+10)
          select           22.5µs ± 1%  10.4µs ± 1%  -53.73%  (p=0.000 n=10+10)
          def              55.0ns ± 0%  55.0ns ± 0%     ~     (all equal)
          func_def         43.6µs ± 1%  43.6µs ± 1%     ~     (p=0.684 n=10+10)
          call             63.0ns ± 0%  64.0ns ± 0%   +1.59%  (p=0.000 n=10+10)
          func_call        1.06µs ± 1%  1.07µs ± 1%   +0.45%  (p=0.045 n=10+9)
          try_finally       135ns ± 0%   137ns ± 0%   +1.48%  (p=0.000 n=10+10)
          defer            2.31µs ± 1%  2.33µs ± 1%   +0.89%  (p=0.000 n=10+10)
          workgroup_empty  70.2µs ± 0%  55.8µs ± 0%  -20.63%  (p=0.000 n=10+10)
          workgroup_raise  90.3µs ± 0%  70.9µs ± 1%  -21.51%  (p=0.000 n=9+10)
      
      The whole Cython/nogil work - starting from 8fa3c15b (Start using Cython
      and providing Cython/nogil API) to this patch - brings in the following
      speedup at Python level:
      
       (on i7@2.6GHz)
      
      thread runtime:
      
          name             old time/op  new time/op  delta
          go               92.9µs ± 1%  15.6µs ± 1%  -83.16%  (p=0.000 n=10+10)
          chan             13.9µs ± 1%   2.9µs ± 6%  -79.14%  (p=0.000 n=10+10)
          select           29.7µs ± 6%   3.4µs ± 5%  -88.55%  (p=0.000 n=10+10)
          def              57.0ns ± 0%  60.0ns ± 0%   +5.26%  (p=0.000 n=10+10)
          func_def         44.0µs ± 1%  43.9µs ± 1%     ~     (p=0.055 n=10+10)
          call             63.5ns ± 1%  63.5ns ± 1%     ~     (p=1.000 n=10+10)
          func_call        1.06µs ± 0%  1.05µs ± 1%   -1.31%  (p=0.000 n=10+10)
          try_finally       139ns ± 0%   137ns ± 0%   -1.44%  (p=0.000 n=10+10)
          defer            2.36µs ± 1%  2.33µs ± 1%   -1.26%  (p=0.000 n=10+10)
          workgroup_empty  98.4µs ± 1%  34.1µs ± 2%  -65.32%  (p=0.000 n=10+10)
          workgroup_raise   135µs ± 1%    46µs ± 1%  -66.35%  (p=0.000 n=10+10)
      
      gevent runtime:
      
          name             old time/op  new time/op  delta
          go               68.8µs ± 1%  15.9µs ± 1%  -76.91%  (p=0.000 n=10+9)
          chan             14.8µs ± 1%   7.3µs ± 1%  -50.67%  (p=0.000 n=10+10)
          select           32.0µs ± 0%  10.4µs ± 1%  -67.57%  (p=0.000 n=10+10)
          def              58.0ns ± 0%  55.0ns ± 0%   -5.17%  (p=0.000 n=10+10)
          func_def         43.9µs ± 1%  43.6µs ± 1%   -0.53%  (p=0.035 n=10+10)
          call             63.5ns ± 1%  64.0ns ± 0%   +0.79%  (p=0.033 n=10+10)
          func_call        1.08µs ± 1%  1.07µs ± 1%   -1.74%  (p=0.000 n=10+9)
          try_finally       142ns ± 0%   137ns ± 0%   -3.52%  (p=0.000 n=10+10)
          defer            2.32µs ± 1%  2.33µs ± 1%   +0.71%  (p=0.005 n=10+10)
          workgroup_empty  90.3µs ± 0%  55.8µs ± 0%  -38.26%  (p=0.000 n=10+10)
          workgroup_raise   108µs ± 1%    71µs ± 1%  -34.64%  (p=0.000 n=10+10)
      
      This patch is the final patch in series to reach the goal of providing
      channels that could be used in Cython/nogil code.
      
      Cython/nogil channels work is dedicated to the memory of Вера Павловна Супрун[4].
      
      [4] https://navytux.spb.ru/memory/%D0%A2%D1%91%D1%82%D1%8F%20%D0%92%D0%B5%D1%80%D0%B0.pdf#page=3
      3b241983
    • Kirill Smelkov's avatar
      golang.pyx: Switch pychan from `class` to `cdef class` · 1bcb8297
      Kirill Smelkov authored
      We will need to add C-level attributes to pychan and this requires it to
      become cdef class. The class is exported because at least
      _golang_test.pyx will also need to have access to those attributes.
      
      If we just do `class pychan` -> `cdef class pychan` e.g. the following
      starts to break:
      
          1.venv/local/lib/python2.7/site-packages/py/_path/local.py:701: in pyimport
              __import__(modname)
          golang/__init__.py:174: in <module>
              from ._golang import    \
          golang/_golang.pyx:455: in init golang._golang
              _pychan_send  = _pychan_send.__func__
          E   AttributeError: 'method_descriptor' object has no attribute '__func__'
      
      and
      
          golang/_golang.pyx:513: in golang._golang.pyselect
              if im_class(recv) is not pychan:
          _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      
          f = <built-in method recv of golang._golang.pychan object at 0x7f7055e57cc8>
      
              def im_class(f):
          >       return f.im_class
          E       AttributeError: 'builtin_function_or_method' object has no attribute 'im_class'
      
          golang/_pycompat.py:28: AttributeError
      
      This is probably because for `cdef class` methods Cython does not
      emulate full method bindings the same way as Python does.  Anyway we can
      check which method is passed to pyselect by chanop.__name__ or by
      inspecting PyCFunction directly. And not having method binding wrapper
      should only remove a bit of overhead.
      
      So we are ok with reworking send/recv chanop detection, and since
      this way im_class provided by golang._pycompat becomes unused, it is
      also removed.
      
      The timings are probably within noise:
      
       (on i7@2.6GHz)
      
      thread runtime:
      
          name             old time/op  new time/op  delta
          go               21.7µs ± 1%  20.0µs ± 1%  -7.60%  (p=0.000 n=10+10)
          chan             9.91µs ± 4%  9.37µs ± 4%  -5.39%  (p=0.000 n=10+10)
          select           19.2µs ± 4%  20.2µs ± 4%  +5.62%  (p=0.001 n=9+8)
          def              58.0ns ± 0%  58.0ns ± 0%    ~     (all equal)
          func_def         44.4µs ± 0%  43.8µs ± 1%  -1.22%  (p=0.000 n=10+10)
          call             63.0ns ± 0%  62.4ns ± 1%  -0.95%  (p=0.011 n=10+10)
          func_call        1.05µs ± 1%  1.06µs ± 1%    ~     (p=0.059 n=10+10)
          try_finally       135ns ± 0%   136ns ± 0%  +0.74%  (p=0.000 n=10+9)
          defer            2.36µs ± 1%  2.28µs ± 1%  -3.59%  (p=0.000 n=10+10)
          workgroup_empty  49.0µs ± 1%  48.2µs ± 1%  -1.63%  (p=0.000 n=10+9)
          workgroup_raise  62.6µs ± 1%  58.9µs ± 1%  -5.96%  (p=0.000 n=10+10)
      
      gevent runtime:
      
          name             old time/op  new time/op  delta
          go               21.7µs ± 1%  20.5µs ± 1%  -5.33%  (p=0.000 n=10+9)
          chan             9.91µs ± 4%  9.72µs ± 5%    ~     (p=0.190 n=10+10)
          select           19.2µs ± 4%  19.5µs ±14%    ~     (p=0.968 n=9+10)
          def              58.0ns ± 0%  58.0ns ± 0%    ~     (all equal)
          func_def         44.4µs ± 0%  45.4µs ± 1%  +2.23%  (p=0.000 n=10+10)
          call             63.0ns ± 0%  64.0ns ± 0%  +1.59%  (p=0.000 n=10+10)
          func_call        1.05µs ± 1%  1.06µs ± 0%  +0.65%  (p=0.002 n=10+10)
          try_finally       135ns ± 0%   137ns ± 0%  +1.48%  (p=0.000 n=10+10)
          defer            2.36µs ± 1%  2.38µs ± 1%  +0.72%  (p=0.006 n=10+10)
          workgroup_empty  49.0µs ± 1%  48.2µs ± 1%  -1.65%  (p=0.000 n=10+10)
          workgroup_raise  62.6µs ± 1%  60.3µs ± 1%  -3.69%  (p=0.000 n=10+10)
      1bcb8297
    • Kirill Smelkov's avatar
      pyx api: Provide go · f971a2a8
      Kirill Smelkov authored
      - Add go functionality to libgolang runtime;
      - Implement go for thread and gevent runtimes.
      
        * Thread runtime uses PyThread_start_new_thread which, if used
          carefully, does not depend on Python GIL and on e.g. POSIX reduces to
          tiny wrapper around pthread_create.
      
        * Gevent runtime uses gevent's Greenlet in Pyx mode. This turns
          gevent to be a build-time dependency.
      
      - Provide low-level _taskgo in C client API;
      - Provide type-safe C++-level go wrapper over _taskgo;
      - Switch golang.go from py implementation into Pyx wrapper over
        Pyx/nogil API.
      
      This is the first patch that adds Pyx/C++/C-level unit tests and hooks
      them into golang_test.py .
      
      This patch brings the following speedup to Python-level interface:
      
       (on i7@2.6GHz)
      
      thread runtime:
      
          name             old time/op  new time/op  delta
          go               93.0µs ± 1%  26.6µs ± 1%  -71.41%  (p=0.000 n=10+10)
          chan             13.6µs ± 2%  13.7µs ± 1%     ~     (p=0.280 n=10+10)
          select           29.9µs ± 4%  29.3µs ± 2%   -1.89%  (p=0.017 n=10+9)
          def              61.0ns ± 0%  55.0ns ± 0%   -9.84%  (p=0.000 n=10+10)
          func_def         43.8µs ± 1%  44.0µs ± 1%   +0.66%  (p=0.006 n=10+10)
          call             62.5ns ± 1%  64.0ns ± 0%   +2.40%  (p=0.000 n=10+8)
          func_call        1.06µs ± 1%  1.09µs ± 1%   +2.72%  (p=0.000 n=10+10)
          try_finally       137ns ± 0%   139ns ± 2%   +1.17%  (p=0.033 n=10+10)
          defer            2.34µs ± 1%  2.36µs ± 1%   +0.84%  (p=0.015 n=10+10)
          workgroup_empty  96.1µs ± 1%  58.1µs ± 1%  -39.55%  (p=0.000 n=10+10)
          workgroup_raise   135µs ± 1%    73µs ± 1%  -45.97%  (p=0.000 n=10+10)
      
      gevent runtime:
      
          name             old time/op  new time/op  delta
          go               68.8µs ± 1%  28.6µs ± 0%  -58.47%  (p=0.000 n=10+8)
          chan             14.8µs ± 1%  15.8µs ± 1%   +6.19%  (p=0.000 n=10+10)
          select           32.0µs ± 0%  33.1µs ± 1%   +3.25%  (p=0.000 n=10+10)
          def              58.0ns ± 0%  55.0ns ± 0%   -5.17%  (p=0.000 n=10+10)
          func_def         43.9µs ± 1%  44.4µs ± 2%   +1.20%  (p=0.007 n=10+10)
          call             63.5ns ± 1%  64.0ns ± 2%     ~     (p=0.307 n=10+10)
          func_call        1.08µs ± 1%  1.06µs ± 0%   -2.55%  (p=0.000 n=10+8)
          try_finally       142ns ± 0%   136ns ± 0%   -4.23%  (p=0.000 n=10+9)
          defer            2.32µs ± 1%  2.29µs ± 1%   -0.96%  (p=0.000 n=10+10)
          workgroup_empty  90.3µs ± 0%  73.8µs ± 1%  -18.29%  (p=0.000 n=10+10)
          workgroup_raise   108µs ± 1%    94µs ± 0%  -13.29%  (p=0.000 n=10+10)
      
      (small changes are probably within noise; "go" and "workgroup_*" should be
      representative)
      f971a2a8
    • Kirill Smelkov's avatar
      Start using Cython and providing Cython/nogil API · 8fa3c15b
      Kirill Smelkov authored
      Start transforming Pygolang from being pure Python project into project
      that uses both Python and Cython and provides both Python and Cython API
      to its users.
      
      For Cython API there is a goal for it to be usable independently of
      Python GIL - i.e. independently of Python runtime and limitations
      imposed by Python's global interpreter lock. The motivation for this
      work is wendelin.core v2, which, due to its design, would deadlock if it
      tries to take the GIL in its pinner thread.
      
      This commit brings bare minimum establishing libraries and build system.
      It:
      
      - Introduces C/C++ library libgolang. Libgolang will be serving most of
        nogil functionality and Pyx API will be a small wrapper over it. At
        this step Libgolang comes only with C-level panic;
      
      - Introduces Pyx package golang that is aliased to Pyx package
        golang._golang . Users can do `from golang cimport ...` similarly to how
        they do py-level `from golang import ...`. At this step golang.pyx
        only wraps panic and provides way to transform C-level panic into Python
        exception;
      
      - Introduces Py package golang.pyx.build . This package, similarly to
        golang.org/pkg/go/build, serves as the system to build Pyx packages -
        both external packages that use Pygolang in pyx mode, and to build
        Pygolang itself. The build, internally, is served by setuptools_dso
        and cython. An example how to use golang.pyx.build from external
        project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
        tests to verify golang.pyx.build functionality;
      
      - Introduces Pygolang build dependency to cython and setuptools_dso.
      8fa3c15b