1. 29 Apr, 2020 1 commit
  2. 28 Feb, 2020 1 commit
  3. 12 Feb, 2020 1 commit
  4. 11 Feb, 2020 1 commit
    • Kirill Smelkov's avatar
      golang, errors, fmt: Error chaining (Python) · 337de0d7
      Kirill Smelkov authored
      Following errors model in Go and fd95c88a (golang, errors, fmt: Error
      chaining (C++/Pyx)) let's add support at Python-level for errors to wrap
      each other and to be inspected/unwrapped:
      
      - an error can additionally provide way to unwrap itself, if it
        provides .Unwrap() method. .__cause__ is not taken into account yet,
        but will be in a follow-up patch;
      - errors.Is(err) tests whether an item in error's chain matches target;
      - `fmt.Errorf("... : %w", ... err)` is similar to `"... : %s" % (..., err)`
        but resulting error, when unwrapped, will return err.
      - errors.Unwrap is not exposed as chaining through both .Unwrap() and
        .__cause__ will need more than just "current element" as unwrapping
        state (i.e. errors.Unwrap API is insufficient - see next patch), and
        in practice users of errors.Unwrap() are very seldom.
      
      Support for error chaining through .__cause__ will follow in the next
      patch.
      
      Top-level documentation is TODO.
      
      See https://blog.golang.org/go1.13-errors for error chaining overview.
      337de0d7
  5. 10 Feb, 2020 1 commit
    • Kirill Smelkov's avatar
      golang: Expose error at Py level · 17798442
      Kirill Smelkov authored
      The first step to expose errors and error chaining to Python:
      
      - Add pyerror that wraps a pyx/nogil C-level error and is exposed as golang.error at py level.
      - py errors must be compared by ==, not by "is"
      - Add (py) errors.New to create a new error from text.
      - a C-level error that has .Unwrap, is exposed with .Unwrap at py level,
        but full py-level chaining will be implemented in a follow-up patch.
      - py error does not support inheritance yet.
      
      Top-level documentation is TODO.
      17798442
  6. 04 Feb, 2020 5 commits
  7. 08 Jan, 2020 1 commit
  8. 27 Nov, 2019 2 commits
    • Kirill Smelkov's avatar
      strings: New package · 0efd4a9a
      Kirill Smelkov authored
      Provide string utilities to verify whether string has suffix/prefix,
      trim it, and split string by a delimiter. The code originated in wcfs
      codebase in wendelin.core .
      
      Pyx/nogil only.
      0efd4a9a
    • Kirill Smelkov's avatar
      fmt: New package · 309963f8
      Kirill Smelkov authored
      With fmt.sprintf() and fmt.errorf() to format strings and errors. The
      code was extracted from wcfs in wendelin.core .
      
      Pyx/nogil only.
      309963f8
  9. 25 Nov, 2019 1 commit
    • Kirill Smelkov's avatar
      libpyxruntime: It should link to libgolang · 6d94fccf
      Kirill Smelkov authored
      It was all working on Debian 10, but running tests on Ubuntu 16.04
      turned out to fail:
      
          $ python -m pytest
          ================================== test session starts ===================================
          platform linux2 -- Python 2.7.12, pytest-4.6.6, py-1.8.0, pluggy-0.13.0
          rootdir: /home/kirr/src/pygolang
          collected 0 items / 10 errors
      
          ========================================= ERRORS =========================================
          ________________________ ERROR collecting golang/_gopath_test.py _________________________
          ImportError while importing test module '/home/kirr/src/pygolang/golang/_gopath_test.py'.
          Hint: make sure your test modules/packages have valid Python names.
          Traceback:
          golang/__init__.py:42: in <module>
              from golang._golang import _pysys_exc_clear as _sys_exc_clear
          golang/_golang.pyx:35: in init golang._golang
              _init_libpyxruntime()
          golang/_golang.pyx:532: in golang._golang._init_libpyxruntime
              import golang.pyx.runtime
          E   ImportError: /home/kirr/src/pygolang/golang/pyx/../runtime/liblibpyxruntime.so.0.1: undefined symbol: _ZN6golang4sync9WaitGroup3addEi
      
          $ c++filt _ZN6golang4sync9WaitGroup3addEi
          golang::sync::WaitGroup::add(int)
      
      So since libpyxruntime is using sync.WaitGroup (and sync.Mutex etc) it should
      be linking to libgolang where those symbols are provided. Fix it.
      
      Fixes: 4fc6e49c (time: Factor-out PyFunc into shared library libpyxruntime.so)
      6d94fccf
  10. 22 Nov, 2019 1 commit
    • Kirill Smelkov's avatar
      sync: Move/Port sync.WorkGroup to C++/Pyx nogil · 33cf3113
      Kirill Smelkov authored
      Provide sync.WorkGroup that can be used directly from C++ and Pyx/nogil codes.
      Python-level sync.WorkGroup becomes a wrapper around pyx/nogil one.
      
      Like with context (2a359791 "context: Move/Port context package to
      C++/Pyx nogil"), timers (b073f6df "time: Move/Port timers to C++/Pyx
      nogil") and interfaces (5a99b769 "libgolang: Start providing
      interfaces") memory for on-heap allocated WorkGroup is managed
      automatically.
      
      Python-level tests should be enough to cover C++/Pyx functionality at
      zero-level approximation.
      33cf3113
  11. 19 Nov, 2019 2 commits
    • Kirill Smelkov's avatar
      libpyxruntime: Fix deadlock in pygil_ensure() · 73182038
      Kirill Smelkov authored
      pygil_ensure() was taking pyexitedMu and then GIL, which can deadlock
      via AB BA scenario if pygil_ensure() is also called in another thread
      with GIL already held.
      
      -> Fix it by always taking GIL with pyexitedMu released.
      
      Without the fix the test added always hangs with thread backend, e.g.
      like this:
      
          golang/pyx/runtime_test.py::test_pyx_pyfunc_vs_gil_deadlock
          STUCK
          terminate called after throwing an instance of 'golang::PanicError'
            what():  STUCK
          Fatal Python error: Aborted
      
      The bug was introduced in b073f6df (time: Move/Port timers to C++/Pyx nogil).
      73182038
    • Kirill Smelkov's avatar
      time: Factor-out PyFunc into shared library libpyxruntime.so · 4fc6e49c
      Kirill Smelkov authored
      Commit b073f6df (time: Move/Port timers to C++/Pyx nogil) added C++
      level PyFunc as a way to call Python function from under nogil code.
      We will soon need this functionality for sync.WorkGroup too, so move it
      into shared place.
      
      We move into regular library - not a pyx module (e.g. golang.pyx) -
      because for pyx modules linking is done at import time by specially
      generated code, while we need the linker support to automatically
      resolve e.g. calls to PyFunc::operator() from inside std::function.
      4fc6e49c
  12. 14 Nov, 2019 3 commits
    • Kirill Smelkov's avatar
      context: Move/Port context package to C++/Pyx nogil · 2a359791
      Kirill Smelkov authored
      Provide context-related functionality that can be used directly from C++
      and Pyx/nogil codes. Python-level classes and functions become small
      wrappers around pyx/nogil ones.
      
      Like with timers (b073f6df "time: Move/Port timers to C++/Pyx nogil")
      and interfaces (5a99b769 "libgolang: Start providing interfaces")
      memory for objects dynamically allocated on heap is managed
      automatically.
      2a359791
    • Kirill Smelkov's avatar
      cxx: New package · 9785f2d3
      Kirill Smelkov authored
      Start new package that provides C++ amendments to be used by libgolang
      and its users. Current functionality: dict and set that wrap
      std::unordered_map and std::unordered_set into ergonomic interface.
      
      The code originates from wendelin.core:
      https://lab.nexedi.com/kirr/wendelin.core/blob/5a045ed1/wcfs/internal/wcfs_misc.h#L214-257
      
      Pyx/nogil only.
      9785f2d3
    • Kirill Smelkov's avatar
      errors: New package · a245ab56
      Kirill Smelkov authored
      With just one function - errors.New() - to create new error with given
      text.
      
      This will be soon used in Pyx/nogil context package to create context.canceled
      and context.deadlineExceeded errors.
      
      Pyx/nogil only for now.
      a245ab56
  13. 07 Nov, 2019 1 commit
    • Kirill Smelkov's avatar
      libgolang: Split time and sync into their own packages · 1a9dae3b
      Kirill Smelkov authored
      We already have some sync functionality implemented in C++ (e.g.
      sync.Once, sync.WaitGroup) and we are going to add more and also move
      timers implementation to C++. It is getting crowded for that
      functionality to still live in libgolang.{h,cpp}
      
      -> Split sync & time functionality into their own C++ packages (still
      built-into libgolang.so)
      1a9dae3b
  14. 04 Nov, 2019 1 commit
    • Kirill Smelkov's avatar
      context: Move code to pyx · cc7069e0
      Kirill Smelkov authored
      In preparation to start migrating, at least partially, context
      functionality to nogil mode, first, similarly to package time (see
      32f34607 "time: Move code to pyx"), move the code from context.py to
      _context.pyx.
      
      This is straight code movement + associated  setup.py & co adjusts.
      
      We don't move just to context.pyx (note no _ prefix), since we will need
      to continue distinguishing pyx/nogil from py objects/functions.
      cc7069e0
  15. 18 Oct, 2019 1 commit
    • Kirill Smelkov's avatar
      golang: Teach defer to chain exceptions (PEP 3134) even on Python2 · bb9a94c3
      Kirill Smelkov authored
      Python3 chains exceptions, so that e.g. if exc1 is raised and, while it
      was not handled, another exc2 is raised, exc2 will be linked to exc1 via
      exc2.__context__ attribute and exc1 will be included into exc2 traceback
      printout. However many projects still use Python2 and there is no
      similar chaining functionality there. This way exc1 is completely lost.
      
      Since defer code is in our hands, we can teach it to implement exception
      chaining even on Python2 by carefully analyzing what happens in
      _GoFrame.__exit__().
      
      Implementing chaining itself is relatively easy, but is only part of the
      story. Even if an exception is chained with its cause, but exception
      dump does not show the cause, the chaining will be practically useless.
      With this in mind this patches settles not only on implementing chaining
      itself, but on also giving a promise that chained cause exceptions will
      be included into traceback dumps as well.
      
      To realize this promise we adjust all exception dumping funcitons in
      traceback module and carefully install adjusted
      traceback.print_exception() into sys.excepthook. This amends python
      interactive sessions and programs run by python interpreter to include
      causes in exception dumps. "Careful" here means that we don't change
      sys.excepthook if on golang module load we see that sys.excepthook was already
      changed by some other module - e.g. due to IPython session running
      because IPython installs its own sys.excepthook. In such cases we don't
      install our sys.excepthook, but we also provide integration patches that
      add exception chaining support for traceback dump functionality in
      popular third-party software. The patches (currently for IPython and
      Pytest) are activated automatically, but only when/if corresponding
      software is imported and actually used. This should give practically
      good implementation of the promise - a user can now rely on seeing
      exception cause in traceback dump whatever way python programs are run.
      
      The implementation takes https://pypi.org/project/pep3134/ experience
      into account [1]. peak.utils.imports [2,3] is used to be notified when/if
      third-party module is imported.
      
      [1] https://github.com/9seconds/pep3134/
      [2] https://pypi.org/project/Importing/
      [3] http://peak.telecommunity.com/DevCenter/Importing
      
      This patch originally started as hacky workaround in wendelin.core
      because in wcfs tests I was frequently hitting situations, where
      exception raised by an assert was hidden by another exception raised in
      further generic teardown check. For example wcfs tests check that wcfs
      is unmounted after every test run [4] and if that fails it was hiding
      problems raised by an assert. As the result I was constantly guessing
      and adding code like [5] to find what was actually breaking. At some
      point I added hacky workaround for defer to print cause exception not to
      loose it [6]. [7] has more context and background discussion on this topic.
      
      [4] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L70
      [5] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L853-857
      [6] kirr/wendelin.core@c00d94c7
      [7] zodbtools!13 (comment 81553)
      
      After this patch, on Python2
      
          defer(cleanup1)
          defer(cleanup2)
          defer(cleanup3)
          ...
      
      is no longer just a syntatic sugar for
      
          try:
              try:
                  try:
                      ...
                  finally:
                      cleanup3()
              finally:
                  cleanup2()
          finally:
              cleanup1()
      bb9a94c3
  16. 04 Oct, 2019 1 commit
    • Kirill Smelkov's avatar
      golang: Expose Sema and Mutex as public Python and Cython/nogil API · 34b7a1f4
      Kirill Smelkov authored
      Until now libgolang had semaphore and mutex functionality, but
      implemented only internally and not exposed to users. Since for its
      pinner thread wendelin.core v2 needs not only nogil channels, but also
      nogil mutexes, timers and so on, it is now time to start providing that.
      
      We start with mutexes:
      
      - Expose Mutex from insides of libgolang.cpp to public API in
        libgolang.h . We actually expose both Mutex and Sema because
        semaphores are sometimes also useful for low-level synchronization,
        and because it is easier to export Mutex and Sema both at the same time.
      
      - Provide pyx/nogil API to those as sync.Mutex and sync.Sema.
      
      - Provide corresponding python wrappers.
      
      - Add Pyx-level test for semaphore wakeup when wakeup is done not by the
        same thread which did the original acquire. This is the test that was
        promised in 5142460d (libgolang/thread: Add links to upstream
        PyThread_release_lock bug), and it used to corrupt memory and deadlock
        on macOS due to CPython & PyPy runtime bugs:
      
          https://bugs.python.org/issue38106
            -> https://github.com/python/cpython/pull/16047
          https://bitbucket.org/pypy/pypy/issues/3072
      
      Note about python wrappers: At present, due to
      https://github.com/cython/cython/issues/3165, C-level panic is not
      properly translated into Py-level exception in (Py)Sema/Mutex constructors.
      This should not pose a real problem in practice though.
      34b7a1f4
  17. 17 Sep, 2019 2 commits
    • Kirill Smelkov's avatar
      pygolang v0.0.4 · 1573d101
      Kirill Smelkov authored
      This release is bugfix-only. Compared to pygolang v0.0.3 (4ca65816)
      the change in speed is likely within noise:
      
           (on i7@2.6GHz)
      
      thread runtime:
      
          name             old time/op  new time/op  delta
          go               18.3µs ± 1%  18.3µs ± 1%    ~     (p=0.218 n=10+10)
          chan             2.97µs ± 5%  2.97µs ± 8%    ~     (p=0.781 n=10+10)
          select           3.59µs ± 2%  3.55µs ± 5%    ~     (p=0.447 n=9+10)
          def              56.0ns ± 0%  55.0ns ± 0%  -1.79%  (p=0.000 n=10+10)
          func_def         43.7µs ± 1%  43.8µs ± 1%  +0.35%  (p=0.029 n=10+10)
          call             65.0ns ± 0%  62.3ns ± 1%  -4.15%  (p=0.000 n=10+10)
          func_call        1.06µs ± 1%  1.04µs ± 0%  -1.26%  (p=0.000 n=10+8)
          try_finally       137ns ± 1%   137ns ± 0%    ~     (p=1.000 n=10+10)
          defer            2.32µs ± 0%  2.33µs ± 1%  +0.43%  (p=0.000 n=9+10)
          workgroup_empty  37.6µs ± 1%  37.1µs ± 2%  -1.29%  (p=0.003 n=10+10)
          workgroup_raise  47.9µs ± 1%  47.6µs ± 0%  -0.63%  (p=0.001 n=9+9)
      
      gevent runtime:
      
          name             old time/op  new time/op  delta
          go               15.8µs ± 0%  16.1µs ± 1%  +2.18%  (p=0.000 n=9+10)
          chan             7.36µs ± 0%  7.21µs ± 0%  -1.97%  (p=0.000 n=8+10)
          select           10.4µs ± 0%  10.5µs ± 0%  +0.71%  (p=0.000 n=10+10)
          def              57.0ns ± 0%  55.0ns ± 0%  -3.51%  (p=0.000 n=10+10)
          func_def         43.3µs ± 1%  44.1µs ± 2%  +1.81%  (p=0.000 n=10+10)
          call             66.0ns ± 0%  65.0ns ± 0%  -1.52%  (p=0.000 n=10+10)
          func_call        1.04µs ± 1%  1.06µs ± 1%  +1.48%  (p=0.000 n=10+10)
          try_finally       137ns ± 1%   136ns ± 0%  -1.31%  (p=0.000 n=10+10)
          defer            2.32µs ± 0%  2.31µs ± 1%    ~     (p=0.472 n=8+10)
          workgroup_empty  56.0µs ± 0%  55.7µs ± 0%  -0.49%  (p=0.000 n=10+10)
          workgroup_raise  71.3µs ± 1%  71.7µs ± 1%  +0.62%  (p=0.001 n=10+10)
      1573d101
    • Kirill Smelkov's avatar
      tox += ThreadSanitizer, AddressSanitizer, Python debug builds · 4dc1a7f0
      Kirill Smelkov authored
      - ThreadSanitizer helps to detect races and some memory errors,
      - AddressSanitizer helps to detect memory errors,
      - Python debug builds help to detect e.g reference counting errors.
      
      Adding all those tools to testing coverage discovers e.g. the following
      bugs (not a full list):
      
      ---- 8< ----
      
      py27-thread-tsan:
      WARNING: ThreadSanitizer: data race (pid=7143)
        Write of size 8 at 0x7b1400000650 by main thread:
          #0 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:649 (libtsan.so.0+0x2b46a)
          #1 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:643 (libtsan.so.0+0x2b46a)
          #2 golang::_chan::decref() golang/runtime/libgolang.cpp:470 (liblibgolang.so.0.1+0x47f2)
          #3 _chanxdecref golang/runtime/libgolang.cpp:452 (liblibgolang.so.0.1+0x484a)
          #4 _test_go_c golang/runtime/libgolang_test_c.c:86 (_golang_test.so+0x13a2e)
          #5 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
          #6 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
          #7 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)
      
        Previous read of size 8 at 0x7b1400000650 by thread T8:
          #0 golang::Sema::acquire() golang/runtime/libgolang.cpp:164 (liblibgolang.so.0.1+0x410a)
          #1 golang::Mutex::lock() golang/runtime/libgolang.cpp:175 (liblibgolang.so.0.1+0x4c82)
          #2 golang::_chan::close() golang/runtime/libgolang.cpp:754 (liblibgolang.so.0.1+0x4c82)
          #3 _chanclose golang/runtime/libgolang.cpp:732 (liblibgolang.so.0.1+0x4d1a)
          #4 _work golang/runtime/libgolang_test_c.c:92 (_golang_test.so+0x136cc)
          #5 <null> <null> (python2.7+0x1929e3)
      
        Thread T8 (tid=7311, finished) created by main thread at:
          #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:915 (libtsan.so.0+0x2be1b)
          #1 PyThread_start_new_thread <null> (python2.7+0x19299f)
          #2 _taskgo golang/runtime/libgolang.cpp:119 (liblibgolang.so.0.1+0x3f68)
          #3 _test_go_c golang/runtime/libgolang_test_c.c:84 (_golang_test.so+0x13a1c)
          #4 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
          #5 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
          #6 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)
      
      py37-thread-asan:
      ==22205==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000002cd0 at pc 0x7fd3732a7679 bp 0x7fd3723c8c50 sp 0x7fd3723c8c48
      READ of size 8 at 0x607000002cd0 thread T7
          #0 0x7fd3732a7678 in golang::Sema::acquire() golang/runtime/libgolang.cpp:164
          #1 0x7fd3732a8644 in golang::Mutex::lock() golang/runtime/libgolang.cpp:175
          #2 0x7fd3732a8644 in golang::_chan::close() golang/runtime/libgolang.cpp:754
          #3 0x7fd3724004b2 in golang::chan<golang::structZ>::close() const golang/libgolang.h:323
          #4 0x7fd3724004b2 in operator() golang/runtime/libgolang_test.cpp:262
          #5 0x7fd3724004b2 in __invoke_impl<void, _test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:60
          #6 0x7fd3724004b2 in __invoke<_test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:95
          #7 0x7fd3724004b2 in __call<void> /usr/include/c++/8/functional:400
          #8 0x7fd3724004b2 in operator()<> /usr/include/c++/8/functional:484
          #9 0x7fd3724004b2 in _M_invoke /usr/include/c++/8/bits/std_function.h:297
          #10 0x7fd3723fdc6e in std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687
          #11 0x7fd3723fdc6e in operator() golang/libgolang.h:273
          #12 0x7fd3723fdc6e in _FUN golang/libgolang.h:271
          #13 0x62ddf3  (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62ddf3)
          #14 0x7fd377393fa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
          #15 0x7fd376eda4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)
      
      0x607000002cd0 is located 16 bytes inside of 72-byte region [0x607000002cc0,0x607000002d08)
      freed by thread T0 here:
          #0 0x7fd377519fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0)
          #1 0x7fd372401335 in golang::chan<golang::structZ>::~chan() golang/libgolang.h:292
          #2 0x7fd372401335 in _test_chan_vs_stackdeadwhileparked() golang/runtime/libgolang_test.cpp:222
      
      previously allocated by thread T0 here:
          #0 0x7fd37751a518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518)
          #1 0x7fd3732a7d0b in zalloc golang/runtime/libgolang.cpp:1185
          #2 0x7fd3732a7d0b in _makechan golang/runtime/libgolang.cpp:413
      
      Thread T7 created by T0 here:
          #0 0x7fd377481db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
          #1 0x62df39 in PyThread_start_new_thread (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62df39)
      
      SUMMARY: AddressSanitizer: heap-use-after-free golang/runtime/libgolang.cpp:164 in golang::Sema::acquire()
      Shadow bytes around the buggy address:
        0x0c0e7fff8540: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
        0x0c0e7fff8550: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
        0x0c0e7fff8560: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
        0x0c0e7fff8570: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
        0x0c0e7fff8580: fd fd fd fd fd fa fa fa fa fa fd fd fd fd fd fd
      =>0x0c0e7fff8590: fd fd fd fa fa fa fa fa fd fd[fd]fd fd fd fd fd
        0x0c0e7fff85a0: fd fa fa fa fa fa 00 00 00 00 00 00 00 00 00 fa
        0x0c0e7fff85b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff85c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff85d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff85e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07
        Heap left redzone:       fa
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      
      ---- 8< ----
      
      The bugs will be addressed in the followup patches.
      4dc1a7f0
  18. 29 Aug, 2019 6 commits
    • Kirill Smelkov's avatar
      pygolang v0.0.3 · 4ca65816
      Kirill Smelkov authored
      4ca65816
    • Kirill Smelkov's avatar
      Hook in list.h from Linux · 9efb6575
      Kirill Smelkov authored
      Copy linux/list.h from wendelin.core which copied it from
      util-linux.git which took it from linux.git at LGPL license state.
      
      Here are corresponding links for wendelin.core and util-linux:
      
          wendelin.core@a0f940ad
          https://git.kernel.org/cgit/utils/util-linux/util-linux.git/tree/include/list.h?id=v2.25-165-g9138d6f
      
      Linked lists will be used in channels implementation in the next patch.
      9efb6575
    • Kirill Smelkov's avatar
      pyx api: Provide go · f971a2a8
      Kirill Smelkov authored
      - Add go functionality to libgolang runtime;
      - Implement go for thread and gevent runtimes.
      
        * Thread runtime uses PyThread_start_new_thread which, if used
          carefully, does not depend on Python GIL and on e.g. POSIX reduces to
          tiny wrapper around pthread_create.
      
        * Gevent runtime uses gevent's Greenlet in Pyx mode. This turns
          gevent to be a build-time dependency.
      
      - Provide low-level _taskgo in C client API;
      - Provide type-safe C++-level go wrapper over _taskgo;
      - Switch golang.go from py implementation into Pyx wrapper over
        Pyx/nogil API.
      
      This is the first patch that adds Pyx/C++/C-level unit tests and hooks
      them into golang_test.py .
      
      This patch brings the following speedup to Python-level interface:
      
       (on i7@2.6GHz)
      
      thread runtime:
      
          name             old time/op  new time/op  delta
          go               93.0µs ± 1%  26.6µs ± 1%  -71.41%  (p=0.000 n=10+10)
          chan             13.6µs ± 2%  13.7µs ± 1%     ~     (p=0.280 n=10+10)
          select           29.9µs ± 4%  29.3µs ± 2%   -1.89%  (p=0.017 n=10+9)
          def              61.0ns ± 0%  55.0ns ± 0%   -9.84%  (p=0.000 n=10+10)
          func_def         43.8µs ± 1%  44.0µs ± 1%   +0.66%  (p=0.006 n=10+10)
          call             62.5ns ± 1%  64.0ns ± 0%   +2.40%  (p=0.000 n=10+8)
          func_call        1.06µs ± 1%  1.09µs ± 1%   +2.72%  (p=0.000 n=10+10)
          try_finally       137ns ± 0%   139ns ± 2%   +1.17%  (p=0.033 n=10+10)
          defer            2.34µs ± 1%  2.36µs ± 1%   +0.84%  (p=0.015 n=10+10)
          workgroup_empty  96.1µs ± 1%  58.1µs ± 1%  -39.55%  (p=0.000 n=10+10)
          workgroup_raise   135µs ± 1%    73µs ± 1%  -45.97%  (p=0.000 n=10+10)
      
      gevent runtime:
      
          name             old time/op  new time/op  delta
          go               68.8µs ± 1%  28.6µs ± 0%  -58.47%  (p=0.000 n=10+8)
          chan             14.8µs ± 1%  15.8µs ± 1%   +6.19%  (p=0.000 n=10+10)
          select           32.0µs ± 0%  33.1µs ± 1%   +3.25%  (p=0.000 n=10+10)
          def              58.0ns ± 0%  55.0ns ± 0%   -5.17%  (p=0.000 n=10+10)
          func_def         43.9µs ± 1%  44.4µs ± 2%   +1.20%  (p=0.007 n=10+10)
          call             63.5ns ± 1%  64.0ns ± 2%     ~     (p=0.307 n=10+10)
          func_call        1.08µs ± 1%  1.06µs ± 0%   -2.55%  (p=0.000 n=10+8)
          try_finally       142ns ± 0%   136ns ± 0%   -4.23%  (p=0.000 n=10+9)
          defer            2.32µs ± 1%  2.29µs ± 1%   -0.96%  (p=0.000 n=10+10)
          workgroup_empty  90.3µs ± 0%  73.8µs ± 1%  -18.29%  (p=0.000 n=10+10)
          workgroup_raise   108µs ± 1%    94µs ± 0%  -13.29%  (p=0.000 n=10+10)
      
      (small changes are probably within noise; "go" and "workgroup_*" should be
      representative)
      f971a2a8
    • Kirill Smelkov's avatar
      pyx api: Provide sleep · ce8152a2
      Kirill Smelkov authored
      - Add sleep functionality to libgolang runtime;
      - Implement sleep for thread and gevent runtimes. Thread runtime
        implements sleep independently of GIL, but only for POSIX for now;
      - Switch golang.time py module into using golang.time pyx module.
      
      As we are adding sleep, related functionality to query system about
      "what is current time?" is also added.
      ce8152a2
    • Kirill Smelkov's avatar
      libgolang: Introduce runtimes · ad00be70
      Kirill Smelkov authored
      Libgolang will be generic library and it will adapt itself to particular
      execution environment by way of runtime plugin. This commit introduces
      stubs for two such runtimes - "thread" and "gevent" - and initializes
      libgolang with particular runtime depending on whether golang is
      imported with gevent preactivated or not.
      
      The runtimes themselves are empty at this step. We'll be adding runtime
      functionality in the following patches.
      ad00be70
    • Kirill Smelkov's avatar
      Start using Cython and providing Cython/nogil API · 8fa3c15b
      Kirill Smelkov authored
      Start transforming Pygolang from being pure Python project into project
      that uses both Python and Cython and provides both Python and Cython API
      to its users.
      
      For Cython API there is a goal for it to be usable independently of
      Python GIL - i.e. independently of Python runtime and limitations
      imposed by Python's global interpreter lock. The motivation for this
      work is wendelin.core v2, which, due to its design, would deadlock if it
      tries to take the GIL in its pinner thread.
      
      This commit brings bare minimum establishing libraries and build system.
      It:
      
      - Introduces C/C++ library libgolang. Libgolang will be serving most of
        nogil functionality and Pyx API will be a small wrapper over it. At
        this step Libgolang comes only with C-level panic;
      
      - Introduces Pyx package golang that is aliased to Pyx package
        golang._golang . Users can do `from golang cimport ...` similarly to how
        they do py-level `from golang import ...`. At this step golang.pyx
        only wraps panic and provides way to transform C-level panic into Python
        exception;
      
      - Introduces Py package golang.pyx.build . This package, similarly to
        golang.org/pkg/go/build, serves as the system to build Pyx packages -
        both external packages that use Pygolang in pyx mode, and to build
        Pygolang itself. The build, internally, is served by setuptools_dso
        and cython. An example how to use golang.pyx.build from external
        project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
        tests to verify golang.pyx.build functionality;
      
      - Introduces Pygolang build dependency to cython and setuptools_dso.
      8fa3c15b
  19. 26 Aug, 2019 1 commit
    • Kirill Smelkov's avatar
      setup: Provide golang.X requirements in pygolang[X] · 89a1061a
      Kirill Smelkov authored
      ... and everything that whole pygolang depends on in pygolang[all].
      
      The reason we do this: currently golang.perf.benchlib needs NumPy, but
      it would be not ok to depend whole pygolang on numpy. Similarly pybench
      needs pytest, but it would be not ok to require pytest unconditionally.
      
      Thus allow users to specify what they need from pygolang via extras,
      like pygolang[x.perf.benchlib] or pygolang[cmd.pybench], with
      pygolang[all] serving as merge point for all subpackages dependencies.
      
      TODO in the future we'll, hopefully, also provide accumulation points,
      like pygolang[x.perf] which contains everything pygolang[x.perf.*]
      depends on.
      
      Note: core functionality and gpython are provided always unconditionally.
      89a1061a
  20. 16 May, 2019 1 commit
  21. 24 Mar, 2019 1 commit
  22. 16 Jan, 2019 3 commits
  23. 13 Dec, 2018 1 commit
  24. 30 Oct, 2018 1 commit