pygolang:4ca65816bf09986bfd51aacb596ed40c2288e8b1 commitshttps://lab.nexedi.com/nexedi/pygolang/-/commits/4ca65816bf09986bfd51aacb596ed40c2288e8b12019-08-29T14:46:38+03:00https://lab.nexedi.com/nexedi/pygolang/-/commit/4ca65816bf09986bfd51aacb596ed40c2288e8b1pygolang v0.0.32019-08-29T14:46:38+03:00Kirill Smelkovkirr@nexedi.comhttps://lab.nexedi.com/nexedi/pygolang/-/commit/3b241983814e5836abae004d8549a68572731328Port/move channels to C/C++/Pyx2019-08-29T14:17:36+03:00Kirill Smelkovkirr@nexedi.com
- Move channels implementation to be done in C++ inside libgolang. The
code and logic is based on previous Python-level channels
implementation, but the new code is just C++ and does not depend on
Python nor GIL at all, and so works without GIL if libgolang
runtime works without GIL(*).
(*) for example "thread" runtime works without GIL, while "gevent" runtime
acquires GIL on every semaphore acquire.
New channels implementation is located in δ(libgolang.cpp).
- Provide low-level C channels API to the implementation. The low-level
C API was inspired by Libtask[1] and Plan9/Libthread[2].
[1] Libtask: a Coroutine Library for C and Unix. <a href="https://swtch.com/libtask" rel="nofollow noreferrer noopener" target="_blank">https://swtch.com/libtask</a>.
[2] <a href="http://9p.io/magic/man2html/2/thread" rel="nofollow noreferrer noopener" target="_blank">http://9p.io/magic/man2html/2/thread</a>.
- Provide high-level C++ channels API that provides type-safety and
automatic channel lifetime management.
Overview of C and C++ APIs are in δ(libgolang.h).
- Expose C++ channels API at Pyx level as Cython/nogil API so that Cython
programs could use channels with ease and without need to care about
lifetime management and low-level details.
Overview of Cython/nogil channels API is in δ(README.rst) and
δ(_golang.pxd).
- Turn Python channels to be tiny wrapper around chan<PyObject>.
Implementation note:
- gevent case needs special care because greenlet, which gevent uses,
swaps coroutine stack from C stack to heap on coroutine park, and
replaces that space on C stack with stack of activated coroutine
copied back from heap. This way if an object on g's stack is accessed
while g is parked it would be memory of another g's stack.
The channels implementation explicitly cares about this issue so that
stack -> * channel send, or * -> stack channel receive work correctly.
It should be noted that greenlet approach, which it inherits from
stackless, is not only a bit tricky, but also comes with overhead
(stack <-> heap copy), and prevents a coroutine to migrate from 1 OS
thread to another OS thread as that would change addresses of on-stack
things for that coroutine.
As the latter property prevents to use multiple CPUs even if the
program / runtime are prepared to work without GIL, it would be more
logical to change gevent/greenlet to use separate stack for each
coroutine. That would remove stack <-> heap copy and the need for
special care in channels implementation for stack - stack sends.
Such approach should be possible to implement with e.g. swapcontext or
similar mechanism, and a proof of concept of such work wrapped into
greenlet-compatible API exists[3]. It would be good if at some point
there would be a chance to explore such approach in Pygolang context.
[3] <a href="https://github.com/python-greenlet/greenlet/issues/113#issuecomment-264529838" rel="nofollow noreferrer noopener" target="_blank">https://github.com/python-greenlet/greenlet/issues/113#issuecomment-264529838</a> and below
Just this patch brings in the following speedup at Python level:
(on i7@2.6GHz)
thread runtime:
name old time/op new time/op delta
go 20.0µs ± 1% 15.6µs ± 1% -21.84% (p=0.000 n=10+10)
chan 9.37µs ± 4% 2.89µs ± 6% -69.12% (p=0.000 n=10+10)
select 20.2µs ± 4% 3.4µs ± 5% -83.20% (p=0.000 n=8+10)
def 58.0ns ± 0% 60.0ns ± 0% +3.45% (p=0.000 n=8+10)
func_def 43.8µs ± 1% 43.9µs ± 1% ~ (p=0.796 n=10+10)
call 62.4ns ± 1% 63.5ns ± 1% +1.76% (p=0.001 n=10+10)
func_call 1.06µs ± 1% 1.05µs ± 1% -0.63% (p=0.002 n=10+10)
try_finally 136ns ± 0% 137ns ± 0% +0.74% (p=0.000 n=9+10)
defer 2.28µs ± 1% 2.33µs ± 1% +2.34% (p=0.000 n=10+10)
workgroup_empty 48.2µs ± 1% 34.1µs ± 2% -29.18% (p=0.000 n=9+10)
workgroup_raise 58.9µs ± 1% 45.5µs ± 1% -22.74% (p=0.000 n=10+10)
gevent runtime:
name old time/op new time/op delta
go 24.7µs ± 1% 15.9µs ± 1% -35.72% (p=0.000 n=9+9)
chan 11.6µs ± 1% 7.3µs ± 1% -36.74% (p=0.000 n=10+10)
select 22.5µs ± 1% 10.4µs ± 1% -53.73% (p=0.000 n=10+10)
def 55.0ns ± 0% 55.0ns ± 0% ~ (all equal)
func_def 43.6µs ± 1% 43.6µs ± 1% ~ (p=0.684 n=10+10)
call 63.0ns ± 0% 64.0ns ± 0% +1.59% (p=0.000 n=10+10)
func_call 1.06µs ± 1% 1.07µs ± 1% +0.45% (p=0.045 n=10+9)
try_finally 135ns ± 0% 137ns ± 0% +1.48% (p=0.000 n=10+10)
defer 2.31µs ± 1% 2.33µs ± 1% +0.89% (p=0.000 n=10+10)
workgroup_empty 70.2µs ± 0% 55.8µs ± 0% -20.63% (p=0.000 n=10+10)
workgroup_raise 90.3µs ± 0% 70.9µs ± 1% -21.51% (p=0.000 n=9+10)
The whole Cython/nogil work - starting from <a href="/kirr/pygolang/-/commit/8fa3c15b25107d9827307749474dc8e3341c6f65" data-original="8fa3c15b" data-link="false" data-link-reference="false" data-project="1158" data-commit="8fa3c15b25107d9827307749474dc8e3341c6f65" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Start using Cython and providing Cython/nogil API" class="gfm gfm-commit has-tooltip">8fa3c15b</a> (Start using Cython
and providing Cython/nogil API) to this patch - brings in the following
speedup at Python level:
(on i7@2.6GHz)
thread runtime:
name old time/op new time/op delta
go 92.9µs ± 1% 15.6µs ± 1% -83.16% (p=0.000 n=10+10)
chan 13.9µs ± 1% 2.9µs ± 6% -79.14% (p=0.000 n=10+10)
select 29.7µs ± 6% 3.4µs ± 5% -88.55% (p=0.000 n=10+10)
def 57.0ns ± 0% 60.0ns ± 0% +5.26% (p=0.000 n=10+10)
func_def 44.0µs ± 1% 43.9µs ± 1% ~ (p=0.055 n=10+10)
call 63.5ns ± 1% 63.5ns ± 1% ~ (p=1.000 n=10+10)
func_call 1.06µs ± 0% 1.05µs ± 1% -1.31% (p=0.000 n=10+10)
try_finally 139ns ± 0% 137ns ± 0% -1.44% (p=0.000 n=10+10)
defer 2.36µs ± 1% 2.33µs ± 1% -1.26% (p=0.000 n=10+10)
workgroup_empty 98.4µs ± 1% 34.1µs ± 2% -65.32% (p=0.000 n=10+10)
workgroup_raise 135µs ± 1% 46µs ± 1% -66.35% (p=0.000 n=10+10)
gevent runtime:
name old time/op new time/op delta
go 68.8µs ± 1% 15.9µs ± 1% -76.91% (p=0.000 n=10+9)
chan 14.8µs ± 1% 7.3µs ± 1% -50.67% (p=0.000 n=10+10)
select 32.0µs ± 0% 10.4µs ± 1% -67.57% (p=0.000 n=10+10)
def 58.0ns ± 0% 55.0ns ± 0% -5.17% (p=0.000 n=10+10)
func_def 43.9µs ± 1% 43.6µs ± 1% -0.53% (p=0.035 n=10+10)
call 63.5ns ± 1% 64.0ns ± 0% +0.79% (p=0.033 n=10+10)
func_call 1.08µs ± 1% 1.07µs ± 1% -1.74% (p=0.000 n=10+9)
try_finally 142ns ± 0% 137ns ± 0% -3.52% (p=0.000 n=10+10)
defer 2.32µs ± 1% 2.33µs ± 1% +0.71% (p=0.005 n=10+10)
workgroup_empty 90.3µs ± 0% 55.8µs ± 0% -38.26% (p=0.000 n=10+10)
workgroup_raise 108µs ± 1% 71µs ± 1% -34.64% (p=0.000 n=10+10)
This patch is the final patch in series to reach the goal of providing
channels that could be used in Cython/nogil code.
Cython/nogil channels work is dedicated to the memory of Вера Павловна Супрун[4].
[4] <a href="https://navytux.spb.ru/memory/%25D0%25A2%25D1%2591%25D1%2582%25D1%258F%2520%25D0%2592%25D0%25B5%25D1%2580%25D0%25B0.pdf#page=3" rel="nofollow noreferrer noopener" target="_blank">https://navytux.spb.ru/memory/%D0%A2%D1%91%D1%82%D1%8F%20%D0%92%D0%B5%D1%80%D0%B0.pdf#page=3</a>https://lab.nexedi.com/nexedi/pygolang/-/commit/9efb6575767ae4db45e11866b16e21a4e625ede0Hook in list.h from Linux2019-08-29T14:08:15+03:00Kirill Smelkovkirr@nexedi.com
Copy linux/list.h from wendelin.core which copied it from
util-linux.git which took it from linux.git at LGPL license state.
Here are corresponding links for wendelin.core and util-linux:
<a href="https://lab.nexedi.com/nexedi/wendelin.core/commit/a0f940ad16cc" data-original="https://lab.nexedi.com/nexedi/wendelin.core/commit/a0f940ad16cc" data-link="false" data-link-reference="true" data-project="21" data-commit="a0f940ad16ccadd036ff1c9080fd9dcbd84c3c6d" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Hook in list.h from Linux" class="gfm gfm-commit has-tooltip">wendelin.core@a0f940ad</a>
<a href="https://git.kernel.org/cgit/utils/util-linux/util-linux.git/tree/include/list.h?id=v2.25-165-g9138d6f" rel="nofollow noreferrer noopener" target="_blank">https://git.kernel.org/cgit/utils/util-linux/util-linux.git/tree/include/list.h?id=v2.25-165-g9138d6f</a>
Linked lists will be used in channels implementation in the next patch.https://lab.nexedi.com/nexedi/pygolang/-/commit/69db91bf7e2fcb300acf2bf7544c49ccda463054libgolang: Add internal semaphores2019-08-29T14:08:15+03:00Kirill Smelkovkirr@nexedi.com
- Add semaphore alloc/free/acquire/release functionality to libgolang runtime;
- Implement semaphores for thread and gevent runtimes.
* Thread runtime uses PyThread_acquire_lock/PyThread_release_lock +
PyThread_acquire_lock/PyThread_release_lock, which, if used
carefully, do not depend on GIL and on e.g. POSIX are tiny wrappers
around sem_init(process-private) + sem_post/sem_wait(*).
* Gevent runtime uses geven't Semaphore in Pyx mode.
- Add Sema and Mutex classes that use semaphores provided by a runtime
in a RAII style.
- Add with_lock(mu) that mimics `with mu` in Python.
Sema and Mutex will be used in channels implementation in the followup
patch.
(*) during late testing a bug was found in CPython2 and PyPy semaphore
implementations on Darwin (technically speaking on POSIX with
_POSIX_SEMAPHORES undefined). Quoting the patch:
FIXME On Darwin, even though this is considered as POSIX, Python uses
mutex+condition variable to implement its lock, and, as of 20190828, Py2.7
implementation, even though similar issue was fixed for Py3 in 2012, contains
synchronization bug: the condition is signalled after mutex unlock while the
correct protocol is to signal condition from under mutex:
<a href="https://github.com/python/cpython/blob/v2.7.16-127-g0229b56d8c0/Python/thread_pthread.h#L486-L506" rel="nofollow noreferrer noopener" target="_blank">https://github.com/python/cpython/blob/v2.7.16-127-g0229b56d8c0/Python/thread_pthread.h#L486-L506</a>
<a href="https://github.com/python/cpython/commit/187aa545165d" rel="nofollow noreferrer noopener" target="_blank">https://github.com/python/cpython/commit/187aa545165d</a> (py3 fix)
PyPy has the same bug for both pypy2 and pypy3:
<a href="https://bitbucket.org/pypy/pypy/src/578667b3fef9/rpython/translator/c/src/thread_pthread.c#lines-443:465" rel="nofollow noreferrer noopener" target="_blank">https://bitbucket.org/pypy/pypy/src/578667b3fef9/rpython/translator/c/src/thread_pthread.c#lines-443:465</a>
<a href="https://bitbucket.org/pypy/pypy/src/5b42890d48c3/rpython/translator/c/src/thread_pthread.c#lines-443:465" rel="nofollow noreferrer noopener" target="_blank">https://bitbucket.org/pypy/pypy/src/5b42890d48c3/rpython/translator/c/src/thread_pthread.c#lines-443:465</a>
This way when Pygolang is used with buggy Python/darwin, the bug leads to
frequently appearing deadlocks, while e.g. CPython3/darwin works ok.
-> TODO maintain our own semaphore code.
So eventually we'll have push down and maintain our own semaphores,
at least for platforms we care, not to be beaten by CPython runtime bugs.https://lab.nexedi.com/nexedi/pygolang/-/commit/e4dddf15a8a982aca713ce545fac7139c0b9cdefgolang.pyx: pyselect: Small cosmetics2019-08-29T14:08:15+03:00Kirill Smelkovkirr@nexedi.com
Move `ch = py{send,recv}.__self__` to right after case __class__ check.
For now both versions - old and new - can work, but when we'll move
channel implementations to C the new version will be required to access
pychan C-level attribute earlier than where ch is currently initialized.https://lab.nexedi.com/nexedi/pygolang/-/commit/2fc715669d71dcca7ea8b3e0d3970c725b8ba9edgolang.pyx: pyselect: Don't accept tuple subclasses; more clear panic on inva...2019-08-29T14:08:15+03:00Kirill Smelkovkirr@nexedi.comhttps://lab.nexedi.com/nexedi/pygolang/-/commit/d5e74947df3754b3792557ac2a521b8d35f7e082golang.pyx: pyselect: * -> py* in logic which analyzes cases2019-08-29T14:08:06+03:00Kirill Smelkovkirr@nexedi.com
For clarity to denote that things work at Python level:
- casev -> pycasev
- case -> pycase
- recv -> pyrecv
- send -> pysend
Channel object is still denoted as `ch` to reduce noise for when chan IO
code will be move into libgolang. `ch` will be renamed to `pych` after
that.https://lab.nexedi.com/nexedi/pygolang/-/commit/b9333e002a219e3e0b68fe7eb1094e99909a4834golang: tests: Rework verifying blockforever2019-08-29T14:05:42+03:00Kirill Smelkovkirr@nexedi.com
- put the logic to test-tweak what happens inside _blockforever into
context manager pypanicWhenBlocked;
- place this manager in pyx code, where it can later be changed to tweak
_blockforever at C level.https://lab.nexedi.com/nexedi/pygolang/-/commit/4166dc6556d25033f7fe44cce6021587f78614dagolang_test.pyx: ch -> pych for Py-level pychan objects2019-08-29T14:05:42+03:00Kirill Smelkovkirr@nexedi.com
For clarity to distinguish where an object is Python-level channel, or
(later) a C-level channel.https://lab.nexedi.com/nexedi/pygolang/-/commit/ef076d3a2cb13b24801db804ccbe8226c6166900golang_test.pyx: Switch to cimport pychan2019-08-29T14:05:42+03:00Kirill Smelkovkirr@nexedi.com
Change `from golang import chan` to `from golang cimport pychan`; add
type annotations where pychan is used. Using pychan at C level will be
needed when test code will need to access C-level pychan attributes.https://lab.nexedi.com/nexedi/pygolang/-/commit/1bcb8297e7121dfea596e87c83897470ad5db4f9golang.pyx: Switch pychan from `class` to `cdef class`2019-08-29T14:05:42+03:00Kirill Smelkovkirr@nexedi.com
We will need to add C-level attributes to pychan and this requires it to
become cdef class. The class is exported because at least
_golang_test.pyx will also need to have access to those attributes.
If we just do `class pychan` -> `cdef class pychan` e.g. the following
starts to break:
1.venv/local/lib/python2.7/site-packages/py/_path/local.py:701: in pyimport
__import__(modname)
golang/__init__.py:174: in <module>
from ._golang import \
golang/_golang.pyx:455: in init golang._golang
_pychan_send = _pychan_send.__func__
E AttributeError: 'method_descriptor' object has no attribute '__func__'
and
golang/_golang.pyx:513: in golang._golang.pyselect
if im_class(recv) is not pychan:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
f = <built-in method recv of golang._golang.pychan object at 0x7f7055e57cc8>
def im_class(f):
> return f.im_class
E AttributeError: 'builtin_function_or_method' object has no attribute 'im_class'
golang/_pycompat.py:28: AttributeError
This is probably because for `cdef class` methods Cython does not
emulate full method bindings the same way as Python does. Anyway we can
check which method is passed to pyselect by chanop.__name__ or by
inspecting PyCFunction directly. And not having method binding wrapper
should only remove a bit of overhead.
So we are ok with reworking send/recv chanop detection, and since
this way im_class provided by golang._pycompat becomes unused, it is
also removed.
The timings are probably within noise:
(on i7@2.6GHz)
thread runtime:
name old time/op new time/op delta
go 21.7µs ± 1% 20.0µs ± 1% -7.60% (p=0.000 n=10+10)
chan 9.91µs ± 4% 9.37µs ± 4% -5.39% (p=0.000 n=10+10)
select 19.2µs ± 4% 20.2µs ± 4% +5.62% (p=0.001 n=9+8)
def 58.0ns ± 0% 58.0ns ± 0% ~ (all equal)
func_def 44.4µs ± 0% 43.8µs ± 1% -1.22% (p=0.000 n=10+10)
call 63.0ns ± 0% 62.4ns ± 1% -0.95% (p=0.011 n=10+10)
func_call 1.05µs ± 1% 1.06µs ± 1% ~ (p=0.059 n=10+10)
try_finally 135ns ± 0% 136ns ± 0% +0.74% (p=0.000 n=10+9)
defer 2.36µs ± 1% 2.28µs ± 1% -3.59% (p=0.000 n=10+10)
workgroup_empty 49.0µs ± 1% 48.2µs ± 1% -1.63% (p=0.000 n=10+9)
workgroup_raise 62.6µs ± 1% 58.9µs ± 1% -5.96% (p=0.000 n=10+10)
gevent runtime:
name old time/op new time/op delta
go 21.7µs ± 1% 20.5µs ± 1% -5.33% (p=0.000 n=10+9)
chan 9.91µs ± 4% 9.72µs ± 5% ~ (p=0.190 n=10+10)
select 19.2µs ± 4% 19.5µs ±14% ~ (p=0.968 n=9+10)
def 58.0ns ± 0% 58.0ns ± 0% ~ (all equal)
func_def 44.4µs ± 0% 45.4µs ± 1% +2.23% (p=0.000 n=10+10)
call 63.0ns ± 0% 64.0ns ± 0% +1.59% (p=0.000 n=10+10)
func_call 1.05µs ± 1% 1.06µs ± 0% +0.65% (p=0.002 n=10+10)
try_finally 135ns ± 0% 137ns ± 0% +1.48% (p=0.000 n=10+10)
defer 2.36µs ± 1% 2.38µs ± 1% +0.72% (p=0.006 n=10+10)
workgroup_empty 49.0µs ± 1% 48.2µs ± 1% -1.65% (p=0.000 n=10+10)
workgroup_raise 62.6µs ± 1% 60.3µs ± 1% -3.69% (p=0.000 n=10+10)https://lab.nexedi.com/nexedi/pygolang/-/commit/a0714b8ead529e59c4746b234a5f3fdc01543d96golang_test.pyx: Rename moved channel utilities * -> py*2019-08-29T14:05:30+03:00Kirill Smelkovkirr@nexedi.com
For consistency to denote that this functions work at Python level:
- len_sendq -> pylen_sendq
- len_recvq -> pylen_recvq
- waitBlocked -> pywaitBlockedhttps://lab.nexedi.com/nexedi/pygolang/-/commit/a508be9a323104a8eb53d2b22104fe34138f77e0golang: tests: Move channel test utilities from golang_test.py -> _golang_tes...2019-08-29T14:05:00+03:00Kirill Smelkovkirr@nexedi.com
Plain test code movement with s/panic/pypanic/ as in golang.pyx panic is
already there and means semantically different thing. We need this code to
live in pyx world, because when channels implementation will be moved to
C, this utilities will need to be adjusted in a way that is not possible
to do from Python.https://lab.nexedi.com/nexedi/pygolang/-/commit/790189e3e252eb0e663a33a7cd59157a484991a2golang.pyx: pychan: return cosmetics2019-08-29T14:01:36+03:00Kirill Smelkovkirr@nexedi.com
Denote what a method returns via `# -> ...` suffix.https://lab.nexedi.com/nexedi/pygolang/-/commit/7e55394d7531a107c1cf4486182159e284d48befgolang.pyx: pychan: self -> ch2019-08-29T14:01:36+03:00Kirill Smelkovkirr@nexedi.com
Use `ch` instead of `self` for pychan methods. This aligns with how
(py)chan objects are currently denoted in (py)select and will reduce the
difference when chan code is moved into libgolang. We are sticking to Go
convention here.
After the code with channels implementation is moved into libgolang, the
rest bits in golang.pyx will be changed to refer to pychan objects as
pych for clarity.https://lab.nexedi.com/nexedi/pygolang/-/commit/311df9f18bc7ca4be26591552fd6769b57ac1efcgolang.pyx: Rename moved channel bits * -> py*2019-08-29T14:01:36+03:00Kirill Smelkovkirr@nexedi.com
To denote that this function/classes work at Python level:
- chan -> pychan
- select -> pyselect
- default -> pydefault
- nilchan -> pynilchanhttps://lab.nexedi.com/nexedi/pygolang/-/commit/83259a1b7c5d73eb7ea8029b654e3322b7ba44aegolang: Move channels implementation from golang.py to golang.pyx2019-08-29T14:01:36+03:00Kirill Smelkovkirr@nexedi.com
Plain code movement with just s/panic/pypanic/ as in golang.pyx panic is
aleady there and means semantically different thing. Moved code, even
though it lives in golang.pyx, is still Python code and requires Python
runtime and GIL. We'll be splitting channels implementation into nogil
world in the following patches.
Just plain movement to Cython brings the following speedup:
(on i7@2.6GHz)
thread runtime:
name old time/op new time/op delta
go 26.6µs ± 1% 21.7µs ± 1% -18.54% (p=0.000 n=10+10)
chan 13.7µs ± 1% 9.9µs ± 4% -27.80% (p=0.000 n=10+10)
select 29.3µs ± 2% 19.2µs ± 4% -34.65% (p=0.000 n=9+9)
def 55.0ns ± 0% 58.0ns ± 0% +5.45% (p=0.000 n=10+10)
func_def 44.0µs ± 1% 44.4µs ± 0% +0.72% (p=0.002 n=10+10)
call 64.0ns ± 0% 63.0ns ± 0% -1.56% (p=0.002 n=8+10)
func_call 1.09µs ± 1% 1.05µs ± 1% -2.96% (p=0.000 n=10+10)
try_finally 139ns ± 2% 135ns ± 0% -2.60% (p=0.000 n=10+10)
defer 2.36µs ± 1% 2.36µs ± 1% ~ (p=0.617 n=10+10)
workgroup_empty 58.1µs ± 1% 49.0µs ± 1% -15.61% (p=0.000 n=10+10)
workgroup_raise 72.7µs ± 1% 62.6µs ± 1% -13.88% (p=0.000 n=10+10)
gevent runtime:
name old time/op new time/op delta
go 28.6µs ± 0% 25.4µs ± 0% -11.20% (p=0.000 n=8+9)
chan 15.8µs ± 1% 12.2µs ± 1% -22.62% (p=0.000 n=10+10)
select 33.1µs ± 1% 23.3µs ± 2% -29.60% (p=0.000 n=10+10)
def 55.0ns ± 0% 56.0ns ± 0% +1.82% (p=0.000 n=10+10)
func_def 44.4µs ± 2% 43.0µs ± 1% -3.06% (p=0.000 n=10+9)
call 64.0ns ± 2% 69.0ns ± 0% +7.81% (p=0.000 n=10+10)
func_call 1.06µs ± 0% 1.06µs ± 1% ~ (p=0.913 n=8+9)
try_finally 136ns ± 0% 139ns ± 0% +2.21% (p=0.000 n=9+10)
defer 2.29µs ± 1% 2.38µs ± 2% +3.58% (p=0.000 n=10+10)
workgroup_empty 73.8µs ± 1% 70.5µs ± 1% -4.48% (p=0.000 n=10+10)
workgroup_raise 94.1µs ± 0% 90.6µs ± 0% -3.69% (p=0.000 n=10+10)https://lab.nexedi.com/nexedi/pygolang/-/commit/f971a2a8e7cc22bb0572104920cb06b5e11ceef5pyx api: Provide go2019-08-29T14:01:36+03:00Kirill Smelkovkirr@nexedi.com
- Add go functionality to libgolang runtime;
- Implement go for thread and gevent runtimes.
* Thread runtime uses PyThread_start_new_thread which, if used
carefully, does not depend on Python GIL and on e.g. POSIX reduces to
tiny wrapper around pthread_create.
* Gevent runtime uses gevent's Greenlet in Pyx mode. This turns
gevent to be a build-time dependency.
- Provide low-level _taskgo in C client API;
- Provide type-safe C++-level go wrapper over _taskgo;
- Switch golang.go from py implementation into Pyx wrapper over
Pyx/nogil API.
This is the first patch that adds Pyx/C++/C-level unit tests and hooks
them into golang_test.py .
This patch brings the following speedup to Python-level interface:
(on i7@2.6GHz)
thread runtime:
name old time/op new time/op delta
go 93.0µs ± 1% 26.6µs ± 1% -71.41% (p=0.000 n=10+10)
chan 13.6µs ± 2% 13.7µs ± 1% ~ (p=0.280 n=10+10)
select 29.9µs ± 4% 29.3µs ± 2% -1.89% (p=0.017 n=10+9)
def 61.0ns ± 0% 55.0ns ± 0% -9.84% (p=0.000 n=10+10)
func_def 43.8µs ± 1% 44.0µs ± 1% +0.66% (p=0.006 n=10+10)
call 62.5ns ± 1% 64.0ns ± 0% +2.40% (p=0.000 n=10+8)
func_call 1.06µs ± 1% 1.09µs ± 1% +2.72% (p=0.000 n=10+10)
try_finally 137ns ± 0% 139ns ± 2% +1.17% (p=0.033 n=10+10)
defer 2.34µs ± 1% 2.36µs ± 1% +0.84% (p=0.015 n=10+10)
workgroup_empty 96.1µs ± 1% 58.1µs ± 1% -39.55% (p=0.000 n=10+10)
workgroup_raise 135µs ± 1% 73µs ± 1% -45.97% (p=0.000 n=10+10)
gevent runtime:
name old time/op new time/op delta
go 68.8µs ± 1% 28.6µs ± 0% -58.47% (p=0.000 n=10+8)
chan 14.8µs ± 1% 15.8µs ± 1% +6.19% (p=0.000 n=10+10)
select 32.0µs ± 0% 33.1µs ± 1% +3.25% (p=0.000 n=10+10)
def 58.0ns ± 0% 55.0ns ± 0% -5.17% (p=0.000 n=10+10)
func_def 43.9µs ± 1% 44.4µs ± 2% +1.20% (p=0.007 n=10+10)
call 63.5ns ± 1% 64.0ns ± 2% ~ (p=0.307 n=10+10)
func_call 1.08µs ± 1% 1.06µs ± 0% -2.55% (p=0.000 n=10+8)
try_finally 142ns ± 0% 136ns ± 0% -4.23% (p=0.000 n=10+9)
defer 2.32µs ± 1% 2.29µs ± 1% -0.96% (p=0.000 n=10+10)
workgroup_empty 90.3µs ± 0% 73.8µs ± 1% -18.29% (p=0.000 n=10+10)
workgroup_raise 108µs ± 1% 94µs ± 0% -13.29% (p=0.000 n=10+10)
(small changes are probably within noise; "go" and "workgroup_*" should be
representative)https://lab.nexedi.com/nexedi/pygolang/-/commit/7ae8c4f3c127325490534fec97985cbb8e70e3a5pyx.build: Allow to combine C and C++ sources in one extension2019-08-29T14:01:32+03:00Kirill Smelkovkirr@nexedi.com
The next patch will add `go` to Pyx API and correspondingly C, C++ and Pyx
level tests for it that will go into _golang_test.pyx extension. For
C-level test we'll use C source file - to verify that libgolang.h could
be used by C projects with C compiler, while for C++ and Pyx-level tests
the sources will be in C++. Thus _golang_test.so will be build from both
C and C++ sources.
This creates a problem: distutils / setuptools use the _same_ compiler
to compile both C and C++ sources and only use C++ compiler at link
stage. Thus, as it is not possible to tune compiler that is used only
for C++ sources, and also as it is not possible to provide per-source
flags, when compiling C-level test, the compiler will be invoked with
`-std=c++11` option that we inject. Gcc tolerates that and only prints a
warning, but Clang considers that an error and gives:
error: invalid argument '-std=c++11' not allowed with 'C'`
A proper fix would be to change the build system from distutils to
something more flexible that uses C++ compiler for C++ sources and C
compiler for C sources and allows to tune per-unit flags. However it is
not a small step at this stage.
-> Use workaround and tweak options that are used when compiling sources
depending on whether it is C or C++.https://lab.nexedi.com/nexedi/pygolang/-/commit/ce8152a23262af76877a3f3e1ce3b9c6fa6d9d34pyx api: Provide sleep2019-08-29T12:45:02+03:00Kirill Smelkovkirr@nexedi.com
- Add sleep functionality to libgolang runtime;
- Implement sleep for thread and gevent runtimes. Thread runtime
implements sleep independently of GIL, but only for POSIX for now;
- Switch golang.time py module into using golang.time pyx module.
As we are adding sleep, related functionality to query system about
"what is current time?" is also added.https://lab.nexedi.com/nexedi/pygolang/-/commit/ad00be703a36572bb8ba13963ff59356e216f8f3libgolang: Introduce runtimes2019-08-29T12:45:02+03:00Kirill Smelkovkirr@nexedi.com
Libgolang will be generic library and it will adapt itself to particular
execution environment by way of runtime plugin. This commit introduces
stubs for two such runtimes - "thread" and "gevent" - and initializes
libgolang with particular runtime depending on whether golang is
imported with gevent preactivated or not.
The runtimes themselves are empty at this step. We'll be adding runtime
functionality in the following patches.https://lab.nexedi.com/nexedi/pygolang/-/commit/8fa3c15b25107d9827307749474dc8e3341c6f65Start using Cython and providing Cython/nogil API2019-08-29T12:44:45+03:00Kirill Smelkovkirr@nexedi.com
Start transforming Pygolang from being pure Python project into project
that uses both Python and Cython and provides both Python and Cython API
to its users.
For Cython API there is a goal for it to be usable independently of
Python GIL - i.e. independently of Python runtime and limitations
imposed by Python's global interpreter lock. The motivation for this
work is wendelin.core v2, which, due to its design, would deadlock if it
tries to take the GIL in its pinner thread.
This commit brings bare minimum establishing libraries and build system.
It:
- Introduces C/C++ library libgolang. Libgolang will be serving most of
nogil functionality and Pyx API will be a small wrapper over it. At
this step Libgolang comes only with C-level panic;
- Introduces Pyx package golang that is aliased to Pyx package
golang._golang . Users can do `from golang cimport ...` similarly to how
they do py-level `from golang import ...`. At this step golang.pyx
only wraps panic and provides way to transform C-level panic into Python
exception;
- Introduces Py package golang.pyx.build . This package, similarly to
golang.org/pkg/go/build, serves as the system to build Pyx packages -
both external packages that use Pygolang in pyx mode, and to build
Pygolang itself. The build, internally, is served by setuptools_dso
and cython. An example how to use golang.pyx.build from external
project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
tests to verify golang.pyx.build functionality;
- Introduces Pygolang build dependency to cython and setuptools_dso.https://lab.nexedi.com/nexedi/pygolang/-/commit/fa9dd0d0ce3a0f4b15403065aa4dee7bd4944f10*: Docstring cosmetics (slight)2019-08-26T12:16:48+03:00Kirill Smelkovkirr@nexedi.comhttps://lab.nexedi.com/nexedi/pygolang/-/commit/21525a8d29777be96dd4585e25dca9eb9fd8707bmanifest: Simplify a bit2019-08-26T12:16:48+03:00Kirill Smelkovkirr@nexedi.com
We can include *.py from top-level directories, not only under
X/testdata/, X/testprog/ ...https://lab.nexedi.com/nexedi/pygolang/-/commit/7ef58207211f12099d68db3cd33a3b19d6a83fffgolang_test: Rename test_go -> test_go_leaked2019-08-26T12:16:48+03:00Kirill Smelkovkirr@nexedi.com
This test is veriyfing only how leaked goroutines behave, not go in
general.https://lab.nexedi.com/nexedi/pygolang/-/commit/62bdb8065a0a73b3318aeb027480ee62ae5e7441golang_test: Switch test_go to pyrun2019-08-26T12:16:48+03:00Kirill Smelkovkirr@nexedi.com
test_go was doing the same what pyrun is doing + adjusting PYTHONPATH a
bit. Merge PYTHONPATH adjustment into pyrun and use pyrun uniformly.https://lab.nexedi.com/nexedi/pygolang/-/commit/88eb8fe08a94bb77045a6f9a818c9edfda8d579egolang_test: Split pyrun into -> pyrun & pyout2019-08-26T12:16:48+03:00Kirill Smelkovkirr@nexedi.com
pyrun runs command without piping output.
pyout runs command and returns its output.https://lab.nexedi.com/nexedi/pygolang/-/commit/f812faa2dd81e8896d3523597c58558618bed5fatests: Factor out pyrun from gpython_test into golang_test2019-08-26T12:16:48+03:00Kirill Smelkovkirr@nexedi.com
We'll need to use this utility in other places; golang_test is the
central place for testing utilities.https://lab.nexedi.com/nexedi/pygolang/-/commit/75169174be71e589322c50d46ab11fd8505df7e3gpython_test: Kill unused imports2019-08-26T12:16:48+03:00Kirill Smelkovkirr@nexedi.com
gpython/gpython_test.py:23: 'subprocess' imported but unused
gpython/gpython_test.py:24: 'six.PY3' imported but unusedhttps://lab.nexedi.com/nexedi/pygolang/-/commit/89a1061a74723d0e3ab1e136860643523cd4d3f1setup: Provide golang.X requirements in pygolang[X]2019-08-26T12:16:03+03:00Kirill Smelkovkirr@nexedi.com
... and everything that whole pygolang depends on in pygolang[all].
The reason we do this: currently golang.perf.benchlib needs NumPy, but
it would be not ok to depend whole pygolang on numpy. Similarly pybench
needs pytest, but it would be not ok to require pytest unconditionally.
Thus allow users to specify what they need from pygolang via extras,
like pygolang[x.perf.benchlib] or pygolang[cmd.pybench], with
pygolang[all] serving as merge point for all subpackages dependencies.
TODO in the future we'll, hopefully, also provide accumulation points,
like pygolang[x.perf] which contains everything pygolang[x.perf.*]
depends on.
Note: core functionality and gpython are provided always unconditionally.https://lab.nexedi.com/nexedi/pygolang/-/commit/02f6991f8ec718301cbec1e86d780efdc1d314aagolang: Add test for blocked select(send|recv) vs close2019-08-23T19:31:10+03:00Kirill Smelkovkirr@nexedi.com
This test passes, but the functionality was not tested before. In
particular all tests were passing even with the following hand-edits:
--- a/golang/__init__.py
+++ b/golang/__init__.py
@@ -627,7 +627,7 @@ def selected():
sel = g.which
if isinstance(sel, _SendWaiting):
if not sel.ok:
- panic("send on closed channel")
+ panic("send on closed channel ZZZ")
return sel.sel_n, None
if isinstance(sel, _RecvWaiting):
with added tests the bug is caught and reported:
def test_select():
N = 1000 # times to do repeated select/chan or select/select interactions
# sync: close vs select(send)
ch = chan()
def _():
waitBlocked(ch.send)
ch.close()
go(_)
> with panics("send on closed channel"): select((ch.send, 0))
golang_test.py:353:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <golang.golang_test.panics instance at 0x7fc1da66e5f0>, exc_type = <class 'golang._PanicError'>, exc_val = _PanicError('send on closed channel ZZZ',)
exc_tb = <traceback object at 0x7fc1dabc33b0>
def __exit__(self, exc_type, exc_val, exc_tb):
ok = self.raises.__exit__(exc_type, exc_val, exc_tb)
if not ok:
return ok
# _PanicError raised - let's check panic argument
> assert self.exc_info.value.args == (self.arg,)
E AssertionError: assert ('send on clo...channel ZZZ',) == ('send on closed channel',)
E At index 0 diff: 'send on closed channel ZZZ' != 'send on closed channel'
E Use -v to get the full diff
golang_test.py:1032: AssertionErrorhttps://lab.nexedi.com/nexedi/pygolang/-/commit/c6bb9eb338c3b48106fff4b58a71808316acc2bfgolang: Fix race in chan._tryrecv2019-08-23T19:31:10+03:00Kirill Smelkovkirr@nexedi.com
For buffered channel _tryrecv, on success, was unlocking ch._mu too
early - before accessing ch._dataq with ch._dataq.append().
Without the fix, newly added test breaks as e.g.
golang/golang_test.py::test_chan_buf_recv_vs_tryrecv_race Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 317, in _
assert (_, _rx) == (1, None), ('i%d' % i)
AssertionError: i30
assert (0, None) == (1, None)
At index 0 diff: 0 != 1
Full diff:
- (0, None)
? ^
+ (1, None)
? ^https://lab.nexedi.com/nexedi/pygolang/-/commit/eb8a1fef30040785fdcd18f09c8eb4928c967d11golang: Fix race in chan._trysend2019-08-23T19:31:10+03:00Kirill Smelkovkirr@nexedi.com
For buffered channel _trysend, on success, was unlocking ch._mu too
early - before accessing ch._dataq with ch._dataq.popleft().
Without the fix, newly added test breaks as e.g.
golang/golang_test.py::test_chan_buf_send_vs_tryrecv_race Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 256, in _
assert (_, _rx) == (1, None)
AssertionError: assert (0, 209) == (1, None)
At index 0 diff: 0 != 1
Full diff:
- (0, 209)
+ (1, None)
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 243, in _
ch.send(i)
File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 340, in send
ok = self._trysend(obj)
File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 417, in _trysend
rx = self._dataq.popleft()
IndexError: pop from an empty dequehttps://lab.nexedi.com/nexedi/pygolang/-/commit/cb5bfdd23d13f568eaaf494bf53fe5e87e1def1bgolang: Test that buffered channel releases objects from buffer on chan GC2019-08-23T19:31:10+03:00Kirill Smelkovkirr@nexedi.com
Currently it behaves correctly, but this aspect will need to be
explicitly cared about when channel implementation moves to C. Add test
to make sure it won't regress.https://lab.nexedi.com/nexedi/pygolang/-/commit/c5810987cb3392b6d98b2d29b71025ef3cd6abf4golang: Run all select tests "more thoroughly"2019-08-23T19:31:10+03:00Kirill Smelkovkirr@nexedi.com
Starting from <a href="/kirr/pygolang/-/commit/b51b8d5d64f020f2b0892eb0f9d2c3a4968c39e3" data-original="b51b8d5d" data-link="false" data-link-reference="false" data-project="1158" data-commit="b51b8d5d64f020f2b0892eb0f9d2c3a4968c39e3" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="select: Run tests more thoroughly" class="gfm gfm-commit has-tooltip">b51b8d5d</a> (select: Run tests more thoroughly) we are
running select subtests in repeated mode with N=1000. However not all
select subtests were run in repeated mode - for example e.g.
"non-blocking try send: not ok" was being run only once.
-> Rework all select subtests to be run in repeated mode to increase the
probability of catching bugs.https://lab.nexedi.com/nexedi/pygolang/-/commit/fa66741271751f075493b129329d7ba42619eb83golang: tests: Use panic when testing "blocks forever"2019-08-23T19:31:08+03:00Kirill Smelkovkirr@nexedi.com
Channels implementation will soon be moved to C and will become
independent from Python runtime. This way while pygolang will still be
providing panic, hooking in and raising arbitrary Python-level exception
will become problematic. -> Rework "blocks forever" tests to rely on
just panic to prepare for that.https://lab.nexedi.com/nexedi/pygolang/-/commit/352628b5310badeeec521d523fa39339f80238f7golang: tests: Factor out retrieving len(ch._recvq) and len(ch._sendq)2019-08-23T19:29:55+03:00Kirill Smelkovkirr@nexedi.com
Chan implementation is going to be moved into C, and this way direct
access to chan internals won't be possible. C implementation will export
dedicated functions for chan tests which will be used.
Prepare for that and factor out retrieving len of chan recv/send queues
into separate functions, which will be adapted at the time of C port.https://lab.nexedi.com/nexedi/pygolang/-/commit/d98e42e3283519dbbc89e8c7f7f0dc1e5e29db61golang: Don't verify ._recvq and ._sendq of nil channel2019-08-23T19:29:55+03:00Kirill Smelkovkirr@nexedi.com
Nil channel will soon be changed to be represented by underlying NULL C
pointer and there won't be channel object for nil and correspondingly
recv/send queues won't be there. -> Prepare for that and don't check for
recvq/sendq on nil channel.https://lab.nexedi.com/nexedi/pygolang/-/commit/049ba6ff77afe27fc9eb9fc55d58074a2c222943golang: Remove outdated TODO2019-08-23T19:29:55+03:00Kirill Smelkovkirr@nexedi.com
We provide gpython since <a href="/nexedi/pygolang/-/commit/32a21d5b36579ac2488edfc675567eeb364a3e13" data-original="32a21d5b" data-link="false" data-link-reference="false" data-project="1156" data-commit="32a21d5b36579ac2488edfc675567eeb364a3e13" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="gpython: Python interpreter with support for lightweight threads" class="gfm gfm-commit has-tooltip">32a21d5b</a> (gpython: Python interpreter with
support for lightweight threads), and golang module, since the
beginning, automatically uses gevent if it was installed via monkey
patching.https://lab.nexedi.com/nexedi/pygolang/-/commit/7f2362dd1d7f9363309ad89b8574318b0a7e71detime: Test for now2019-08-23T19:29:55+03:00Kirill Smelkovkirr@nexedi.com
This currently pass trivially (since time.now = stdtime.now), but will
be useful once we get time.now implementation into our hands and move it into Cython.