The new home is https://lab.nexedi.com/nexedi/pygolang

[1] https://stack.nexedi.com

A build fix wrt gevent-1.5 + benchmarks for nogil go and channels.

This is top-level documentation for error chaining that was promised
and marked as TODO in

- fd95c88a (golang, errors, fmt: Error chaining (C++/Pyx))
- 17798442 (golang: Expose error at Py level)
- 78d0c76f (golang: Teach pyerror to be a base class)
- 337de0d7 (golang, errors, fmt: Error chaining (Python))
- 03f88c0b (errors: Take .__cause__ into account)

This provides top-level documentation for b and u that was promised and
marked as TODO in bcb95cd5 (golang: Provide b, u for strings).

Provide sync.RWMutex that can be useful for cases when there are
multiple simultaneous readers and more seldom writer(s).

This implements readers-writer mutex with preference for writers
similarly to Go version.

For base functionality we have overview in the readme itself, but for
packages we have only their listing with brief overview and no
documentation for in-package functionality.

Let's have at least links to .h/.pxd/.py where package functionality is
documented.

This will be needed in Python-wrappers for C-level code which exposes
channels. For example in time package whose implementation will soon be
moved to nogil world fully.

3121b290 (golang: Teach pychan to work with channels of C types, not
only PyObjects) added ability to convert pychan to chan[T], but
described that ability only briefly in README. Provide more details in
that description before we add description and implementation for
reflexive pychan <- chan[T] wrapping.

Python3 chains exceptions, so that e.g. if exc1 is raised and, while it
was not handled, another exc2 is raised, exc2 will be linked to exc1 via
exc2.__context__ attribute and exc1 will be included into exc2 traceback
printout. However many projects still use Python2 and there is no
similar chaining functionality there. This way exc1 is completely lost.

Since defer code is in our hands, we can teach it to implement exception
chaining even on Python2 by carefully analyzing what happens in
_GoFrame.__exit__().

Implementing chaining itself is relatively easy, but is only part of the
story. Even if an exception is chained with its cause, but exception
dump does not show the cause, the chaining will be practically useless.
With this in mind this patches settles not only on implementing chaining
itself, but on also giving a promise that chained cause exceptions will
be included into traceback dumps as well.

To realize this promise we adjust all exception dumping funcitons in
traceback module and carefully install adjusted
traceback.print_exception() into sys.excepthook. This amends python
interactive sessions and programs run by python interpreter to include
causes in exception dumps. "Careful" here means that we don't change
sys.excepthook if on golang module load we see that sys.excepthook was already
changed by some other module - e.g. due to IPython session running
because IPython installs its own sys.excepthook. In such cases we don't
install our sys.excepthook, but we also provide integration patches that
add exception chaining support for traceback dump functionality in
popular third-party software. The patches (currently for IPython and
Pytest) are activated automatically, but only when/if corresponding
software is imported and actually used. This should give practically
good implementation of the promise - a user can now rely on seeing
exception cause in traceback dump whatever way python programs are run.

The implementation takes https://pypi.org/project/pep3134/ experience
into account [1]. peak.utils.imports [2,3] is used to be notified when/if
third-party module is imported.

[1] https://github.com/9seconds/pep3134/
[2] https://pypi.org/project/Importing/
[3] http://peak.telecommunity.com/DevCenter/Importing

This patch originally started as hacky workaround in wendelin.core
because in wcfs tests I was frequently hitting situations, where
exception raised by an assert was hidden by another exception raised in
further generic teardown check. For example wcfs tests check that wcfs
is unmounted after every test run [4] and if that fails it was hiding
problems raised by an assert. As the result I was constantly guessing
and adding code like [5] to find what was actually breaking. At some
point I added hacky workaround for defer to print cause exception not to
loose it [6]. [7] has more context and background discussion on this topic.

[4] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L70
[5] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L853-857
[6] kirr/wendelin.core@c00d94c7
[7] zodbtools!13 (comment 81553)

After this patch, on Python2

    defer(cleanup1)
    defer(cleanup2)
    defer(cleanup3)
    ...

is no longer just a syntatic sugar for

    try:
        try:
            try:
                ...
            finally:
                cleanup3()
        finally:
            cleanup2()
    finally:
        cleanup1()

Introduce notion of data type (dtype similarly to NumPy) into pychan and
teach it to accept for send objects only matching that dtype. Likewise
teach pychan to decode raw bytes received from underlying channel into
Python object correspodningg to pychan dtype. For C dtypes, e.g.
'C.int', 'C.double' etc, contrary to chan of python objects, the
transfer can be done without depending on Python GIL. This way channels
of such C-level dtypes can be used to implement interaction in between
Python and nogil worlds.

In a followup commit we are going to add channel types (e.g. chan of
double, chan of int, etc) and accordingly there will be several nil
channel objects, e.g. nil(dtype=int), nil(dtype=double) etc, which will
be separate python objects. Even without data types, another planned
change is to add directional channels, e.g. a channel instance that can
only send, but not receive and vice versa(*).

This way for the same underlying channel object, there can be several
pychan objects that point to it - even for nil channel - e.g. nilchan
and `niltx = nilchan.txonly()` that creates another pychan object
pointing to the same underlying nil.

Since we want all channels (of the same type) that point to the same
underlying channel to compare as same, we cannot use "is" for comparison
and have to use ==. In other words channels, both at C and Python level,
should be perceived as pointers, of which there can be multiple ones
pointing to the same location, and thus == has to be used to compare
them.

(*) see https://golang.org/ref/spec#Channel_types for details.

Until now libgolang had semaphore and mutex functionality, but
implemented only internally and not exposed to users. Since for its
pinner thread wendelin.core v2 needs not only nogil channels, but also
nogil mutexes, timers and so on, it is now time to start providing that.

We start with mutexes:

- Expose Mutex from insides of libgolang.cpp to public API in
  libgolang.h . We actually expose both Mutex and Sema because
  semaphores are sometimes also useful for low-level synchronization,
  and because it is easier to export Mutex and Sema both at the same time.

- Provide pyx/nogil API to those as sync.Mutex and sync.Sema.

- Provide corresponding python wrappers.

- Add Pyx-level test for semaphore wakeup when wakeup is done not by the
  same thread which did the original acquire. This is the test that was
  promised in 5142460d (libgolang/thread: Add links to upstream
  PyThread_release_lock bug), and it used to corrupt memory and deadlock
  on macOS due to CPython & PyPy runtime bugs:

    https://bugs.python.org/issue38106
      -> https://github.com/python/cpython/pull/16047
    https://bitbucket.org/pypy/pypy/issues/3072

Note about python wrappers: At present, due to
https://github.com/cython/cython/issues/3165, C-level panic is not
properly translated into Py-level exception in (Py)Sema/Mutex constructors.
This should not pose a real problem in practice though.

- Move channels implementation to be done in C++ inside libgolang. The
  code and logic is based on previous Python-level channels
  implementation, but the new code is just C++ and does not depend on
  Python nor GIL at all, and so works without GIL if libgolang
  runtime works without GIL(*).

  (*) for example "thread" runtime works without GIL, while "gevent" runtime
      acquires GIL on every semaphore acquire.

  New channels implementation is located in δ(libgolang.cpp).

- Provide low-level C channels API to the implementation. The low-level
  C API was inspired by Libtask[1] and Plan9/Libthread[2].

  [1] Libtask: a Coroutine Library for C and Unix. https://swtch.com/libtask.
  [2] http://9p.io/magic/man2html/2/thread.

- Provide high-level C++ channels API that provides type-safety and
  automatic channel lifetime management.

  Overview of C and C++ APIs are in δ(libgolang.h).

- Expose C++ channels API at Pyx level as Cython/nogil API so that Cython
  programs could use channels with ease and without need to care about
  lifetime management and low-level details.

  Overview of Cython/nogil channels API is in δ(README.rst) and
  δ(_golang.pxd).

- Turn Python channels to be tiny wrapper around chan<PyObject>.

Implementation note:

- gevent case needs special care because greenlet, which gevent uses,
  swaps coroutine stack from C stack to heap on coroutine park, and
  replaces that space on C stack with stack of activated coroutine
  copied back from heap. This way if an object on g's stack is accessed
  while g is parked it would be memory of another g's stack.

  The channels implementation explicitly cares about this issue so that
  stack -> * channel send, or * -> stack channel receive work correctly.

  It should be noted that greenlet approach, which it inherits from
  stackless, is not only a bit tricky, but also comes with overhead
  (stack <-> heap copy), and prevents a coroutine to migrate from 1 OS
  thread to another OS thread as that would change addresses of on-stack
  things for that coroutine.

  As the latter property prevents to use multiple CPUs even if the
  program / runtime are prepared to work without GIL, it would be more
  logical to change gevent/greenlet to use separate stack for each
  coroutine. That would remove stack <-> heap copy and the need for
  special care in channels implementation for stack - stack sends.
  Such approach should be possible to implement with e.g. swapcontext or
  similar mechanism, and a proof of concept of such work wrapped into
  greenlet-compatible API exists[3]. It would be good if at some point
  there would be a chance to explore such approach in Pygolang context.

  [3] https://github.com/python-greenlet/greenlet/issues/113#issuecomment-264529838 and below

Just this patch brings in the following speedup at Python level:

 (on i7@2.6GHz)

thread runtime:

    name             old time/op  new time/op  delta
    go               20.0µs ± 1%  15.6µs ± 1%  -21.84%  (p=0.000 n=10+10)
    chan             9.37µs ± 4%  2.89µs ± 6%  -69.12%  (p=0.000 n=10+10)
    select           20.2µs ± 4%   3.4µs ± 5%  -83.20%  (p=0.000 n=8+10)
    def              58.0ns ± 0%  60.0ns ± 0%   +3.45%  (p=0.000 n=8+10)
    func_def         43.8µs ± 1%  43.9µs ± 1%     ~     (p=0.796 n=10+10)
    call             62.4ns ± 1%  63.5ns ± 1%   +1.76%  (p=0.001 n=10+10)
    func_call        1.06µs ± 1%  1.05µs ± 1%   -0.63%  (p=0.002 n=10+10)
    try_finally       136ns ± 0%   137ns ± 0%   +0.74%  (p=0.000 n=9+10)
    defer            2.28µs ± 1%  2.33µs ± 1%   +2.34%  (p=0.000 n=10+10)
    workgroup_empty  48.2µs ± 1%  34.1µs ± 2%  -29.18%  (p=0.000 n=9+10)
    workgroup_raise  58.9µs ± 1%  45.5µs ± 1%  -22.74%  (p=0.000 n=10+10)

gevent runtime:

    name             old time/op  new time/op  delta
    go               24.7µs ± 1%  15.9µs ± 1%  -35.72%  (p=0.000 n=9+9)
    chan             11.6µs ± 1%   7.3µs ± 1%  -36.74%  (p=0.000 n=10+10)
    select           22.5µs ± 1%  10.4µs ± 1%  -53.73%  (p=0.000 n=10+10)
    def              55.0ns ± 0%  55.0ns ± 0%     ~     (all equal)
    func_def         43.6µs ± 1%  43.6µs ± 1%     ~     (p=0.684 n=10+10)
    call             63.0ns ± 0%  64.0ns ± 0%   +1.59%  (p=0.000 n=10+10)
    func_call        1.06µs ± 1%  1.07µs ± 1%   +0.45%  (p=0.045 n=10+9)
    try_finally       135ns ± 0%   137ns ± 0%   +1.48%  (p=0.000 n=10+10)
    defer            2.31µs ± 1%  2.33µs ± 1%   +0.89%  (p=0.000 n=10+10)
    workgroup_empty  70.2µs ± 0%  55.8µs ± 0%  -20.63%  (p=0.000 n=10+10)
    workgroup_raise  90.3µs ± 0%  70.9µs ± 1%  -21.51%  (p=0.000 n=9+10)

The whole Cython/nogil work - starting from 8fa3c15b (Start using Cython
and providing Cython/nogil API) to this patch - brings in the following
speedup at Python level:

 (on i7@2.6GHz)

thread runtime:

    name             old time/op  new time/op  delta
    go               92.9µs ± 1%  15.6µs ± 1%  -83.16%  (p=0.000 n=10+10)
    chan             13.9µs ± 1%   2.9µs ± 6%  -79.14%  (p=0.000 n=10+10)
    select           29.7µs ± 6%   3.4µs ± 5%  -88.55%  (p=0.000 n=10+10)
    def              57.0ns ± 0%  60.0ns ± 0%   +5.26%  (p=0.000 n=10+10)
    func_def         44.0µs ± 1%  43.9µs ± 1%     ~     (p=0.055 n=10+10)
    call             63.5ns ± 1%  63.5ns ± 1%     ~     (p=1.000 n=10+10)
    func_call        1.06µs ± 0%  1.05µs ± 1%   -1.31%  (p=0.000 n=10+10)
    try_finally       139ns ± 0%   137ns ± 0%   -1.44%  (p=0.000 n=10+10)
    defer            2.36µs ± 1%  2.33µs ± 1%   -1.26%  (p=0.000 n=10+10)
    workgroup_empty  98.4µs ± 1%  34.1µs ± 2%  -65.32%  (p=0.000 n=10+10)
    workgroup_raise   135µs ± 1%    46µs ± 1%  -66.35%  (p=0.000 n=10+10)

gevent runtime:

    name             old time/op  new time/op  delta
    go               68.8µs ± 1%  15.9µs ± 1%  -76.91%  (p=0.000 n=10+9)
    chan             14.8µs ± 1%   7.3µs ± 1%  -50.67%  (p=0.000 n=10+10)
    select           32.0µs ± 0%  10.4µs ± 1%  -67.57%  (p=0.000 n=10+10)
    def              58.0ns ± 0%  55.0ns ± 0%   -5.17%  (p=0.000 n=10+10)
    func_def         43.9µs ± 1%  43.6µs ± 1%   -0.53%  (p=0.035 n=10+10)
    call             63.5ns ± 1%  64.0ns ± 0%   +0.79%  (p=0.033 n=10+10)
    func_call        1.08µs ± 1%  1.07µs ± 1%   -1.74%  (p=0.000 n=10+9)
    try_finally       142ns ± 0%   137ns ± 0%   -3.52%  (p=0.000 n=10+10)
    defer            2.32µs ± 1%  2.33µs ± 1%   +0.71%  (p=0.005 n=10+10)
    workgroup_empty  90.3µs ± 0%  55.8µs ± 0%  -38.26%  (p=0.000 n=10+10)
    workgroup_raise   108µs ± 1%    71µs ± 1%  -34.64%  (p=0.000 n=10+10)

This patch is the final patch in series to reach the goal of providing
channels that could be used in Cython/nogil code.

Cython/nogil channels work is dedicated to the memory of Вера Павловна Супрун[4].

[4] https://navytux.spb.ru/memory/%D0%A2%D1%91%D1%82%D1%8F%20%D0%92%D0%B5%D1%80%D0%B0.pdf#page=3

- Add go functionality to libgolang runtime;
- Implement go for thread and gevent runtimes.

  * Thread runtime uses PyThread_start_new_thread which, if used
    carefully, does not depend on Python GIL and on e.g. POSIX reduces to
    tiny wrapper around pthread_create.

  * Gevent runtime uses gevent's Greenlet in Pyx mode. This turns
    gevent to be a build-time dependency.

- Provide low-level _taskgo in C client API;
- Provide type-safe C++-level go wrapper over _taskgo;
- Switch golang.go from py implementation into Pyx wrapper over
  Pyx/nogil API.

This is the first patch that adds Pyx/C++/C-level unit tests and hooks
them into golang_test.py .

This patch brings the following speedup to Python-level interface:

 (on i7@2.6GHz)

thread runtime:

    name             old time/op  new time/op  delta
    go               93.0µs ± 1%  26.6µs ± 1%  -71.41%  (p=0.000 n=10+10)
    chan             13.6µs ± 2%  13.7µs ± 1%     ~     (p=0.280 n=10+10)
    select           29.9µs ± 4%  29.3µs ± 2%   -1.89%  (p=0.017 n=10+9)
    def              61.0ns ± 0%  55.0ns ± 0%   -9.84%  (p=0.000 n=10+10)
    func_def         43.8µs ± 1%  44.0µs ± 1%   +0.66%  (p=0.006 n=10+10)
    call             62.5ns ± 1%  64.0ns ± 0%   +2.40%  (p=0.000 n=10+8)
    func_call        1.06µs ± 1%  1.09µs ± 1%   +2.72%  (p=0.000 n=10+10)
    try_finally       137ns ± 0%   139ns ± 2%   +1.17%  (p=0.033 n=10+10)
    defer            2.34µs ± 1%  2.36µs ± 1%   +0.84%  (p=0.015 n=10+10)
    workgroup_empty  96.1µs ± 1%  58.1µs ± 1%  -39.55%  (p=0.000 n=10+10)
    workgroup_raise   135µs ± 1%    73µs ± 1%  -45.97%  (p=0.000 n=10+10)

gevent runtime:

    name             old time/op  new time/op  delta
    go               68.8µs ± 1%  28.6µs ± 0%  -58.47%  (p=0.000 n=10+8)
    chan             14.8µs ± 1%  15.8µs ± 1%   +6.19%  (p=0.000 n=10+10)
    select           32.0µs ± 0%  33.1µs ± 1%   +3.25%  (p=0.000 n=10+10)
    def              58.0ns ± 0%  55.0ns ± 0%   -5.17%  (p=0.000 n=10+10)
    func_def         43.9µs ± 1%  44.4µs ± 2%   +1.20%  (p=0.007 n=10+10)
    call             63.5ns ± 1%  64.0ns ± 2%     ~     (p=0.307 n=10+10)
    func_call        1.08µs ± 1%  1.06µs ± 0%   -2.55%  (p=0.000 n=10+8)
    try_finally       142ns ± 0%   136ns ± 0%   -4.23%  (p=0.000 n=10+9)
    defer            2.32µs ± 1%  2.29µs ± 1%   -0.96%  (p=0.000 n=10+10)
    workgroup_empty  90.3µs ± 0%  73.8µs ± 1%  -18.29%  (p=0.000 n=10+10)
    workgroup_raise   108µs ± 1%    94µs ± 0%  -13.29%  (p=0.000 n=10+10)

(small changes are probably within noise; "go" and "workgroup_*" should be
representative)

Start transforming Pygolang from being pure Python project into project
that uses both Python and Cython and provides both Python and Cython API
to its users.

For Cython API there is a goal for it to be usable independently of
Python GIL - i.e. independently of Python runtime and limitations
imposed by Python's global interpreter lock. The motivation for this
work is wendelin.core v2, which, due to its design, would deadlock if it
tries to take the GIL in its pinner thread.

This commit brings bare minimum establishing libraries and build system.
It:

- Introduces C/C++ library libgolang. Libgolang will be serving most of
  nogil functionality and Pyx API will be a small wrapper over it. At
  this step Libgolang comes only with C-level panic;

- Introduces Pyx package golang that is aliased to Pyx package
  golang._golang . Users can do `from golang cimport ...` similarly to how
  they do py-level `from golang import ...`. At this step golang.pyx
  only wraps panic and provides way to transform C-level panic into Python
  exception;

- Introduces Py package golang.pyx.build . This package, similarly to
  golang.org/pkg/go/build, serves as the system to build Pyx packages -
  both external packages that use Pygolang in pyx mode, and to build
  Pygolang itself. The build, internally, is served by setuptools_dso
  and cython. An example how to use golang.pyx.build from external
  project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
  tests to verify golang.pyx.build functionality;

- Introduces Pygolang build dependency to cython and setuptools_dso.

Implement deadlines / timeouts using timers added recently in 9c260fde
(time: New package that mirrors Go's time).

Add time.Timer, time.Ticker and convenience functions time.tick,
time.after and time.after_func. These will be used in context to support
deadlines and timeouts.

While at time topic, also provide sleep and now from golang.time, so
that there is no need to import both golang.time and stdlib's time in a
file.

Provide time constants in the module as they are useful to have and
mirror constants provided by Go's time.

Note: timers implementation is very suboptimal for now.

- correct CHANGELOG title levels: since readme uses "----" for the
  first level and "~~~~" for the second level, "====" was interpreted as
  the third title level and "Pygolang change history" became
  sub-sub-section of "Additional packages and utilities" with section
  for each version conversely becoming first level. It was not very
  noticeable until 0c5f9d06 (readme: Push "Additional packages and
  utilities" into its own section) started to use "~~~~".
  -> Rework CHANGELOG titling to align with the one in README.

- fix minor markup bits in README.

WorkGroup provides way to spawn goroutines that work on a common task
and wait for their completion. It is modelled after
https://godoc.org/golang.org/x/sync/errgroup but is not equal to it.

See WorkGroup docstring for details.

Add sync.Once and sync.WaitGroup.

Add .Context, .background .with_cancel and .with_value. There is no
support for deadline/timeout yet, since Go time package is not yet
mirrored.

Contrary to Go stdlib version, we also support context.merge that takes
https://godoc.org/lab.nexedi.com/kirr/go123/xcontext#hdr-Merging_contexts
for its inspiration.

See https://blog.golang.org/context for overview of contexts.

We are going to add more packages to mirror Go stdlib (e.g. next is
context). Organize a dedicated section for what is additionally provided
besides mirroring Go language and refer to that place only once from the
top-level abstract.

Send/recv on the nil channel block forever; close panics.
If a nil channel is used in select - corresponding case is never selected.

Setting channel to nil is a usual idiom in Go to disable processing some
cases in select. Nil channel is also used as "done" for e.g.
context.Background() - for contexts that can be never canceled.

Since we already have func (see 5146eb0b "Add support for defer &
recover") we can use @func for both plain functions and for methods.

For example instead of

	@func
	def my_function(...):
	    ...

	@method(MyClass)			<-- NOTE
	def my_method(self, ...):
	    ...

have it as

	@func
	def my_function(...):
	    ...

	@func(MyClass)				<-- NOTE
	def my_method(self, ...):
	    ...

which looks more similar to Go and exposes less golang keywords to a user.

Provide gpython interpreter, that sets UTF-8 as default encoding,
integrates gevent and puts go, chan, select etc into builtin namespace.
Please see details in the documentation added by this patch.

pymain was taken from kirr/go123@7a476082
(go123: tracing/pyruntraced: New tool to run Python code with tracepoints activated (draft))

This are functions to decode quotation that was produced by
strconv.quote().

Move quote implementation from gcompat to strconv. Make quote work on both
unicode|bytes string input and produce the same output type. Preserve qq
(still remaining in gcompat) to always produce str, since that is used
in prints.

Stop hiding qq and benchmarking bits in changelog only - add their
description into additional sections in readme.

Since py.bench description is now in readme, there is no need to keep
py.bench example in the changelog.

`defer` allows to schedule a cleanup to be executed when current function
returns. It is similar to `try`/`finally` but does not force the cleanup part
to be far away in the end. For example::

   wc = wcfs.join(zurl)    │     wc = wcfs.join(zurl)
   defer(wc.close)         │     try:
                           │        ...
   ...                     │        ...
   ...                     │        ...
   ...                     │     finally:
                           │        wc.close()

For completeness there is `recover` and `panic` that allow to program with
Go-style error handling, for example::

   def _():
      r = recover()
      if r is not None:
         print("recovered. error was: %s" % (r,))
   defer(_)

   ...

   panic("aaa")

But `recover` and `panic` are probably of less utility since they can be
practically natively modelled with `try`/`except`.

If `defer` is used, the function that uses it must be wrapped with `@func` or
`@method` decorators.

The implementation is partly inspired by work of Denis Kolodin:

- https://habr.com/post/191786
- https://stackoverflow.com/a/43028386/9456786

Not only we can import modules by full path, but now we can also spawn
threads/coroutines and exchange data in between them with the same
primitives and semantic as in Go.

The bulk of new functionality is copied from here:

	kirr/go123@9e1aa6ab

Original commit description follows:

"""
golang: New _Python_ package to provide Go-like features to Python language
- `go` spawns lightweight thread.
- `chan` and `select` provide channels with Go semantic.
- `method` allows to define methods separate from class.
- `gimport` allows to import python modules by full path in a Go workspace.

The focus of first draft was on usage interface and on correctness, not speed.
In particular select should be fully working.

If there is a chance I will maybe try to followup with gevent-based
implementation in the future.
Hide whitespace changes
"""

Module gopath provides way to import python modules by full path in a Go workspace.

For example

    lonet = gopath.gimport('lab.nexedi.com/kirr/go123/xnet/lonet')

will import either

    lab.nexedi.com/kirr/go123/xnet/lonet.py, or
    lab.nexedi.com/kirr/go123/xnet/lonet/__init__.py

located somewhere under $GOPATH.