For consistency to denote that this functions work at Python level:

    - len_sendq    -> pylen_sendq
    - len_recvq    -> pylen_recvq
    - waitBlocked  -> pywaitBlocked

Plain test code movement with s/panic/pypanic/ as in golang.pyx panic is
already there and means semantically different thing. We need this code to
live in pyx world, because when channels implementation will be moved to
C, this utilities will need to be adjusted in a way that is not possible
to do from Python.

Denote what a method returns via `# -> ...` suffix.

Use `ch` instead of `self` for pychan methods. This aligns with how
(py)chan objects are currently denoted in (py)select and will reduce the
difference when chan code is moved into libgolang. We are sticking to Go
convention here.

After the code with channels implementation is moved into libgolang, the
rest bits in golang.pyx will be changed to refer to pychan objects as
pych for clarity.

To denote that this function/classes work at Python level:

    - chan    -> pychan
    - select  -> pyselect
    - default -> pydefault
    - nilchan -> pynilchan

Plain code movement with just s/panic/pypanic/ as in golang.pyx panic is
aleady there and means semantically different thing. Moved code, even
though it lives in golang.pyx, is still Python code and requires Python
runtime and GIL. We'll be splitting channels implementation into nogil
world in the following patches.

Just plain movement to Cython brings the following speedup:

 (on i7@2.6GHz)

thread runtime:

    name             old time/op  new time/op  delta
    go               26.6µs ± 1%  21.7µs ± 1%  -18.54%  (p=0.000 n=10+10)
    chan             13.7µs ± 1%   9.9µs ± 4%  -27.80%  (p=0.000 n=10+10)
    select           29.3µs ± 2%  19.2µs ± 4%  -34.65%  (p=0.000 n=9+9)
    def              55.0ns ± 0%  58.0ns ± 0%   +5.45%  (p=0.000 n=10+10)
    func_def         44.0µs ± 1%  44.4µs ± 0%   +0.72%  (p=0.002 n=10+10)
    call             64.0ns ± 0%  63.0ns ± 0%   -1.56%  (p=0.002 n=8+10)
    func_call        1.09µs ± 1%  1.05µs ± 1%   -2.96%  (p=0.000 n=10+10)
    try_finally       139ns ± 2%   135ns ± 0%   -2.60%  (p=0.000 n=10+10)
    defer            2.36µs ± 1%  2.36µs ± 1%     ~     (p=0.617 n=10+10)
    workgroup_empty  58.1µs ± 1%  49.0µs ± 1%  -15.61%  (p=0.000 n=10+10)
    workgroup_raise  72.7µs ± 1%  62.6µs ± 1%  -13.88%  (p=0.000 n=10+10)

gevent runtime:

    name             old time/op  new time/op  delta
    go               28.6µs ± 0%  25.4µs ± 0%  -11.20%  (p=0.000 n=8+9)
    chan             15.8µs ± 1%  12.2µs ± 1%  -22.62%  (p=0.000 n=10+10)
    select           33.1µs ± 1%  23.3µs ± 2%  -29.60%  (p=0.000 n=10+10)
    def              55.0ns ± 0%  56.0ns ± 0%   +1.82%  (p=0.000 n=10+10)
    func_def         44.4µs ± 2%  43.0µs ± 1%   -3.06%  (p=0.000 n=10+9)
    call             64.0ns ± 2%  69.0ns ± 0%   +7.81%  (p=0.000 n=10+10)
    func_call        1.06µs ± 0%  1.06µs ± 1%     ~     (p=0.913 n=8+9)
    try_finally       136ns ± 0%   139ns ± 0%   +2.21%  (p=0.000 n=9+10)
    defer            2.29µs ± 1%  2.38µs ± 2%   +3.58%  (p=0.000 n=10+10)
    workgroup_empty  73.8µs ± 1%  70.5µs ± 1%   -4.48%  (p=0.000 n=10+10)
    workgroup_raise  94.1µs ± 0%  90.6µs ± 0%   -3.69%  (p=0.000 n=10+10)

- Add go functionality to libgolang runtime;
- Implement go for thread and gevent runtimes.

  * Thread runtime uses PyThread_start_new_thread which, if used
    carefully, does not depend on Python GIL and on e.g. POSIX reduces to
    tiny wrapper around pthread_create.

  * Gevent runtime uses gevent's Greenlet in Pyx mode. This turns
    gevent to be a build-time dependency.

- Provide low-level _taskgo in C client API;
- Provide type-safe C++-level go wrapper over _taskgo;
- Switch golang.go from py implementation into Pyx wrapper over
  Pyx/nogil API.

This is the first patch that adds Pyx/C++/C-level unit tests and hooks
them into golang_test.py .

This patch brings the following speedup to Python-level interface:

 (on i7@2.6GHz)

thread runtime:

    name             old time/op  new time/op  delta
    go               93.0µs ± 1%  26.6µs ± 1%  -71.41%  (p=0.000 n=10+10)
    chan             13.6µs ± 2%  13.7µs ± 1%     ~     (p=0.280 n=10+10)
    select           29.9µs ± 4%  29.3µs ± 2%   -1.89%  (p=0.017 n=10+9)
    def              61.0ns ± 0%  55.0ns ± 0%   -9.84%  (p=0.000 n=10+10)
    func_def         43.8µs ± 1%  44.0µs ± 1%   +0.66%  (p=0.006 n=10+10)
    call             62.5ns ± 1%  64.0ns ± 0%   +2.40%  (p=0.000 n=10+8)
    func_call        1.06µs ± 1%  1.09µs ± 1%   +2.72%  (p=0.000 n=10+10)
    try_finally       137ns ± 0%   139ns ± 2%   +1.17%  (p=0.033 n=10+10)
    defer            2.34µs ± 1%  2.36µs ± 1%   +0.84%  (p=0.015 n=10+10)
    workgroup_empty  96.1µs ± 1%  58.1µs ± 1%  -39.55%  (p=0.000 n=10+10)
    workgroup_raise   135µs ± 1%    73µs ± 1%  -45.97%  (p=0.000 n=10+10)

gevent runtime:

    name             old time/op  new time/op  delta
    go               68.8µs ± 1%  28.6µs ± 0%  -58.47%  (p=0.000 n=10+8)
    chan             14.8µs ± 1%  15.8µs ± 1%   +6.19%  (p=0.000 n=10+10)
    select           32.0µs ± 0%  33.1µs ± 1%   +3.25%  (p=0.000 n=10+10)
    def              58.0ns ± 0%  55.0ns ± 0%   -5.17%  (p=0.000 n=10+10)
    func_def         43.9µs ± 1%  44.4µs ± 2%   +1.20%  (p=0.007 n=10+10)
    call             63.5ns ± 1%  64.0ns ± 2%     ~     (p=0.307 n=10+10)
    func_call        1.08µs ± 1%  1.06µs ± 0%   -2.55%  (p=0.000 n=10+8)
    try_finally       142ns ± 0%   136ns ± 0%   -4.23%  (p=0.000 n=10+9)
    defer            2.32µs ± 1%  2.29µs ± 1%   -0.96%  (p=0.000 n=10+10)
    workgroup_empty  90.3µs ± 0%  73.8µs ± 1%  -18.29%  (p=0.000 n=10+10)
    workgroup_raise   108µs ± 1%    94µs ± 0%  -13.29%  (p=0.000 n=10+10)

(small changes are probably within noise; "go" and "workgroup_*" should be
representative)

The next patch will add `go` to Pyx API and correspondingly C, C++ and Pyx
level tests for it that will go into _golang_test.pyx extension. For
C-level test we'll use C source file - to verify that libgolang.h could
be used by C projects with C compiler, while for C++ and Pyx-level tests
the sources will be in C++. Thus _golang_test.so will be build from both
C and C++ sources.

This creates a problem: distutils / setuptools use the _same_ compiler
to compile both C and C++ sources and only use C++ compiler at link
stage. Thus, as it is not possible to tune compiler that is used only
for C++ sources, and also as it is not possible to provide per-source
flags, when compiling C-level test, the compiler will be invoked with
`-std=c++11` option that we inject. Gcc tolerates that and only prints a
warning, but Clang considers that an error and gives:

	error: invalid argument '-std=c++11' not allowed with 'C'`

A proper fix would be to change the build system from distutils to
something more flexible that uses C++ compiler for C++ sources and C
compiler for C sources and allows to tune per-unit flags. However it is
not a small step at this stage.

-> Use workaround and tweak options that are used when compiling sources
depending on whether it is C or C++.

- Add sleep functionality to libgolang runtime;
- Implement sleep for thread and gevent runtimes. Thread runtime
  implements sleep independently of GIL, but only for POSIX for now;
- Switch golang.time py module into using golang.time pyx module.

As we are adding sleep, related functionality to query system about
"what is current time?" is also added.

Libgolang will be generic library and it will adapt itself to particular
execution environment by way of runtime plugin. This commit introduces
stubs for two such runtimes - "thread" and "gevent" - and initializes
libgolang with particular runtime depending on whether golang is
imported with gevent preactivated or not.

The runtimes themselves are empty at this step. We'll be adding runtime
functionality in the following patches.

Start transforming Pygolang from being pure Python project into project
that uses both Python and Cython and provides both Python and Cython API
to its users.

For Cython API there is a goal for it to be usable independently of
Python GIL - i.e. independently of Python runtime and limitations
imposed by Python's global interpreter lock. The motivation for this
work is wendelin.core v2, which, due to its design, would deadlock if it
tries to take the GIL in its pinner thread.

This commit brings bare minimum establishing libraries and build system.
It:

- Introduces C/C++ library libgolang. Libgolang will be serving most of
  nogil functionality and Pyx API will be a small wrapper over it. At
  this step Libgolang comes only with C-level panic;

- Introduces Pyx package golang that is aliased to Pyx package
  golang._golang . Users can do `from golang cimport ...` similarly to how
  they do py-level `from golang import ...`. At this step golang.pyx
  only wraps panic and provides way to transform C-level panic into Python
  exception;

- Introduces Py package golang.pyx.build . This package, similarly to
  golang.org/pkg/go/build, serves as the system to build Pyx packages -
  both external packages that use Pygolang in pyx mode, and to build
  Pygolang itself. The build, internally, is served by setuptools_dso
  and cython. An example how to use golang.pyx.build from external
  project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
  tests to verify golang.pyx.build functionality;

- Introduces Pygolang build dependency to cython and setuptools_dso.

We can include *.py from top-level directories, not only under
X/testdata/, X/testprog/ ...

This test is veriyfing only how leaked goroutines behave, not go in
general.

test_go was doing the same what pyrun is doing + adjusting PYTHONPATH a
bit. Merge PYTHONPATH adjustment into pyrun and use pyrun uniformly.

pyrun runs command without piping output.
pyout runs command and returns its output.

We'll need to use this utility in other places; golang_test is the
central place for testing utilities.

gpython/gpython_test.py:23: 'subprocess' imported but unused
gpython/gpython_test.py:24: 'six.PY3' imported but unused

... and everything that whole pygolang depends on in pygolang[all].

The reason we do this: currently golang.perf.benchlib needs NumPy, but
it would be not ok to depend whole pygolang on numpy. Similarly pybench
needs pytest, but it would be not ok to require pytest unconditionally.

Thus allow users to specify what they need from pygolang via extras,
like pygolang[x.perf.benchlib] or pygolang[cmd.pybench], with
pygolang[all] serving as merge point for all subpackages dependencies.

TODO in the future we'll, hopefully, also provide accumulation points,
like pygolang[x.perf] which contains everything pygolang[x.perf.*]
depends on.

Note: core functionality and gpython are provided always unconditionally.

This test passes, but the functionality was not tested before. In
particular all tests were passing even with the following hand-edits:

    --- a/golang/__init__.py
    +++ b/golang/__init__.py
    @@ -627,7 +627,7 @@ def selected():
             sel = g.which
             if isinstance(sel, _SendWaiting):
                 if not sel.ok:
    -                panic("send on closed channel")
    +                panic("send on closed channel ZZZ")
                 return sel.sel_n, None

             if isinstance(sel, _RecvWaiting):

with added tests the bug is caught and reported:

    def test_select():
        N = 1000 # times to do repeated select/chan or select/select interactions

        # sync: close vs select(send)
        ch = chan()
        def _():
            waitBlocked(ch.send)
            ch.close()
        go(_)
>       with panics("send on closed channel"): select((ch.send, 0))

golang_test.py:353:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <golang.golang_test.panics instance at 0x7fc1da66e5f0>, exc_type = <class 'golang._PanicError'>, exc_val = _PanicError('send on closed channel ZZZ',)
exc_tb = <traceback object at 0x7fc1dabc33b0>

    def __exit__(self, exc_type, exc_val, exc_tb):
        ok = self.raises.__exit__(exc_type, exc_val, exc_tb)
        if not ok:
            return ok
        # _PanicError raised - let's check panic argument
>       assert self.exc_info.value.args == (self.arg,)
E       AssertionError: assert ('send on clo...channel ZZZ',) == ('send on closed channel',)
E         At index 0 diff: 'send on closed channel ZZZ' != 'send on closed channel'
E         Use -v to get the full diff

golang_test.py:1032: AssertionError

For buffered channel _tryrecv, on success, was unlocking ch._mu too
early - before accessing ch._dataq with ch._dataq.append().

Without the fix, newly added test breaks as e.g.

    golang/golang_test.py::test_chan_buf_recv_vs_tryrecv_race Exception in thread Thread-3:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 317, in _
        assert (_, _rx) == (1, None), ('i%d' % i)
    AssertionError: i30
    assert (0, None) == (1, None)
      At index 0 diff: 0 != 1
      Full diff:
      - (0, None)
      ?  ^
      + (1, None)
      ?  ^

For buffered channel _trysend, on success, was unlocking ch._mu too
early - before accessing ch._dataq with ch._dataq.popleft().

Without the fix, newly added test breaks as e.g.

    golang/golang_test.py::test_chan_buf_send_vs_tryrecv_race Exception in thread Thread-3:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 256, in _
        assert (_, _rx) == (1, None)
    AssertionError: assert (0, 209) == (1, None)
      At index 0 diff: 0 != 1
      Full diff:
      - (0, 209)
      + (1, None)

    Exception in thread Thread-2:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 243, in _
        ch.send(i)
      File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 340, in send
        ok = self._trysend(obj)
      File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 417, in _trysend
        rx = self._dataq.popleft()
    IndexError: pop from an empty deque

Currently it behaves correctly, but this aspect will need to be
explicitly cared about when channel implementation moves to C. Add test
to make sure it won't regress.

Starting from b51b8d5d (select: Run tests more thoroughly) we are
running select subtests in repeated mode with N=1000. However not all
select subtests were run in repeated mode - for example e.g.
"non-blocking try send: not ok" was being run only once.

-> Rework all select subtests to be run in repeated mode to increase the
probability of catching bugs.

Channels implementation will soon be moved to C and will become
independent from Python runtime. This way while pygolang will still be
providing panic, hooking in and raising arbitrary Python-level exception
will become problematic. -> Rework "blocks forever" tests to rely on
just panic to prepare for that.

Chan implementation is going to be moved into C, and this way direct
access to chan internals won't be possible. C implementation will export
dedicated functions for chan tests which will be used.

Prepare for that and factor out retrieving len of chan recv/send queues
into separate functions, which will be adapted at the time of C port.

Nil channel will soon be changed to be represented by underlying NULL C
pointer and there won't be channel object for nil and correspondingly
recv/send queues won't be there. -> Prepare for that and don't check for
recvq/sendq on nil channel.

We provide gpython since 32a21d5b (gpython: Python interpreter with
support for lightweight threads), and golang module, since the
beginning, automatically uses gevent if it was installed via monkey
patching.

This currently pass trivially (since time.now = stdtime.now), but will
be useful once we get time.now implementation into our hands and move it into Cython.

After 9c260fde (time: New package that mirrors Go's time) we have
golang.time.now and golang.time.sleep and it makes it a more
self-dependent system if timing facility is used through golang.time
instead of outside std time module.

For now this is only a "cleanness" change, but will become important once
we start adding pyx-level nogil API to pygolang - there it will be
important to use golang.time.* for correctness.

Starting from e6bea2cf (sync: New package that mirrors Go's sync) there
is sync.WaitGroup and it was using threading.Event to signal that
waitgroup's counter dropped to zero. threading.Event is implemented in
Python's stdlib via threading.Condition, which in turn uses
threading.Lock and list of waiters to implement its functionality.
Which in turn is very similar to what golang.chan does internally with
semaphore and e.g. recv queue.

I noticed this while debugging sync test deadlock (see previous patch)
and suspecting bug in threading.Event for a moment. While it turned
there is no bug in threading.Event, it is better to rely on the common
functionality for similar tasks, and pygolang's channel perfectly
matches here the need to signal an event. By using our own code instead
of stdlib's threading we are likely becoming less bug-prone.

This also brings a small speedup:

    (on i5@1.80GHz)
    name             old time/op  new time/op  delta
    workgroup_empty   239µs ± 1%   220µs ± 1%  -7.62%  (p=0.000 n=10+9)
    workgroup_raise   275µs ± 2%   264µs ± 2%  -3.85%  (p=0.000 n=10+10)

sync.WorkGroup test was doing ctx.done() wait from under test mutex,
something like

    def _(ctx, i):
        with mu:
            ...
            if i == 0:
                raise RuntimeError()    # to cause ctx cancel
            ctx.done().recv()           # i=1 -> wait till ctx is canceled

but it can be a deadlock if T(i=1) runs first and enters
ctx.done().recv() before T(i=0) is run - then T(i=0) will block forever
waiting to lock mu.

This failure was not seen so far, probably because the time to go a new
thread/goroutine is relatively high. However one of upcoming patches,
where go is made faster, revealed this problem and, without the fix,
sync.WorkGroup test was regularly deadlocking.

The problem was there from sync.WorkGroup beginning - from
9ee7ba91 (sync += WorkGroup).

Fix the deadlock by waiting for ctx.done() outside of mu.

9c61f254 (pygopath: Initial draft) has a "TODO py3 support", probably
because https://stackoverflow.com/a/67692 shows different codes for
Python2 and Python3.

However today we have tox coverage for all Python2.7, Python3.6 and
Python3.7 and everything works including gimport tests.

-> Remove the TODO.

Currently gimport tests depend on module-level setup_module /
teardown_module, which require for gimport tests to be in separate file.

Redo the test not to depend on global module-level state and
setup/teardown. This allows e.g. to merge gimport tests into
golang_test.py if/when we want/need to.

Should be in 2aad64bb (golang: Add support for nil channel).

In tests in places where the code checks that something panics, verify
not only that _PanicError is raised, but also what was the argument
passed to panic.

This makes sure that tested code panics in expected place, not just
panics "somewhere".

To keep signal/noise ratio high introduce `panics` which is similar to
`pytest.raises` and asserts that wrapped code panics with expected
argument.

-> Use .[test] to refer to them.
https://stackoverflow.com/a/41398850/9456786

A problem was hit with pytest.fail with raises Failed exception not
being propagated to .wait. As it turned out it was not propagated
because pytest's Failed derives from BaseException, not Exception, and
we were catching only Exception and its children.

Rework the code to propagate all exception types from workers.

Performance change is with noise (it is either a bit faster for one set
of runs, or a bit slower for another set of runs).

It is true for any decorator, that it makes things faster if
def+decorator is used globally instead of at runtime, but for @func it
is especially true since @func, using decorator.decorate, has relatively
high overhead if it is used not only once at program startup, but
instead every time something is run.

In particular moving @func out of WorkGroup.go() make things
considerably faster:

	name             old time/op  new time/op  delta
	workgroup_empty   195µs ± 0%   113µs ± 1%  -41.91%  (p=0.008 n=5+5)
	workgroup_raise   221µs ± 1%   137µs ± 1%  -38.29%  (p=0.008 n=5+5)

See

	https://lab.nexedi.com/kirr/misc/raw/009c4fee/pygolang/prof_workgroup_empty.svg

for bench_workgroup_empty profile, where it is seen that
decorator.decorate was using ~ half of the whole WorkGroup.go() time.

Start adding benchmarks to pygolang.

Measure how much sync chan send/recv take and how much select takes for
synchronous channels. Also measure how much overhead @func adds at both
def and call times, and the overhead of defer compared to try/finally.
For std (non-gevent'ed) Python2.7 we are currently at:

	name         time/op
	go           91.6µs ± 1%
	chan         13.7µs ± 3%
	select       30.1µs ± 4%
	def          55.0ns ± 0%
	func_def     43.6µs ± 0%
	call         63.0ns ± 0%
	func_call    1.06µs ± 0%
	try_finally   136ns ± 1%
	defer        2.33µs ± 0%