bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on
Python2) added integration patches for IPython and Pytest to properly
dump tracebacks for chained exceptions even on Python2. However the
functionality of patches was tested only manually.

-> Add corresponding tests to verify how IPython and Pytest behaves
when dumping tracebacks.

Only io.EOF and io.ErrUnexpectedEOF for now.
Moved here from wcfs from wendelin.core.

errors.New was added in a245ab56 (errors: New package) without test.

Currently libgolang_test.cpp contains tests for code in libgolang.cpp and
for code that lives in other libgolang packages - sync, fmt, etc. It is
becoming tight and we are going to split libgolang_test.cpp and move
package tests to their corresponing files - e.g. to sync_test.cpp and
the like.

Move common assertion utilities into shared header before that as a
preparatory step.

Providing pygolang-specific DSO is needed because using just
setuptools_dso.DSO in external project will result in that e.g.
"<golang/libgolang.h>" won't be found.

Provide string utilities to verify whether string has suffix/prefix,
trim it, and split string by a delimiter. The code originated in wcfs
codebase in wendelin.core .

Pyx/nogil only.

With fmt.sprintf() and fmt.errorf() to format strings and errors. The
code was extracted from wcfs in wendelin.core .

Pyx/nogil only.

Commit b073f6df (time: Move/Port timers to C++/Pyx nogil) added C++
level PyFunc as a way to call Python function from under nogil code.
We will soon need this functionality for sync.WorkGroup too, so move it
into shared place.

We move into regular library - not a pyx module (e.g. golang.pyx) -
because for pyx modules linking is done at import time by specially
generated code, while we need the linker support to automatically
resolve e.g. calls to PyFunc::operator() from inside std::function.

Provide context-related functionality that can be used directly from C++
and Pyx/nogil codes. Python-level classes and functions become small
wrappers around pyx/nogil ones.

Like with timers (b073f6df "time: Move/Port timers to C++/Pyx nogil")
and interfaces (5a99b769 "libgolang: Start providing interfaces")
memory for objects dynamically allocated on heap is managed
automatically.

Start new package that provides C++ amendments to be used by libgolang
and its users. Current functionality: dict and set that wrap
std::unordered_map and std::unordered_set into ergonomic interface.

The code originates from wendelin.core:
https://lab.nexedi.com/kirr/wendelin.core/blob/5a045ed1/wcfs/internal/wcfs_misc.h#L214-257

Pyx/nogil only.

With just one function - errors.New() - to create new error with given
text.

This will be soon used in Pyx/nogil context package to create context.canceled
and context.deadlineExceeded errors.

Pyx/nogil only for now.

We already have some sync functionality implemented in C++ (e.g.
sync.Once, sync.WaitGroup) and we are going to add more and also move
timers implementation to C++. It is getting crowded for that
functionality to still live in libgolang.{h,cpp}

-> Split sync & time functionality into their own C++ packages (still
built-into libgolang.so)

Python3 chains exceptions, so that e.g. if exc1 is raised and, while it
was not handled, another exc2 is raised, exc2 will be linked to exc1 via
exc2.__context__ attribute and exc1 will be included into exc2 traceback
printout. However many projects still use Python2 and there is no
similar chaining functionality there. This way exc1 is completely lost.

Since defer code is in our hands, we can teach it to implement exception
chaining even on Python2 by carefully analyzing what happens in
_GoFrame.__exit__().

Implementing chaining itself is relatively easy, but is only part of the
story. Even if an exception is chained with its cause, but exception
dump does not show the cause, the chaining will be practically useless.
With this in mind this patches settles not only on implementing chaining
itself, but on also giving a promise that chained cause exceptions will
be included into traceback dumps as well.

To realize this promise we adjust all exception dumping funcitons in
traceback module and carefully install adjusted
traceback.print_exception() into sys.excepthook. This amends python
interactive sessions and programs run by python interpreter to include
causes in exception dumps. "Careful" here means that we don't change
sys.excepthook if on golang module load we see that sys.excepthook was already
changed by some other module - e.g. due to IPython session running
because IPython installs its own sys.excepthook. In such cases we don't
install our sys.excepthook, but we also provide integration patches that
add exception chaining support for traceback dump functionality in
popular third-party software. The patches (currently for IPython and
Pytest) are activated automatically, but only when/if corresponding
software is imported and actually used. This should give practically
good implementation of the promise - a user can now rely on seeing
exception cause in traceback dump whatever way python programs are run.

The implementation takes https://pypi.org/project/pep3134/ experience
into account [1]. peak.utils.imports [2,3] is used to be notified when/if
third-party module is imported.

[1] https://github.com/9seconds/pep3134/
[2] https://pypi.org/project/Importing/
[3] http://peak.telecommunity.com/DevCenter/Importing

This patch originally started as hacky workaround in wendelin.core
because in wcfs tests I was frequently hitting situations, where
exception raised by an assert was hidden by another exception raised in
further generic teardown check. For example wcfs tests check that wcfs
is unmounted after every test run [4] and if that fails it was hiding
problems raised by an assert. As the result I was constantly guessing
and adding code like [5] to find what was actually breaking. At some
point I added hacky workaround for defer to print cause exception not to
loose it [6]. [7] has more context and background discussion on this topic.

[4] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L70
[5] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L853-857
[6] kirr/wendelin.core@c00d94c7
[7] zodbtools!13 (comment 81553)

After this patch, on Python2

    defer(cleanup1)
    defer(cleanup2)
    defer(cleanup3)
    ...

is no longer just a syntatic sugar for

    try:
        try:
            try:
                ...
            finally:
                cleanup3()
        finally:
            cleanup2()
    finally:
        cleanup1()

- ThreadSanitizer helps to detect races and some memory errors,
- AddressSanitizer helps to detect memory errors,
- Python debug builds help to detect e.g reference counting errors.

Adding all those tools to testing coverage discovers e.g. the following
bugs (not a full list):

---- 8< ----

py27-thread-tsan:
WARNING: ThreadSanitizer: data race (pid=7143)
  Write of size 8 at 0x7b1400000650 by main thread:
    #0 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:649 (libtsan.so.0+0x2b46a)
    #1 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:643 (libtsan.so.0+0x2b46a)
    #2 golang::_chan::decref() golang/runtime/libgolang.cpp:470 (liblibgolang.so.0.1+0x47f2)
    #3 _chanxdecref golang/runtime/libgolang.cpp:452 (liblibgolang.so.0.1+0x484a)
    #4 _test_go_c golang/runtime/libgolang_test_c.c:86 (_golang_test.so+0x13a2e)
    #5 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
    #6 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
    #7 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)

  Previous read of size 8 at 0x7b1400000650 by thread T8:
    #0 golang::Sema::acquire() golang/runtime/libgolang.cpp:164 (liblibgolang.so.0.1+0x410a)
    #1 golang::Mutex::lock() golang/runtime/libgolang.cpp:175 (liblibgolang.so.0.1+0x4c82)
    #2 golang::_chan::close() golang/runtime/libgolang.cpp:754 (liblibgolang.so.0.1+0x4c82)
    #3 _chanclose golang/runtime/libgolang.cpp:732 (liblibgolang.so.0.1+0x4d1a)
    #4 _work golang/runtime/libgolang_test_c.c:92 (_golang_test.so+0x136cc)
    #5 <null> <null> (python2.7+0x1929e3)

  Thread T8 (tid=7311, finished) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:915 (libtsan.so.0+0x2be1b)
    #1 PyThread_start_new_thread <null> (python2.7+0x19299f)
    #2 _taskgo golang/runtime/libgolang.cpp:119 (liblibgolang.so.0.1+0x3f68)
    #3 _test_go_c golang/runtime/libgolang_test_c.c:84 (_golang_test.so+0x13a1c)
    #4 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
    #5 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
    #6 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)

py37-thread-asan:
==22205==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000002cd0 at pc 0x7fd3732a7679 bp 0x7fd3723c8c50 sp 0x7fd3723c8c48
READ of size 8 at 0x607000002cd0 thread T7
    #0 0x7fd3732a7678 in golang::Sema::acquire() golang/runtime/libgolang.cpp:164
    #1 0x7fd3732a8644 in golang::Mutex::lock() golang/runtime/libgolang.cpp:175
    #2 0x7fd3732a8644 in golang::_chan::close() golang/runtime/libgolang.cpp:754
    #3 0x7fd3724004b2 in golang::chan<golang::structZ>::close() const golang/libgolang.h:323
    #4 0x7fd3724004b2 in operator() golang/runtime/libgolang_test.cpp:262
    #5 0x7fd3724004b2 in __invoke_impl<void, _test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:60
    #6 0x7fd3724004b2 in __invoke<_test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:95
    #7 0x7fd3724004b2 in __call<void> /usr/include/c++/8/functional:400
    #8 0x7fd3724004b2 in operator()<> /usr/include/c++/8/functional:484
    #9 0x7fd3724004b2 in _M_invoke /usr/include/c++/8/bits/std_function.h:297
    #10 0x7fd3723fdc6e in std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687
    #11 0x7fd3723fdc6e in operator() golang/libgolang.h:273
    #12 0x7fd3723fdc6e in _FUN golang/libgolang.h:271
    #13 0x62ddf3  (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62ddf3)
    #14 0x7fd377393fa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
    #15 0x7fd376eda4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)

0x607000002cd0 is located 16 bytes inside of 72-byte region [0x607000002cc0,0x607000002d08)
freed by thread T0 here:
    #0 0x7fd377519fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0)
    #1 0x7fd372401335 in golang::chan<golang::structZ>::~chan() golang/libgolang.h:292
    #2 0x7fd372401335 in _test_chan_vs_stackdeadwhileparked() golang/runtime/libgolang_test.cpp:222

previously allocated by thread T0 here:
    #0 0x7fd37751a518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518)
    #1 0x7fd3732a7d0b in zalloc golang/runtime/libgolang.cpp:1185
    #2 0x7fd3732a7d0b in _makechan golang/runtime/libgolang.cpp:413

Thread T7 created by T0 here:
    #0 0x7fd377481db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x62df39 in PyThread_start_new_thread (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62df39)

SUMMARY: AddressSanitizer: heap-use-after-free golang/runtime/libgolang.cpp:164 in golang::Sema::acquire()
Shadow bytes around the buggy address:
  0x0c0e7fff8540: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
  0x0c0e7fff8550: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c0e7fff8560: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
  0x0c0e7fff8570: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
  0x0c0e7fff8580: fd fd fd fd fd fa fa fa fa fa fd fd fd fd fd fd
=>0x0c0e7fff8590: fd fd fd fa fa fa fa fa fd fd[fd]fd fd fd fd fd
  0x0c0e7fff85a0: fd fa fa fa fa fa 00 00 00 00 00 00 00 00 00 fa
  0x0c0e7fff85b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff85c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff85d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff85e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

---- 8< ----

The bugs will be addressed in the followup patches.

As suggested by https://github.com/jayvdb run manifest checker and oops,
it finds that we forgot to include pyproject.toml for testprog/golang_pyx_user:

    missing from sdist:
      golang/pyx/testprog/golang_pyx_user/pyproject.toml
    suggested MANIFEST.in rules:
      recursive-include golang *.toml

-> Fix MANIFEST.in in generic way to include golang/*/*.toml

The bug was not affecting pygolang usage - only golang.pyx.build tests were
failing for sdist installed pygolang.

Copy linux/list.h from wendelin.core which copied it from
util-linux.git which took it from linux.git at LGPL license state.

Here are corresponding links for wendelin.core and util-linux:

    wendelin.core@a0f940ad
    https://git.kernel.org/cgit/utils/util-linux/util-linux.git/tree/include/list.h?id=v2.25-165-g9138d6f

Linked lists will be used in channels implementation in the next patch.

Start transforming Pygolang from being pure Python project into project
that uses both Python and Cython and provides both Python and Cython API
to its users.

For Cython API there is a goal for it to be usable independently of
Python GIL - i.e. independently of Python runtime and limitations
imposed by Python's global interpreter lock. The motivation for this
work is wendelin.core v2, which, due to its design, would deadlock if it
tries to take the GIL in its pinner thread.

This commit brings bare minimum establishing libraries and build system.
It:

- Introduces C/C++ library libgolang. Libgolang will be serving most of
  nogil functionality and Pyx API will be a small wrapper over it. At
  this step Libgolang comes only with C-level panic;

- Introduces Pyx package golang that is aliased to Pyx package
  golang._golang . Users can do `from golang cimport ...` similarly to how
  they do py-level `from golang import ...`. At this step golang.pyx
  only wraps panic and provides way to transform C-level panic into Python
  exception;

- Introduces Py package golang.pyx.build . This package, similarly to
  golang.org/pkg/go/build, serves as the system to build Pyx packages -
  both external packages that use Pygolang in pyx mode, and to build
  Pygolang itself. The build, internally, is served by setuptools_dso
  and cython. An example how to use golang.pyx.build from external
  project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
  tests to verify golang.pyx.build functionality;

- Introduces Pygolang build dependency to cython and setuptools_dso.

We can include *.py from top-level directories, not only under
X/testdata/, X/testprog/ ...

Python adds dirname of run program to sys.path . This way when
golang_test_goleaked.py runs it can import modules located under golang/
just by their name. Until now this was not noticed, but in the next
patch we are going to add golang.time module and if test program is run
with golang/ in sys.path just plain `import time` won't import time from
stdlib and instead import time from golang/ .

Such behaviour can be mitigated by doing `from __future__ import
absolute_import` and we do that, including in golang_test_goleaked.py
(see 81dfefa0 "*: __future__ += absolute_imports; Use unified __future__
everywhere"). However that does not prevent modules - even modules from
stdlib - who are doing `import time` and not doing future
absolute_import to import golang's time instead of stdlib. For example
on PyPy2 threading imports time and then the test breaks:

  Traceback (most recent call last):
    File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/golang_test_goleaked.py", line 24, in <module>
      from golang import go, chan
    File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/__init__.py", line 38, in <module>
      import inspect, threading, collections, random, sys
    File "/usr/lib/pypy/lib-python/2.7/threading.py", line 15, in <module>
      from time import time as _time, sleep as _sleep
    File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/time.py", line 30, in <module>
      from golang import go, chan, select, default, nilchan, panic
  ImportError: cannot import name 'go'

-> Move the test program into a directory different from golang/ to
avoid this trap.

Pymain was not adding e.g. directory of executed file to sys.path, and
as the result if there were e.g. 2 files

	dir/hello.py	# imports world
	dir/world.py

running `gpython dir/hello.py` would fail to import world.

The case for interactive console was also failing to setup sys.argv as
empty, so it was containing ['/path/to/gpython']. It was also, contrary
to standard python, unconditionally showing '>>>' prompt even when stdin
was not a tty.

Fix all that and add a test to cover pymain functionality.

Until now it was not included by `python setup.py sdist`.

On this topic, also include README.rst explicitly, since even if it is
currently implicitly included, that might change some day and break us.

Not only we can import modules by full path, but now we can also spawn
threads/coroutines and exchange data in between them with the same
primitives and semantic as in Go.

The bulk of new functionality is copied from here:

	kirr/go123@9e1aa6ab

Original commit description follows:

"""
golang: New _Python_ package to provide Go-like features to Python language
- `go` spawns lightweight thread.
- `chan` and `select` provide channels with Go semantic.
- `method` allows to define methods separate from class.
- `gimport` allows to import python modules by full path in a Go workspace.

The focus of first draft was on usage interface and on correctness, not speed.
In particular select should be fully working.

If there is a chance I will maybe try to followup with gevent-based
implementation in the future.
Hide whitespace changes
"""

Module gopath provides way to import python modules by full path in a Go workspace.

For example

    lonet = gopath.gimport('lab.nexedi.com/kirr/go123/xnet/lonet')

will import either

    lab.nexedi.com/kirr/go123/xnet/lonet.py, or
    lab.nexedi.com/kirr/go123/xnet/lonet/__init__.py

located somewhere under $GOPATH.