tox += ThreadSanitizer, AddressSanitizer, Python debug builds

- ThreadSanitizer helps to detect races and some memory errors,
- AddressSanitizer helps to detect memory errors,
- Python debug builds help to detect e.g reference counting errors.

Adding all those tools to testing coverage discovers e.g. the following
bugs (not a full list):

---- 8< ----

py27-thread-tsan:
WARNING: ThreadSanitizer: data race (pid=7143)
  Write of size 8 at 0x7b1400000650 by main thread:
    #0 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:649 (libtsan.so.0+0x2b46a)
    #1 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:643 (libtsan.so.0+0x2b46a)
    #2 golang::_chan::decref() golang/runtime/libgolang.cpp:470 (liblibgolang.so.0.1+0x47f2)
    #3 _chanxdecref golang/runtime/libgolang.cpp:452 (liblibgolang.so.0.1+0x484a)
    #4 _test_go_c golang/runtime/libgolang_test_c.c:86 (_golang_test.so+0x13a2e)
    #5 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
    #6 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
    #7 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)

  Previous read of size 8 at 0x7b1400000650 by thread T8:
    #0 golang::Sema::acquire() golang/runtime/libgolang.cpp:164 (liblibgolang.so.0.1+0x410a)
    #1 golang::Mutex::lock() golang/runtime/libgolang.cpp:175 (liblibgolang.so.0.1+0x4c82)
    #2 golang::_chan::close() golang/runtime/libgolang.cpp:754 (liblibgolang.so.0.1+0x4c82)
    #3 _chanclose golang/runtime/libgolang.cpp:732 (liblibgolang.so.0.1+0x4d1a)
    #4 _work golang/runtime/libgolang_test_c.c:92 (_golang_test.so+0x136cc)
    #5 <null> <null> (python2.7+0x1929e3)

  Thread T8 (tid=7311, finished) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:915 (libtsan.so.0+0x2be1b)
    #1 PyThread_start_new_thread <null> (python2.7+0x19299f)
    #2 _taskgo golang/runtime/libgolang.cpp:119 (liblibgolang.so.0.1+0x3f68)
    #3 _test_go_c golang/runtime/libgolang_test_c.c:84 (_golang_test.so+0x13a1c)
    #4 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
    #5 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
    #6 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)

py37-thread-asan:
==22205==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000002cd0 at pc 0x7fd3732a7679 bp 0x7fd3723c8c50 sp 0x7fd3723c8c48
READ of size 8 at 0x607000002cd0 thread T7
    #0 0x7fd3732a7678 in golang::Sema::acquire() golang/runtime/libgolang.cpp:164
    #1 0x7fd3732a8644 in golang::Mutex::lock() golang/runtime/libgolang.cpp:175
    #2 0x7fd3732a8644 in golang::_chan::close() golang/runtime/libgolang.cpp:754
    #3 0x7fd3724004b2 in golang::chan<golang::structZ>::close() const golang/libgolang.h:323
    #4 0x7fd3724004b2 in operator() golang/runtime/libgolang_test.cpp:262
    #5 0x7fd3724004b2 in __invoke_impl<void, _test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:60
    #6 0x7fd3724004b2 in __invoke<_test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:95
    #7 0x7fd3724004b2 in __call<void> /usr/include/c++/8/functional:400
    #8 0x7fd3724004b2 in operator()<> /usr/include/c++/8/functional:484
    #9 0x7fd3724004b2 in _M_invoke /usr/include/c++/8/bits/std_function.h:297
    #10 0x7fd3723fdc6e in std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687
    #11 0x7fd3723fdc6e in operator() golang/libgolang.h:273
    #12 0x7fd3723fdc6e in _FUN golang/libgolang.h:271
    #13 0x62ddf3  (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62ddf3)
    #14 0x7fd377393fa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
    #15 0x7fd376eda4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)

0x607000002cd0 is located 16 bytes inside of 72-byte region [0x607000002cc0,0x607000002d08)
freed by thread T0 here:
    #0 0x7fd377519fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0)
    #1 0x7fd372401335 in golang::chan<golang::structZ>::~chan() golang/libgolang.h:292
    #2 0x7fd372401335 in _test_chan_vs_stackdeadwhileparked() golang/runtime/libgolang_test.cpp:222

previously allocated by thread T0 here:
    #0 0x7fd37751a518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518)
    #1 0x7fd3732a7d0b in zalloc golang/runtime/libgolang.cpp:1185
    #2 0x7fd3732a7d0b in _makechan golang/runtime/libgolang.cpp:413

Thread T7 created by T0 here:
    #0 0x7fd377481db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x62df39 in PyThread_start_new_thread (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62df39)

SUMMARY: AddressSanitizer: heap-use-after-free golang/runtime/libgolang.cpp:164 in golang::Sema::acquire()
Shadow bytes around the buggy address:
  0x0c0e7fff8540: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
  0x0c0e7fff8550: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c0e7fff8560: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
  0x0c0e7fff8570: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
  0x0c0e7fff8580: fd fd fd fd fd fa fa fa fa fa fd fd fd fd fd fd
=>0x0c0e7fff8590: fd fd fd fa fa fa fa fa fd fd[fd]fd fd fd fd fd
  0x0c0e7fff85a0: fd fa fa fa fa fa 00 00 00 00 00 00 00 00 00 fa
  0x0c0e7fff85b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff85c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff85d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff85e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

---- 8< ----

The bugs will be addressed in the followup patches.

MANIFEST.in

-include COPYING README.rst CHANGELOG.rst tox.ini pyproject.toml
+include COPYING README.rst CHANGELOG.rst tox.ini pyproject.toml trun
 include golang/libgolang.h
 include golang/runtime/libgolang.cpp
 recursive-include golang    *.py *.pxd *.pyx *.toml

setup.py

@@ -28,23 +28,19 @@ from setuptools.command.develop import develop as _develop
 from os.path import dirname, join
 import sys, re
 
+# read file content
+def readfile(path):
+    with open(path, 'r') as f:
+        return f.read()
+
 # reuse golang.pyx.build to build pygolang extensions.
 # we have to be careful and inject synthetic golang package in order to be
 # able to import golang.pyx.build without built/working golang.
-import imp, pkgutil
-golang = imp.new_module('golang')
-golang.__package__ = 'golang'
-golang.__path__    = ['golang']
-golang.__file__    = 'golang/__init__.py'
-golang.__loader__  = pkgutil.ImpLoader('golang', None, 'golang/__init__.py',
-                                       [None, None, imp.PY_SOURCE])
-sys.modules['golang'] = golang
+trun = {}
+exec(readfile('trun'), trun)
+trun['ximport_empty_golangmod']()
 from golang.pyx.build import setup, Extension as Ext
 
-# read file content
-def readfile(path):
-    with open(path, 'r') as f:
-        return f.read()
 
 # grep searches text for pattern.
 # return re.Match object or raises if pattern was not found.

tox.ini

 [tox]
-envlist = {py27,py36,py37,pypy,pypy3}-{thread,gevent}
+envlist =
+    {      py27,py36,py37d,py37,pypy,pypy3}-{thread,gevent}
+# XXX gevent fails to install with python2-dbg (https://github.com/gevent/gevent/issues/1461)
+    {py27d                                }-{thread       }
+
+
+# ThreadSanitizer
+
+# XXX under pypy tsan does not observe the GIL lock/release(*), and this
+# way reports even plain T1.Py_INCREF / T2.Py_DECREF as a race.
+# (*) PyPy locks its GIL (see RPyGilAcquire) by manually doing atomic cmpxchg
+# and other games, which TSAN cannot see if PyPy itself was not compiled with
+# -fsanitize=thread.
+    {      py27,py36,py37d,py37           }-{thread       }-tsan
+# XXX see ^^^ about gevent vs python2-dbg
+    {py27d                                }-{thread       }-tsan
+# XXX py*-gevent-tsan would be nice to have, but at present TSAN is not
+# effective with gevent, because it does not understand greenlet "thread"
+# switching and so perceives the program as having only one thread where races
+# are impossible. Disabled to save time.
+#   {      py27,py36,py37d,py37           }-{       gevent}-tsan
+
+
+# AddressSanitizer
+
+# XXX asan does not work with gevent: https://github.com/python-greenlet/greenlet/issues/113
+    {py27d,py27,py36,py37d,py37,pypy,pypy3}-{thread       }-asan
 
 [testenv]
+basepython =
+    py27d:  python2.7-dbg
+    py27:   python2.7
+    py36:   python3.6
+    py37d:  python3.7-dbg
+    py37:   python3.7
+    pypy:   pypy
+    pypy3:  pypy3
+
+setenv =
+# distutils take CFLAGS for both C and C++.
+# distutils use  CFLAGS also at link stage -> we don't need to set LDFLAGS separately.
+    tsan: CFLAGS=-g -fsanitize=thread
+    asan: CFLAGS=-g -fsanitize=address
+# XXX however distutils' try_link, which is used by numpy.distutils use only CC
+# as linker without CFLAGS and _without_ LDFLAGS, which fails if *.o were
+# compiled with -fsanitize=X and linked without that option. Work it around
+# with also adjusting CC.
+# XXX better arrange to pass CFLAGS to pygolang only, e.g. by adding --race or
+# --sanitize=thread to `setup.py build_ext`.
+    tsan: CC=cc -fsanitize=thread
+    asan: CC=cc -fsanitize=address
+
+# always compile pygolang from source and don't reuse binary pygolang wheels as
+# we compile each case with different CFLAGS.
+install_command =
+    python -m pip install --no-binary pygolang {opts} {packages}
+
 deps    =
     .[all]
 
@@ -10,5 +64,12 @@ deps    =
 changedir = {envsitepackagesdir}
 
 commands=
-    thread: {envpython} -m pytest gpython/ golang/
-    gevent: gpython     -m pytest gpython/ golang/
+    {toxinidir}/trun        \
+    thread: {envpython}     \
+    gevent: gpython         \
+        -m pytest           \
+# asan/tsan: tell pytest not to capture output - else it is not possible to see
+# reports from sanitizers because they crash tested process on error.
+# likewise for python debug builds.
+        asan,tsan,py{27,37}d: -s    \
+        gpython/ golang/

trun

+#!/usr/bin/env python
+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+""" `trun ...` - run `...` while testing pygolang
+
+For example it is not possible to import sanitized libgolang.so if non-sanitized
+python was used to start tests - it will fail with
+
+    ImportError: /usr/lib/x86_64-linux-gnu/libtsan.so.0: cannot allocate memory in static TLS block
+
+trun cares to run python with LD_PRELOAD set appropriately to /path/to/libtsan.so
+"""
+
+# TODO integrate startup code into e.g. `gpython -race` which should arrange
+# for _golang.xxx-race.so to be loaded instead of _golang.xxx.so, and also
+# arrange for initial $LD_PRELOAD. Always build both _golang.xxx.so and
+# _golang.xxx-race.so as part of standard pygolang build.
+
+from __future__ import print_function, absolute_import
+
+import os, sys, re, subprocess, imp, pkgutil
+PY3 = (bytes is not str)
+
+# env_prepend prepends value to ${name} environment variable.
+#
+# the value is prepended with " " separator.
+# the value is prepended - not appended - so that defaults set by trun can be overridden by user.
+def env_prepend(name, value):
+    _ = os.environ.get(name, "")
+    if _ != "":
+        value += " " + _
+    os.environ[name] = value
+
+# grep1 searches for the first line, that matches pattern, from text.
+def grep1(pattern, text): # -> re.Match|None
+    p = re.compile(pattern)
+    for l in text.splitlines():
+        m = p.search(l)
+        if m is not None:
+            return m
+    return None
+
+# ximport_empty_golangmod injects synthetic golang package in order to be able
+# to import e.g. golang.pyx.build, or locate golang._golang, without built/working golang.
+def ximport_empty_golangmod():
+    assert 'golang' not in sys.modules
+    golang = imp.new_module('golang')
+    golang.__package__ = 'golang'
+    golang.__path__    = ['golang']
+    golang.__file__    = 'golang/__init__.py'
+    golang.__loader__  = pkgutil.ImpLoader('golang', None, 'golang/__init__.py',
+                                           [None, None, imp.PY_SOURCE])
+    sys.modules['golang'] = golang
+
+
+def main():
+    # install synthetic golang package so that we can import golang.X even if
+    # golang._golang (which is imported by golang/__init__.py) is not built/functional.
+    # Use golang.pyx.build._findpkg to locate _golang.so that corresponds to our python.
+    ximport_empty_golangmod()
+    from golang.pyx import build
+    _golang_so = build._findpkg('golang._golang')
+
+    # determine if _golang.so is linked to a sanitizer, and if yes, to which
+    # particular sanitizer DSO. Set LD_PRELOAD appropriately.
+    ld_preload = None
+    if 'linux' in sys.platform:
+        p = subprocess.Popen(["ldd", _golang_so.path], stdout=subprocess.PIPE)
+        out, _ = p.communicate()
+        if PY3:
+            out = out.decode('utf-8')
+        _ = grep1(r"lib.san\.so\.. => ([^\s]+)", out)
+        if _ is not None:
+            ld_preload = ("LD_PRELOAD", _.group(1))
+
+    elif 'darwin' in sys.platform:
+        # on darwin there is no ready out-of-the box analog of ldd, but
+        # sanitizer runtimes print instruction what to preload, e.g.
+        #   ==973==ERROR: Interceptors are not working. This may be because ThreadSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
+        #   DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.tsan_osx_dynamic.dylib
+        #   "interceptors not installed" && 0./test.sh: line 6:   973 Abort trap: 6           ./trun python -m pytest "$@"
+        # try to `import golang` to retrieve that.
+        p = subprocess.Popen(["python", "-c", "import golang"], stderr=subprocess.PIPE)
+        _, err = p.communicate()
+        if p.returncode != 0:
+            if PY3:
+                err = err.decode('utf-8')
+
+            _ = grep1("DYLD_INSERT_LIBRARIES=(.*)$", err)
+            if _ is not None:
+                ld_preload = ("DYLD_INSERT_LIBRARIES", _.group(1))
+            else:
+                print("trun %r: `import golang` failed with unexpected error:" % sys.argv[1:], file=sys.stderr)
+                print(err, file=sys.stderr)
+                sys.exit(2)
+
+    # ld_preload has e.g. ("LD_PRELOAD", "/usr/lib/x86_64-linux-gnu/libtsan.so.0")
+    if ld_preload is not None:
+        #print('env <-', ld_preload)
+        env_prepend(*ld_preload)
+
+    # $LD_PRELOAD setup; ready to exec `...`
+
+    # if TSAN/ASAN detects a bug - make it fail loudly on the first bug
+    env_prepend("TSAN_OPTIONS", "halt_on_error=1")
+    env_prepend("ASAN_OPTIONS", "halt_on_error=1")
+
+    # tweak TSAN/ASAN defaults:
+
+    # enable TSAN deadlock detector
+    # (unfortunately it caughts only few _potential_ deadlocks and actually
+    # gets stuck on any real deadlock)
+    env_prepend("TSAN_OPTIONS", "detect_deadlocks=1")
+    env_prepend("TSAN_OPTIONS", "second_deadlock_stack=1")
+
+    # many python allocations, whose lifetime coincides with python interpreter
+    # lifetime and which are not explicitly freed on python shutdown, are
+    # reported as leaks. Disable leak reporting to avoid huge non-pygolang
+    # related printouts.
+    env_prepend("ASAN_OPTIONS", "detect_leaks=0")
+
+    # tune ASAN to check more aggressively by default
+    env_prepend("ASAN_OPTIONS", "detect_stack_use_after_return=1")
+
+    # exec `...`
+    os.execvp(sys.argv[1], sys.argv[1:])
+
+
+if __name__ == '__main__':
+    main()