1. 17 Sep, 2019 1 commit
    • Kirill Smelkov's avatar
      tox += ThreadSanitizer, AddressSanitizer, Python debug builds · 4dc1a7f0
      Kirill Smelkov authored
      - ThreadSanitizer helps to detect races and some memory errors,
      - AddressSanitizer helps to detect memory errors,
      - Python debug builds help to detect e.g reference counting errors.
      Adding all those tools to testing coverage discovers e.g. the following
      bugs (not a full list):
      ---- 8< ----
      WARNING: ThreadSanitizer: data race (pid=7143)
        Write of size 8 at 0x7b1400000650 by main thread:
          #0 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:649 (libtsan.so.0+0x2b46a)
          #1 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:643 (libtsan.so.0+0x2b46a)
          #2 golang::_chan::decref() golang/runtime/libgolang.cpp:470 (liblibgolang.so.0.1+0x47f2)
          #3 _chanxdecref golang/runtime/libgolang.cpp:452 (liblibgolang.so.0.1+0x484a)
          #4 _test_go_c golang/runtime/libgolang_test_c.c:86 (_golang_test.so+0x13a2e)
          #5 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
          #6 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
          #7 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)
        Previous read of size 8 at 0x7b1400000650 by thread T8:
          #0 golang::Sema::acquire() golang/runtime/libgolang.cpp:164 (liblibgolang.so.0.1+0x410a)
          #1 golang::Mutex::lock() golang/runtime/libgolang.cpp:175 (liblibgolang.so.0.1+0x4c82)
          #2 golang::_chan::close() golang/runtime/libgolang.cpp:754 (liblibgolang.so.0.1+0x4c82)
          #3 _chanclose golang/runtime/libgolang.cpp:732 (liblibgolang.so.0.1+0x4d1a)
          #4 _work golang/runtime/libgolang_test_c.c:92 (_golang_test.so+0x136cc)
          #5 <null> <null> (python2.7+0x1929e3)
        Thread T8 (tid=7311, finished) created by main thread at:
          #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:915 (libtsan.so.0+0x2be1b)
          #1 PyThread_start_new_thread <null> (python2.7+0x19299f)
          #2 _taskgo golang/runtime/libgolang.cpp:119 (liblibgolang.so.0.1+0x3f68)
          #3 _test_go_c golang/runtime/libgolang_test_c.c:84 (_golang_test.so+0x13a1c)
          #4 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
          #5 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
          #6 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)
      ==22205==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000002cd0 at pc 0x7fd3732a7679 bp 0x7fd3723c8c50 sp 0x7fd3723c8c48
      READ of size 8 at 0x607000002cd0 thread T7
          #0 0x7fd3732a7678 in golang::Sema::acquire() golang/runtime/libgolang.cpp:164
          #1 0x7fd3732a8644 in golang::Mutex::lock() golang/runtime/libgolang.cpp:175
          #2 0x7fd3732a8644 in golang::_chan::close() golang/runtime/libgolang.cpp:754
          #3 0x7fd3724004b2 in golang::chan<golang::structZ>::close() const golang/libgolang.h:323
          #4 0x7fd3724004b2 in operator() golang/runtime/libgolang_test.cpp:262
          #5 0x7fd3724004b2 in __invoke_impl<void, _test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:60
          #6 0x7fd3724004b2 in __invoke<_test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:95
          #7 0x7fd3724004b2 in __call<void> /usr/include/c++/8/functional:400
          #8 0x7fd3724004b2 in operator()<> /usr/include/c++/8/functional:484
          #9 0x7fd3724004b2 in _M_invoke /usr/include/c++/8/bits/std_function.h:297
          #10 0x7fd3723fdc6e in std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687
          #11 0x7fd3723fdc6e in operator() golang/libgolang.h:273
          #12 0x7fd3723fdc6e in _FUN golang/libgolang.h:271
          #13 0x62ddf3  (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62ddf3)
          #14 0x7fd377393fa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
          #15 0x7fd376eda4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)
      0x607000002cd0 is located 16 bytes inside of 72-byte region [0x607000002cc0,0x607000002d08)
      freed by thread T0 here:
          #0 0x7fd377519fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0)
          #1 0x7fd372401335 in golang::chan<golang::structZ>::~chan() golang/libgolang.h:292
          #2 0x7fd372401335 in _test_chan_vs_stackdeadwhileparked() golang/runtime/libgolang_test.cpp:222
      previously allocated by thread T0 here:
          #0 0x7fd37751a518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518)
          #1 0x7fd3732a7d0b in zalloc golang/runtime/libgolang.cpp:1185
          #2 0x7fd3732a7d0b in _makechan golang/runtime/libgolang.cpp:413
      Thread T7 created by T0 here:
          #0 0x7fd377481db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
          #1 0x62df39 in PyThread_start_new_thread (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62df39)
      SUMMARY: AddressSanitizer: heap-use-after-free golang/runtime/libgolang.cpp:164 in golang::Sema::acquire()
      Shadow bytes around the buggy address:
        0x0c0e7fff8540: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
        0x0c0e7fff8550: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
        0x0c0e7fff8560: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
        0x0c0e7fff8570: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
        0x0c0e7fff8580: fd fd fd fd fd fa fa fa fa fa fd fd fd fd fd fd
      =>0x0c0e7fff8590: fd fd fd fa fa fa fa fa fd fd[fd]fd fd fd fd fd
        0x0c0e7fff85a0: fd fa fa fa fa fa 00 00 00 00 00 00 00 00 00 fa
        0x0c0e7fff85b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff85c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff85d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff85e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07
        Heap left redzone:       fa
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      ---- 8< ----
      The bugs will be addressed in the followup patches.
  2. 26 Aug, 2019 1 commit
    • Kirill Smelkov's avatar
      setup: Provide golang.X requirements in pygolang[X] · 89a1061a
      Kirill Smelkov authored
      ... and everything that whole pygolang depends on in pygolang[all].
      The reason we do this: currently golang.perf.benchlib needs NumPy, but
      it would be not ok to depend whole pygolang on numpy. Similarly pybench
      needs pytest, but it would be not ok to require pytest unconditionally.
      Thus allow users to specify what they need from pygolang via extras,
      like pygolang[x.perf.benchlib] or pygolang[cmd.pybench], with
      pygolang[all] serving as merge point for all subpackages dependencies.
      TODO in the future we'll, hopefully, also provide accumulation points,
      like pygolang[x.perf] which contains everything pygolang[x.perf.*]
      depends on.
      Note: core functionality and gpython are provided always unconditionally.
  3. 08 Jul, 2019 1 commit
  4. 24 May, 2019 1 commit
    • Kirill Smelkov's avatar
      Revert "tox -= Python3.6" · 08ec7950
      Kirill Smelkov authored
      This reverts commit 469f21a9.
      Even though Debian stopped shipping Python3.6 I'm now building it myself
      to have test coverage for two latest Python releases.
  5. 19 Mar, 2019 2 commits
    • Kirill Smelkov's avatar
      tox -= Python3.6 · 469f21a9
      Kirill Smelkov authored
      Debian dropped Python3.6 from python3-distutils. Without it I'm no
      longer being able to run tests on that python version:
      	(neo) (z-dev) (g.env) kirr@deco:~/src/tools/go/pygolang$ tox -e py36-thread
      	execution failed:  -- Traceback (most recent call last):
      	  File "<string>", line 1, in <module>
      	ModuleNotFoundError: No module named 'distutils.sysconfig'
      	GLOB sdist-make: /home/kirr/src/tools/go/pygolang/setup.py
      	py36-thread create: /home/kirr/src/tools/go/pygolang/.tox/py36-thread
      	ERROR: invocation failed (exit code 1), logfile: /home/kirr/src/tools/go/pygolang/.tox/py36-thread/log/py36-thread-0.log
      	ERROR: actionid: py36-thread
      	msg: getenv
      	cmdargs: '/usr/bin/python3 -m virtualenv --python /usr/bin/python3.6 py36-thread'
      	Traceback (most recent call last):
      	  File "/usr/lib/python3/dist-packages/virtualenv.py", line 25, in <module>
      	    import distutils.sysconfig
      	ModuleNotFoundError: No module named 'distutils.sysconfig'
      	Running virtualenv with interpreter /usr/bin/python3.6
      	ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python3 -m virtualenv --python /usr/bin/python3.6 py36-thread (see /home/kirr/src/tools/go/pygolang/.tox/py36-thread/log/py36-thread-0.log)', 1)
    • Kirill Smelkov's avatar
      tox += PyPy3 · da68a8ae
      Kirill Smelkov authored
      Debian started to ship PyPy3:
  6. 16 Jan, 2019 1 commit
  7. 31 Oct, 2018 1 commit
  8. 02 Jul, 2018 1 commit