1. 19 May, 2020 1 commit
    • Kirill Smelkov's avatar
      golang: tests: Add tests for IPython and Pytest integration patches · 09629367
      Kirill Smelkov authored
      bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on
      Python2) added integration patches for IPython and Pytest to properly
      dump tracebacks for chained exceptions even on Python2. However the
      functionality of patches was tested only manually.
      
      -> Add corresponding tests to verify how IPython and Pytest behaves
      when dumping tracebacks.
      09629367
  2. 12 Feb, 2020 1 commit
  3. 04 Feb, 2020 5 commits
  4. 06 Dec, 2019 1 commit
  5. 27 Nov, 2019 2 commits
    • Kirill Smelkov's avatar
      strings: New package · 0efd4a9a
      Kirill Smelkov authored
      Provide string utilities to verify whether string has suffix/prefix,
      trim it, and split string by a delimiter. The code originated in wcfs
      codebase in wendelin.core .
      
      Pyx/nogil only.
      0efd4a9a
    • Kirill Smelkov's avatar
      fmt: New package · 309963f8
      Kirill Smelkov authored
      With fmt.sprintf() and fmt.errorf() to format strings and errors. The
      code was extracted from wcfs in wendelin.core .
      
      Pyx/nogil only.
      309963f8
  6. 19 Nov, 2019 1 commit
    • Kirill Smelkov's avatar
      time: Factor-out PyFunc into shared library libpyxruntime.so · 4fc6e49c
      Kirill Smelkov authored
      Commit b073f6df (time: Move/Port timers to C++/Pyx nogil) added C++
      level PyFunc as a way to call Python function from under nogil code.
      We will soon need this functionality for sync.WorkGroup too, so move it
      into shared place.
      
      We move into regular library - not a pyx module (e.g. golang.pyx) -
      because for pyx modules linking is done at import time by specially
      generated code, while we need the linker support to automatically
      resolve e.g. calls to PyFunc::operator() from inside std::function.
      4fc6e49c
  7. 14 Nov, 2019 3 commits
    • Kirill Smelkov's avatar
      context: Move/Port context package to C++/Pyx nogil · 2a359791
      Kirill Smelkov authored
      Provide context-related functionality that can be used directly from C++
      and Pyx/nogil codes. Python-level classes and functions become small
      wrappers around pyx/nogil ones.
      
      Like with timers (b073f6df "time: Move/Port timers to C++/Pyx nogil")
      and interfaces (5a99b769 "libgolang: Start providing interfaces")
      memory for objects dynamically allocated on heap is managed
      automatically.
      2a359791
    • Kirill Smelkov's avatar
      cxx: New package · 9785f2d3
      Kirill Smelkov authored
      Start new package that provides C++ amendments to be used by libgolang
      and its users. Current functionality: dict and set that wrap
      std::unordered_map and std::unordered_set into ergonomic interface.
      
      The code originates from wendelin.core:
      https://lab.nexedi.com/kirr/wendelin.core/blob/5a045ed1/wcfs/internal/wcfs_misc.h#L214-257
      
      Pyx/nogil only.
      9785f2d3
    • Kirill Smelkov's avatar
      errors: New package · a245ab56
      Kirill Smelkov authored
      With just one function - errors.New() - to create new error with given
      text.
      
      This will be soon used in Pyx/nogil context package to create context.canceled
      and context.deadlineExceeded errors.
      
      Pyx/nogil only for now.
      a245ab56
  8. 07 Nov, 2019 1 commit
    • Kirill Smelkov's avatar
      libgolang: Split time and sync into their own packages · 1a9dae3b
      Kirill Smelkov authored
      We already have some sync functionality implemented in C++ (e.g.
      sync.Once, sync.WaitGroup) and we are going to add more and also move
      timers implementation to C++. It is getting crowded for that
      functionality to still live in libgolang.{h,cpp}
      
      -> Split sync & time functionality into their own C++ packages (still
      built-into libgolang.so)
      1a9dae3b
  9. 18 Oct, 2019 1 commit
    • Kirill Smelkov's avatar
      golang: Teach defer to chain exceptions (PEP 3134) even on Python2 · bb9a94c3
      Kirill Smelkov authored
      Python3 chains exceptions, so that e.g. if exc1 is raised and, while it
      was not handled, another exc2 is raised, exc2 will be linked to exc1 via
      exc2.__context__ attribute and exc1 will be included into exc2 traceback
      printout. However many projects still use Python2 and there is no
      similar chaining functionality there. This way exc1 is completely lost.
      
      Since defer code is in our hands, we can teach it to implement exception
      chaining even on Python2 by carefully analyzing what happens in
      _GoFrame.__exit__().
      
      Implementing chaining itself is relatively easy, but is only part of the
      story. Even if an exception is chained with its cause, but exception
      dump does not show the cause, the chaining will be practically useless.
      With this in mind this patches settles not only on implementing chaining
      itself, but on also giving a promise that chained cause exceptions will
      be included into traceback dumps as well.
      
      To realize this promise we adjust all exception dumping funcitons in
      traceback module and carefully install adjusted
      traceback.print_exception() into sys.excepthook. This amends python
      interactive sessions and programs run by python interpreter to include
      causes in exception dumps. "Careful" here means that we don't change
      sys.excepthook if on golang module load we see that sys.excepthook was already
      changed by some other module - e.g. due to IPython session running
      because IPython installs its own sys.excepthook. In such cases we don't
      install our sys.excepthook, but we also provide integration patches that
      add exception chaining support for traceback dump functionality in
      popular third-party software. The patches (currently for IPython and
      Pytest) are activated automatically, but only when/if corresponding
      software is imported and actually used. This should give practically
      good implementation of the promise - a user can now rely on seeing
      exception cause in traceback dump whatever way python programs are run.
      
      The implementation takes https://pypi.org/project/pep3134/ experience
      into account [1]. peak.utils.imports [2,3] is used to be notified when/if
      third-party module is imported.
      
      [1] https://github.com/9seconds/pep3134/
      [2] https://pypi.org/project/Importing/
      [3] http://peak.telecommunity.com/DevCenter/Importing
      
      This patch originally started as hacky workaround in wendelin.core
      because in wcfs tests I was frequently hitting situations, where
      exception raised by an assert was hidden by another exception raised in
      further generic teardown check. For example wcfs tests check that wcfs
      is unmounted after every test run [4] and if that fails it was hiding
      problems raised by an assert. As the result I was constantly guessing
      and adding code like [5] to find what was actually breaking. At some
      point I added hacky workaround for defer to print cause exception not to
      loose it [6]. [7] has more context and background discussion on this topic.
      
      [4] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L70
      [5] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L853-857
      [6] kirr/wendelin.core@c00d94c7
      [7] zodbtools!13 (comment 81553)
      
      After this patch, on Python2
      
          defer(cleanup1)
          defer(cleanup2)
          defer(cleanup3)
          ...
      
      is no longer just a syntatic sugar for
      
          try:
              try:
                  try:
                      ...
                  finally:
                      cleanup3()
              finally:
                  cleanup2()
          finally:
              cleanup1()
      bb9a94c3
  10. 17 Sep, 2019 1 commit
    • Kirill Smelkov's avatar
      tox += ThreadSanitizer, AddressSanitizer, Python debug builds · 4dc1a7f0
      Kirill Smelkov authored
      - ThreadSanitizer helps to detect races and some memory errors,
      - AddressSanitizer helps to detect memory errors,
      - Python debug builds help to detect e.g reference counting errors.
      
      Adding all those tools to testing coverage discovers e.g. the following
      bugs (not a full list):
      
      ---- 8< ----
      
      py27-thread-tsan:
      WARNING: ThreadSanitizer: data race (pid=7143)
        Write of size 8 at 0x7b1400000650 by main thread:
          #0 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:649 (libtsan.so.0+0x2b46a)
          #1 free ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:643 (libtsan.so.0+0x2b46a)
          #2 golang::_chan::decref() golang/runtime/libgolang.cpp:470 (liblibgolang.so.0.1+0x47f2)
          #3 _chanxdecref golang/runtime/libgolang.cpp:452 (liblibgolang.so.0.1+0x484a)
          #4 _test_go_c golang/runtime/libgolang_test_c.c:86 (_golang_test.so+0x13a2e)
          #5 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
          #6 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
          #7 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)
      
        Previous read of size 8 at 0x7b1400000650 by thread T8:
          #0 golang::Sema::acquire() golang/runtime/libgolang.cpp:164 (liblibgolang.so.0.1+0x410a)
          #1 golang::Mutex::lock() golang/runtime/libgolang.cpp:175 (liblibgolang.so.0.1+0x4c82)
          #2 golang::_chan::close() golang/runtime/libgolang.cpp:754 (liblibgolang.so.0.1+0x4c82)
          #3 _chanclose golang/runtime/libgolang.cpp:732 (liblibgolang.so.0.1+0x4d1a)
          #4 _work golang/runtime/libgolang_test_c.c:92 (_golang_test.so+0x136cc)
          #5 <null> <null> (python2.7+0x1929e3)
      
        Thread T8 (tid=7311, finished) created by main thread at:
          #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors.cc:915 (libtsan.so.0+0x2be1b)
          #1 PyThread_start_new_thread <null> (python2.7+0x19299f)
          #2 _taskgo golang/runtime/libgolang.cpp:119 (liblibgolang.so.0.1+0x3f68)
          #3 _test_go_c golang/runtime/libgolang_test_c.c:84 (_golang_test.so+0x13a1c)
          #4 __pyx_pf_6golang_12_golang_test_12test_go_c golang/_golang_test.cpp:3340 (_golang_test.so+0xcbaa)
          #5 __pyx_pw_6golang_12_golang_test_13test_go_c golang/_golang_test.cpp:3305 (_golang_test.so+0xcbaa)
          #6 PyEval_EvalFrameEx <null> (python2.7+0xf68b4)
      
      py37-thread-asan:
      ==22205==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000002cd0 at pc 0x7fd3732a7679 bp 0x7fd3723c8c50 sp 0x7fd3723c8c48
      READ of size 8 at 0x607000002cd0 thread T7
          #0 0x7fd3732a7678 in golang::Sema::acquire() golang/runtime/libgolang.cpp:164
          #1 0x7fd3732a8644 in golang::Mutex::lock() golang/runtime/libgolang.cpp:175
          #2 0x7fd3732a8644 in golang::_chan::close() golang/runtime/libgolang.cpp:754
          #3 0x7fd3724004b2 in golang::chan<golang::structZ>::close() const golang/libgolang.h:323
          #4 0x7fd3724004b2 in operator() golang/runtime/libgolang_test.cpp:262
          #5 0x7fd3724004b2 in __invoke_impl<void, _test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:60
          #6 0x7fd3724004b2 in __invoke<_test_chan_vs_stackdeadwhileparked()::<lambda()>&> /usr/include/c++/8/bits/invoke.h:95
          #7 0x7fd3724004b2 in __call<void> /usr/include/c++/8/functional:400
          #8 0x7fd3724004b2 in operator()<> /usr/include/c++/8/functional:484
          #9 0x7fd3724004b2 in _M_invoke /usr/include/c++/8/bits/std_function.h:297
          #10 0x7fd3723fdc6e in std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687
          #11 0x7fd3723fdc6e in operator() golang/libgolang.h:273
          #12 0x7fd3723fdc6e in _FUN golang/libgolang.h:271
          #13 0x62ddf3  (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62ddf3)
          #14 0x7fd377393fa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
          #15 0x7fd376eda4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)
      
      0x607000002cd0 is located 16 bytes inside of 72-byte region [0x607000002cc0,0x607000002d08)
      freed by thread T0 here:
          #0 0x7fd377519fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0)
          #1 0x7fd372401335 in golang::chan<golang::structZ>::~chan() golang/libgolang.h:292
          #2 0x7fd372401335 in _test_chan_vs_stackdeadwhileparked() golang/runtime/libgolang_test.cpp:222
      
      previously allocated by thread T0 here:
          #0 0x7fd37751a518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518)
          #1 0x7fd3732a7d0b in zalloc golang/runtime/libgolang.cpp:1185
          #2 0x7fd3732a7d0b in _makechan golang/runtime/libgolang.cpp:413
      
      Thread T7 created by T0 here:
          #0 0x7fd377481db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
          #1 0x62df39 in PyThread_start_new_thread (/home/kirr/src/tools/go/pygolang-master/.tox/py37-thread-asan/bin/python3+0x62df39)
      
      SUMMARY: AddressSanitizer: heap-use-after-free golang/runtime/libgolang.cpp:164 in golang::Sema::acquire()
      Shadow bytes around the buggy address:
        0x0c0e7fff8540: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
        0x0c0e7fff8550: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
        0x0c0e7fff8560: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
        0x0c0e7fff8570: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
        0x0c0e7fff8580: fd fd fd fd fd fa fa fa fa fa fd fd fd fd fd fd
      =>0x0c0e7fff8590: fd fd fd fa fa fa fa fa fd fd[fd]fd fd fd fd fd
        0x0c0e7fff85a0: fd fa fa fa fa fa 00 00 00 00 00 00 00 00 00 fa
        0x0c0e7fff85b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff85c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff85d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff85e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07
        Heap left redzone:       fa
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      
      ---- 8< ----
      
      The bugs will be addressed in the followup patches.
      4dc1a7f0
  11. 29 Aug, 2019 3 commits
    • Kirill Smelkov's avatar
      check-manifest · 59892def
      Kirill Smelkov authored
      As suggested by https://github.com/jayvdb run manifest checker and oops,
      it finds that we forgot to include pyproject.toml for testprog/golang_pyx_user:
      
          missing from sdist:
            golang/pyx/testprog/golang_pyx_user/pyproject.toml
          suggested MANIFEST.in rules:
            recursive-include golang *.toml
      
      -> Fix MANIFEST.in in generic way to include golang/*/*.toml
      
      The bug was not affecting pygolang usage - only golang.pyx.build tests were
      failing for sdist installed pygolang.
      59892def
    • Kirill Smelkov's avatar
      Hook in list.h from Linux · 9efb6575
      Kirill Smelkov authored
      Copy linux/list.h from wendelin.core which copied it from
      util-linux.git which took it from linux.git at LGPL license state.
      
      Here are corresponding links for wendelin.core and util-linux:
      
          wendelin.core@a0f940ad
          https://git.kernel.org/cgit/utils/util-linux/util-linux.git/tree/include/list.h?id=v2.25-165-g9138d6f
      
      Linked lists will be used in channels implementation in the next patch.
      9efb6575
    • Kirill Smelkov's avatar
      Start using Cython and providing Cython/nogil API · 8fa3c15b
      Kirill Smelkov authored
      Start transforming Pygolang from being pure Python project into project
      that uses both Python and Cython and provides both Python and Cython API
      to its users.
      
      For Cython API there is a goal for it to be usable independently of
      Python GIL - i.e. independently of Python runtime and limitations
      imposed by Python's global interpreter lock. The motivation for this
      work is wendelin.core v2, which, due to its design, would deadlock if it
      tries to take the GIL in its pinner thread.
      
      This commit brings bare minimum establishing libraries and build system.
      It:
      
      - Introduces C/C++ library libgolang. Libgolang will be serving most of
        nogil functionality and Pyx API will be a small wrapper over it. At
        this step Libgolang comes only with C-level panic;
      
      - Introduces Pyx package golang that is aliased to Pyx package
        golang._golang . Users can do `from golang cimport ...` similarly to how
        they do py-level `from golang import ...`. At this step golang.pyx
        only wraps panic and provides way to transform C-level panic into Python
        exception;
      
      - Introduces Py package golang.pyx.build . This package, similarly to
        golang.org/pkg/go/build, serves as the system to build Pyx packages -
        both external packages that use Pygolang in pyx mode, and to build
        Pygolang itself. The build, internally, is served by setuptools_dso
        and cython. An example how to use golang.pyx.build from external
        project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
        tests to verify golang.pyx.build functionality;
      
      - Introduces Pygolang build dependency to cython and setuptools_dso.
      8fa3c15b
  12. 26 Aug, 2019 1 commit
  13. 16 May, 2019 1 commit
    • Kirill Smelkov's avatar
      golang: Don't keep test programs inside golang/ dir · 6e3b3ff4
      Kirill Smelkov authored
      Python adds dirname of run program to sys.path . This way when
      golang_test_goleaked.py runs it can import modules located under golang/
      just by their name. Until now this was not noticed, but in the next
      patch we are going to add golang.time module and if test program is run
      with golang/ in sys.path just plain `import time` won't import time from
      stdlib and instead import time from golang/ .
      
      Such behaviour can be mitigated by doing `from __future__ import
      absolute_import` and we do that, including in golang_test_goleaked.py
      (see 81dfefa0 "*: __future__ += absolute_imports; Use unified __future__
      everywhere"). However that does not prevent modules - even modules from
      stdlib - who are doing `import time` and not doing future
      absolute_import to import golang's time instead of stdlib. For example
      on PyPy2 threading imports time and then the test breaks:
      
        Traceback (most recent call last):
          File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/golang_test_goleaked.py", line 24, in <module>
            from golang import go, chan
          File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/__init__.py", line 38, in <module>
            import inspect, threading, collections, random, sys
          File "/usr/lib/pypy/lib-python/2.7/threading.py", line 15, in <module>
            from time import time as _time, sleep as _sleep
          File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/time.py", line 30, in <module>
            from golang import go, chan, select, default, nilchan, panic
        ImportError: cannot import name 'go'
      
      -> Move the test program into a directory different from golang/ to
      avoid this trap.
      6e3b3ff4
  14. 13 Mar, 2019 1 commit
    • Kirill Smelkov's avatar
      gpython: Fix pymain to properly setup sys.path[0] · 6b4990f6
      Kirill Smelkov authored
      Pymain was not adding e.g. directory of executed file to sys.path, and
      as the result if there were e.g. 2 files
      
      	dir/hello.py	# imports world
      	dir/world.py
      
      running `gpython dir/hello.py` would fail to import world.
      
      The case for interactive console was also failing to setup sys.argv as
      empty, so it was containing ['/path/to/gpython']. It was also, contrary
      to standard python, unconditionally showing '>>>' prompt even when stdin
      was not a tty.
      
      Fix all that and add a test to cover pymain functionality.
      6b4990f6
  15. 30 Dec, 2018 1 commit
  16. 02 Jul, 2018 2 commits
  17. 20 Jun, 2018 1 commit
    • Kirill Smelkov's avatar
      Turn pygopath into full pygolang · afa46cf5
      Kirill Smelkov authored
      Not only we can import modules by full path, but now we can also spawn
      threads/coroutines and exchange data in between them with the same
      primitives and semantic as in Go.
      
      The bulk of new functionality is copied from here:
      
      	kirr/go123@9e1aa6ab
      
      Original commit description follows:
      
      """
      golang: New _Python_ package to provide Go-like features to Python language
      - `go` spawns lightweight thread.
      - `chan` and `select` provide channels with Go semantic.
      - `method` allows to define methods separate from class.
      - `gimport` allows to import python modules by full path in a Go workspace.
      
      The focus of first draft was on usage interface and on correctness, not speed.
      In particular select should be fully working.
      
      If there is a chance I will maybe try to followup with gevent-based
      implementation in the future.
      Hide whitespace changes
      """
      afa46cf5
  18. 21 May, 2018 1 commit
    • Kirill Smelkov's avatar
      pygopath: Initial draft · 9c61f254
      Kirill Smelkov authored
      Module gopath provides way to import python modules by full path in a Go workspace.
      
      For example
      
          lonet = gopath.gimport('lab.nexedi.com/kirr/go123/xnet/lonet')
      
      will import either
      
          lab.nexedi.com/kirr/go123/xnet/lonet.py, or
          lab.nexedi.com/kirr/go123/xnet/lonet/__init__.py
      
      located somewhere under $GOPATH.
      9c61f254