1. 29 Aug, 2019 1 commit
    • Kirill Smelkov's avatar
      Start using Cython and providing Cython/nogil API · 8fa3c15b
      Kirill Smelkov authored
      Start transforming Pygolang from being pure Python project into project
      that uses both Python and Cython and provides both Python and Cython API
      to its users.
      
      For Cython API there is a goal for it to be usable independently of
      Python GIL - i.e. independently of Python runtime and limitations
      imposed by Python's global interpreter lock. The motivation for this
      work is wendelin.core v2, which, due to its design, would deadlock if it
      tries to take the GIL in its pinner thread.
      
      This commit brings bare minimum establishing libraries and build system.
      It:
      
      - Introduces C/C++ library libgolang. Libgolang will be serving most of
        nogil functionality and Pyx API will be a small wrapper over it. At
        this step Libgolang comes only with C-level panic;
      
      - Introduces Pyx package golang that is aliased to Pyx package
        golang._golang . Users can do `from golang cimport ...` similarly to how
        they do py-level `from golang import ...`. At this step golang.pyx
        only wraps panic and provides way to transform C-level panic into Python
        exception;
      
      - Introduces Py package golang.pyx.build . This package, similarly to
        golang.org/pkg/go/build, serves as the system to build Pyx packages -
        both external packages that use Pygolang in pyx mode, and to build
        Pygolang itself. The build, internally, is served by setuptools_dso
        and cython. An example how to use golang.pyx.build from external
        project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
        tests to verify golang.pyx.build functionality;
      
      - Introduces Pygolang build dependency to cython and setuptools_dso.
      8fa3c15b
  2. 26 Aug, 2019 8 commits
  3. 23 Aug, 2019 16 commits
    • Kirill Smelkov's avatar
      golang: Add test for blocked select(send|recv) vs close · 02f6991f
      Kirill Smelkov authored
      This test passes, but the functionality was not tested before. In
      particular all tests were passing even with the following hand-edits:
      
          --- a/golang/__init__.py
          +++ b/golang/__init__.py
          @@ -627,7 +627,7 @@ def selected():
                   sel = g.which
                   if isinstance(sel, _SendWaiting):
                       if not sel.ok:
          -                panic("send on closed channel")
          +                panic("send on closed channel ZZZ")
                       return sel.sel_n, None
      
                   if isinstance(sel, _RecvWaiting):
      
      with added tests the bug is caught and reported:
      
          def test_select():
              N = 1000 # times to do repeated select/chan or select/select interactions
      
              # sync: close vs select(send)
              ch = chan()
              def _():
                  waitBlocked(ch.send)
                  ch.close()
              go(_)
      >       with panics("send on closed channel"): select((ch.send, 0))
      
      golang_test.py:353:
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      
      self = <golang.golang_test.panics instance at 0x7fc1da66e5f0>, exc_type = <class 'golang._PanicError'>, exc_val = _PanicError('send on closed channel ZZZ',)
      exc_tb = <traceback object at 0x7fc1dabc33b0>
      
          def __exit__(self, exc_type, exc_val, exc_tb):
              ok = self.raises.__exit__(exc_type, exc_val, exc_tb)
              if not ok:
                  return ok
              # _PanicError raised - let's check panic argument
      >       assert self.exc_info.value.args == (self.arg,)
      E       AssertionError: assert ('send on clo...channel ZZZ',) == ('send on closed channel',)
      E         At index 0 diff: 'send on closed channel ZZZ' != 'send on closed channel'
      E         Use -v to get the full diff
      
      golang_test.py:1032: AssertionError
      02f6991f
    • Kirill Smelkov's avatar
      golang: Fix race in chan._tryrecv · c6bb9eb3
      Kirill Smelkov authored
      For buffered channel _tryrecv, on success, was unlocking ch._mu too
      early - before accessing ch._dataq with ch._dataq.append().
      
      Without the fix, newly added test breaks as e.g.
      
          golang/golang_test.py::test_chan_buf_recv_vs_tryrecv_race Exception in thread Thread-3:
          Traceback (most recent call last):
            File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
              self.run()
            File "/usr/lib/python2.7/threading.py", line 754, in run
              self.__target(*self.__args, **self.__kwargs)
            File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 317, in _
              assert (_, _rx) == (1, None), ('i%d' % i)
          AssertionError: i30
          assert (0, None) == (1, None)
            At index 0 diff: 0 != 1
            Full diff:
            - (0, None)
            ?  ^
            + (1, None)
            ?  ^
      c6bb9eb3
    • Kirill Smelkov's avatar
      golang: Fix race in chan._trysend · eb8a1fef
      Kirill Smelkov authored
      For buffered channel _trysend, on success, was unlocking ch._mu too
      early - before accessing ch._dataq with ch._dataq.popleft().
      
      Without the fix, newly added test breaks as e.g.
      
          golang/golang_test.py::test_chan_buf_send_vs_tryrecv_race Exception in thread Thread-3:
          Traceback (most recent call last):
            File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
              self.run()
            File "/usr/lib/python2.7/threading.py", line 754, in run
              self.__target(*self.__args, **self.__kwargs)
            File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 256, in _
              assert (_, _rx) == (1, None)
          AssertionError: assert (0, 209) == (1, None)
            At index 0 diff: 0 != 1
            Full diff:
            - (0, 209)
            + (1, None)
      
          Exception in thread Thread-2:
          Traceback (most recent call last):
            File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
              self.run()
            File "/usr/lib/python2.7/threading.py", line 754, in run
              self.__target(*self.__args, **self.__kwargs)
            File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 243, in _
              ch.send(i)
            File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 340, in send
              ok = self._trysend(obj)
            File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 417, in _trysend
              rx = self._dataq.popleft()
          IndexError: pop from an empty deque
      eb8a1fef
    • Kirill Smelkov's avatar
      golang: Test that buffered channel releases objects from buffer on chan GC · cb5bfdd2
      Kirill Smelkov authored
      Currently it behaves correctly, but this aspect will need to be
      explicitly cared about when channel implementation moves to C. Add test
      to make sure it won't regress.
      cb5bfdd2
    • Kirill Smelkov's avatar
      golang: Run all select tests "more thoroughly" · c5810987
      Kirill Smelkov authored
      Starting from b51b8d5d (select: Run tests more thoroughly) we are
      running select subtests in repeated mode with N=1000. However not all
      select subtests were run in repeated mode - for example e.g.
      "non-blocking try send: not ok" was being run only once.
      
      -> Rework all select subtests to be run in repeated mode to increase the
      probability of catching bugs.
      c5810987
    • Kirill Smelkov's avatar
      golang: tests: Use panic when testing "blocks forever" · fa667412
      Kirill Smelkov authored
      Channels implementation will soon be moved to C and will become
      independent from Python runtime. This way while pygolang will still be
      providing panic, hooking in and raising arbitrary Python-level exception
      will become problematic. -> Rework "blocks forever" tests to rely on
      just panic to prepare for that.
      fa667412
    • Kirill Smelkov's avatar
      golang: tests: Factor out retrieving len(ch._recvq) and len(ch._sendq) · 352628b5
      Kirill Smelkov authored
      Chan implementation is going to be moved into C, and this way direct
      access to chan internals won't be possible. C implementation will export
      dedicated functions for chan tests which will be used.
      
      Prepare for that and factor out retrieving len of chan recv/send queues
      into separate functions, which will be adapted at the time of C port.
      352628b5
    • Kirill Smelkov's avatar
      golang: Don't verify ._recvq and ._sendq of nil channel · d98e42e3
      Kirill Smelkov authored
      Nil channel will soon be changed to be represented by underlying NULL C
      pointer and there won't be channel object for nil and correspondingly
      recv/send queues won't be there. -> Prepare for that and don't check for
      recvq/sendq on nil channel.
      d98e42e3
    • Kirill Smelkov's avatar
      golang: Remove outdated TODO · 049ba6ff
      Kirill Smelkov authored
      We provide gpython since 32a21d5b (gpython: Python interpreter with
      support for lightweight threads), and golang module, since the
      beginning, automatically uses gevent if it was installed via monkey
      patching.
      049ba6ff
    • Kirill Smelkov's avatar
      time: Test for now · 7f2362dd
      Kirill Smelkov authored
      This currently pass trivially (since time.now = stdtime.now), but will
      be useful once we get time.now implementation into our hands and move it into Cython.
      7f2362dd
    • Kirill Smelkov's avatar
      *: Use golang.time universally · c3bd2c50
      Kirill Smelkov authored
      After 9c260fde (time: New package that mirrors Go's time) we have
      golang.time.now and golang.time.sleep and it makes it a more
      self-dependent system if timing facility is used through golang.time
      instead of outside std time module.
      
      For now this is only a "cleanness" change, but will become important once
      we start adding pyx-level nogil API to pygolang - there it will be
      important to use golang.time.* for correctness.
      c3bd2c50
    • Kirill Smelkov's avatar
      sync: threading.Event -> chan · 78d85cdc
      Kirill Smelkov authored
      Starting from e6bea2cf (sync: New package that mirrors Go's sync) there
      is sync.WaitGroup and it was using threading.Event to signal that
      waitgroup's counter dropped to zero. threading.Event is implemented in
      Python's stdlib via threading.Condition, which in turn uses
      threading.Lock and list of waiters to implement its functionality.
      Which in turn is very similar to what golang.chan does internally with
      semaphore and e.g. recv queue.
      
      I noticed this while debugging sync test deadlock (see previous patch)
      and suspecting bug in threading.Event for a moment. While it turned
      there is no bug in threading.Event, it is better to rely on the common
      functionality for similar tasks, and pygolang's channel perfectly
      matches here the need to signal an event. By using our own code instead
      of stdlib's threading we are likely becoming less bug-prone.
      
      This also brings a small speedup:
      
          (on i5@1.80GHz)
          name             old time/op  new time/op  delta
          workgroup_empty   239µs ± 1%   220µs ± 1%  -7.62%  (p=0.000 n=10+9)
          workgroup_raise   275µs ± 2%   264µs ± 2%  -3.85%  (p=0.000 n=10+10)
      78d85cdc
    • Kirill Smelkov's avatar
      sync.WorkGroup: Fix deadlock thinko in tests · b8b042c5
      Kirill Smelkov authored
      sync.WorkGroup test was doing ctx.done() wait from under test mutex,
      something like
      
          def _(ctx, i):
              with mu:
                  ...
                  if i == 0:
                      raise RuntimeError()    # to cause ctx cancel
                  ctx.done().recv()           # i=1 -> wait till ctx is canceled
      
      but it can be a deadlock if T(i=1) runs first and enters
      ctx.done().recv() before T(i=0) is run - then T(i=0) will block forever
      waiting to lock mu.
      
      This failure was not seen so far, probably because the time to go a new
      thread/goroutine is relatively high. However one of upcoming patches,
      where go is made faster, revealed this problem and, without the fix,
      sync.WorkGroup test was regularly deadlocking.
      
      The problem was there from sync.WorkGroup beginning - from
      9ee7ba91 (sync += WorkGroup).
      
      Fix the deadlock by waiting for ctx.done() outside of mu.
      b8b042c5
    • Kirill Smelkov's avatar
      golang: gimport works on Py3 · 086d4296
      Kirill Smelkov authored
      9c61f254 (pygopath: Initial draft) has a "TODO py3 support", probably
      because https://stackoverflow.com/a/67692 shows different codes for
      Python2 and Python3.
      
      However today we have tox coverage for all Python2.7, Python3.6 and
      Python3.7 and everything works including gimport tests.
      
      -> Remove the TODO.
      086d4296
    • Kirill Smelkov's avatar
      golang/_gopath_test: Redo $GOPATH set/restore via fixture · 7d08f078
      Kirill Smelkov authored
      Currently gimport tests depend on module-level setup_module /
      teardown_module, which require for gimport tests to be in separate file.
      
      Redo the test not to depend on global module-level state and
      setup/teardown. This allows e.g. to merge gimport tests into
      golang_test.py if/when we want/need to.
      7d08f078
    • Kirill Smelkov's avatar
      golang: Test len(nilchan) and repr(nilchan) · 3e5b5f01
      Kirill Smelkov authored
      Should be in 2aad64bb (golang: Add support for nil channel).
      3e5b5f01
  4. 20 Jul, 2019 1 commit
    • Kirill Smelkov's avatar
      *_test: Verify panic argument · 773d8fb2
      Kirill Smelkov authored
      In tests in places where the code checks that something panics, verify
      not only that _PanicError is raised, but also what was the argument
      passed to panic.
      
      This makes sure that tested code panics in expected place, not just
      panics "somewhere".
      
      To keep signal/noise ratio high introduce `panics` which is similar to
      `pytest.raises` and asserts that wrapped code panics with expected
      argument.
      773d8fb2
  5. 08 Jul, 2019 1 commit
  6. 26 Jun, 2019 4 commits
    • Kirill Smelkov's avatar
      sync.WorkGroup: Propagate all exception types, not only those derived from Exception · 79aab7df
      Kirill Smelkov authored
      A problem was hit with pytest.fail with raises Failed exception not
      being propagated to .wait. As it turned out it was not propagated
      because pytest's Failed derives from BaseException, not Exception, and
      we were catching only Exception and its children.
      
      Rework the code to propagate all exception types from workers.
      
      Performance change is with noise (it is either a bit faster for one set
      of runs, or a bit slower for another set of runs).
      79aab7df
    • Kirill Smelkov's avatar
      sync.Workgroup: Don't use @func at runtime · 94c6160b
      Kirill Smelkov authored
      It is true for any decorator, that it makes things faster if
      def+decorator is used globally instead of at runtime, but for @func it
      is especially true since @func, using decorator.decorate, has relatively
      high overhead if it is used not only once at program startup, but
      instead every time something is run.
      
      In particular moving @func out of WorkGroup.go() make things
      considerably faster:
      
      	name             old time/op  new time/op  delta
      	workgroup_empty   195µs ± 0%   113µs ± 1%  -41.91%  (p=0.008 n=5+5)
      	workgroup_raise   221µs ± 1%   137µs ± 1%  -38.29%  (p=0.008 n=5+5)
      
      See
      
      	https://lab.nexedi.com/kirr/misc/raw/009c4fee/pygolang/prof_workgroup_empty.svg
      
      for bench_workgroup_empty profile, where it is seen that
      decorator.decorate was using ~ half of the whole WorkGroup.go() time.
      94c6160b
    • Kirill Smelkov's avatar
      golang: Add benchmarks for chan, select, @func and defer · 3c55ca59
      Kirill Smelkov authored
      Start adding benchmarks to pygolang.
      
      Measure how much sync chan send/recv take and how much select takes for
      synchronous channels. Also measure how much overhead @func adds at both
      def and call times, and the overhead of defer compared to try/finally.
      For std (non-gevent'ed) Python2.7 we are currently at:
      
      	name         time/op
      	go           91.6µs ± 1%
      	chan         13.7µs ± 3%
      	select       30.1µs ± 4%
      	def          55.0ns ± 0%
      	func_def     43.6µs ± 0%
      	call         63.0ns ± 0%
      	func_call    1.06µs ± 0%
      	try_finally   136ns ± 1%
      	defer        2.33µs ± 0%
      3c55ca59
    • Kirill Smelkov's avatar
      golang: test_method -> test_func · c3ea8f51
      Kirill Smelkov authored
      @method(cls) was deprecated and removed in favour of @func(cls) in
      942ee900 (golang: Deprecate @method(cls) in favour of @func(cls)) and
      262f8986 (golang: Kill @method). The test name was overlooked.
      c3ea8f51
  7. 21 Jun, 2019 1 commit
    • Kirill Smelkov's avatar
      golang: Fix `@func(cls) def name` not to override `name` in calling context · 924a808c
      Kirill Smelkov authored
      With @func being a decorator, the following
      
      	@func(cls)
      	def name():
      		...
      
      is always processed by python as
      
      	name = func(cls)(def name(): ...)
      
      Before this patch it was leading to name being overridden with None:
      
      	def f():
      		print 'hello'
      
      	class C:
      		pass
      
      	@func(C)
      	def f(c):
      		print 'C.f', c
      
      	f()
      	Traceback (most recent call last):
      	  File "<console>", line 1, in <module>
      	TypeError: 'NoneType' object is not callable
      
      We can fix it by returning from `func(cls)(def name(): ...)` the
      original `name` object from the calling context.
      
      Unfortunately if `name` was not previously set I did not find a way(*) to
      avoid polluting the calling namespace where it is set to what @func(cls)
      returns (None) by the hardcoded way how python processes decorators:
      
      	In [2]: c = """
      	   ...: @fff
      	   ...: def ccc():
      	   ...:     return 1
      	   ...: """
      
      	In [3]: cc = compile(c, "file", "exec")
      
      	In [4]: dis(cc)
      	  2           0 LOAD_NAME                0 (fff)
      	              3 LOAD_CONST               0 (<code object ccc at 0x7fafe58d0130, file "file", line 2>)
      	              6 MAKE_FUNCTION            0
      	              9 CALL_FUNCTION            1
      	             12 STORE_NAME               1 (ccc)	<-- NOTE means: ccc = what fff() call returns
      	             15 LOAD_CONST               1 (None)
      	             18 RETURN_VALUE
      
      At least with no overriding taking place the situation is better now.
      
      NOTE: it is only @func(cls) which potentially pollutes calling
      namespace. Just @func (without class) is always clean because by
      definition it works as a regular decorator.
      
      (*) there is a very low-level and potentially fragile way to disable
      STORE_NAME after CALL_FUNCTION by dynamically patching caller's bytecode
      at runtime and replacing STORE_NAME with POP_TOP + NOP...
      924a808c
  8. 24 May, 2019 1 commit
    • Kirill Smelkov's avatar
      Revert "tox -= Python3.6" · 08ec7950
      Kirill Smelkov authored
      This reverts commit 469f21a9.
      
      Even though Debian stopped shipping Python3.6 I'm now building it myself
      to have test coverage for two latest Python releases.
      08ec7950
  9. 16 May, 2019 7 commits
    • Kirill Smelkov's avatar
      pygolang v0.0.2 · d3bfb1bf
      Kirill Smelkov authored
      d3bfb1bf
    • Kirill Smelkov's avatar
      context: Add support for deadlines · b2450310
      Kirill Smelkov authored
      Implement deadlines / timeouts using timers added recently in 9c260fde
      (time: New package that mirrors Go's time).
      b2450310
    • Kirill Smelkov's avatar
      context: tests: Factor out common bits from test_context · 27f91b78
      Kirill Smelkov authored
      In the next patch we'll be adding deadlines support with another test.
      Move common infrastructure that will be used in all context tests to be
      test-module global.
      27f91b78
    • Kirill Smelkov's avatar
      context: Don't hardcode that contexts are always done with canceled error · e5687f2f
      Kirill Smelkov authored
      We'll need to mark contexts that miss their deadline with
      deadlineExceeded error in a follow-up patch.
      
      In this patch: transform _BaseCtx._cancel to accept an error with which
      a context has to be canceled, but for now always use canceled in places
      where _cancel is called.
      e5687f2f
    • Kirill Smelkov's avatar
      context: Note that a context has to be explicitly canceled to release resources · 58ba1765
      Kirill Smelkov authored
      Same story as in Go, but we were lacking documentation notice about this.
      58ba1765
    • Kirill Smelkov's avatar
      time: New package that mirrors Go's time · 9c260fde
      Kirill Smelkov authored
      Add time.Timer, time.Ticker and convenience functions time.tick,
      time.after and time.after_func. These will be used in context to support
      deadlines and timeouts.
      
      While at time topic, also provide sleep and now from golang.time, so
      that there is no need to import both golang.time and stdlib's time in a
      file.
      
      Provide time constants in the module as they are useful to have and
      mirror constants provided by Go's time.
      
      Note: timers implementation is very suboptimal for now.
      9c260fde
    • Kirill Smelkov's avatar
      golang: Don't keep test programs inside golang/ dir · 6e3b3ff4
      Kirill Smelkov authored
      Python adds dirname of run program to sys.path . This way when
      golang_test_goleaked.py runs it can import modules located under golang/
      just by their name. Until now this was not noticed, but in the next
      patch we are going to add golang.time module and if test program is run
      with golang/ in sys.path just plain `import time` won't import time from
      stdlib and instead import time from golang/ .
      
      Such behaviour can be mitigated by doing `from __future__ import
      absolute_import` and we do that, including in golang_test_goleaked.py
      (see 81dfefa0 "*: __future__ += absolute_imports; Use unified __future__
      everywhere"). However that does not prevent modules - even modules from
      stdlib - who are doing `import time` and not doing future
      absolute_import to import golang's time instead of stdlib. For example
      on PyPy2 threading imports time and then the test breaks:
      
        Traceback (most recent call last):
          File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/golang_test_goleaked.py", line 24, in <module>
            from golang import go, chan
          File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/__init__.py", line 38, in <module>
            import inspect, threading, collections, random, sys
          File "/usr/lib/pypy/lib-python/2.7/threading.py", line 15, in <module>
            from time import time as _time, sleep as _sleep
          File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/time.py", line 30, in <module>
            from golang import go, chan, select, default, nilchan, panic
        ImportError: cannot import name 'go'
      
      -> Move the test program into a directory different from golang/ to
      avoid this trap.
      6e3b3ff4