1. 08 Jul, 2019 1 commit
  2. 26 Jun, 2019 4 commits
    • sync.WorkGroup: Propagate all exception types, not only those derived from Exception · 79aab7df
      A problem was hit with pytest.fail with raises Failed exception not
      being propagated to .wait. As it turned out it was not propagated
      because pytest's Failed derives from BaseException, not Exception, and
      we were catching only Exception and its children.
      
      Rework the code to propagate all exception types from workers.
      
      Performance change is with noise (it is either a bit faster for one set
      of runs, or a bit slower for another set of runs).
      Kirill Smelkov committed
    • sync.Workgroup: Don't use @func at runtime · 94c6160b
      It is true for any decorator, that it makes things faster if
      def+decorator is used globally instead of at runtime, but for @func it
      is especially true since @func, using decorator.decorate, has relatively
      high overhead if it is used not only once at program startup, but
      instead every time something is run.
      
      In particular moving @func out of WorkGroup.go() make things
      considerably faster:
      
      	name             old time/op  new time/op  delta
      	workgroup_empty   195µs ± 0%   113µs ± 1%  -41.91%  (p=0.008 n=5+5)
      	workgroup_raise   221µs ± 1%   137µs ± 1%  -38.29%  (p=0.008 n=5+5)
      
      See
      
      	https://lab.nexedi.com/kirr/misc/raw/009c4fee/pygolang/prof_workgroup_empty.svg
      
      for bench_workgroup_empty profile, where it is seen that
      decorator.decorate was using ~ half of the whole WorkGroup.go() time.
      Kirill Smelkov committed
    • golang: Add benchmarks for chan, select, @func and defer · 3c55ca59
      Start adding benchmarks to pygolang.
      
      Measure how much sync chan send/recv take and how much select takes for
      synchronous channels. Also measure how much overhead @func adds at both
      def and call times, and the overhead of defer compared to try/finally.
      For std (non-gevent'ed) Python2.7 we are currently at:
      
      	name         time/op
      	go           91.6µs ± 1%
      	chan         13.7µs ± 3%
      	select       30.1µs ± 4%
      	def          55.0ns ± 0%
      	func_def     43.6µs ± 0%
      	call         63.0ns ± 0%
      	func_call    1.06µs ± 0%
      	try_finally   136ns ± 1%
      	defer        2.33µs ± 0%
      Kirill Smelkov committed
    • golang: test_method -> test_func · c3ea8f51
      @method(cls) was deprecated and removed in favour of @func(cls) in
      942ee900 (golang: Deprecate @method(cls) in favour of @func(cls)) and
      262f8986 (golang: Kill @method). The test name was overlooked.
      Kirill Smelkov committed
  3. 21 Jun, 2019 1 commit
    • golang: Fix `@func(cls) def name` not to override `name` in calling context · 924a808c
      With @func being a decorator, the following
      
      	@func(cls)
      	def name():
      		...
      
      is always processed by python as
      
      	name = func(cls)(def name(): ...)
      
      Before this patch it was leading to name being overridden with None:
      
      	def f():
      		print 'hello'
      
      	class C:
      		pass
      
      	@func(C)
      	def f(c):
      		print 'C.f', c
      
      	f()
      	Traceback (most recent call last):
      	  File "<console>", line 1, in <module>
      	TypeError: 'NoneType' object is not callable
      
      We can fix it by returning from `func(cls)(def name(): ...)` the
      original `name` object from the calling context.
      
      Unfortunately if `name` was not previously set I did not find a way(*) to
      avoid polluting the calling namespace where it is set to what @func(cls)
      returns (None) by the hardcoded way how python processes decorators:
      
      	In [2]: c = """
      	   ...: @fff
      	   ...: def ccc():
      	   ...:     return 1
      	   ...: """
      
      	In [3]: cc = compile(c, "file", "exec")
      
      	In [4]: dis(cc)
      	  2           0 LOAD_NAME                0 (fff)
      	              3 LOAD_CONST               0 (<code object ccc at 0x7fafe58d0130, file "file", line 2>)
      	              6 MAKE_FUNCTION            0
      	              9 CALL_FUNCTION            1
      	             12 STORE_NAME               1 (ccc)	<-- NOTE means: ccc = what fff() call returns
      	             15 LOAD_CONST               1 (None)
      	             18 RETURN_VALUE
      
      At least with no overriding taking place the situation is better now.
      
      NOTE: it is only @func(cls) which potentially pollutes calling
      namespace. Just @func (without class) is always clean because by
      definition it works as a regular decorator.
      
      (*) there is a very low-level and potentially fragile way to disable
      STORE_NAME after CALL_FUNCTION by dynamically patching caller's bytecode
      at runtime and replacing STORE_NAME with POP_TOP + NOP...
      Kirill Smelkov committed
  4. 24 May, 2019 1 commit
  5. 16 May, 2019 7 commits
    • pygolang v0.0.2 · d3bfb1bf
      Kirill Smelkov committed
    • context: Add support for deadlines · b2450310
      Implement deadlines / timeouts using timers added recently in 9c260fde
      (time: New package that mirrors Go's time).
      Kirill Smelkov committed
    • context: tests: Factor out common bits from test_context · 27f91b78
      In the next patch we'll be adding deadlines support with another test.
      Move common infrastructure that will be used in all context tests to be
      test-module global.
      Kirill Smelkov committed
    • context: Don't hardcode that contexts are always done with canceled error · e5687f2f
      We'll need to mark contexts that miss their deadline with
      deadlineExceeded error in a follow-up patch.
      
      In this patch: transform _BaseCtx._cancel to accept an error with which
      a context has to be canceled, but for now always use canceled in places
      where _cancel is called.
      Kirill Smelkov committed
    • context: Note that a context has to be explicitly canceled to release resources · 58ba1765
      Same story as in Go, but we were lacking documentation notice about this.
      Kirill Smelkov committed
    • time: New package that mirrors Go's time · 9c260fde
      Add time.Timer, time.Ticker and convenience functions time.tick,
      time.after and time.after_func. These will be used in context to support
      deadlines and timeouts.
      
      While at time topic, also provide sleep and now from golang.time, so
      that there is no need to import both golang.time and stdlib's time in a
      file.
      
      Provide time constants in the module as they are useful to have and
      mirror constants provided by Go's time.
      
      Note: timers implementation is very suboptimal for now.
      Kirill Smelkov committed
    • golang: Don't keep test programs inside golang/ dir · 6e3b3ff4
      Python adds dirname of run program to sys.path . This way when
      golang_test_goleaked.py runs it can import modules located under golang/
      just by their name. Until now this was not noticed, but in the next
      patch we are going to add golang.time module and if test program is run
      with golang/ in sys.path just plain `import time` won't import time from
      stdlib and instead import time from golang/ .
      
      Such behaviour can be mitigated by doing `from __future__ import
      absolute_import` and we do that, including in golang_test_goleaked.py
      (see 81dfefa0 "*: __future__ += absolute_imports; Use unified __future__
      everywhere"). However that does not prevent modules - even modules from
      stdlib - who are doing `import time` and not doing future
      absolute_import to import golang's time instead of stdlib. For example
      on PyPy2 threading imports time and then the test breaks:
      
        Traceback (most recent call last):
          File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/golang_test_goleaked.py", line 24, in <module>
            from golang import go, chan
          File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/__init__.py", line 38, in <module>
            import inspect, threading, collections, random, sys
          File "/usr/lib/pypy/lib-python/2.7/threading.py", line 15, in <module>
            from time import time as _time, sleep as _sleep
          File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/time.py", line 30, in <module>
            from golang import go, chan, select, default, nilchan, panic
        ImportError: cannot import name 'go'
      
      -> Move the test program into a directory different from golang/ to
      avoid this trap.
      Kirill Smelkov committed
  6. 14 May, 2019 1 commit
  7. 10 May, 2019 1 commit
    • fixup! pygolang v0.0.1 · 1172e39f
      - correct CHANGELOG title levels: since readme uses "----" for the
        first level and "~~~~" for the second level, "====" was interpreted as
        the third title level and "Pygolang change history" became
        sub-sub-section of "Additional packages and utilities" with section
        for each version conversely becoming first level. It was not very
        noticeable until 0c5f9d06 (readme: Push "Additional packages and
        utilities" into its own section) started to use "~~~~".
        -> Rework CHANGELOG titling to align with the one in README.
      
      - fix minor markup bits in README.
      Kirill Smelkov committed
  8. 09 May, 2019 1 commit
  9. 03 May, 2019 3 commits
  10. 02 May, 2019 2 commits
  11. 24 Mar, 2019 3 commits
    • golang: Kill @method · 262f8986
      It was deprecated and scheduled to be removed in 942ee900 (golang:
      Deprecate @method(cls) in favour of @func(cls)).
      Kirill Smelkov committed
    • pygolang v0.0.0.dev8 · 0f60aa77
      Kirill Smelkov committed
    • gpython: Workaround PyPy2 vs Gevent patch_thread crash · 704d99f0
      	pypy-gevent runtests: commands[0] | gpython -m pytest gpython/ golang/
      	Traceback (most recent call last):
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/bin/gpython", line 10, in <module>
      	    sys.exit(main())
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gpython/__init__.py", line 153, in main
      	    _ = monkey.patch_all()      # XXX sys=True ?
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gevent/monkey.py", line 976, in patch_all
      	    patch_thread(Event=Event, _warnings=_warnings)
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gevent/monkey.py", line 178, in ignores
      	    return func(*args, **kwargs)
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gevent/monkey.py", line 588, in patch_thread
      	    _patch_existing_locks(threading_mod)
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy-gevent/site-packages/gevent/monkey.py", line 492, in _patch_existing_locks
      	    if o._RLock__owner is not None:
      	AttributeError: 'thread.RLock' object has no attribute '_RLock__owner'
      
      It will be fixed on next Gevent release, however until then the crash is there
      and we have to workaround: we can skip patching existing locks as it will be
      the same behaviour of next Gevent release, since its just not possible to patch
      locks created via instantiated C-level classes:
      
      https://github.com/gevent/gevent/commit/d0e04658
      
      We are doing monkey-patching very early, so it should be safe.
      Kirill Smelkov committed
  12. 20 Mar, 2019 1 commit
    • golang: Deflake channel tests · 731f39e3
      Channel tests are passing on an idle machine. However if there is
      another load - e.g. in the presence of simultaneous Firefox start,
      channel tests break, for example:
      
      	    def test_select():
      	        ...
      
      	        # non-blocking try recv: ok
      	        ch = chan()
      	        done = chan()
      	        def _():
      	            for i in range(N):
      	                ch.send(i)
      	            done.close()
      	        go(_)
      
      	        for i in range(N):
      	            tdelay()
      	            if i % 2:
      	                _, _rx = select(
      	                        ch.recv,
      	                        default,
      	                )
      	                assert (_, _rx) == (0, i)
      	            else:
      	                _, _rx = select(
      	                        ch.recv_,
      	                        default,
      	                )
      	>               assert (_, _rx) == (0, (i, True))
      	E               assert (1, None) == (0, (320, True))
      	E                 At index 0 diff: 1 != 0
      	E                 Use -v to get the full diff
      
      	golang_test.py:209: AssertionError
      
      The failure here is that it was default case selected, not ch.recv_. The
      default was selected because the sending goroutine was not fast to
      enqueue next send before we tried to receive. We were trying to make
      sure that the sender will be enqueued first via adding 1ms time delay
      before trying to receive, but in the presence of concurrent load spikes
      that turns out to be not enough.
      
      We could try to fix the test by increasing the time to sleep in tdelay,
      make the tests more slow and still not 100% reliable. However we can
      change the tests to instead actually wait for the condition that is
      semantically required: a sender enqueued on the channel.
      
      Do that everywhere where tdelay was used.
      
      Now tests are faster (it was ~3s total, now it is ~ 0.5s total) and pass
      irregardless of whether the machine is idle or otherwise loaded.
      Kirill Smelkov committed
  13. 19 Mar, 2019 3 commits
    • tox -= Python3.6 · 469f21a9
      Debian dropped Python3.6 from python3-distutils. Without it I'm no
      longer being able to run tests on that python version:
      
      	(neo) (z-dev) (g.env) kirr@deco:~/src/tools/go/pygolang$ tox -e py36-thread
      	execution failed:  -- Traceback (most recent call last):
      	  File "<string>", line 1, in <module>
      	ModuleNotFoundError: No module named 'distutils.sysconfig'
      
      	GLOB sdist-make: /home/kirr/src/tools/go/pygolang/setup.py
      	py36-thread create: /home/kirr/src/tools/go/pygolang/.tox/py36-thread
      	ERROR: invocation failed (exit code 1), logfile: /home/kirr/src/tools/go/pygolang/.tox/py36-thread/log/py36-thread-0.log
      	ERROR: actionid: py36-thread
      	msg: getenv
      	cmdargs: '/usr/bin/python3 -m virtualenv --python /usr/bin/python3.6 py36-thread'
      
      	Traceback (most recent call last):
      	  File "/usr/lib/python3/dist-packages/virtualenv.py", line 25, in <module>
      	    import distutils.sysconfig
      	ModuleNotFoundError: No module named 'distutils.sysconfig'
      	Running virtualenv with interpreter /usr/bin/python3.6
      
      	ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python3 -m virtualenv --python /usr/bin/python3.6 py36-thread (see /home/kirr/src/tools/go/pygolang/.tox/py36-thread/log/py36-thread-0.log)', 1)
      
      https://bugs.debian.org/918881#19
      Kirill Smelkov committed
    • gpython: Don't check for time not being pre-imported on PyPy · e847c550
      PyPy7 always pre-imports the time module. Without the change tests were
      failing:
      
      	Traceback (most recent call last):
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy3-gevent/bin/gpython", line 10, in <module>
      	    sys.exit(main())
      	  File "/home/kirr/src/tools/go/pygolang/.tox/pypy3-gevent/site-packages/gpython/__init__.py", line 145, in main
      	    '\n\n\t%s\n\nsys.modules:\n\n\t%s' % (bad, sysmodv))
      	RuntimeError: gpython: internal error: the following modules are pre-imported, but must be not:
      
      	        ['time']
      
      	sys.modules:
      
      	        ['__future__', '__main__', '__pypy__', '__pypy__._pypydatetime', '__pypy__.builders', '__pypy__.intop', '__pypy__.os', '__pypy__.thread', '__pypy__.time', '_ast', '_bootlocale', '_codecs', '_collections', '_collections_abc', '_continuation', '_csv', '_frozen_importlib', '_frozen_importlib_external', '_imp', '_io', '_locale', '_multibytecodec', '_operator', '_rawffi', '_rawffi.alt', '_signal', '_sre', '_structseq', '_thread', '_warnings', '_weakref', '_weakrefset', 'abc', 'array', 'builtins', 'codecs', 'copyreg', 'encodings', 'encodings.aliases', 'encodings.ascii', 'encodings.latin_1', 'encodings.utf_8', 'errno', 'gc', 'genericpath', 'gpython', 'marshal', 'os', 'os.path', 'posix', 'posixpath', 'pwd', 're', 'site', 'sre_compile', 'sre_constants', 'sre_parse', 'stat', 'sys', 'time', 'unicodedata']
      Kirill Smelkov committed
    • tox += PyPy3 · da68a8ae
      Debian started to ship PyPy3:
      
      	https://packages.debian.org/search?keywords=pypy3
      Kirill Smelkov committed
  14. 15 Mar, 2019 1 commit
    • golang: Deprecate @method(cls) in favour of @func(cls) · 942ee900
      Since we already have func (see 5146eb0b "Add support for defer &
      recover") we can use @func for both plain functions and for methods.
      
      For example instead of
      
      	@func
      	def my_function(...):
      	    ...
      
      	@method(MyClass)			<-- NOTE
      	def my_method(self, ...):
      	    ...
      
      have it as
      
      	@func
      	def my_function(...):
      	    ...
      
      	@func(MyClass)				<-- NOTE
      	def my_method(self, ...):
      	    ...
      
      which looks more similar to Go and exposes less golang keywords to a user.
      Kirill Smelkov committed
  15. 13 Mar, 2019 1 commit
    • gpython: Fix pymain to properly setup sys.path[0] · 6b4990f6
      Pymain was not adding e.g. directory of executed file to sys.path, and
      as the result if there were e.g. 2 files
      
      	dir/hello.py	# imports world
      	dir/world.py
      
      running `gpython dir/hello.py` would fail to import world.
      
      The case for interactive console was also failing to setup sys.argv as
      empty, so it was containing ['/path/to/gpython']. It was also, contrary
      to standard python, unconditionally showing '>>>' prompt even when stdin
      was not a tty.
      
      Fix all that and add a test to cover pymain functionality.
      Kirill Smelkov committed
  16. 16 Jan, 2019 6 commits
  17. 30 Dec, 2018 1 commit
  18. 13 Dec, 2018 2 commits