- 14 Nov, 2019 3 commits
-
-
Kirill Smelkov authored
It will be some extra work to define general equality when moving context code into C. However that is unnecessary as usually a key is specific to one particular package, and value associated with that key is struct, or some other aggregate, that is holding all package-specific context values. Switching to compare keys by identity preserves that use-case while making transition to C easier.
-
Kirill Smelkov authored
We always store just one key in there. This will simplify transition to C.
-
Kirill Smelkov authored
This continues 2c8063f4 (*: Channels must be compared by ==, not by "is" even for nilchan) and changes comparision of .done() for different contexts from "is" to ==. The reason is that soon we are going to move contexts into C, and this way in context tree there potentially can be PyCtx -> CCtx -> PyCtx chains, and while all those contexts could have the same C-level channel, it would be difficult to propagate the knowledge that Python-level pychan wrapper should be also exactly the same. We already treat channel objects as pointers and require to compare them by ==, so there should be no loss here. However to keep performance of repeated select( ctx.done(), ... ) we continue to require that .done() for a particular python context object returns exactly the same channel object every time.
-
- 13 Nov, 2019 1 commit
-
-
Kirill Smelkov authored
Rearrange time C++ API so that e.g. time.Ticker is already refptr<time._Ticker>. This way time users don't need to write refptr all the time.
-
- 11 Nov, 2019 2 commits
-
-
Kirill Smelkov authored
https://bitbucket.org/pypy/pypy/commits/251b47698e8e https://bitbucket.org/pypy/pypy/issues/3096#comment-54707695 Which in turn fixes recover to clear current exception (see 9e6ff8bd "golang: Fix recover to clear current exception") for context.
-
Kirill Smelkov authored
e.g. Context -> PyContext, with_cancel -> pywith_cancel, etc. This will make the followup transition of context to pyx/nogil more clear.
-
- 07 Nov, 2019 10 commits
-
-
Kirill Smelkov authored
As of 20191107 imp is present in Python 3.8 source tree and is _used_ there - see e.g. https://github.com/python/cpython/blob/v3.8.0-66-g7c20888e713/Lib/pkgutil.py#L188-L192 And they say imp will be there until at least Python 4 - see e.g. https://github.com/python/cpython/commit/e4f41deccf94 This fixes test_defer_excchain_dump failure under python3-dbg, where E Failed: not equal: E Differences (unified diff with -expected +actual): E @@ -1,4 +1,6 @@ E +PYGOLANG/golang/_gopath.py:37: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses E + import imp was added into traceback printed on stderr. Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)
-
Kirill Smelkov authored
Provide time.Ticker, time.Timer and friends that can be used directly from C++ and Pyx/nogil codes. Python-level classes become small wrapper around pyx/nogil ones. This is the first patch that moves to Pyx/nogil classes that are dynamically allocated on heap. refptr<T> is used to automatically manage lifetime of such objects. At Pyx level exposed API is very similar to Python-one, while internally it uses refptr<T> and friends.
-
Kirill Smelkov authored
Refobj was just added in previous patch as the class that provides base functionality for reference-counted objects.
-
Kirill Smelkov authored
Since we are going to move more pygolang functionality into C++ (timers, context, ...), and since C++ does not have garbage collector, we will need to find a way to automatically manage memory in leak/error free way. Let's do this via refptr<T> smart pointer (inspired by WebKit's RefPtr<T>), which, similarly to chan<T> automatically manages pointed object's lifetime through reference counting. refptr<T> will be used in follow-up patches. Top-level documentation is TODO.
-
Kirill Smelkov authored
This will be needed in Python-wrappers for C-level code which exposes channels. For example in time package whose implementation will soon be moved to nogil world fully.
-
Kirill Smelkov authored
3121b290 (golang: Teach pychan to work with channels of C types, not only PyObjects) added ability to convert pychan to chan[T], but described that ability only briefly in README. Provide more details in that description before we add description and implementation for reflexive pychan <- chan[T] wrapping.
-
Kirill Smelkov authored
We already have some sync functionality implemented in C++ (e.g. sync.Once, sync.WaitGroup) and we are going to add more and also move timers implementation to C++. It is getting crowded for that functionality to still live in libgolang.{h,cpp} -> Split sync & time functionality into their own C++ packages (still built-into libgolang.so)
-
Kirill Smelkov authored
Briefly describe provided functionality instead of only referring to https://golang.org/pkg/time.
-
Kirill Smelkov authored
Briefly describe provided functionality instead of only referring to https://golang.org/pkg/sync.
-
Kirill Smelkov authored
Make built extensions depend on golang/{sync,time}.pxd and rebuild if those files change. A bit overkill to depend globally, but more correct, since an extension that is using pygolang can be using those packages. Before the patch - without those depends - since distutils does not implement proper dependency tracking, extensions - even those that actually depend on sync/time pxd, were not rebuilt. sync.pxd was missed to be added in 34b7a1f4 (golang: Expose Sema and Mutex as public Python and Cython/nogil API). time.pxd was missed to be added in ce8152a2 (pyx api: Provide sleep).
-
- 06 Nov, 2019 3 commits
-
-
Kirill Smelkov authored
Just cosmetics - easier to oversee the code.
-
Kirill Smelkov authored
Ticker: self -> pytx Timer: self -> pyt
-
Kirill Smelkov authored
e.g. tick -> pytick, Timer -> PyTimer. etc. This will make the followup transition of time to pyx/nogil more clear.
-
- 04 Nov, 2019 6 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
e.g. go -> pygo, select -> pyselect, etc. This will make the followup transition to non-py mode more clear.
-
Kirill Smelkov authored
Similarly to 0e838833 (time: Turn Ticker and Timer into cdef classes) rework all classes in context package to be cdef classes: - cdef their attributes (else accessing any of them raises AttributeError). But don't change any of them to be used in cimport mode yet - this will be done later as separate steps.
-
Kirill Smelkov authored
In preparation to start migrating, at least partially, context functionality to nogil mode, first, similarly to package time (see 32f34607 "time: Move code to pyx"), move the code from context.py to _context.pyx. This is straight code movement + associated setup.py & co adjusts. We don't move just to context.pyx (note no _ prefix), since we will need to continue distinguishing pyx/nogil from py objects/functions.
-
- 31 Oct, 2019 1 commit
-
-
Kirill Smelkov authored
Gcc 5.4 from Ubuntu 16.04 LTS complains e.g. ./golang/libgolang.h: In function ‘golang::_selcase golang::_selrecv_(golang::_chan*, void*, bool*)’: ./golang/libgolang.h:227:5: sorry, unimplemented: non-trivial designated initializers not supported }; ^ The problem started to appear after 47111d3e (libgolang: Teach select to accept inplace tx data) when we moved _selcase.ptxrx inside union. Let's add workaround for older compilers, even though e.g. gcc 8.3 from Debian 10 accepts existing code just fine.
-
- 30 Oct, 2019 1 commit
-
-
Kirill Smelkov authored
Provide sync.WaitGroup that can be used directly from C++ and Pyx/nogil codes. Python-level sync.WaitGroup becomes small wrapper around pyx/nogil one. Python-level tests should be enough to cover C++/Pyx functionality at zero-level approximation.
-
- 24 Oct, 2019 1 commit
-
-
Kirill Smelkov authored
Provide sync.Once that can be used directly from C++ and Pyx/nogil worlds. Python-level sync.Once becomes small wrapper around pyx/nogil sync.Once .
-
- 22 Oct, 2019 1 commit
-
-
Kirill Smelkov authored
Libgolang, since 3b241983 (Port/move channels to C/C++/Pyx), already had defer macro implemented and used, but only internally. The reason it was not yet exposed as public API is that there is a difference with Go's defer in that deferred function is called at end of current scope instead of end of current function, and I was a bit reluctant to expose defer with different-than-Go semantic. However even with this difference defer is useful, and the difference can be documented. Unfortunately it is not easy to correctly fix the difference, so the most practical way for now is to expose defer as it is. I've also contemplated how to avoid using macro, but without a macro, users will have to explicitly declare placeholder variable for every defer call which goes against usability. Since defer is exposed as macro, I've also contemplated to expose it as something like `libgolang_defer` with the idea to avoid name conflicts, and so that users - that are using defer - will be doing `#define defer libgolang_defer`. However I ended up not doing that and exposing `defer` macro with its own name. My rationale is: - grepping /usr/include/ for \<defer\> on my system did not showed any real usage. - Qt also #defines `slots` and `signals` and that does not cause problems in practice. -> expose `defer` macro into public C++ API as is, so that it can be used not only inside libgolang.cpp . For example I myself need defer functionality in C++ part of wendelin.core.
-
- 18 Oct, 2019 3 commits
-
-
Kirill Smelkov authored
Else the exception, even if it was recovered from, will be included as cause for next raised exception. See added test for details.
-
Kirill Smelkov authored
Python3 chains exceptions, so that e.g. if exc1 is raised and, while it was not handled, another exc2 is raised, exc2 will be linked to exc1 via exc2.__context__ attribute and exc1 will be included into exc2 traceback printout. However many projects still use Python2 and there is no similar chaining functionality there. This way exc1 is completely lost. Since defer code is in our hands, we can teach it to implement exception chaining even on Python2 by carefully analyzing what happens in _GoFrame.__exit__(). Implementing chaining itself is relatively easy, but is only part of the story. Even if an exception is chained with its cause, but exception dump does not show the cause, the chaining will be practically useless. With this in mind this patches settles not only on implementing chaining itself, but on also giving a promise that chained cause exceptions will be included into traceback dumps as well. To realize this promise we adjust all exception dumping funcitons in traceback module and carefully install adjusted traceback.print_exception() into sys.excepthook. This amends python interactive sessions and programs run by python interpreter to include causes in exception dumps. "Careful" here means that we don't change sys.excepthook if on golang module load we see that sys.excepthook was already changed by some other module - e.g. due to IPython session running because IPython installs its own sys.excepthook. In such cases we don't install our sys.excepthook, but we also provide integration patches that add exception chaining support for traceback dump functionality in popular third-party software. The patches (currently for IPython and Pytest) are activated automatically, but only when/if corresponding software is imported and actually used. This should give practically good implementation of the promise - a user can now rely on seeing exception cause in traceback dump whatever way python programs are run. The implementation takes https://pypi.org/project/pep3134/ experience into account [1]. peak.utils.imports [2,3] is used to be notified when/if third-party module is imported. [1] https://github.com/9seconds/pep3134/ [2] https://pypi.org/project/Importing/ [3] http://peak.telecommunity.com/DevCenter/Importing This patch originally started as hacky workaround in wendelin.core because in wcfs tests I was frequently hitting situations, where exception raised by an assert was hidden by another exception raised in further generic teardown check. For example wcfs tests check that wcfs is unmounted after every test run [4] and if that fails it was hiding problems raised by an assert. As the result I was constantly guessing and adding code like [5] to find what was actually breaking. At some point I added hacky workaround for defer to print cause exception not to loose it [6]. [7] has more context and background discussion on this topic. [4] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L70 [5] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L853-857 [6] kirr/wendelin.core@c00d94c7 [7] zodbtools!13 (comment 81553) After this patch, on Python2 defer(cleanup1) defer(cleanup2) defer(cleanup3) ... is no longer just a syntatic sugar for try: try: try: ... finally: cleanup3() finally: cleanup2() finally: cleanup1()
-
Kirill Smelkov authored
- _pyrun runs the command and returns full information: exitcode, stdout, stderr. - pyrun runs the command and raises exception if ran command fails. We will need _pyrun in the next patch to test that particular command fails and access its stderr.
-
- 15 Oct, 2019 2 commits
-
-
Kirill Smelkov authored
_makesema declaration was giving a warning when compiling with C: In file included from golang/runtime/libgolang_test_c.c:26: ./golang/libgolang.h:237:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes] LIBGOLANG_API _sema *_makesema(); ^~~~~~~~~~~~~ Fix the prototype to be valid for both C and C++. Fixes: 34b7a1f4.
-
Kirill Smelkov authored
E.g. wendelin.core amends build_ext class and so it needs a way to import it.
-
- 14 Oct, 2019 6 commits
-
-
Kirill Smelkov authored
- use .c.chan_double() which gives chan[double] pyx/nogil way to access Ticker.c and Timer.c. Use the channels via pyx/nogil API from inside. - use pyx/nogil sleep and now; This gets time.pyx codebase closer to be used from pyx/nogil mode. NOTE: unless something like pyx/nogil memory management emerges[1] we are relying on Python to manage memory of Ticker and Timer classes. If we just spawn e.g. Ticker.__tick via pyx/nogil go, the thread that is spawned won't be holding a reference to Ticker object, and once the ticker goes out of scope in original thread (while its channel .c might be still in scope), __tick will segfault accessing freed Ticker object. To workaround it we use the following pattern: nogilready = chan(dtype='C.structZ') pygo(mymeth) nogilready.recv() def mymeth(MyObject self, pychan nogilready) with nogil: nogilready.chan_structZ().close() self._mymeth() cdef void _mymeth(MyObject self) nogil: ... where python reference to MyObject will be held in spawned thread during its lifetime, while the service provided by mymeth will be done under nogil. [1] https://www.nexedi.com/blog/NXD-Document.Blog.Cypclass
-
Kirill Smelkov authored
This will allow to use the channels in nogil mode including in followup patches touching time.pyx codebase.
-
Kirill Smelkov authored
C.structZ is empty structure and chan[C.structZ] can be used in pyx/nogil world. This way context tree initially created from Python can be extended in pyx/nogil and e.g. root context can be canceled from Python, which will correctly transfer the cancelation signal to pyx/nogil world too.
-
Kirill Smelkov authored
Introduce notion of data type (dtype similarly to NumPy) into pychan and teach it to accept for send objects only matching that dtype. Likewise teach pychan to decode raw bytes received from underlying channel into Python object correspodningg to pychan dtype. For C dtypes, e.g. 'C.int', 'C.double' etc, contrary to chan of python objects, the transfer can be done without depending on Python GIL. This way channels of such C-level dtypes can be used to implement interaction in between Python and nogil worlds.
-
Kirill Smelkov authored
In a followup commit we are going to add channel types (e.g. chan of double, chan of int, etc) and accordingly there will be several nil channel objects, e.g. nil(dtype=int), nil(dtype=double) etc, which will be separate python objects. Even without data types, another planned change is to add directional channels, e.g. a channel instance that can only send, but not receive and vice versa(*). This way for the same underlying channel object, there can be several pychan objects that point to it - even for nil channel - e.g. nilchan and `niltx = nilchan.txonly()` that creates another pychan object pointing to the same underlying nil. Since we want all channels (of the same type) that point to the same underlying channel to compare as same, we cannot use "is" for comparison and have to use ==. In other words channels, both at C and Python level, should be perceived as pointers, of which there can be multiple ones pointing to the same location, and thus == has to be used to compare them. (*) see https://golang.org/ref/spec#Channel_types for details.
-
Kirill Smelkov authored
For example - def send(pych, obj): + def send(pychan pych, obj): Even though Cython allows not to annotate self with type, this makes the code more clear.
-