Commit afa46cf5 authored by Kirill Smelkov's avatar Kirill Smelkov

Turn pygopath into full pygolang

Not only we can import modules by full path, but now we can also spawn
threads/coroutines and exchange data in between them with the same
primitives and semantic as in Go.

The bulk of new functionality is copied from here:

	go123@9e1aa6ab

Original commit description follows:

"""
golang: New _Python_ package to provide Go-like features to Python language
- `go` spawns lightweight thread.
- `chan` and `select` provide channels with Go semantic.
- `method` allows to define methods separate from class.
- `gimport` allows to import python modules by full path in a Go workspace.

The focus of first draft was on usage interface and on correctness, not speed.
In particular select should be fully working.

If there is a chance I will maybe try to followup with gevent-based
implementation in the future.
Hide whitespace changes
"""
parent 9c61f254
Pygopath is free software licensed under GPLv3+ with additional permission Pygolang is free software licensed under GPLv3+ with additional permission
to link, combine and redistribute it with other free or open source software. to link, combine and redistribute it with other free or open source software.
Please see https://www.nexedi.com/licensing for rationale and options. Please see https://www.nexedi.com/licensing for rationale and options.
......
recursive-include testdata *.py recursive-include golang/testdata *.py
================================================================= ========================================
Pygopath - Import python modules by full path in a Go workspace Pygolang - Go-like features for Python
================================================================= ========================================
Module `gopath` provides way to import python modules by full path in a Go workspace. Package golang provides Go-like features for Python:
- `go` spawns lightweight thread.
- `chan` and `select` provide channels with Go semantic.
- `method` allows to define methods separate from class.
- `gimport` allows to import python modules by full path in a Go workspace.
Goroutines and channels
-----------------------
`go` spawns a thread, or a coroutine if gevent was activated. It is possible to
exchange data in between either threads or coroutines via channels. `chan`
creates a new channel with Go semantic - either synchronous or buffered. Use
`chan.recv`, `chan.send` and `chan.close` for communication. `select` can be
used to multiplex on several channels. For example::
ch1 = chan() # synchronous channel
ch2 = chan(3) # channel with buffer of size 3
def _():
ch1.send('a')
ch2.send('b')
go(_)
ch1.recv() # will give 'a'
ch2.recv_() # will give ('b', True)
_, _rx = select(
ch1.recv, # 0
ch2.recv_, # 1
(ch2.send, obj2), # 2
default, # 3
)
if _ == 0:
# _rx is what was received from ch1
...
if _ == 1:
# _rx is (rx, ok) of what was received from ch2
...
if _ == 2:
# we know obj2 was sent to ch2
...
if _ == 3:
# default case
...
Methods
-------
`method` decorator allows to define methods separate from class.
For example::
@method(MyClass)
def my_method(self, ...):
...
will define `MyClass.my_method()`.
Import
------
`gimport` provides way to import python modules by full path in a Go workspace.
For example For example
:: ::
lonet = gopath.gimport('lab.nexedi.com/kirr/go123/xnet/lonet') lonet = gimport('lab.nexedi.com/kirr/go123/xnet/lonet')
will import either will import either
......
This diff is collapsed.
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# #
# See COPYING file for full licensing terms. # See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options. # See https://www.nexedi.com/licensing for rationale and options.
"""Module gopath provides way to import python modules by full path in a Go workspace. """Module _gopath provides way to import python modules by full path in a Go workspace.
For example For example
...@@ -56,8 +56,8 @@ def _gimport(name): ...@@ -56,8 +56,8 @@ def _gimport(name):
# reason: if we leave dots in place, python emits warning: # reason: if we leave dots in place, python emits warning:
# RuntimeWarning: Parent module 'lab.nexedi' not found while handling absolute import # RuntimeWarning: Parent module 'lab.nexedi' not found while handling absolute import
# #
# we put every imported module under `gopath.` namespace with '.' changed to '_' # we put every imported module under `golang._gopath.` namespace with '.' changed to '_'
modname = 'gopath.' + name.replace('.', '_') modname = 'golang._gopath.' + name.replace('.', '_')
try: try:
return sys.modules[modname] return sys.modules[modname]
......
...@@ -18,7 +18,7 @@ ...@@ -18,7 +18,7 @@
# See https://www.nexedi.com/licensing for rationale and options. # See https://www.nexedi.com/licensing for rationale and options.
import os, os.path import os, os.path
from gopath import gimport from golang._gopath import gimport
GOPATH_orig = None GOPATH_orig = None
......
# -*- coding: utf-8 -*-
# Copyright (C) 2018 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
# option) any later version, as published by the Free Software Foundation.
#
# You can also Link and Combine this program with other software covered by
# the terms of any of the Free Software licenses or any of the Open Source
# Initiative approved licenses and Convey the resulting work. Corresponding
# source of such a combination shall include the source code for all other
# software used.
#
# This program is distributed WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
"""Package gcompat provides Go-compatibility layer for Python"""
# qq is substitute for %q, which is missing in python.
#
# (python's automatic escape uses smartquotes quoting with either ' or ").
def qq(obj):
# go: like %s, %q automatically converts to string
if not isinstance(obj, basestring):
obj = str(obj)
return _quote(obj)
# _quote quotes string into valid "..." string always quoted with ".
def _quote(s):
# TODO also accept unicode as input.
# TODO output printable UTF-8 characters as-is, but escape non-printable UTF-8 and invalid UTF-8 bytes.
outv = []
# we don't want ' to be escaped
for _ in s.split("'"):
# this escape almost everything except " character
# NOTE string_escape does not do smartquotes and always uses ' for quoting
# (repr(str) is the same except it does smartquoting picking ' or " automatically)
q = _.encode("string_escape")
q = q.replace('"', r'\"')
outv.append(q)
return '"' + "'".join(outv) + '"'
# -*- coding: utf-8 -*-
# Copyright (C) 2018 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
# option) any later version, as published by the Free Software Foundation.
#
# You can also Link and Combine this program with other software covered by
# the terms of any of the Free Software licenses or any of the Open Source
# Initiative approved licenses and Convey the resulting work. Corresponding
# source of such a combination shall include the source code for all other
# software used.
#
# This program is distributed WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
from golang.gcompat import qq
def test_qq():
testv = (
# in want without leading/trailing "
('', r""),
('\'', r"'"),
('"', r"\""),
('abc\ndef', r"abc\ndef"),
('a\'c\ndef', r"a'c\ndef"),
('a\"c\ndef', r"a\"c\ndef"),
# ('привет', r"привет"), TODO
)
for tin, twant in testv:
twant = '"' + twant + '"' # add lead/trail "
assert qq(tin) == twant
# -*- coding: utf-8 -*-
# Copyright (C) 2018 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
# option) any later version, as published by the Free Software Foundation.
#
# You can also Link and Combine this program with other software covered by
# the terms of any of the Free Software licenses or any of the Open Source
# Initiative approved licenses and Convey the resulting work. Corresponding
# source of such a combination shall include the source code for all other
# software used.
#
# This program is distributed WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
from golang import go, chan, select, default, _PanicError
from pytest import raises
import time, threading
# tdelay delays a bit.
#
# XXX needed in situations when we need to start with known ordering but do not
# have a way to wait properly for ordering event.
def tdelay():
time.sleep(1E-3) # 1ms
def test_chan():
# sync: pre-close vs send/recv
ch = chan()
ch.close()
assert ch.recv() == None
assert ch.recv_() == (None, False)
assert ch.recv_() == (None, False)
raises(_PanicError, "ch.send(0)")
raises(_PanicError, "ch.close()")
# sync: send vs recv
ch = chan()
def _():
ch.send(1)
assert ch.recv() == 2
ch.close()
go(_)
assert ch.recv() == 1
ch.send(2)
assert ch.recv_() == (None, False)
assert ch.recv_() == (None, False)
# sync: close vs send
ch = chan()
def _():
tdelay()
ch.close()
go(_)
raises(_PanicError, "ch.send(0)")
# close vs recv
ch = chan()
def _():
tdelay()
ch.close()
go(_)
assert ch.recv_() == (None, False)
# sync: close vs multiple recv
ch = chan()
done = chan()
mu = threading.Lock()
s = set()
def _():
assert ch.recv_() == (None, False)
with mu:
x = len(s)
s.add(x)
done.send(x)
for i in range(3):
go(_)
ch.close()
for i in range(3):
done.recv()
assert s == {0,1,2}
# buffered
ch = chan(3)
done = chan()
for _ in range(2):
for i in range(3):
assert len(ch) == i
ch.send(i)
assert len(ch) == i+1
for i in range(3):
assert ch.recv_() == (i, True)
assert len(ch) == 0
for i in range(3):
ch.send(i)
assert len(ch) == 3
def _():
tdelay()
assert ch.recv_() == (0, True)
done.send('a')
for i in range(1,4):
assert ch.recv_() == (i, True)
assert ch.recv_() == (None, False)
done.send('b')
go(_)
ch.send(3) # will block without receiver
assert done.recv() == 'a'
ch.close()
assert done.recv() == 'b'
def test_select():
# non-blocking try send: not ok
ch = chan()
_, _rx = select(
(ch.send, 0),
default,
)
assert (_, _rx) == (1, None)
# non-blocking try recv: not ok
_, _rx = select(
ch.recv,
default,
)
assert (_, _rx) == (1, None)
_, _rx = select(
ch.recv_,
default,
)
assert (_, _rx) == (1, None)
# non-blocking try send: ok
ch = chan()
done = chan()
def _():
i = 0
while 1:
x = ch.recv()
if x == 'stop':
break
assert x == i
i += 1
done.close()
go(_)
for i in range(10):
tdelay()
_, _rx = select(
(ch.send, i),
default,
)
assert (_, _rx) == (0, None)
ch.send('stop')
done.recv()
# non-blocking try recv: ok
ch = chan()
done = chan()
def _():
for i in range(10):
ch.send(i)
done.close()
go(_)
for i in range(10):
tdelay()
if i % 2:
_, _rx = select(
ch.recv,
default,
)
assert (_, _rx) == (0, i)
else:
_, _rx = select(
ch.recv_,
default,
)
assert (_, _rx) == (0, (i, True))
done.recv()
# blocking 2·send
ch1 = chan()
ch2 = chan()
done = chan()
def _():
tdelay()
assert ch1.recv() == 'a'
done.close()
go(_)
_, _rx = select(
(ch1.send, 'a'),
(ch2.send, 'b'),
)
assert (_, _rx) == (0, None)
done.recv()
assert len(ch1._sendq) == len(ch1._recvq) == 0
assert len(ch2._sendq) == len(ch2._recvq) == 0
# blocking 2·recv
ch1 = chan()
ch2 = chan()
done = chan()
def _():
tdelay()
ch1.send('a')
done.close()
go(_)
_, _rx = select(
ch1.recv,
ch2.recv,
)
assert (_, _rx) == (0, 'a')
done.recv()
assert len(ch1._sendq) == len(ch1._recvq) == 0
assert len(ch2._sendq) == len(ch2._recvq) == 0
# blocking send/recv
ch1 = chan()
ch2 = chan()
done = chan()
def _():
tdelay()
assert ch1.recv() == 'a'
done.close()
go(_)
_, _rx = select(
(ch1.send, 'a'),
ch2.recv,
)
assert (_, _rx) == (0, None)
done.recv()
assert len(ch1._sendq) == len(ch1._recvq) == 0
assert len(ch2._sendq) == len(ch2._recvq) == 0
# blocking recv/send
ch1 = chan()
ch2 = chan()
done = chan()
def _():
tdelay()
ch1.send('a')
done.close()
go(_)
_, _rx = select(
ch1.recv,
(ch2.send, 'b'),
)
assert (_, _rx) == (0, 'a')
done.recv()
assert len(ch1._sendq) == len(ch1._recvq) == 0
assert len(ch2._sendq) == len(ch2._recvq) == 0
# buffered ping-pong
ch = chan(1)
for i in range(10):
_, _rx = select(
(ch.send, i),
ch.recv,
)
assert _ == (i % 2)
assert _rx == (i - 1 if i % 2 else None)
# select vs select
for i in range(10):
ch1 = chan()
ch2 = chan()
done = chan()
def _():
_, _rx = select(
(ch1.send, 'a'),
(ch2.send, 'xxx2'),
)
assert (_, _rx) == (0, None)
_, _rx = select(
(ch1.send, 'yyy2'),
ch2.recv,
)
assert (_, _rx) == (1, 'b')
done.close()
go(_)
_, _rx = select(
ch1.recv,
(ch2.send, 'xxx1'),
)
assert (_, _rx) == (0, 'a')
_, _rx = select(
(ch1.send, 'yyy1'),
(ch2.send, 'b'),
)
assert (_, _rx) == (1, None)
done.recv()
assert len(ch1._sendq) == len(ch1._recvq) == 0
assert len(ch2._sendq) == len(ch2._recvq) == 0
# pygopath | pythonic package setup # pygolang | pythonic package setup
from setuptools import setup, find_packages from setuptools import setup, find_packages
# read file content # read file content
...@@ -7,20 +7,18 @@ def readfile(path): ...@@ -7,20 +7,18 @@ def readfile(path):
return f.read() return f.read()
setup( setup(
name = 'pygopath', name = 'pygolang',
version = '0.0.0.dev1', version = '0.0.0.dev2',
description = 'Import python modules by full-path in Go workspace', description = 'Go-like features for Python',
long_description = readfile('README.rst'), long_description = readfile('README.rst'),
url = 'https://lab.nexedi.com/kirr/pygopath', url = 'https://lab.nexedi.com/kirr/pygolang',
license = 'GPLv3+ with wide exception for Open-Source', license = 'GPLv3+ with wide exception for Open-Source',
author = 'Kirill Smelkov', author = 'Kirill Smelkov',
author_email= 'kirr@nexedi.com', author_email= 'kirr@nexedi.com',
keywords = 'go GOPATH python import', keywords = 'go channel goroutine GOPATH python import',
# XXX find_packages does not find top-level *.py packages = find_packages(),
#packages = find_packages(),
packages = [''],
extras_require = { extras_require = {
'test': ['pytest'], 'test': ['pytest'],
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment