Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gevent
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
Kirill Smelkov
gevent
Commits
692e8547
Commit
692e8547
authored
Oct 22, 2015
by
Jason Madden
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Document hang with multiprocessing.Queue and monkey-patched thread module. [skip ci]
parent
643cb4c8
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
45 additions
and
9 deletions
+45
-9
doc/whatsnew_1_1.rst
doc/whatsnew_1_1.rst
+14
-8
gevent/monkey.py
gevent/monkey.py
+31
-1
No files found.
doc/whatsnew_1_1.rst
View file @
692e8547
...
...
@@ -90,14 +90,20 @@ downstream libraries, notably `gunicorn`_.
In addition, simple use of :class:`multiprocessing.Process` is now
possible in a monkey patched system, at least on POSIX platforms.
.. note:: All of the above entail forking a child process. Forking
a child process that uses gevent, greenlets, and libev
can have some unexpected consequences if the child
doesn'
t
immediately
``
exec
``
a
new
binary
.
Be
sure
you
understand
these
consequences
before
using
this
functionality
,
especially
late
in
a
program
's lifecycle.
For a more robust solution to certain uses of child
process, consider `gipc`_.
.. caution:: Use of :class:`multiprocessing.Queue` when :mod:`thread`
has been monkey-patched will lead to a hang due to
``Queue``'
s
internal
use
of
a
blocking
pipe
and
threads
.
For
the
same
reason
,
:
class
:`
concurrent
.
futures
.
ProcessPoolExecutor
`,
which
internally
uses
a
``
Queue
``,
will
hang
.
..
tip
::
All
of
the
above
entail
forking
a
child
process
.
Forking
a
child
process
that
uses
gevent
,
greenlets
,
and
libev
can
have
some
unexpected
consequences
if
the
child
doesn
't immediately ``exec`` a new binary. Be sure you
understand these consequences before using this
functionality, especially late in a program'
s
lifecycle
.
For
a
more
robust
solution
to
certain
uses
of
child
process
,
consider
`
gipc
`
_
.
..
_gunicorn
:
http
://
gunicorn
.
org
..
_gipc
:
https
://
gehrcke
.
de
/
gipc
/
...
...
gevent/monkey.py
View file @
692e8547
...
...
@@ -46,6 +46,9 @@ Sometimes it is useful to run existing python scripts or modules that
were not built to be gevent aware under gevent. To do so, this module
can be run as the main module, passing the script and its arguments.
For details, see the :func:`main` function.
Functions
=========
"""
from
__future__
import
absolute_import
from
__future__
import
print_function
...
...
@@ -250,7 +253,8 @@ def _patch_existing_locks(threading):
def
patch_thread
(
threading
=
True
,
_threading_local
=
True
,
Event
=
False
,
logging
=
True
,
existing_locks
=
True
):
"""Replace the standard :mod:`thread` module to make it greenlet-based.
"""
Replace the standard :mod:`thread` module to make it greenlet-based.
- If *threading* is true (the default), also patch ``threading``.
- If *_threading_local* is true (the default), also patch ``_threading_local.local``.
...
...
@@ -259,7 +263,33 @@ def patch_thread(threading=True, _threading_local=True, Event=False, logging=Tru
- If *existing_locks* is True (the default), and the process is still single threaded,
make sure than any :class:`threading.RLock` (and, under Python 3, :class:`importlib._bootstrap._ModuleLock`)
instances that are currently locked can be properly unlocked.
.. caution::
Monkey-patching :mod:`thread` and using
:class:`multiprocessing.Queue` or
:class:`concurrent.futures.ProcessPoolExecutor` (which uses a
``Queue``) will hang the process.
"""
# Description of the hang:
# There is an incompatibility with patching 'thread' and the 'multiprocessing' module:
# The problem is that multiprocessing.queues.Queue uses a half-duplex multiprocessing.Pipe,
# which is implemented with os.pipe() and _multiprocessing.Connection. os.pipe isn't patched
# by gevent, as it returns just a fileno. _multiprocessing.Connection is an internal implementation
# class implemented in C, which exposes a 'poll(timeout)' method; under the covers, this issues a
# (blocking) select() call: hence the need for a real thread. Except for that method, we could
# almost replace Connection with gevent.fileobject.SocketAdapter, plus a trivial
# patch to os.pipe (below). Sigh, so close. (With a little work, we could replicate that method)
# import os
# import fcntl
# os_pipe = os.pipe
# def _pipe():
# r, w = os_pipe()
# fcntl.fcntl(r, fcntl.F_SETFL, os.O_NONBLOCK)
# fcntl.fcntl(w, fcntl.F_SETFL, os.O_NONBLOCK)
# return r, w
# os.pipe = _pipe
patch_module
(
'thread'
)
if
threading
:
patch_module
(
'threading'
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment