Commit 692e8547 authored by Jason Madden's avatar Jason Madden

Document hang with multiprocessing.Queue and monkey-patched thread module. [skip ci]

parent 643cb4c8
......@@ -90,14 +90,20 @@ downstream libraries, notably `gunicorn`_.
In addition, simple use of :class:`multiprocessing.Process` is now
possible in a monkey patched system, at least on POSIX platforms.
.. note:: All of the above entail forking a child process. Forking
a child process that uses gevent, greenlets, and libev
can have some unexpected consequences if the child
doesn't immediately ``exec`` a new binary. Be sure you
understand these consequences before using this
functionality, especially late in a program's lifecycle.
For a more robust solution to certain uses of child
process, consider `gipc`_.
.. caution:: Use of :class:`multiprocessing.Queue` when :mod:`thread`
has been monkey-patched will lead to a hang due to
``Queue``'s internal use of a blocking pipe and threads. For the same
reason, :class:`concurrent.futures.ProcessPoolExecutor`,
which internally uses a ``Queue``, will hang.
.. tip:: All of the above entail forking a child process. Forking
a child process that uses gevent, greenlets, and libev
can have some unexpected consequences if the child
doesn't immediately ``exec`` a new binary. Be sure you
understand these consequences before using this
functionality, especially late in a program's lifecycle.
For a more robust solution to certain uses of child
process, consider `gipc`_.
.. _gunicorn: http://gunicorn.org
.. _gipc: https://gehrcke.de/gipc/
......
......@@ -46,6 +46,9 @@ Sometimes it is useful to run existing python scripts or modules that
were not built to be gevent aware under gevent. To do so, this module
can be run as the main module, passing the script and its arguments.
For details, see the :func:`main` function.
Functions
=========
"""
from __future__ import absolute_import
from __future__ import print_function
......@@ -250,7 +253,8 @@ def _patch_existing_locks(threading):
def patch_thread(threading=True, _threading_local=True, Event=False, logging=True,
existing_locks=True):
"""Replace the standard :mod:`thread` module to make it greenlet-based.
"""
Replace the standard :mod:`thread` module to make it greenlet-based.
- If *threading* is true (the default), also patch ``threading``.
- If *_threading_local* is true (the default), also patch ``_threading_local.local``.
......@@ -259,7 +263,33 @@ def patch_thread(threading=True, _threading_local=True, Event=False, logging=Tru
- If *existing_locks* is True (the default), and the process is still single threaded,
make sure than any :class:`threading.RLock` (and, under Python 3, :class:`importlib._bootstrap._ModuleLock`)
instances that are currently locked can be properly unlocked.
.. caution::
Monkey-patching :mod:`thread` and using
:class:`multiprocessing.Queue` or
:class:`concurrent.futures.ProcessPoolExecutor` (which uses a
``Queue``) will hang the process.
"""
# Description of the hang:
# There is an incompatibility with patching 'thread' and the 'multiprocessing' module:
# The problem is that multiprocessing.queues.Queue uses a half-duplex multiprocessing.Pipe,
# which is implemented with os.pipe() and _multiprocessing.Connection. os.pipe isn't patched
# by gevent, as it returns just a fileno. _multiprocessing.Connection is an internal implementation
# class implemented in C, which exposes a 'poll(timeout)' method; under the covers, this issues a
# (blocking) select() call: hence the need for a real thread. Except for that method, we could
# almost replace Connection with gevent.fileobject.SocketAdapter, plus a trivial
# patch to os.pipe (below). Sigh, so close. (With a little work, we could replicate that method)
# import os
# import fcntl
# os_pipe = os.pipe
# def _pipe():
# r, w = os_pipe()
# fcntl.fcntl(r, fcntl.F_SETFL, os.O_NONBLOCK)
# fcntl.fcntl(w, fcntl.F_SETFL, os.O_NONBLOCK)
# return r, w
# os.pipe = _pipe
patch_module('thread')
if threading:
patch_module('threading')
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment