docs about #704 [skip ci]

changelog.rst

@@ -12,6 +12,14 @@
   shipped with. Some Linux distributions, including RedHat/CentOS and
   Amazon have backported the changes to older versions. Reported in
   :issue:`702`.
+- PyPy: An interaction between Cython compiled code and the garbage
+  collector caused PyPy to crash when a previously-allocated Semaphore
+  was used in a ``__del__`` method, something done in the popular
+  libraries ``requests`` and ``urllib3``. Due to this and other Cython
+  related issues, the Semaphore class is no longer compiled by Cython.
+  This means that it is now traceable and not exactly as atomic as the
+  Cython version, though the overall semantics should remain the same.
+  Reported in :issue:`704` by Shaun Crampton.
 
 1.1rc2 (Dec 11, 2015)
 =====================

doc/whatsnew_1_1.rst

@@ -50,25 +50,24 @@ through 4.0.0 and 4.0.1.
   of PyPy. The benchmarks distributed with gevent typically perform as
   well or better on PyPy than on CPython at least on some platforms.
   Things that are known or expected to be (relatively) slower under
-  PyPy include the :mod:`c-ares resolver <gevent.resolver_ares>` and
-  :class:`gevent.lock.Semaphore`. Whether or not these matter will
-  depend on the workload of each application.
-
-.. note:: Released versions of PyPy through at least 4.0.0 have `a
-          bug`_ that can cause a memory leak when subclassing
-          objects that are implemented in Cython, as are the two
-          things mentioned above. The `Semaphore` class is
-          subclassed to become :class:`gevent.thread.LockType`,
-          which in turn is used as the basis for
-          :class:`threading.Lock`. The `Semaphore` object is coded
-          carefully to avoid this leak, assuming it is deallocated
-          when not acquired (which should be the typical case). The
-          ``c-ares`` package has not been audited for this issue.
+  PyPy include the :mod:`c-ares resolver <gevent.resolver_ares>`.
+  Whether or not these matter will depend on the workload of each
+  application.
+
+.. caution:: The ``c-ares`` resolver is not recommended for use under
+             PyPy. Released versions of PyPy through at least 4.0.1
+             have `a bug`_ that can cause a memory leak when
+             subclassing objects that are implemented in Cython, as is
+             the c-ares resolver. things mentioned above. The
+             ``c-ares`` package has not been audited for this issue.
+             In addition, thanks to reports like :issue:`704`, we know
+             that the PyPy garbage collector can interact badly with
+             Cython-compiled code, leading to crashes.
 
 .. note:: PyPy 4.0.x on Linux is known to *rarely* (once per 24 hours)
           encounter crashes when running heavily loaded, heavily
-          networked gevent programs. The exact cause is unknown and is
-          being tracked in :issue:`677`.
+          networked gevent programs (even without ``c-ares``). The
+          exact cause is unknown and is being tracked in :issue:`677`.
 
 .. _cffi 1.3.0: https://bitbucket.org/cffi/cffi/src/ad3140a30a7b0ca912185ef500546a9fb5525ece/doc/source/whatsnew.rst?at=default
 .. _1.2.0: https://cffi.readthedocs.org/en/latest/whatsnew.html#v1-2-0

gevent/resolver_ares.py

@@ -27,8 +27,9 @@ class Resolver(object):
     reports of it not properly honoring certain system configurations.
     However, because it does not use threads, it may scale better.
 
-    .. note:: This module is considered experimental on PyPy, and
-       due to its implementation in cython, it may be slower.
+    .. caution:: This module is considered extremely experimental on PyPy, and
+       due to its implementation in cython, it may be slower. It may also lead to
+       interpreter crashes.
 
     .. _c-ares: http://c-ares.haxx.se
     """

greentest/test__semaphore.py

@@ -34,7 +34,8 @@ class TestSemaphore(greentest.TestCase):
 
     def test_semaphore_in_class_with_del(self):
         # Issue #704. This used to crash the process
-        # under PyPy through at least 4.0.1
+        # under PyPy through at least 4.0.1 if the Semaphore
+        # was implemented with Cython.
         class X(object):
             def __init__(self):
                 self.s = Semaphore()

greentest/test__threading_vs_settrace.py

@@ -48,6 +48,12 @@ sys.stdout.write("..finishing..")
 
 class TestTrace(unittest.TestCase):
     def test_untraceable_lock(self):
+        # Untraceable locks were part of the solution to https://bugs.python.org/issue1733757
+        # which details a deadlock that could happen if a trace function invoked
+        # threading.currentThread at shutdown time---the cleanup lock would be held
+        # by the VM, and calling currentThread would try to acquire it again. The interpreter
+        # changed in 2.6 to use the `with` statement (https://hg.python.org/cpython/rev/76f577a9ec03/),
+        # which apparently doesn't trace in quite the same way.
         if hasattr(sys, 'gettrace'):
             old = sys.gettrace()
         else:
@@ -57,7 +63,7 @@ class TestTrace(unittest.TestCase):
         try:
             def trace(frame, ev, arg):
                 lst.append((frame.f_code.co_filename, frame.f_lineno, ev))
-                if not PYPY:
+                if not PYPY: # because we expect to trace on PyPy
                     print("TRACE: %s:%s %s" % lst[-1])
                 return trace