Add a 'timeout' argument to subprocess.Popen.

If the timeout expires before the subprocess exits, the wait method and the
communicate method will raise a subprocess.TimeoutExpired exception.  When used
with communicate, it is possible to catch the exception, kill the process, and
retry the communicate and receive any output written to stdout or stderr.

Doc/library/subprocess.rst

@@ -249,15 +249,21 @@ Convenience Functions
 This module also defines four shortcut functions:
 
 
-.. function:: call(*popenargs, **kwargs)
+.. function:: call(*popenargs, timeout=None, **kwargs)
 
    Run command with arguments.  Wait for command to complete, then return the
    :attr:`returncode` attribute.
 
-   The arguments are the same as for the :class:`Popen` constructor.  Example::
+   The arguments are the same as for the :class:`Popen` constructor, with the
+   exception of the *timeout* argument, which is given to :meth:`Popen.wait`.
+   Example::
 
       >>> retcode = subprocess.call(["ls", "-l"])
 
+   If the timeout expires, the child process will be killed and then waited for
+   again.  The :exc:`TimeoutExpired` exception will be re-raised after the child
+   process has terminated.
+
    .. warning::
 
       Like :meth:`Popen.wait`, this will deadlock when using
@@ -265,34 +271,43 @@ This module also defines four shortcut functions:
       generates enough output to a pipe such that it blocks waiting
       for the OS pipe buffer to accept more data.
 
+   .. versionchanged:: 3.2
+      *timeout* was added.
+
 
-.. function:: check_call(*popenargs, **kwargs)
+.. function:: check_call(*popenargs, timeout=None, **kwargs)
 
    Run command with arguments.  Wait for command to complete. If the exit code was
    zero then return, otherwise raise :exc:`CalledProcessError`. The
    :exc:`CalledProcessError` object will have the return code in the
    :attr:`returncode` attribute.
 
-   The arguments are the same as for the :class:`Popen` constructor.  Example::
+   The arguments are the same as for the :func:`call` function.  Example::
 
       >>> subprocess.check_call(["ls", "-l"])
       0
 
+   As in the :func:`call` function, if the timeout expires, the child process
+   will be killed and the wait retried.  The :exc:`TimeoutExpired` exception
+   will be re-raised after the child process has terminated.
+
    .. warning::
 
       See the warning for :func:`call`.
 
+   .. versionchanged:: 3.2
+      *timeout* was added.
 
-.. function:: check_output(*popenargs, **kwargs)
+
+.. function:: check_output(*popenargs, timeout=None, **kwargs)
 
    Run command with arguments and return its output as a byte string.
 
    If the exit code was non-zero it raises a :exc:`CalledProcessError`.  The
    :exc:`CalledProcessError` object will have the return code in the
-   :attr:`returncode`
-   attribute and output in the :attr:`output` attribute.
+   :attr:`returncode` attribute and output in the :attr:`output` attribute.
 
-   The arguments are the same as for the :class:`Popen` constructor.  Example::
+   The arguments are the same as for the :func:`call` function.  Example::
 
       >>> subprocess.check_output(["ls", "-l", "/dev/null"])
       b'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'
@@ -305,8 +320,17 @@ This module also defines four shortcut functions:
       ...     stderr=subprocess.STDOUT)
       b'ls: non_existent_file: No such file or directory\n'
 
+   As in the :func:`call` function, if the timeout expires, the child process
+   will be killed and the wait retried.  The :exc:`TimeoutExpired` exception
+   will be re-raised after the child process has terminated.  The output from
+   the child process so far will be in the :attr:`output` attribute of the
+   exception.
+
    .. versionadded:: 3.1
 
+   .. versionchanged:: 3.2
+      *timeout* was added.
+
 
 .. function:: getstatusoutput(cmd)
 
@@ -359,6 +383,10 @@ arguments.
 check_call() will raise :exc:`CalledProcessError`, if the called process returns
 a non-zero return code.
 
+All of the functions and methods that accept a *timeout* parameter, such as
+:func:`call` and :meth:`Popen.communicate` will raise :exc:`TimeoutExpired` if
+the timeout expires before the process exits.
+
 
 Security
 ^^^^^^^^
@@ -380,11 +408,15 @@ Instances of the :class:`Popen` class have the following methods:
    attribute.
 
 
-.. method:: Popen.wait()
+.. method:: Popen.wait(timeout=None)
 
    Wait for child process to terminate.  Set and return :attr:`returncode`
    attribute.
 
+   If the process does not terminate after *timeout* seconds, raise a
+   :exc:`TimeoutExpired` exception.  It is safe to catch this exception and
+   retry the wait.
+
    .. warning::
 
       This will deadlock when using ``stdout=PIPE`` and/or
@@ -392,11 +424,14 @@ Instances of the :class:`Popen` class have the following methods:
       a pipe such that it blocks waiting for the OS pipe buffer to
       accept more data.  Use :meth:`communicate` to avoid that.
 
+   .. versionchanged:: 3.2
+      *timeout* was added.
 
-.. method:: Popen.communicate(input=None)
+
+.. method:: Popen.communicate(input=None, timeout=None)
 
    Interact with process: Send data to stdin.  Read data from stdout and stderr,
-   until end-of-file is reached.  Wait for process to terminate. The optional
+   until end-of-file is reached.  Wait for process to terminate.  The optional
    *input* argument should be a byte string to be sent to the child process, or
    ``None``, if no data should be sent to the child.
 
@@ -407,11 +442,29 @@ Instances of the :class:`Popen` class have the following methods:
    ``None`` in the result tuple, you need to give ``stdout=PIPE`` and/or
    ``stderr=PIPE`` too.
 
+   If the process does not terminate after *timeout* seconds, a
+   :exc:`TimeoutExpired` exception will be raised.  Catching this exception and
+   retrying communication will not lose any output.
+
+   The child process is not killed if the timeout expires, so in order to
+   cleanup properly a well-behaved application should kill the child process and
+   finish communication::
+
+      proc = subprocess.Popen(...)
+      try:
+          outs, errs = proc.communicate(timeout=15)
+      except TimeoutExpired:
+          proc.kill()
+          outs, errs = proc.communicate()
+
    .. note::
 
       The data read is buffered in memory, so do not use this method if the data
       size is large or unlimited.
 
+   .. versionchanged:: 3.2
+      *timeout* was added.
+
 
 .. method:: Popen.send_signal(signal)
 

Lib/test/test_subprocess.py

@@ -56,6 +56,8 @@ class BaseTestCase(unittest.TestCase):
         # shutdown time.  That frustrates tests trying to check stderr produced
         # from a spawned Python process.
         actual = support.strip_python_stderr(stderr)
+        # strip_python_stderr also strips whitespace, so we do too.
+        expected = expected.strip()
         self.assertEqual(actual, expected, msg)
 
 
@@ -67,6 +69,15 @@ class ProcessTestCase(BaseTestCase):
                               "import sys; sys.exit(47)"])
         self.assertEqual(rc, 47)
 
+    def test_call_timeout(self):
+        # call() function with timeout argument; we want to test that the child
+        # process gets killed when the timeout expires.  If the child isn't
+        # killed, this call will deadlock since subprocess.call waits for the
+        # child.
+        self.assertRaises(subprocess.TimeoutExpired, subprocess.call,
+                          [sys.executable, "-c", "while True: pass"],
+                          timeout=0.1)
+
     def test_check_call_zero(self):
         # check_call() function with zero return code
         rc = subprocess.check_call([sys.executable, "-c",
@@ -109,6 +120,18 @@ class ProcessTestCase(BaseTestCase):
             self.fail("Expected ValueError when stdout arg supplied.")
         self.assertIn('stdout', c.exception.args[0])
 
+    def test_check_output_timeout(self):
+        # check_output() function with timeout arg
+        with self.assertRaises(subprocess.TimeoutExpired) as c:
+            output = subprocess.check_output(
+                    [sys.executable, "-c",
+                     "import sys; sys.stdout.write('BDFL')\n"
+                     "sys.stdout.flush()\n"
+                     "while True: pass"],
+                    timeout=0.5)
+            self.fail("Expected TimeoutExpired.")
+        self.assertEqual(c.exception.output, b'BDFL')
+
     def test_call_kwargs(self):
         # call() function with keyword args
         newenv = os.environ.copy()
@@ -366,6 +389,41 @@ class ProcessTestCase(BaseTestCase):
         self.assertEqual(stdout, b"banana")
         self.assertStderrEqual(stderr, b"pineapple")
 
+    def test_communicate_timeout(self):
+        p = subprocess.Popen([sys.executable, "-c",
+                              'import sys,os,time;'
+                              'sys.stderr.write("pineapple\\n");'
+                              'time.sleep(1);'
+                              'sys.stderr.write("pear\\n");'
+                              'sys.stdout.write(sys.stdin.read())'],
+                             universal_newlines=True,
+                             stdin=subprocess.PIPE,
+                             stdout=subprocess.PIPE,
+                             stderr=subprocess.PIPE)
+        self.assertRaises(subprocess.TimeoutExpired, p.communicate, "banana",
+                          timeout=0.3)
+        # Make sure we can keep waiting for it, and that we get the whole output
+        # after it completes.
+        (stdout, stderr) = p.communicate()
+        self.assertEqual(stdout, "banana")
+        self.assertStderrEqual(stderr.encode(), b"pineapple\npear\n")
+
+    def test_communicate_timeout_large_ouput(self):
+        # Test a expring timeout while the child is outputting lots of data.
+        p = subprocess.Popen([sys.executable, "-c",
+                              'import sys,os,time;'
+                              'sys.stdout.write("a" * (64 * 1024));'
+                              'time.sleep(0.2);'
+                              'sys.stdout.write("a" * (64 * 1024));'
+                              'time.sleep(0.2);'
+                              'sys.stdout.write("a" * (64 * 1024));'
+                              'time.sleep(0.2);'
+                              'sys.stdout.write("a" * (64 * 1024));'],
+                             stdout=subprocess.PIPE)
+        self.assertRaises(subprocess.TimeoutExpired, p.communicate, timeout=0.4)
+        (stdout, _) = p.communicate()
+        self.assertEqual(len(stdout), 4 * 64 * 1024)
+
     # Test for the fd leak reported in http://bugs.python.org/issue2791.
     def test_communicate_pipe_fd_leak(self):
         for stdin_pipe in (False, True):
@@ -561,6 +619,13 @@ class ProcessTestCase(BaseTestCase):
         self.assertEqual(p.wait(), 0)
 
 
+    def test_wait_timeout(self):
+        p = subprocess.Popen([sys.executable,
+                              "-c", "import time; time.sleep(1)"])
+        self.assertRaises(subprocess.TimeoutExpired, p.wait, timeout=0.1)
+        self.assertEqual(p.wait(timeout=2), 0)
+
+
     def test_invalid_bufsize(self):
         # an invalid type of the bufsize argument should raise
         # TypeError.

PC/_subprocess.c

@@ -682,6 +682,7 @@ PyInit__subprocess()
     defint(d, "SW_HIDE", SW_HIDE);
     defint(d, "INFINITE", INFINITE);
     defint(d, "WAIT_OBJECT_0", WAIT_OBJECT_0);
+    defint(d, "WAIT_TIMEOUT", WAIT_TIMEOUT);
     defint(d, "CREATE_NEW_CONSOLE", CREATE_NEW_CONSOLE);
     defint(d, "CREATE_NEW_PROCESS_GROUP", CREATE_NEW_PROCESS_GROUP);