merge heads

Doc/glossary.rst

@@ -385,7 +385,7 @@ Glossary
       :meth:`str.lower` method can serve as a key function for case insensitive
       sorts.  Alternatively, an ad-hoc key function can be built from a
       :keyword:`lambda` expression such as ``lambda r: (r[0], r[2])``.  Also,
-      the :mod:`operator` module provides three key function constuctors:
+      the :mod:`operator` module provides three key function constructors:
       :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and
       :func:`~operator.methodcaller`.  See the :ref:`Sorting HOW TO
       <sortinghowto>` for examples of how to create and use key functions.

Doc/library/argparse.rst

@@ -1642,8 +1642,8 @@ Argument groups
 
        --bar BAR  bar help
 
-   Note that any arguments not your user defined groups will end up back in the
-   usual "positional arguments" and "optional arguments" sections.
+   Note that any arguments not in your user-defined groups will end up back
+   in the usual "positional arguments" and "optional arguments" sections.
 
 
 Mutual exclusion
@@ -1833,9 +1833,10 @@ A partial upgrade path from :mod:`optparse` to :mod:`argparse`:
 * Replace all :meth:`optparse.OptionParser.add_option` calls with
   :meth:`ArgumentParser.add_argument` calls.
 
-* Replace ``options, args = parser.parse_args()`` with ``args =
+* Replace ``(options, args) = parser.parse_args()`` with ``args =
   parser.parse_args()`` and add additional :meth:`ArgumentParser.add_argument`
-  calls for the positional arguments.
+  calls for the positional arguments. Keep in mind that what was previously
+  called ``options``, now in :mod:`argparse` context is called ``args``.
 
 * Replace callback actions and the ``callback_*`` keyword arguments with
   ``type`` or ``action`` arguments.

Doc/library/logging.handlers.rst

@@ -654,7 +654,7 @@ event of a certain severity or greater is seen.
 :class:`BufferingHandler`, which is an abstract class. This buffers logging
 records in memory. Whenever each record is added to the buffer, a check is made
 by calling :meth:`shouldFlush` to see if the buffer should be flushed.  If it
-should, then :meth:`flush` is expected to do the needful.
+should, then :meth:`flush` is expected to do the flushing.
 
 
 .. class:: BufferingHandler(capacity)

Doc/library/signal.rst

@@ -5,46 +5,58 @@
    :synopsis: Set handlers for asynchronous events.
 
 
-This module provides mechanisms to use signal handlers in Python. Some general
-rules for working with signals and their handlers:
-
-* A handler for a particular signal, once set, remains installed until it is
-  explicitly reset (Python emulates the BSD style interface regardless of the
-  underlying implementation), with the exception of the handler for
-  :const:`SIGCHLD`, which follows the underlying implementation.
-
-* There is no way to "block" signals temporarily from critical sections (since
-  this is not supported by all Unix flavors).
-
-* Although Python signal handlers are called asynchronously as far as the Python
-  user is concerned, they can only occur between the "atomic" instructions of the
-  Python interpreter.  This means that signals arriving during long calculations
-  implemented purely in C (such as regular expression matches on large bodies of
-  text) may be delayed for an arbitrary amount of time.
-
-* When a signal arrives during an I/O operation, it is possible that the I/O
-  operation raises an exception after the signal handler returns. This is
-  dependent on the underlying Unix system's semantics regarding interrupted system
-  calls.
-
-* Because the C signal handler always returns, it makes little sense to catch
-  synchronous errors like :const:`SIGFPE` or :const:`SIGSEGV`.
-
-* Python installs a small number of signal handlers by default: :const:`SIGPIPE`
-  is ignored (so write errors on pipes and sockets can be reported as ordinary
-  Python exceptions) and :const:`SIGINT` is translated into a
-  :exc:`KeyboardInterrupt` exception.  All of these can be overridden.
-
-* Some care must be taken if both signals and threads are used in the same
-  program.  The fundamental thing to remember in using signals and threads
-  simultaneously is: always perform :func:`signal` operations in the main thread
-  of execution.  Any thread can perform an :func:`alarm`, :func:`getsignal`,
-  :func:`pause`, :func:`setitimer` or :func:`getitimer`; only the main thread
-  can set a new signal handler, and the main thread will be the only one to
-  receive signals (this is enforced by the Python :mod:`signal` module, even
-  if the underlying thread implementation supports sending signals to
-  individual threads).  This means that signals can't be used as a means of
-  inter-thread communication.  Use locks instead.
+This module provides mechanisms to use signal handlers in Python.
+
+
+General rules
+-------------
+
+The :func:`signal.signal` function allows to define custom handlers to be
+executed when a signal is received.  A small number of default handlers are
+installed: :const:`SIGPIPE` is ignored (so write errors on pipes and sockets
+can be reported as ordinary Python exceptions) and :const:`SIGINT` is
+translated into a :exc:`KeyboardInterrupt` exception.
+
+A handler for a particular signal, once set, remains installed until it is
+explicitly reset (Python emulates the BSD style interface regardless of the
+underlying implementation), with the exception of the handler for
+:const:`SIGCHLD`, which follows the underlying implementation.
+
+There is no way to "block" signals temporarily from critical sections (since
+this is not supported by all Unix flavors).
+
+
+Execution of Python signal handlers
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A Python signal handler does not get executed inside the low-level (C) signal
+handler.  Instead, the low-level signal handler sets a flag which tells the
+:term:`virtual machine` to execute the corresponding Python signal handler
+at a later point(for example at the next :term:`bytecode` instruction).
+This has consequences:
+
+* It makes little sense to catch synchronous errors like :const:`SIGFPE` or
+  :const:`SIGSEGV`.
+
+* A long-running calculation implemented purely in C (such as regular
+  expression matching on a large body of text) may run uninterrupted for an
+  arbitrary amount of time, regardless of any signals received.  The Python
+  signal handlers will be called when the calculation finishes.
+
+
+Signals and threads
+^^^^^^^^^^^^^^^^^^^
+
+Python signal handlers are always executed in the main Python thread,
+even if the signal was received in another thread.  This means that signals
+can't be used as a means of inter-thread communication.  You can use
+the synchronization primitives from the :mod:`threading` module instead.
+
+Besides, only the main thread is allowed to set a new signal handler.
+
+
+Module contents
+---------------
 
 The variables defined in the :mod:`signal` module are:
 

Doc/library/syslog.rst

@@ -78,7 +78,8 @@ Priority levels (high to low):
 Facilities:
    :const:`LOG_KERN`, :const:`LOG_USER`, :const:`LOG_MAIL`, :const:`LOG_DAEMON`,
    :const:`LOG_AUTH`, :const:`LOG_LPR`, :const:`LOG_NEWS`, :const:`LOG_UUCP`,
-   :const:`LOG_CRON` and :const:`LOG_LOCAL0` to :const:`LOG_LOCAL7`.
+   :const:`LOG_CRON`, :const:`LOG_SYSLOG` and :const:`LOG_LOCAL0` to
+   :const:`LOG_LOCAL7`.
 
 Log options:
    :const:`LOG_PID`, :const:`LOG_CONS`, :const:`LOG_NDELAY`, :const:`LOG_NOWAIT`

Doc/library/unittest.rst

@@ -640,7 +640,7 @@ This is the output of running the example above in verbose mode: ::
 
 Classes can be skipped just like methods: ::
 
-   @skip("showing class skipping")
+   @unittest.skip("showing class skipping")
    class MySkippedTestCase(unittest.TestCase):
        def test_not_run(self):
            pass

Doc/library/webbrowser.rst

@@ -137,6 +137,8 @@ for the controller classes, all defined in this module.
 +-----------------------+-----------------------------------------+-------+
 | ``'macosx'``          | :class:`MacOSX('default')`              | \(4)  |
 +-----------------------+-----------------------------------------+-------+
+| ``'safari'``          | :class:`MacOSX('safari')`               | \(4)  |
++-----------------------+-----------------------------------------+-------+
 
 Notes:
 

Lib/concurrent/futures/_base.py

@@ -112,9 +112,11 @@ class _AllCompletedWaiter(_Waiter):
     def __init__(self, num_pending_calls, stop_on_exception):
         self.num_pending_calls = num_pending_calls
         self.stop_on_exception = stop_on_exception
+        self.lock = threading.Lock()
         super().__init__()
 
     def _decrement_pending_calls(self):
+        with self.lock:
             self.num_pending_calls -= 1
             if not self.num_pending_calls:
                 self.event.set()

Lib/idlelib/NEWS.txt

 What's New in IDLE 3.2.3?
 =========================
 
+- Issue #14409: IDLE now properly executes commands in the Shell window
+  when it cannot read the normal config files on startup and
+  has to use the built-in default key bindings.
+  There was previously a bug in one of the defaults.
+
 - Issue #3573: IDLE hangs when passing invalid command line args
   (directory(ies) instead of file(s)).
 

Lib/idlelib/configHandler.py

@@ -595,7 +595,7 @@ class IdleConf:
             '<<replace>>': ['<Control-h>'],
             '<<goto-line>>': ['<Alt-g>'],
             '<<smart-backspace>>': ['<Key-BackSpace>'],
-            '<<newline-and-indent>>': ['<Key-Return> <Key-KP_Enter>'],
+            '<<newline-and-indent>>': ['<Key-Return>', '<Key-KP_Enter>'],
             '<<smart-indent>>': ['<Key-Tab>'],
             '<<indent-region>>': ['<Control-Key-bracketright>'],
             '<<dedent-region>>': ['<Control-Key-bracketleft>'],

Lib/logging/handlers.py

@@ -519,11 +519,16 @@ class SocketHandler(logging.Handler):
         """
         ei = record.exc_info
         if ei:
-            dummy = self.format(record) # just to get traceback text into record.exc_text
-            record.exc_info = None  # to avoid Unpickleable error
-        s = pickle.dumps(record.__dict__, 1)
-        if ei:
-            record.exc_info = ei  # for next handler
+            # just to get traceback text into record.exc_text ...
+            dummy = self.format(record)
+        # See issue #14436: If msg or args are objects, they may not be
+        # available on the receiving end. So we convert the msg % args
+        # to a string, save it as msg and zap the args.
+        d = dict(record.__dict__)
+        d['msg'] = record.getMessage()
+        d['args'] = None
+        d['exc_info'] = None
+        s = pickle.dumps(d, 1)
         slen = struct.pack(">L", len(s))
         return slen + s
 

Lib/multiprocessing/connection.py

@@ -94,6 +94,13 @@ def arbitrary_address(family):
     else:
         raise ValueError('unrecognized family')
 
+def _validate_family(family):
+    '''
+    Checks if the family is valid for the current environment.
+    '''
+    if sys.platform != 'win32' and family == 'AF_PIPE':
+        raise ValueError('Family %s is not recognized.' % family)
+
 
 def address_type(address):
     '''
@@ -126,6 +133,7 @@ class Listener(object):
                  or default_family
         address = address or arbitrary_address(family)
 
+        _validate_family(family)
         if family == 'AF_PIPE':
             self._listener = PipeListener(address, backlog)
         else:
@@ -163,6 +171,7 @@ def Client(address, family=None, authkey=None):
     Returns a connection to the address of a `Listener`
     '''
     family = family or address_type(address)
+    _validate_family(family)
     if family == 'AF_PIPE':
         c = PipeClient(address)
     else:

Lib/pydoc.py

@@ -1829,7 +1829,7 @@ has the same effect as typing a particular string at the help> prompt.
 Welcome to Python %s!  This is the online help utility.
 
 If this is your first time using Python, you should definitely check out
-the tutorial on the Internet at http://docs.python.org/tutorial/.
+the tutorial on the Internet at http://docs.python.org/%s/tutorial/.
 
 Enter the name of any module, keyword, or topic to get help on writing
 Python programs and using Python modules.  To quit this help utility and
@@ -1839,7 +1839,7 @@ To get a list of available modules, keywords, or topics, type "modules",
 "keywords", or "topics".  Each module also comes with a one-line summary
 of what it does; to list the modules whose summaries contain a given word
 such as "spam", type "modules spam".
-''' % sys.version[:3])
+''' % tuple([sys.version[:3]]*2))
 
     def list(self, items, columns=4, width=80):
         items = list(sorted(items))

Lib/rlcompleter.py

-"""Word completion for GNU readline 2.0.
+"""Word completion for GNU readline.
 
-This requires the latest extension to the readline module. The completer
-completes keywords, built-ins and globals in a selectable namespace (which
-defaults to __main__); when completing NAME.NAME..., it evaluates (!) the
-expression up to the last dot and completes its attributes.
+The completer completes keywords, built-ins and globals in a selectable
+namespace (which defaults to __main__); when completing NAME.NAME..., it
+evaluates (!) the expression up to the last dot and completes its attributes.
 
-It's very cool to do "import sys" type "sys.", hit the
-completion key (twice), and see the list of names defined by the
-sys module!
+It's very cool to do "import sys" type "sys.", hit the completion key (twice),
+and see the list of names defined by the sys module!
 
 Tip: to use the tab key as the completion key, call
 
@@ -15,21 +13,19 @@ Tip: to use the tab key as the completion key, call
 
 Notes:
 
-- Exceptions raised by the completer function are *ignored* (and
-generally cause the completion to fail).  This is a feature -- since
-readline sets the tty device in raw (or cbreak) mode, printing a
-traceback wouldn't work well without some complicated hoopla to save,
-reset and restore the tty state.
+- Exceptions raised by the completer function are *ignored* (and generally cause
+  the completion to fail).  This is a feature -- since readline sets the tty
+  device in raw (or cbreak) mode, printing a traceback wouldn't work well
+  without some complicated hoopla to save, reset and restore the tty state.
 
-- The evaluation of the NAME.NAME... form may cause arbitrary
-application defined code to be executed if an object with a
-__getattr__ hook is found.  Since it is the responsibility of the
-application (or the user) to enable this feature, I consider this an
-acceptable risk.  More complicated expressions (e.g. function calls or
-indexing operations) are *not* evaluated.
+- The evaluation of the NAME.NAME... form may cause arbitrary application
+  defined code to be executed if an object with a __getattr__ hook is found.
+  Since it is the responsibility of the application (or the user) to enable this
+  feature, I consider this an acceptable risk.  More complicated expressions
+  (e.g. function calls or indexing operations) are *not* evaluated.
 
 - When the original stdin is not a tty device, GNU readline is never
-used, and this module (and the readline module) are silently inactive.
+  used, and this module (and the readline module) are silently inactive.
 
 """
 

Lib/socket.py

@@ -197,6 +197,17 @@ class socket(_socket.socket):
         if self._io_refs <= 0:
             self._real_close()
 
+    def detach(self):
+        """detach() -> file descriptor
+
+        Close the socket object without closing the underlying file descriptor.
+        The object cannot be used after this call, but the file descriptor
+        can be reused for other purposes.  The file descriptor is returned.
+        """
+        self._closed = True
+        return super().detach()
+
+
 def fromfd(fd, family, type, proto=0):
     """ fromfd(fd, family, type[, proto]) -> socket object
 

Lib/test/test_concurrent_futures.py

@@ -183,7 +183,9 @@ class ProcessPoolShutdownTest(ProcessPoolMixin, ExecutorShutdownTest):
         for p in processes:
             p.join()
 
+
 class WaitTests(unittest.TestCase):
+
     def test_first_completed(self):
         future1 = self.executor.submit(mul, 21, 2)
         future2 = self.executor.submit(time.sleep, 1.5)
@@ -284,7 +286,21 @@ class WaitTests(unittest.TestCase):
 
 
 class ThreadPoolWaitTests(ThreadPoolMixin, WaitTests):
-    pass
+
+    def test_pending_calls_race(self):
+        # Issue #14406: multi-threaded race condition when waiting on all
+        # futures.
+        event = threading.Event()
+        def future_func():
+            event.wait()
+        oldswitchinterval = sys.getswitchinterval()
+        sys.setswitchinterval(1e-6)
+        try:
+            fs = {self.executor.submit(future_func) for i in range(100)}
+            event.set()
+            futures.wait(fs, return_when=futures.ALL_COMPLETED)
+        finally:
+            sys.setswitchinterval(oldswitchinterval)
 
 
 class ProcessPoolWaitTests(ProcessPoolMixin, WaitTests):

Lib/test/test_multiprocessing.py

@@ -2319,8 +2319,20 @@ class TestStdinBadfiledescriptor(unittest.TestCase):
         flike.flush()
         assert sio.getvalue() == 'foo'
 
+
+#
+# Issue 14151: Test invalid family on invalid environment
+#
+
+class TestInvalidFamily(unittest.TestCase):
+
+    @unittest.skipIf(WIN32, "skipped on Windows")
+    def test_invalid_family(self):
+        with self.assertRaises(ValueError):
+            multiprocessing.connection.Listener(r'\\.\test')
+
 testcases_other = [OtherTest, TestInvalidHandle, TestInitializers,
-                   TestStdinBadfiledescriptor]
+                   TestStdinBadfiledescriptor, TestInvalidFamily]
 
 #
 #

Lib/test/test_socket.py

@@ -951,6 +951,7 @@ class BasicTCPTest(SocketConnectedTest):
         f = self.cli_conn.detach()
         self.assertEqual(f, fileno)
         # cli_conn cannot be used anymore...
+        self.assertTrue(self.cli_conn._closed)
         self.assertRaises(socket.error, self.cli_conn.recv, 1024)
         self.cli_conn.close()
         # ...but we can create another socket using the (still open)

Misc/NEWS

@@ -10,6 +10,9 @@ What's New in Python 3.2.4
 Core and Builtins
 -----------------
 
+- Issue #13019: Fix potential reference leaks in bytearray.extend().  Patch
+  by Suman Saha.
+
 - Issue #14378: Fix compiling ast.ImportFrom nodes with a "__future__" string as
   the module name that was not interned.
 
@@ -31,6 +34,21 @@ Core and Builtins
 Library
 -------
 
+- Issue #14151: Raise a ValueError, not a NameError, when trying to create
+  a multiprocessing Client or Listener with an AF_PIPE type address under
+  non-Windows platforms.  Patch by Popa Claudiu.
+
+- Issue #13872: socket.detach() now marks the socket closed (as mirrored
+  in the socket repr()).  Patch by Matt Joiner.
+
+- Issue #14406: Fix a race condition when using ``concurrent.futures.wait(
+  return_when=ALL_COMPLETED)``.  Patch by Matt Joiner.
+
+- Issue #14409: IDLE now properly executes commands in the Shell window
+  when it cannot read the normal config files on startup and
+  has to use the built-in default key bindings.
+  There was previously a bug in one of the defaults.
+
 - Issue #10340: asyncore - properly handle EINVAL in dispatcher constructor on
   OSX; avoid to call handle_connect in case of a disconnected socket which
   was not meant to connect.
@@ -97,6 +115,8 @@ Extension Modules
 Build
 -----
 
+- Issue #14437: Fix building the _io module under Cygwin.
+
 - Issue #14387: Do not include accu.h from Python.h.
 
 - Issue #14359: Only use O_CLOEXEC in _posixmodule.c if it is defined.

Modules/_io/_iomodule.h

@@ -67,7 +67,7 @@ typedef struct {
     PyObject *filename; /* Not used, but part of the IOError object */
     Py_ssize_t written;
 } PyBlockingIOErrorObject;
-PyAPI_DATA(PyObject *) PyExc_BlockingIOError;
+extern PyObject *PyExc_BlockingIOError;
 
 /*
  * Offset type for positioning.

Modules/python.c

@@ -22,9 +22,9 @@ extern wchar_t* _Py_DecodeUTF8_surrogateescape(const char *s, Py_ssize_t size);
 int
 main(int argc, char **argv)
 {
-    wchar_t **argv_copy = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*)*argc);
+    wchar_t **argv_copy = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*)*(argc+1));
     /* We need a second copies, as Python might modify the first one. */
-    wchar_t **argv_copy2 = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*)*argc);
+    wchar_t **argv_copy2 = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*)*(argc+1));
     int i, res;
     char *oldloc;
     /* 754 requires that FP exceptions run in "no stop" mode by default,
@@ -58,6 +58,8 @@ main(int argc, char **argv)
         }
         argv_copy2[i] = argv_copy[i];
     }
+    argv_copy2[argc] = argv_copy[argc] = NULL;
+
     setlocale(LC_ALL, oldloc);
     free(oldloc);
     res = Py_Main(argc, argv_copy);

Objects/bytearrayobject.c

@@ -2234,8 +2234,10 @@ bytearray_extend(PyByteArrayObject *self, PyObject *arg)
     }
 
     bytearray_obj = PyByteArray_FromStringAndSize(NULL, buf_size);
-    if (bytearray_obj == NULL)
+    if (bytearray_obj == NULL) {
+        Py_DECREF(it);
         return NULL;
+    }
     buf = PyByteArray_AS_STRING(bytearray_obj);
 
     while ((item = PyIter_Next(it)) != NULL) {
@@ -2268,8 +2270,10 @@ bytearray_extend(PyByteArrayObject *self, PyObject *arg)
         return NULL;
     }
 
-    if (bytearray_setslice(self, Py_SIZE(self), Py_SIZE(self), bytearray_obj) == -1)
+    if (bytearray_setslice(self, Py_SIZE(self), Py_SIZE(self), bytearray_obj) == -1) {
+        Py_DECREF(bytearray_obj);
         return NULL;
+    }
     Py_DECREF(bytearray_obj);
 
     Py_RETURN_NONE;