Branch merge

Doc/c-api/init.rst

@@ -122,7 +122,7 @@ Process-wide parameters
    program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The
    returned string points into static storage; the caller should not modify its
    value.  This corresponds to the :makevar:`prefix` variable in the top-level
-   :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure`
+   :file:`Makefile` and the ``--prefix`` argument to the :program:`configure`
    script at build time.  The value is available to Python code as ``sys.prefix``.
    It is only useful on Unix.  See also the next function.
 
@@ -135,7 +135,7 @@ Process-wide parameters
    program name is ``'/usr/local/bin/python'``, the exec-prefix is
    ``'/usr/local'``.  The returned string points into static storage; the caller
    should not modify its value.  This corresponds to the :makevar:`exec_prefix`
-   variable in the top-level :file:`Makefile` and the :option:`--exec-prefix`
+   variable in the top-level :file:`Makefile` and the ``--exec-prefix``
    argument to the :program:`configure` script at build  time.  The value is
    available to Python code as ``sys.exec_prefix``.  It is only useful on Unix.
 

Doc/documenting/markup.rst

@@ -513,7 +513,10 @@ in a different style:
 
 .. describe:: keyword
 
-   The name of a keyword in Python.
+   The name of a Python keyword.  Using this role will generate a link to the
+   documentation of the keyword.  ``True``, ``False`` and ``None`` do not use
+   this role, but simple code markup (````True````), given that they're
+   fundamental to the language and should be known to any programmer.
 
 .. describe:: mailheader
 

Doc/faq/design.rst

@@ -667,7 +667,7 @@ construction of large programs.
 Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
 (ABCs).  You can then use :func:`isinstance` and :func:`issubclass` to check
 whether an instance or a class implements a particular ABC.  The
-:mod:`collections` modules defines a set of useful ABCs such as
+:mod:`collections.abc` module defines a set of useful ABCs such as
 :class:`Iterable`, :class:`Container`, and :class:`MutableMapping`.
 
 For Python, many of the advantages of interface specifications can be obtained

Doc/faq/programming.rst

@@ -473,15 +473,6 @@ calling another function by using ``*`` and ``**``::
        ...
        g(x, *args, **kwargs)
 
-In the unlikely case that you care about Python versions older than 2.0, use
-:func:`apply`::
-
-   def f(x, *args, **kwargs):
-       ...
-       kwargs['width'] = '14.3c'
-       ...
-       apply(g, (x,)+args, kwargs)
-
 
 How do I write a function with output parameters (call by reference)?
 ---------------------------------------------------------------------

Doc/faq/windows.rst

@@ -543,10 +543,10 @@ with multithreading-DLL options (``/MD``).
 
 If you can't change compilers or flags, try using :c:func:`Py_RunSimpleString`.
 A trick to get it to run an arbitrary file is to construct a call to
-:func:`execfile` with the name of your file as argument.
+:func:`exec` and :func:`open` with the name of your file as argument.
 
 Also note that you can not mix-and-match Debug and Release versions.  If you
-wish to use the Debug Multithreaded DLL, then your module *must* have an "_d"
+wish to use the Debug Multithreaded DLL, then your module *must* have ``_d``
 appended to the base name.
 
 

Doc/glossary.rst

@@ -434,8 +434,8 @@ Glossary
 
    mapping
       A container object that supports arbitrary key lookups and implements the
-      methods specified in the :class:`~collections.Mapping` or
-      :class:`~collections.MutableMapping`
+      methods specified in the :class:`~collections.abc.Mapping` or
+      :class:`~collections.abc.MutableMapping`
       :ref:`abstract base classes <collections-abstract-base-classes>`.  Examples
       include :class:`dict`, :class:`collections.defaultdict`,
       :class:`collections.OrderedDict` and :class:`collections.Counter`.
@@ -492,7 +492,7 @@ Glossary
       :func:`builtins.open` and :func:`os.open` are distinguished by their
       namespaces.  Namespaces also aid readability and maintainability by making
       it clear which module implements a function.  For instance, writing
-      :func:`random.seed` or :func:`itertools.izip` makes it clear that those
+      :func:`random.seed` or :func:`itertools.islice` makes it clear that those
       functions are implemented by the :mod:`random` and :mod:`itertools`
       modules, respectively.
 

Doc/howto/logging.rst

@@ -412,10 +412,10 @@ With the logger object configured, the following methods create log messages:
   :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with
   a message and a level that corresponds to their respective method names. The
   message is actually a format string, which may contain the standard string
-  substitution syntax of :const:`%s`, :const:`%d`, :const:`%f`, and so on.  The
+  substitution syntax of ``%s``, ``%d``, ``%f``, and so on.  The
   rest of their arguments is a list of objects that correspond with the
-  substitution fields in the message.  With regard to :const:`**kwargs`, the
-  logging methods care only about a keyword of :const:`exc_info` and use it to
+  substitution fields in the message.  With regard to ``**kwargs``, the
+  logging methods care only about a keyword of ``exc_info`` and use it to
   determine whether to log exception information.
 
 * :meth:`Logger.exception` creates a log message similar to

Doc/library/argparse.rst

@@ -155,7 +155,7 @@ ArgumentParser objects
      conflicting optionals.
 
    * prog_ - The name of the program (default:
-     :data:`sys.argv[0]`)
+     ``sys.argv[0]``)
 
    * usage_ - The string describing the program usage (default: generated)
 

Doc/library/base64.rst

@@ -45,8 +45,8 @@ The modern interface provides:
    at least length 2 (additional characters are ignored) which specifies the
    alternative alphabet used instead of the ``+`` and ``/`` characters.
 
-   The decoded string is returned.  A `binascii.Error` is raised if *s* is
-   incorrectly padded.
+   The decoded string is returned.  A :exc:`binascii.Error` exception is raised
+   if *s* is incorrectly padded.
 
    If *validate* is ``False`` (the default), non-base64-alphabet characters are
    discarded prior to the padding check.  If *validate* is ``True``,

Doc/library/configparser.rst

@@ -806,17 +806,17 @@ To get interpolation, use :class:`ConfigParser`::
    cfg = configparser.ConfigParser()
    cfg.read('example.cfg')
 
-   # Set the optional `raw` argument of get() to True if you wish to disable
+   # Set the optional *raw* argument of get() to True if you wish to disable
    # interpolation in a single get operation.
    print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!"
    print(cfg.get('Section1', 'foo', raw=True))  # -> "%(bar)s is %(baz)s!"
 
-   # The optional `vars` argument is a dict with members that will take
+   # The optional *vars* argument is a dict with members that will take
    # precedence in interpolation.
    print(cfg.get('Section1', 'foo', vars={'bar': 'Documentation',
                                              'baz': 'evil'}))
 
-   # The optional `fallback` argument can be used to provide a fallback value
+   # The optional *fallback* argument can be used to provide a fallback value
    print(cfg.get('Section1', 'foo'))
          # -> "Python is fun!"
 

Doc/library/email.header.rst

@@ -141,11 +141,11 @@ Here is the :class:`Header` class description:
       Returns an approximation of the :class:`Header` as a string, using an
       unlimited line length.  All pieces are converted to unicode using the
       specified encoding and joined together appropriately.  Any pieces with a
-      charset of `unknown-8bit` are decoded as `ASCII` using the `replace`
+      charset of ``'unknown-8bit'`` are decoded as ASCII using the ``'replace'``
       error handler.
 
       .. versionchanged:: 3.2
-         Added handling for the `unknown-8bit` charset.
+         Added handling for the ``'unknown-8bit'`` charset.
 
 
    .. method:: __eq__(other)

Doc/library/functions.rst

@@ -10,7 +10,7 @@ are always available.  They are listed here in alphabetical order.
 ===================  =================  ==================  ================  ====================
 ..                   ..                 Built-in Functions  ..                ..
 ===================  =================  ==================  ================  ====================
-:func:`abs`          :func:`dict`       :func:`help`        :func:`min`       :func:`setattr`
+:func:`abs`          |func-dict|_       :func:`help`        :func:`min`       :func:`setattr`
 :func:`all`          :func:`dir`        :func:`hex`         :func:`next`      :func:`slice`
 :func:`any`          :func:`divmod`     :func:`id`          :func:`object`    :func:`sorted`
 :func:`ascii`        :func:`enumerate`  :func:`input`       :func:`oct`       :func:`staticmethod`
@@ -19,13 +19,22 @@ are always available.  They are listed here in alphabetical order.
 :func:`bytearray`    :func:`filter`     :func:`issubclass`  :func:`pow`       :func:`super`
 :func:`bytes`        :func:`float`      :func:`iter`        :func:`print`     :func:`tuple`
 :func:`callable`     :func:`format`     :func:`len`         :func:`property`  :func:`type`
-:func:`chr`          :func:`frozenset`  :func:`list`        :func:`range`     :func:`vars`
+:func:`chr`          |func-frozenset|_  :func:`list`        :func:`range`     :func:`vars`
 :func:`classmethod`  :func:`getattr`    :func:`locals`      :func:`repr`      :func:`zip`
 :func:`compile`      :func:`globals`    :func:`map`         :func:`reversed`  :func:`__import__`
 :func:`complex`      :func:`hasattr`    :func:`max`         :func:`round`
-:func:`delattr`      :func:`hash`       :func:`memoryview`  :func:`set`
+:func:`delattr`      :func:`hash`       |func-memoryview|_  |func-set|_
 ===================  =================  ==================  ================  ====================
 
+.. using :func:`dict` would create a link to another page, so local targets are
+   used, with replacement texts to make the output in the table consistent
+
+.. |func-dict| replace:: ``dict()``
+.. |func-frozenset| replace:: ``frozenset()``
+.. |func-memoryview| replace:: ``memoryview()``
+.. |func-set| replace:: ``set()``
+
+
 .. function:: abs(x)
 
    Return the absolute value of a number.  The argument may be an
@@ -74,11 +83,12 @@ are always available.  They are listed here in alphabetical order.
 
 .. function:: bool([x])
 
-   Convert a value to a Boolean, using the standard truth testing procedure.  If
-   *x* is false or omitted, this returns :const:`False`; otherwise it returns
-   :const:`True`. :class:`bool` is also a class, which is a subclass of
-   :class:`int`. Class :class:`bool` cannot be subclassed further.  Its only
-   instances are :const:`False` and :const:`True`.
+   Convert a value to a Boolean, using the standard :ref:`truth testing
+   procedure <truth>`.  If *x* is false or omitted, this returns ``False``;
+   otherwise it returns ``True``. :class:`bool` is also a class, which is a
+   subclass of :class:`int` (see :ref:`typesnumeric`).  Class :class:`bool`
+   cannot be subclassed further.  Its only instances are ``False`` and
+   ``True`` (see :ref:`bltin-boolean-values`).
 
    .. index:: pair: Boolean; type
 
@@ -248,6 +258,7 @@ are always available.  They are listed here in alphabetical order.
    example, ``delattr(x, 'foobar')`` is equivalent to ``del x.foobar``.
 
 
+.. _func-dict:
 .. function:: dict([arg])
    :noindex:
 
@@ -491,6 +502,7 @@ are always available.  They are listed here in alphabetical order.
 
    The float type is described in :ref:`typesnumeric`.
 
+
 .. function:: format(value[, format_spec])
 
    .. index::
@@ -511,6 +523,8 @@ are always available.  They are listed here in alphabetical order.
    :exc:`TypeError` exception is raised if the method is not found or if either
    the *format_spec* or the return value are not strings.
 
+
+.. _func-frozenset:
 .. function:: frozenset([iterable])
    :noindex:
 
@@ -717,6 +731,8 @@ are always available.  They are listed here in alphabetical order.
    such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and
    ``heapq.nlargest(1, iterable, key=keyfunc)``.
 
+
+.. _func-memoryview:
 .. function:: memoryview(obj)
    :noindex:
 
@@ -1040,7 +1056,7 @@ are always available.  They are listed here in alphabetical order.
 
    Range objects implement the :class:`collections.Sequence` ABC, and provide
    features such as containment tests, element index lookup, slicing and
-   support for negative indices:
+   support for negative indices (see :ref:`typesseq`):
 
       >>> r = range(0, 20, 2)
       >>> r
@@ -1108,6 +1124,8 @@ are always available.  They are listed here in alphabetical order.
       can't be represented exactly as a float.  See :ref:`tut-fp-issues` for
       more information.
 
+
+.. _func-set:
 .. function:: set([iterable])
    :noindex:
 

Doc/library/inspect.rst

@@ -575,13 +575,13 @@ properties, will be invoked and :meth:`__getattr__` and :meth:`__getattribute__`
 may be called.
 
 For cases where you want passive introspection, like documentation tools, this
-can be inconvenient. `getattr_static` has the same signature as :func:`getattr`
+can be inconvenient. :func:`getattr_static` has the same signature as :func:`getattr`
 but avoids executing code when it fetches attributes.
 
 .. function:: getattr_static(obj, attr, default=None)
 
    Retrieve attributes without triggering dynamic lookup via the
-   descriptor protocol, `__getattr__` or `__getattribute__`.
+   descriptor protocol, :meth:`__getattr__` or :meth:`__getattribute__`.
 
    Note: this function may not be able to retrieve all attributes
    that getattr can fetch (like dynamically created attributes)
@@ -589,12 +589,12 @@ but avoids executing code when it fetches attributes.
    that raise AttributeError). It can also return descriptors objects
    instead of instance members.
 
-   If the instance `__dict__` is shadowed by another member (for example a
+   If the instance :attr:`__dict__` is shadowed by another member (for example a
    property) then this function will be unable to find instance members.
 
    .. versionadded:: 3.2
 
-`getattr_static` does not resolve descriptors, for example slot descriptors or
+:func:`getattr_static` does not resolve descriptors, for example slot descriptors or
 getset descriptors on objects implemented in C. The descriptor object
 is returned instead of the underlying attribute.
 

Doc/library/packaging.util.rst

@@ -90,34 +90,6 @@ This module contains various helpers for the other modules.
    Search the path for a given executable name.
 
 
-.. function:: subst_vars(s, local_vars)
-
-   Perform shell/Perl-style variable substitution on *s*.  Every occurrence of
-   ``$`` followed by a name is considered a variable, and variable is
-   substituted by the value found in the *local_vars* dictionary, or in
-   ``os.environ`` if it's not in *local_vars*. *os.environ* is first
-   checked/augmented to guarantee that it contains certain values: see
-   :func:`check_environ`.  Raise :exc:`ValueError` for any variables not found
-   in either *local_vars* or ``os.environ``.
-
-   Note that this is not a fully-fledged string interpolation function. A valid
-   ``$variable`` can consist only of upper and lower case letters, numbers and
-   an underscore. No { } or ( ) style quoting is available.
-
-
-.. function:: split_quoted(s)
-
-   Split a string up according to Unix shell-like rules for quotes and
-   backslashes. In short: words are delimited by spaces, as long as those spaces
-   are not escaped by a backslash, or inside a quoted string. Single and double
-   quotes are equivalent, and the quote characters can be backslash-escaped.
-   The backslash is stripped from any two-character escape sequence, leaving
-   only the escaped character.  The quote characters are stripped from any
-   quoted string.  Returns a list of words.
-
-   .. TODO Should probably be moved into the standard library.
-
-
 .. function:: execute(func, args[, msg=None, verbose=0, dry_run=0])
 
    Perform some action that affects the outside world (for instance, writing to
@@ -175,12 +147,3 @@ This module contains various helpers for the other modules.
    figure out to use direct compilation or not (see the source for details).
    The *direct* flag is used by the script generated in indirect mode; unless
    you know what you're doing, leave it set to ``None``.
-
-
-.. function:: rfc822_escape(header)
-
-   Return a version of *header* escaped for inclusion in an :rfc:`822` header, by
-   ensuring there are 8 spaces space after each newline.  Note that it does no
-   other modification of the string.
-
-   .. TODO this _can_ be replaced

Doc/library/stdtypes.rst

@@ -2724,6 +2724,8 @@ special operations.  There is exactly one ellipsis object, named
 It is written as ``Ellipsis`` or ``...``.
 
 
+.. _bltin-notimplemented-object:
+
 The NotImplemented Object
 -------------------------
 
@@ -2735,6 +2737,8 @@ information.  There is exactly one ``NotImplemented`` object.
 It is written as ``NotImplemented``.
 
 
+.. _bltin-boolean-values:
+
 Boolean Values
 --------------
 

Doc/library/string.rst

@@ -216,6 +216,8 @@ keyword.  If it's a number, it refers to a positional argument, and if it's a ke
 it refers to a named keyword argument.  If the numerical arg_names in a format string
 are 0, 1, 2, ... in sequence, they can all be omitted (not just some)
 and the numbers 0, 1, 2, ... will be automatically inserted in that order.
+Because *arg_name* is not quote-delimited, it is not possible to specify arbitrary
+dictionary keys (e.g., the strings ``'10'`` or ``':-]'``) within a format string.
 The *arg_name* can be followed by any number of index or
 attribute expressions. An expression of the form ``'.name'`` selects the named
 attribute using :func:`getattr`, while an expression of the form ``'[index]'``

Doc/library/unittest.rst

@@ -293,7 +293,7 @@ used from the command line. The basic command-line usage is::
 
    As a shortcut, ``python -m unittest`` is the equivalent of
    ``python -m unittest discover``. If you want to pass arguments to test
-   discovery the `discover` sub-command must be used explicitly.
+   discovery the ``discover`` sub-command must be used explicitly.
 
 The ``discover`` sub-command has the following options:
 
@@ -305,11 +305,11 @@ The ``discover`` sub-command has the following options:
 
 .. cmdoption:: -s directory
 
-   Directory to start discovery ('.' default)
+   Directory to start discovery (``.`` default)
 
 .. cmdoption:: -p pattern
 
-   Pattern to match test files ('test*.py' default)
+   Pattern to match test files (``test*.py`` default)
 
 .. cmdoption:: -t directory
 
@@ -724,8 +724,8 @@ Test cases
    single test.
 
    .. versionchanged:: 3.2
-      `TestCase` can be instantiated successfully without providing a method
-      name. This makes it easier to experiment with `TestCase` from the
+      :class:`TestCase` can be instantiated successfully without providing a method
+      name. This makes it easier to experiment with :class:`TestCase` from the
       interactive interpreter.
 
    *methodName* defaults to :meth:`runTest`.
@@ -944,17 +944,17 @@ Test cases
    +---------------------------------------------------------+--------------------------------------+------------+
    | Method                                                  | Checks that                          | New in     |
    +=========================================================+======================================+============+
-   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `exc`  |            |
+   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *exc*  |            |
    | <TestCase.assertRaises>`                                |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `exc`  | 3.1        |
-   | <TestCase.assertRaisesRegex>`                           | and the message matches `re`         |            |
+   | :meth:`assertRaisesRegex(exc, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *exc*  | 3.1        |
+   | <TestCase.assertRaisesRegex>`                           | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
+   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
    | <TestCase.assertWarns>`                                 |                                      |            |
    +---------------------------------------------------------+--------------------------------------+------------+
-   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises `warn` | 3.2        |
-   | <TestCase.assertWarnsRegex>`                            | and the message matches `re`         |            |
+   | :meth:`assertWarnsRegex(warn, re, fun, *args, **kwds)   | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
+   | <TestCase.assertWarnsRegex>`                            | and the message matches *re*         |            |
    +---------------------------------------------------------+--------------------------------------+------------+
 
    .. method:: assertRaises(exception, callable, *args, **kwds)
@@ -1116,7 +1116,7 @@ Test cases
    | :meth:`assertNotRegex(s, re)          | ``not regex.search(s)``        | 3.2          |
    | <TestCase.assertNotRegex>`            |                                |              |
    +---------------------------------------+--------------------------------+--------------+
-   | :meth:`assertCountEqual(a, b)         | `a` and `b` have the same      | 3.2          |
+   | :meth:`assertCountEqual(a, b)         | *a* and *b* have the same      | 3.2          |
    | <TestCase.assertCountEqual>`          | elements in the same number,   |              |
    |                                       | regardless of their order      |              |
    +---------------------------------------+--------------------------------+--------------+
@@ -1911,7 +1911,7 @@ Loading and running tests
 .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, runnerclass=None, warnings=None)
 
    A basic test runner implementation that outputs results to a stream. If *stream*
-   is `None`, the default, `sys.stderr` is used as the output stream. This class
+   is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class
    has a few configurable parameters, but is essentially very simple.  Graphical
    applications which run test suites should provide alternate implementations.
 
@@ -1928,7 +1928,7 @@ Loading and running tests
       Added the ``warnings`` argument.
 
    .. versionchanged:: 3.2
-      The default stream is set to `sys.stderr` at instantiation time rather
+      The default stream is set to :data:`sys.stderr` at instantiation time rather
       than import time.
 
    .. method:: _makeResult()

Doc/packaging/setupcfg.rst

@@ -756,8 +756,7 @@ needs to have its options defined in a dedicated section.  Here's an example::
    [files]
    packages = coconut
 
-   [extension=_fastcoconut]
-   name = coconut._fastcoconut
+   [extension: coconut._fastcoconut]
    language = cxx
    sources = cxx_src/cononut_utils.cxx
              cxx_src/python_module.cxx
@@ -768,8 +767,12 @@ needs to have its options defined in a dedicated section.  Here's an example::
        -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
        /DGECODE_VERSION='win32' -- sys.platform == 'win32'
 
-The section name must start with ``extension=``; the righ-hand part is currently
-discarded.  Valid fields and their values are listed in the documentation of the
+The section name must start with ``extension:``; the right-hand part is used as
+the full name (including a parent package, if any) of the extension.  Whitespace
+around the extension name is allowed.  If the extension module is not standalone
+(e.g. ``_bisect``) but part of a package (e.g. ``thing._speedups``), the parent
+package must be listed in the ``packages`` field.
+Valid fields and their values are listed in the documentation of the
 :class:`packaging.compiler.extension.Extension` class; values documented as
 Python lists translate to multi-line values in the configuration file.  In
 addition, multi-line values accept environment markers on each line, after a

Lib/compileall.py

@@ -142,7 +142,7 @@ def compile_path(skip_curdir=1, maxlevels=0, force=False, quiet=False,
 
     Arguments (all optional):
 
-    skip_curdir: if true, skip current directory (default true)
+    skip_curdir: if true, skip current directory (default True)
     maxlevels:   max recursion level (default 0)
     force: as for compile_dir() (default False)
     quiet: as for compile_dir() (default False)
@@ -177,17 +177,17 @@ def main():
                         help='use legacy (pre-PEP3147) compiled file locations')
     parser.add_argument('-d', metavar='DESTDIR',  dest='ddir', default=None,
                         help=('directory to prepend to file paths for use in '
-                              'compile time tracebacks and in runtime '
+                              'compile-time tracebacks and in runtime '
                               'tracebacks in cases where the source file is '
                               'unavailable'))
     parser.add_argument('-x', metavar='REGEXP', dest='rx', default=None,
-                        help=('skip files matching the regular expression. '
-                              'The regexp is searched for in the full path '
-                              'to each file considered for compilation.'))
+                        help=('skip files matching the regular expression; '
+                              'the regexp is searched for in the full path '
+                              'of each file considered for compilation'))
     parser.add_argument('-i', metavar='FILE', dest='flist',
                         help=('add all the files and directories listed in '
-                              'FILE to the list considered for compilation. '
-                              'If "-", names are read from stdin.'))
+                              'FILE to the list considered for compilation; '
+                              'if "-", names are read from stdin'))
     parser.add_argument('compile_dest', metavar='FILE|DIR', nargs='*',
                         help=('zero or more file and directory names '
                               'to compile; if no arguments given, defaults '

Lib/distutils/tests/support.py

@@ -175,10 +175,9 @@ def _get_xxmodule_path():
 def fixup_build_ext(cmd):
     """Function needed to make build_ext tests pass.
 
-    When Python was build with --enable-shared on Unix, -L. is not good
-    enough to find the libpython<blah>.so.  This is because regrtest runs
-    it under a tempdir, not in the top level where the .so lives.  By the
-    time we've gotten here, Python's already been chdir'd to the tempdir.
+    When Python was built with --enable-shared on Unix, -L. is not enough to
+    find libpython<blah>.so, because regrtest runs in a tempdir, not in the
+    source directory where the .so lives.
 
     When Python was built with in debug mode on Windows, build_ext commands
     need their debug attribute set, and it is not done automatically for

Lib/packaging/command/build_py.py

@@ -393,7 +393,7 @@ class build_py(Command, Mixin2to3):
                            self.get_command_name())
             return
 
-        from packaging.util import byte_compile
+        from packaging.util import byte_compile  # FIXME use compileall
         prefix = self.build_lib
         if prefix[-1] != os.sep:
             prefix = prefix + os.sep

Lib/packaging/command/install_lib.py

@@ -122,7 +122,7 @@ class install_lib(Command):
                            self.get_command_name())
             return
 
-        from packaging.util import byte_compile
+        from packaging.util import byte_compile  # FIXME use compileall
 
         # Get the "--root" directory supplied to the "install_dist" command,
         # and use it as a prefix to strip off the purported filename

Lib/packaging/command/register.py

@@ -2,7 +2,6 @@
 
 # Contributed by Richard Jones
 
-import io
 import getpass
 import urllib.error
 import urllib.parse

Lib/packaging/command/upload.py

@@ -5,7 +5,6 @@ import socket
 import logging
 import platform
 import urllib.parse
-from io import BytesIO
 from base64 import standard_b64encode
 from hashlib import md5
 from urllib.error import HTTPError

Lib/packaging/config.py

@@ -16,6 +16,19 @@ from packaging.command import set_command
 from packaging.markers import interpret
 
 
+def _check_name(name, packages):
+    if '.' not in name:
+        return
+    parts = name.split('.')
+    modname = parts[-1]
+    parent = '.'.join(parts[:-1])
+    if parent not in packages:
+        # we could log a warning instead of raising, but what's the use
+        # of letting people build modules they can't import?
+        raise PackagingOptionError(
+            'parent package for extension %r not found' % name)
+
+
 def _pop_values(values_dct, key):
     """Remove values from the dictionary and convert them as a list"""
     vals_str = values_dct.pop(key, '')
@@ -142,7 +155,8 @@ class Config:
                         try:
                             hook = resolve_name(line)
                         except ImportError as e:
-                            logger.warning('cannot find setup hook: %s', e.args[0])
+                            logger.warning('cannot find setup hook: %s',
+                                           e.args[0])
                         else:
                             self.setup_hooks.append(hook)
                     self.run_hooks(content)
@@ -216,7 +230,7 @@ class Config:
             for data in files.get('package_data', []):
                 data = data.split('=')
                 if len(data) != 2:
-                    continue  # XXX error should never pass silently
+                    continue  # FIXME errors should never pass silently
                 key, value = data
                 self.dist.package_data[key.strip()] = value.strip()
 
@@ -251,13 +265,18 @@ class Config:
 
         ext_modules = self.dist.ext_modules
         for section_key in content:
-            labels = section_key.split('=')
+            # no str.partition in 2.4 :(
+            labels = section_key.split(':')
             if len(labels) == 2 and labels[0] == 'extension':
-                # labels[1] not used from now but should be implemented
-                # for extension build dependency
                 values_dct = content[section_key]
+                if 'name' in values_dct:
+                    raise PackagingOptionError(
+                        'extension name should be given as [extension: name], '
+                        'not as key')
+                name = labels[1].strip()
+                _check_name(name, self.dist.packages)
                 ext_modules.append(Extension(
-                    values_dct.pop('name'),
+                    name,
                     _pop_values(values_dct, 'sources'),
                     _pop_values(values_dct, 'include_dirs'),
                     _pop_values(values_dct, 'define_macros'),

Lib/packaging/create.py

@@ -36,7 +36,7 @@ from packaging._trove import all_classifiers as _CLASSIFIERS_LIST
 from packaging.version import is_valid_version
 
 _FILENAME = 'setup.cfg'
-_DEFAULT_CFG = '.pypkgcreate'
+_DEFAULT_CFG = '.pypkgcreate'  # FIXME use a section in user .pydistutils.cfg
 
 _helptext = {
     'name': '''
@@ -127,6 +127,10 @@ def ask_yn(question, default=None, helptext=None):
         print('\nERROR: You must select "Y" or "N".\n')
 
 
+# XXX use util.ask
+# FIXME: if prompt ends with '?', don't add ':'
+
+
 def ask(question, default=None, helptext=None, required=True,
         lengthy=False, multiline=False):
     prompt = '%s: ' % (question,)

Lib/packaging/tests/support.py

 """Support code for packaging test cases.
 
+*This module should not be considered public: its content and API may
+change in incompatible ways.*
+
 A few helper classes are provided: LoggingCatcher, TempdirManager and
 EnvironRestorer. They are written to be used as mixins::
 
@@ -7,6 +10,7 @@ EnvironRestorer. They are written to be used as mixins::
     from packaging.tests.support import LoggingCatcher
 
     class SomeTestCase(LoggingCatcher, unittest.TestCase):
+        ...
 
 If you need to define a setUp method on your test class, you have to
 call the mixin class' setUp method or it won't work (same thing for
@@ -14,17 +18,18 @@ tearDown):
 
         def setUp(self):
             super(SomeTestCase, self).setUp()
-            ... # other setup code
+            ...  # other setup code
 
 Also provided is a DummyCommand class, useful to mock commands in the
-tests of another command that needs them, a create_distribution function
-and a skip_unless_symlink decorator.
+tests of another command that needs them, for example to fake
+compilation in build_ext (this requires that the mock build_ext command
+be injected into the distribution object's command_obj dictionary).
 
-Also provided is a DummyCommand class, useful to mock commands in the
-tests of another command that needs them, a create_distribution function
-and a skip_unless_symlink decorator.
+For tests that need to compile an extension module, use the
+copy_xxmodule_c and fixup_build_ext functions.
 
 Each class or function has a docstring to explain its purpose and usage.
+Existing tests should also be used as examples.
 """
 
 import os
@@ -39,9 +44,17 @@ from packaging.dist import Distribution
 from packaging.tests import unittest
 from test.support import requires_zlib, unlink
 
-__all__ = ['LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
-           'DummyCommand', 'unittest', 'create_distribution',
-           'skip_unless_symlink', 'requires_zlib', 'copy_xxmodule_c']
+# define __all__ to make pydoc more useful
+__all__ = [
+    # TestCase mixins
+    'LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
+    # mocks
+    'DummyCommand', 'TestDistribution',
+    # misc. functions and decorators
+    'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
+    # imported from this module for backport purposes
+    'unittest', 'requires_zlib', 'skip_unless_symlink',
+]
 
 
 logger = logging.getLogger('packaging')
@@ -233,6 +246,8 @@ class DummyCommand:
     Useful for mocking one dependency command in the tests for another
     command, see e.g. the dummy build command in test_build_scripts.
     """
+    # XXX does not work with dist.get_reinitialized_command, which typechecks
+    # and wants a finalized attribute
 
     def __init__(self, **kwargs):
         for kw, val in kwargs.items():
@@ -308,10 +323,9 @@ def _get_xxmodule_path():
 def fixup_build_ext(cmd):
     """Function needed to make build_ext tests pass.
 
-    When Python was build with --enable-shared on Unix, -L. is not good
-    enough to find the libpython<blah>.so.  This is because regrtest runs
-    it under a tempdir, not in the top level where the .so lives.  By the
-    time we've gotten here, Python's already been chdir'd to the tempdir.
+    When Python was built with --enable-shared on Unix, -L. is not enough to
+    find libpython<blah>.so, because regrtest runs in a tempdir, not in the
+    source directory where the .so lives.
 
     When Python was built with in debug mode on Windows, build_ext commands
     need their debug attribute set, and it is not done automatically for

Lib/packaging/tests/test_config.py

@@ -6,7 +6,7 @@ from io import StringIO
 
 from packaging import command
 from packaging.dist import Distribution
-from packaging.errors import PackagingFileError
+from packaging.errors import PackagingFileError, PackagingOptionError
 from packaging.compiler import new_compiler, _COMPILERS
 from packaging.command.sdist import sdist
 
@@ -100,21 +100,21 @@ sub_commands = foo
 
 # Can not be merged with SETUP_CFG else install_dist
 # command will fail when trying to compile C sources
+# TODO use a DummyCommand to mock build_ext
 EXT_SETUP_CFG = """
 [files]
 packages = one
            two
+           parent.undeclared
 
-[extension=speed_coconuts]
-name = one.speed_coconuts
+[extension:one.speed_coconuts]
 sources = c_src/speed_coconuts.c
 extra_link_args = "`gcc -print-file-name=libgcc.a`" -shared
 define_macros = HAVE_CAIRO HAVE_GTK2
 libraries = gecodeint gecodekernel -- sys.platform != 'win32'
     GecodeInt GecodeKernel -- sys.platform == 'win32'
 
-[extension=fast_taunt]
-name = two.fast_taunt
+[extension: two.fast_taunt]
 sources = cxx_src/utils_taunt.cxx
           cxx_src/python_module.cxx
 include_dirs = /usr/include/gecode
@@ -124,6 +124,30 @@ extra_compile_args = -fPIC -O2
     /DGECODE_VERSION='win32' -- sys.platform == 'win32'
 language = cxx
 
+# corner case: if the parent package of an extension is declared but
+# not its grandparent, it's legal
+[extension: parent.undeclared._speed]
+sources = parent/undeclared/_speed.c
+"""
+
+EXT_SETUP_CFG_BUGGY_1 = """
+[extension: realname]
+name = crash_here
+"""
+
+EXT_SETUP_CFG_BUGGY_2 = """
+[files]
+packages = ham
+
+[extension: spam.eggs]
+"""
+
+EXT_SETUP_CFG_BUGGY_3 = """
+[files]
+packages = ok
+           ok.works
+
+[extension: ok.works.breaks._ext]
 """
 
 HOOKS_MODULE = """
@@ -311,7 +335,7 @@ class ConfigTestCase(support.TempdirManager,
         dist = self.get_dist()
 
         ext_modules = dict((mod.name, mod) for mod in dist.ext_modules)
-        self.assertEqual(len(ext_modules), 2)
+        self.assertEqual(len(ext_modules), 3)
         ext = ext_modules.get('one.speed_coconuts')
         self.assertEqual(ext.sources, ['c_src/speed_coconuts.c'])
         self.assertEqual(ext.define_macros, ['HAVE_CAIRO', 'HAVE_GTK2'])
@@ -335,6 +359,15 @@ class ConfigTestCase(support.TempdirManager,
         self.assertEqual(ext.extra_compile_args, cargs)
         self.assertEqual(ext.language, 'cxx')
 
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_1)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_2)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
+        self.write_file('setup.cfg', EXT_SETUP_CFG_BUGGY_3)
+        self.assertRaises(PackagingOptionError, self.get_dist)
+
     def test_project_setup_hook_works(self):
         # Bug #11637: ensure the project directory is on sys.path to allow
         # project-specific hooks
@@ -358,7 +391,7 @@ class ConfigTestCase(support.TempdirManager,
         self.write_setup({
             'setup-hooks': '\n  packaging.tests.test_config.first_hook'
                            '\n  packaging.tests.test_config.missing_hook'
-                           '\n  packaging.tests.test_config.third_hook'
+                           '\n  packaging.tests.test_config.third_hook',
         })
         self.write_file('README', 'yeah')
         dist = self.get_dist()

Lib/packaging/tests/test_util.py

@@ -15,7 +15,7 @@ from packaging.errors import (
 from packaging import util
 from packaging.dist import Distribution
 from packaging.util import (
-    convert_path, change_root, split_quoted, strtobool, rfc822_escape,
+    convert_path, change_root, split_quoted, strtobool,
     get_compiler_versions, _MAC_OS_X_LD_VERSION, byte_compile, find_packages,
     spawn, get_pypirc_path, generate_pypirc, read_pypirc, resolve_name, iglob,
     RICH_GLOB, egginfo_to_distinfo, is_setuptools, is_distutils, is_packaging,
@@ -255,13 +255,6 @@ class UtilTestCase(support.EnvironRestorer,
         for n in no:
             self.assertFalse(strtobool(n))
 
-    def test_rfc822_escape(self):
-        header = 'I am a\npoor\nlonesome\nheader\n'
-        res = rfc822_escape(header)
-        wanted = ('I am a%(8s)spoor%(8s)slonesome%(8s)s'
-                  'header%(8s)s') % {'8s': '\n' + 8 * ' '}
-        self.assertEqual(res, wanted)
-
     def test_find_exe_version(self):
         # the ld version scheme under MAC OS is:
         #   ^@(#)PROGRAM:ld  PROJECT:ld64-VERSION

Lib/packaging/util.py

@@ -8,8 +8,6 @@ import errno
 import shutil
 import string
 import hashlib
-import tarfile
-import zipfile
 import posixpath
 import subprocess
 import sysconfig
@@ -23,6 +21,30 @@ from packaging.errors import (PackagingPlatformError, PackagingFileError,
                               PackagingByteCompileError, PackagingExecError,
                               InstallationException, PackagingInternalError)
 
+__all__ = [
+    # file dependencies
+    'newer', 'newer_group',
+    # helpers for commands (dry-run system)
+    'execute', 'write_file',
+    # spawning programs
+    'find_executable', 'spawn',
+    # path manipulation
+    'convert_path', 'change_root',
+    # 2to3 conversion
+    'Mixin2to3', 'run_2to3',
+    # packaging compatibility helpers
+    'cfg_to_args', 'generate_setup_py',
+    'egginfo_to_distinfo',
+    'get_install_method',
+    # misc
+    'ask', 'check_environ', 'encode_multipart', 'resolve_name',
+    # querying for information  TODO move to sysconfig
+    'get_compiler_versions', 'get_platform', 'set_platform',
+    # configuration  TODO move to packaging.config
+    'get_pypirc_path', 'read_pypirc', 'generate_pypirc',
+    'strtobool', 'split_multiline',
+]
+
 _PLATFORM = None
 _DEFAULT_INSTALLER = 'packaging'
 
@@ -152,31 +174,6 @@ def check_environ():
     _environ_checked = True
 
 
-def subst_vars(s, local_vars):
-    """Perform shell/Perl-style variable substitution on 'string'.
-
-    Every occurrence of '$' followed by a name is considered a variable, and
-    variable is substituted by the value found in the 'local_vars'
-    dictionary, or in 'os.environ' if it's not in 'local_vars'.
-    'os.environ' is first checked/augmented to guarantee that it contains
-    certain values: see 'check_environ()'.  Raise ValueError for any
-    variables not found in either 'local_vars' or 'os.environ'.
-    """
-    check_environ()
-
-    def _subst(match, local_vars=local_vars):
-        var_name = match.group(1)
-        if var_name in local_vars:
-            return str(local_vars[var_name])
-        else:
-            return os.environ[var_name]
-
-    try:
-        return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
-    except KeyError as var:
-        raise ValueError("invalid variable '$%s'" % var)
-
-
 # Needed by 'split_quoted()'
 _wordchars_re = _squote_re = _dquote_re = None
 
@@ -188,6 +185,8 @@ def _init_regex():
     _dquote_re = re.compile(r'"(?:[^"\\]|\\.)*"')
 
 
+# TODO replace with shlex.split after testing
+
 def split_quoted(s):
     """Split a string up according to Unix shell-like rules for quotes and
     backslashes.
@@ -435,15 +434,6 @@ byte_compile(files, optimize=%r, force=%r,
                               file, cfile_base)
 
 
-def rfc822_escape(header):
-    """Return a form of *header* suitable for inclusion in an RFC 822-header.
-
-    This function ensures there are 8 spaces after each newline.
-    """
-    lines = header.split('\n')
-    sep = '\n' + 8 * ' '
-    return sep.join(lines)
-
 _RE_VERSION = re.compile('(\d+\.\d+(\.\d+)*)')
 _MAC_OS_X_LD_VERSION = re.compile('^@\(#\)PROGRAM:ld  '
                                   'PROJECT:ld64-((\d+)(\.\d+)*)')
@@ -543,6 +533,10 @@ def write_file(filename, contents):
     """Create *filename* and write *contents* to it.
 
     *contents* is a sequence of strings without line terminators.
+
+    This functions is not intended to replace the usual with open + write
+    idiom in all cases, only with Command.execute, which runs depending on
+    the dry_run argument and also logs its arguments).
     """
     with open(filename, "w") as f:
         for line in contents:
@@ -562,6 +556,7 @@ def _is_archive_file(name):
 
 
 def _under(path, root):
+    # XXX use os.path
     path = path.split(os.sep)
     root = root.split(os.sep)
     if len(root) > len(path):
@@ -664,103 +659,11 @@ def splitext(path):
     return base, ext
 
 
-def unzip_file(filename, location, flatten=True):
-    """Unzip the file *filename* into the *location* directory."""
-    if not os.path.exists(location):
-        os.makedirs(location)
-    with open(filename, 'rb') as zipfp:
-        zip = zipfile.ZipFile(zipfp)
-        leading = has_leading_dir(zip.namelist()) and flatten
-        for name in zip.namelist():
-            data = zip.read(name)
-            fn = name
-            if leading:
-                fn = split_leading_dir(name)[1]
-            fn = os.path.join(location, fn)
-            dir = os.path.dirname(fn)
-            if not os.path.exists(dir):
-                os.makedirs(dir)
-            if fn.endswith('/') or fn.endswith('\\'):
-                # A directory
-                if not os.path.exists(fn):
-                    os.makedirs(fn)
-            else:
-                with open(fn, 'wb') as fp:
-                    fp.write(data)
-
-
-def untar_file(filename, location):
-    """Untar the file *filename* into the *location* directory."""
-    if not os.path.exists(location):
-        os.makedirs(location)
-    if filename.lower().endswith('.gz') or filename.lower().endswith('.tgz'):
-        mode = 'r:gz'
-    elif (filename.lower().endswith('.bz2')
-          or filename.lower().endswith('.tbz')):
-        mode = 'r:bz2'
-    elif filename.lower().endswith('.tar'):
-        mode = 'r'
-    else:
-        mode = 'r:*'
-    with tarfile.open(filename, mode) as tar:
-        leading = has_leading_dir(member.name for member in tar.getmembers())
-        for member in tar.getmembers():
-            fn = member.name
-            if leading:
-                fn = split_leading_dir(fn)[1]
-            path = os.path.join(location, fn)
-            if member.isdir():
-                if not os.path.exists(path):
-                    os.makedirs(path)
-            else:
-                try:
-                    fp = tar.extractfile(member)
-                except (KeyError, AttributeError):
-                    # Some corrupt tar files seem to produce this
-                    # (specifically bad symlinks)
-                    continue
-                try:
-                    if not os.path.exists(os.path.dirname(path)):
-                        os.makedirs(os.path.dirname(path))
-                        with open(path, 'wb') as destfp:
-                            shutil.copyfileobj(fp, destfp)
-                finally:
-                    fp.close()
-
-
-def has_leading_dir(paths):
-    """Return true if all the paths have the same leading path name.
-
-    In other words, check that everything is in one subdirectory in an
-    archive.
-    """
-    common_prefix = None
-    for path in paths:
-        prefix, rest = split_leading_dir(path)
-        if not prefix:
-            return False
-        elif common_prefix is None:
-            common_prefix = prefix
-        elif prefix != common_prefix:
-            return False
-    return True
-
-
-def split_leading_dir(path):
-    path = str(path)
-    path = path.lstrip('/').lstrip('\\')
-    if '/' in path and (('\\' in path and path.find('/') < path.find('\\'))
-                        or '\\' not in path):
-        return path.split('/', 1)
-    elif '\\' in path:
-        return path.split('\\', 1)
-    else:
-        return path, ''
-
 if sys.platform == 'darwin':
     _cfg_target = None
     _cfg_target_split = None
 
+
 def spawn(cmd, search_path=True, verbose=0, dry_run=False, env=None):
     """Run another program specified as a command list 'cmd' in a new process.
 
@@ -1510,7 +1413,7 @@ def encode_multipart(fields, files, boundary=None):
     for key, values in fields:
         # handle multiple entries for the same name
         if not isinstance(values, (tuple, list)):
-            values=[values]
+            values = [values]
 
         for value in values:
             l.extend((

Lib/pipes.py

@@ -54,8 +54,6 @@ for the built-in function open() or for os.popen().
 
 To create a new template object initialized to a given one:
    t2 = t.clone()
-
-For an example, see the function test() at the end of the file.
 """                                     # '
 
 

Lib/shutil.py

@@ -35,7 +35,7 @@ __all__ = ["copyfileobj", "copyfile", "copymode", "copystat", "copy", "copy2",
            "register_archive_format", "unregister_archive_format",
            "get_unpack_formats", "register_unpack_format",
            "unregister_unpack_format", "unpack_archive",
-           "ignore_patterns"]
+           "ignore_patterns", "chown"]
            # disk_usage is added later, if available on the platform
 
 class Error(EnvironmentError):
@@ -791,6 +791,7 @@ elif os.name == 'nt':
         used = total - free
         return _ntuple_diskusage(total, used, free)
 
+
 def chown(path, user=None, group=None):
     """Change owner user and group of the given path.