Issue #22827: Backport the new Distributing and Instaling Docs from 3.4

Doc/conf.py

@@ -94,11 +94,11 @@ _stdauthor = r'Guido van Rossum\\and the Python development team'
 latex_documents = [
     ('c-api/index', 'c-api.tex',
      'The Python/C API', _stdauthor, 'manual'),
-    ('distutils/index', 'distutils.tex',
+    ('distributing/index', 'distributing.tex',
      'Distributing Python Modules', _stdauthor, 'manual'),
     ('extending/index', 'extending.tex',
      'Extending and Embedding Python', _stdauthor, 'manual'),
-    ('install/index', 'install.tex',
+    ('installing/index', 'installing.tex',
      'Installing Python Modules', _stdauthor, 'manual'),
     ('library/index', 'library.tex',
      'The Python Library Reference', _stdauthor, 'manual'),

Doc/contents.rst

@@ -11,8 +11,8 @@
    library/index.rst
    extending/index.rst
    c-api/index.rst
-   distutils/index.rst
-   install/index.rst
+   distributing/index.rst
+   installing/index.rst
    howto/index.rst
    faq/index.rst
    glossary.rst
@@ -21,3 +21,11 @@
    bugs.rst
    copyright.rst
    license.rst
+
+.. to include legacy packaging docs in build
+
+.. toctree::
+   :hidden:
+
+   distutils/index.rst
+   install/index.rst

Doc/distributing/index.rst

+.. _distributing-index:
+
+###############################
+  Distributing Python Modules
+###############################
+
+:Email: distutils-sig@python.org
+
+
+As a popular open source development project, Python has an active
+supporting community of contributors and users that also make their software
+available for other Python developers to use under open source license terms.
+
+This allows Python users to share and collaborate effectively, benefiting
+from the solutions others have already created to common (and sometimes
+even rare!) problems, as well as potentially contributing their own
+solutions to the common pool.
+
+This guide covers the distribution part of the process. For a guide to
+installing other Python projects, refer to the
+:ref:`installation guide <installing-index>`.
+
+.. note::
+
+   For corporate and other institutional users, be aware that many
+   organisations have their own policies around using and contributing to
+   open source software. Please take such policies into account when making
+   use of the distribution and installation tools provided with Python.
+
+
+Key terms
+=========
+
+* the `Python Packaging Index <https://pypi.python.org/pypi>`__ is a public
+  repository of open source licensed packages made available for use by
+  other Python users
+* the `Python Packaging Authority
+  <https://packaging.python.org/en/latest/future.html>`__ are the group of
+  developers and documentation authors responsible for the maintenance and
+  evolution of the standard packaging tools and the associated metadata and
+  file format standards. They maintain a variety of tools, documentation
+  and issue trackers on both `GitHub <https://github.com/pypa>`__ and
+  `BitBucket <https://bitbucket.org/pypa/>`__.
+* :mod:`distutils` is the original build and distribution system first added
+  to the Python standard library in 1998. While direct use of :mod:`distutils`
+  is being phased out, it still laid the foundation for the current packaging
+  and distribution infrastructure, and it not only remains part of the
+  standard library, but its name lives on in other ways (such as the name
+  of the mailing list used to coordinate Python packaging standards
+  development).
+* `setuptools`_ is a (largely) drop-in replacement for :mod:`distutils` first
+  published in 2004. Its most notable addition over the unmodified
+  :mod:`distutils` tools was the ability to declare dependencies on other
+  packages. It is currently recommended as a more regularly updated
+  alternative to :mod:`distutils` that offers consistent support for more
+  recent packaging standards across a wide range of Python versions.
+* `wheel`_ (in this context) is a project that adds the ``bdist_wheel``
+  command to :mod:`distutils`/`setuptools`_. This produces a cross platform
+  binary packaging format (called "wheels" or "wheel files" and defined in
+  :pep:`427`) that allows Python libraries, even those including binary
+  extensions, to be installed on a system without needing to be built
+  locally.
+
+.. _setuptools: https://setuptools.pypa.io/en/latest/setuptools.html
+.. _wheel: http://wheel.readthedocs.org
+
+Open source licensing and collaboration
+=======================================
+
+In most parts of the world, software is automatically covered by copyright.
+This means that other developers require explicit permission to copy, use,
+modify and redistribute the software.
+
+Open source licensing is a way of explicitly granting such permission in a
+relatively consistent way, allowing developers to share and collaborate
+efficiently by making common solutions to various problems freely available.
+This leaves many developers free to spend more time focusing on the problems
+that are relatively unique to their specific situation.
+
+The distribution tools provided with Python are designed to make it
+reasonably straightforward for developers to make their own contributions
+back to that common pool of software if they choose to do so.
+
+The same distribution tools can also be used to distribute software within
+an organisation, regardless of whether that software is published as open
+source software or not.
+
+
+Installing the tools
+====================
+
+The standard library does not include build tools that support modern
+Python packaging standards, as the core development team has found that it
+is important to have standard tools that work consistently, even on older
+versions of Python.
+
+The currently recommended build and distribution tools can be installed
+by invoking the ``pip`` module at the command line::
+
+    python -m pip install setuptools wheel twine
+
+.. note::
+
+   For POSIX users (including Mac OS X and Linux users), these instructions
+   assume the use of a :term:`virtual environment`.
+
+   For Windows users, these instructions assume that the option to
+   adjust the system PATH environment variable was selected when installing
+   Python.
+
+The Python Packaging User Guide includes more details on the `currently
+recommended tools`_.
+
+.. _currently recommended tools: https://packaging.python.org/en/latest/current.html#packaging-tool-recommendations
+
+Reading the guide
+=================
+
+The Python Packaging User Guide covers the various key steps and elements
+involved in creating a project:
+
+* `Project structure`_
+* `Building and packaging the project`_
+* `Uploading the project to the Python Packaging Index`_
+
+.. _Project structure: \
+   https://packaging.python.org/en/latest/distributing.html#creating-your-own-project
+.. _Building and packaging the project: \
+   https://packaging.python.org/en/latest/distributing.html#packaging-your-project
+.. _Uploading the project to the Python Packaging Index: \
+   https://packaging.python.org/en/latest/distributing.html#uploading-your-project-to-pypi
+
+
+How do I...?
+============
+
+These are quick answers or links for some common tasks.
+
+... choose a name for my project?
+---------------------------------
+
+This isn't an easy topic, but here are a few tips:
+
+* check the Python Packaging Index to see if the name is already in use
+* check popular hosting sites like GitHub, BitBucket, etc to see if there
+  is already a project with that name
+* check what comes up in a web search for the name you're considering
+* avoid particularly common words, especially ones with multiple meanings,
+  as they can make it difficult for users to find your software when
+  searching for it
+
+
+... create and distribute binary extensions?
+--------------------------------------------
+
+This is actually quite a complex topic, with a variety of alternatives
+available depending on exactly what you're aiming to achieve. See the
+Python Packaging User Guide for more information and recommendations.
+
+.. seealso::
+
+   `Python Packaging User Guide: Binary Extensions
+   <https://packaging.python.org/en/latest/extensions.html>`__
+
+.. other topics:
+
+   Once the Development & Deployment part of PPUG is fleshed out, some of
+   those sections should be linked from new questions here (most notably,
+   we should have a question about avoiding depending on PyPI that links to
+   https://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches)

Doc/glossary.rst

@@ -720,6 +720,12 @@ Glossary
       the dictionary view to become a full list use ``list(dictview)``.  See
       :ref:`dict-views`.
 
+   virtual environment
+      A cooperatively isolated runtime environment that allows Python users
+      and applications to install and upgrade Python distribution packages
+      without interfering with the behaviour of other Python applications
+      running on the same system.
+
    virtual machine
       A computer defined entirely in software.  Python's virtual machine
       executes the :term:`bytecode` emitted by the bytecode compiler.

Doc/installing/index.rst

+.. highlightlang:: none
+
+.. _installing-index:
+
+*****************************
+  Installing Python Modules
+*****************************
+
+:Email: distutils-sig@python.org
+
+As a popular open source development project, Python has an active
+supporting community of contributors and users that also make their software
+available for other Python developers to use under open source license terms.
+
+This allows Python users to share and collaborate effectively, benefiting
+from the solutions others have already created to common (and sometimes
+even rare!) problems, as well as potentially contributing their own
+solutions to the common pool.
+
+This guide covers the installation part of the process. For a guide to
+creating and sharing your own Python projects, refer to the
+:ref:`distribution guide <distributing-index>`.
+
+.. note::
+
+   For corporate and other institutional users, be aware that many
+   organisations have their own policies around using and contributing to
+   open source software. Please take such policies into account when making
+   use of the distribution and installation tools provided with Python.
+
+
+Key terms
+=========
+
+* ``pip`` is the preferred installer program. Starting with Python 2.7.9, it
+  is included by default with the Python binary installers.
+* a virtual environment is a semi-isolated Python environment that allows
+  packages to be installed for use by a particular application, rather than
+  being installed system wide
+* ``virtualenv`` is a third party tools for creating virtual environments, it
+  is defaults to installing ``pip`` into all created virtual environments.
+* the `Python Packaging Index <https://pypi.python.org/pypi>`__ is a public
+  repository of open source licensed packages made available for use by
+  other Python users
+* the `Python Packaging Authority
+  <https://packaging.python.org/en/latest/future.html>`__ are the group of
+  developers and documentation authors responsible for the maintenance and
+  evolution of the standard packaging tools and the associated metadata and
+  file format standards. They maintain a variety of tools, documentation
+  and issue trackers on both `GitHub <https://github.com/pypa>`__ and
+  `BitBucket <https://bitbucket.org/pypa/>`__.
+* ``distutils`` is the original build and distribution system first added to
+  the Python standard library in 1998. While direct use of ``distutils`` is
+  being phased out, it still laid the foundation for the current packaging
+  and distribution infrastructure, and it not only remains part of the
+  standard library, but its name lives on in other ways (such as the name
+  of the mailing list used to coordinate Python packaging standards
+  development).
+
+
+Basic usage
+===========
+
+The standard packaging tools are all designed to be used from the command
+line.
+
+The following command will install the latest version of a module and its
+dependencies from the Python Packaging Index::
+
+    python -m pip install SomePackage
+
+.. note::
+
+   For POSIX users (including Mac OS X and Linux users), the examples in
+   this guide assume the use of a :term:`virtual environment`. You may install
+   ``virtualenv`` to provide such environments using either pip
+   (``pip install virtualenv``) or through your system package manager
+   (commonly called ``virtualenv`` or ``python-virtualenv``).
+
+   For Windows users, the examples in this guide assume that the option to
+   adjust the system PATH environment variable was selected when installing
+   Python.
+
+It's also possible to specify an exact or minimum version directly on the
+command line::
+
+    python -m pip install SomePackage==1.0.4    # specific version
+    python -m pip install 'SomePackage>=1.0.4'  # minimum version
+
+Normally, if a suitable module is already installed, attempting to install
+it again will have no effect. Upgrading existing modules must be requested
+explicitly::
+
+    python -m pip install --upgrade SomePackage
+
+More information and resources regarding ``pip`` and its capabilities can be
+found in the `Python Packaging User Guide <https://packaging.python.org>`__.
+
+.. seealso::
+
+    `Python Packaging User Guide: Installing Python Distribution Packages
+    <https://packaging.python.org/en/latest/installing.html#installing-python-distribution-packages>`__
+
+
+How do I ...?
+=============
+
+These are quick answers or links for some common tasks.
+
+... install ``pip`` in versions of Python prior to Python 2.7.9?
+----------------------------------------------------------------
+
+Python only started bundling ``pip`` with Python 2.7.9. For earlier versions,
+``pip`` needs to be "bootstrapped" as described in the Python Packaging
+User Guide.
+
+.. seealso::
+
+   `Python Packaging User Guide: Setup for Installing Distribution Packages
+   <https://packaging.python.org/en/latest/installing.html#setup-for-installing-distribution-packages>`__
+
+
+.. installing-per-user-installation:
+
+... install packages just for the current user?
+-----------------------------------------------
+
+Passing the ``--user`` option to ``python -m pip install`` will install a
+package just for the current user, rather than for all users of the system.
+
+
+... install scientific Python packages?
+---------------------------------------
+
+A number of scientific Python packages have complex binary dependencies, and
+aren't currently easy to install using ``pip`` directly. At this point in
+time, it will often be easier for users to install these packages by
+`other means
+<https://packaging.python.org/en/latest/science.html>`__
+rather than attempting to install them with ``pip``.
+
+.. seealso::
+
+   `Python Packaging User Guide: Installing Scientific Packages
+   <https://packaging.python.org/en/latest/science.html>`__
+
+
+... work with multiple versions of Python installed in parallel?
+----------------------------------------------------------------
+
+On Linux, Mac OS X and other POSIX systems, use the versioned Python commands
+in combination with the ``-m`` switch to run the appropriate copy of
+``pip``::
+
+   python2   -m pip install SomePackage  # default Python 2
+   python2.7 -m pip install SomePackage  # specifically Python 2.7
+   python3   -m pip install SomePackage  # default Python 3
+   python3.4 -m pip install SomePackage  # specifically Python 3.4
+
+(appropriately versioned ``pip`` commands may also be available)
+
+On Windows, use the ``py`` Python launcher in combination with the ``-m``
+switch::
+
+   py -2   -m pip install SomePackage  # default Python 2
+   py -2.7 -m pip install SomePackage  # specifically Python 2.7
+   py -3   -m pip install SomePackage  # default Python 3
+   py -3.4 -m pip install SomePackage  # specifically Python 3.4
+
+.. other questions:
+
+   Once the Development & Deployment part of PPUG is fleshed out, some of
+   those sections should be linked from new questions here (most notably,
+   we should have a question about avoiding depending on PyPI that links to
+   https://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches)
+
+
+Common installation issues
+==========================
+
+Installing into the system Python on Linux
+------------------------------------------
+
+On Linux systems, a Python installation will typically be included as part
+of the distribution. Installing into this Python installation requires
+root access to the system, and may interfere with the operation of the
+system package manager and other components of the system if a component
+is unexpectedly upgraded using ``pip``.
+
+On such systems, it is often better to use a virtual environment or a
+per-user installation when installing packages with ``pip``.
+
+
+Installing binary extensions
+----------------------------
+
+Python has typically relied heavily on source based distribution, with end
+users being expected to compile extension modules from source as part of
+the installation process.
+
+With the introduction of support for the binary ``wheel`` format, and the
+ability to publish wheels for at least Windows and Mac OS X through the
+Python Packaging Index, this problem is expected to diminish over time,
+as users are more regularly able to install pre-built extensions rather
+than needing to build them themselves.
+
+Some of the solutions for installing `scientific software
+<https://packaging.python.org/en/latest/science.html>`__
+that is not yet available as pre-built ``wheel`` files may also help with
+obtaining other binary extensions without needing to build them locally.
+
+.. seealso::
+
+   `Python Packaging User Guide: Binary Extensions
+   <https://packaging.python.org/en/latest/extensions.html>`__