Commit 3564c78c authored by Jason Madden's avatar Jason Madden

Documentation updates and clarifications.

Fixes #1567, fixes #1562, fixes #1559 and fixes #1566
parent 93ac3bed
......@@ -18,7 +18,6 @@ Read the documentation online at http://www.gevent.org.
Post feedback and issues on the `bug tracker`_, `mailing list`_, blog_
and `twitter (@gevent)`_.
.. include:: docs/install.rst
.. _bug tracker: https://github.com/gevent/gevent/wiki/Projects
......
......@@ -11,6 +11,8 @@ astroid >= 2.2.5 ; python_version >= "3.4" and platform_python_implementation ==
# For generating CHANGES.rst
towncrier
# For making releases
zest.releaser[recommended]
# benchmarks use this
pyperf >= 1.6.1
......
......@@ -36,11 +36,9 @@ Developing and Packaging gevent
===============================
.. toctree::
:maxdepth: 2
:maxdepth: 3
development/installing_from_source
development/index
development/release_process
Related Information
......
========================
Continuous integration
========================
A test suite is run for every push and pull request submitted. Travis
CI is used to test on Linux and macOS, and `AppVeyor`_ runs the builds on
Windows.
.. image:: https://travis-ci.org/gevent/gevent.svg?branch=master
:target: https://travis-ci.org/gevent/gevent
.. image:: https://ci.appveyor.com/api/projects/status/q4kl21ng2yo2ixur?svg=true
:target: https://ci.appveyor.com/project/denik/gevent
Builds on Travis CI automatically submit updates to `coveralls.io`_ to
monitor test coverage.
.. image:: https://coveralls.io/repos/gevent/gevent/badge.svg?branch=master&service=github
:target: https://coveralls.io/github/gevent/gevent?branch=master
.. _coverage.py: https://pypi.python.org/pypi/coverage/
.. _coveralls.io: https://coveralls.io/github/gevent/gevent
.. _AppVeyor: https://ci.appveyor.com/project/denik/gevent
=================
Getting Started
=================
Developing gevent requires being able to install gevent from source.
See :doc:`installing_from_source` for general information about that.
Use A Virtual Environment
=========================
It is recommended to install the development copy of gevent in a
`virtual environment <https://docs.python.org/3/tutorial/venv.html>`_;
you can use the :mod:`venv` module distributed with Python 3, or
`virtualenv <https://pypi.org/project/virtualenv/>`_, possibly with
`virtualenvwrapper <https://pypi.org/project/virtualenvwrapper/>`_.
You may want a different virtual environment for each Python
implementation and version that you'll be working with. gevent
includes a `tox <http://tox.readthedocs.org/>`_ configuration for
automating the process of testing across multiple Python versions, but
that can be slow.
The rest of this document will assume working in an isolated virtual
environment, but usually won't show that in the prompt. An example of
creating a virtual environment is shown here::
$ python3 -m venv gevent-env
$ cd gevent-env
$ . bin/activate
(gevent-env) $
Installing Dependencies
=======================
To work on gevent, we'll need to get the source, install gevent's
dependencies, including test dependencies, and install gevent as an
`editable install
<https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs>`_
using pip's ``-e`` option (also known as `development mode
<https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode>`_,
this is mostly the same as running ``python setup.py develop``).
Getting the source means cloning the git repository::
(gevent-env) $ git clone https://github.com/gevent/gevent.git
(gevent-env) $ cd gevent
Installing gevent's dependencies, test dependencies, and gevent itself
can be done in one line by installing the ``dev-requirements.txt`` file::
(gevent-env) $ pip install -r dev-requirements.txt
.. warning::
This pip command does not work with pip 19.1. Either use pip 19.0
or below, or use pip 19.1.1 with ``--no-use-pep517``. See `issue
1412 <https://github.com/gevent/gevent/issues/1412>`_.
......@@ -8,231 +8,18 @@ including information about running tests.
More information is in the ``CONTRIBUTING.rst`` document in the root
of the gevent repository.
.. toctree::
:maxdepth: 2
getting_started
installing_from_source
running_tests
ci
release_process
..
The contributor guide (CONTRIBUTING.rst) references this document.
Things to include:
- Custom commands in ``setup.py``
- Writing tests and the gevent test framework:
- Avoiding hard test dependencies.
- Resource usage.
- test files must be executable
- Maybe these things belong in a README in the gevent.tests directory?
Getting Started
===============
Developing gevent requires being able to install gevent from source.
See :doc:`installing_from_source` for general information about that.
It is recommended to install the development copy of gevent in a
`virtual environment <https://docs.python.org/3/tutorial/venv.html>`_;
you can use the :mod:`venv` module distributed with Python 3, or
`virtualenv <https://pypi.org/project/virtualenv/>`_, possibly with
`virtualenvwrapper <https://pypi.org/project/virtualenvwrapper/>`_.
You may want a different virtual environment for each Python
implementation and version that you'll be working with. gevent
includes a `tox <http://tox.readthedocs.org/>`_ configuration for
automating the process of testing across multiple Python versions, but
that can be slow.
The rest of this document will assume working in an isolated virtual
environment, but usually won't show that in the prompt. An example of
creating a virtual environment is shown here::
$ python3 -m venv gevent-env
$ cd gevent-env
$ . bin/activate
(gevent-env) $
To work on gevent, we'll need to get the source, install gevent's
dependencies, including test dependencies, and install gevent as an
`editable install
<https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs>`_
using pip's `-e option` (also known as `development mode
<https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode>`_,
this is mostly the same as running ``python setup.py develop``).
Getting the source means cloning the git repository::
(gevent-env) $ git clone https://github.com/gevent/gevent.git
(gevent-env) $ cd gevent
Installing gevent's dependencies, test dependencies, and gevent itself
can be done in one line by installing the ``dev-requirements.txt`` file::
(gevent-env) $ pip install -r dev-requirements.txt
.. warning::
This pip command does not work with pip 19.1. Either use pip 19.0
or below, or use pip 19.1.1 with ``--no-use-pep517``. See `issue
1412 <https://github.com/gevent/gevent/issues/1412>`_.
Running Tests
=============
gevent has an extensive regression test suite, implemented using the
standard :mod:`unittest` module. It uses a :mod:`custom testrunner
<gevent.testing.testrunner>` that provides enhanced test isolation
(important for monkey-patching), runs tests in parallel, and takes
care of other gevent-specific quirks.
The test runner has a number of options:
.. command-output:: python -mgevent.tests --help
The simplest way to run all the tests is just to invoke the test
runner, typically from the root of the source checkout::
(gevent-env) $ python -mgevent.tests
Running tests in parallel with concurrency 7
...
Ran 3107 tests (skipped=333) in 132 files in 01:52
You can also run an individual gevent test file using the test runner::
(gevent-env) $ python -m gevent.tests test__util.py
Running tests in parallel with concurrency 1
+ /.../python -u -mgevent.tests.test__util
- /.../python -u -mgevent.tests.test__util [Ran 9 tests in 1.1s]
Longest-running tests:
1.1 seconds: /.../python -u -mgevent.tests.test__util
Ran 9 tests in 1 files in 1.1s
Or you can run a monkey-patched standard library test::
(gevent-env) $ python -m gevent.tests.test___monkey_patching test_socket.py
Running tests in parallel with concurrency 1
+ /.../python -u -W ignore -m gevent.testing.monkey_test test_socket.py
Running with patch_all(Event=False): test_socket.py
Added imports 1
Skipped testEmptyFileSend (1)
...
Ran 555 tests in 23.042s
OK (skipped=172)
- /.../python -u -W ignore -m gevent.testing.monkey_test test_socket.py [took 26.7s]
Longest-running tests:
26.7 seconds: /.../python -u -W ignore -m gevent.testing.monkey_test test_socket.py
Ran 0 tests in 1 files in 00:27
Environment Variables
---------------------
Some testrunner options have equivalent environment variables.
Notably, ``--quiet`` is ``GEVENTTEST_QUIET`` and ``-u`` is
``GEVENTTEST_USE_RESOURCES``.
Using tox
---------
Before submitting a pull request, it's a good idea to run the tests
across all supported versions of Python, and to check the code quality
using prospector. This is what is done on Travis CI. Locally it
can be done using tox::
pip install tox
tox
Measuring Code Coverage
-----------------------
This is done on CI so it's not often necessary to do locally.
The testrunner accepts a ``--coverage`` argument to enable code
coverage metrics through the `coverage.py`_ package. That would go
something like this::
python -m gevent.tests --coverage
coverage combine
coverage html -i
<open htmlcov/index.html>
.. _limiting-test-resource-usage:
Limiting Resource Usage
-----------------------
gevent supports the standard library test suite's resources. All
resources are enabled by default. Disabling resources disables the
tests that use those resources. For example, to disable tests that
access the external network (the Internet), disable the ``network``
resource. There's an option for this::
$ python -m gevent.tests -u-network
And an environment variable::
$ GEVENTTEST_USE_RESOURCES=-network python -m gevent.tests
Continuous integration
======================
A test suite is run for every push and pull request submitted. Travis
CI is used to test on Linux, and `AppVeyor`_ runs the builds on
Windows.
.. image:: https://travis-ci.org/gevent/gevent.svg?branch=master
:target: https://travis-ci.org/gevent/gevent
.. image:: https://ci.appveyor.com/api/projects/status/q4kl21ng2yo2ixur?svg=true
:target: https://ci.appveyor.com/project/denik/gevent
Builds on Travis CI automatically submit updates to `coveralls.io`_ to
monitor test coverage.
.. image:: https://coveralls.io/repos/gevent/gevent/badge.svg?branch=master&service=github
:target: https://coveralls.io/github/gevent/gevent?branch=master
.. note:: On Debian, you will probably need ``libpythonX.Y-testsuite``
installed to run all the tests.
.. _coverage.py: https://pypi.python.org/pypi/coverage/
.. _coveralls.io: https://coveralls.io/github/gevent/gevent
.. _AppVeyor: https://ci.appveyor.com/project/denik/gevent
Releasing gevent
================
.. note:: This is a semi-organized collection of notes for gevent
maintainers.
gevent is released using `zest.releaser
<https://pypi.org/project/zest.releaser/>`_. The general flow is
something like this:
1. Push all relevant changes to master.
2. From the gevent working copy, run ``prerelease``. Fix any issues it
brings up. Let it bump the version number (or enter the correct
one) and commit.
3. Run ``release``. Let it create the tag and commit it; let it create
an sdist, but **do not** let it upload it.
4. Push the tag and master to github.
5. Let appveyor build the tag. Download all the built wheels from that
release. The easiest way to do that is with Ned Batchelder's
`appveyor-download.py script
<https://bitbucket.org/ned/coveragepy/src/tip/ci/download_appveyor.py>`_.
6. Meanwhile, spin up docker and from the root of the gevent checkout
run ``scripts/releases/make-manylinux``. This creates wheels in
``wheelhouse/``.
7. If on a mac, ``cd scripts/releases && ./geventreleases.sh``. This
creates wheels in ``/tmp/gevent/``.
8. Upload the Appveyor, manylinux, and mac wheels to pypi using
``twine``. Also be sure to upload the sdist!
9. Run ``postrelease``, let it bump the version and push the changes
to github.
......@@ -9,6 +9,7 @@ fallback to doing this if no binary wheel is available. (If you'll be
:ref:`developing <development>` gevent, you'll need to install from
source also; follow that link for more details.)
General Notes
=============
......@@ -18,9 +19,34 @@ General Notes
options, such as to use a system version of libuv instead of the
embedded version. See :ref:`build-config`.
- You'll need a working C compiler that can build Python extensions.
On some platforms, you may need to install Python development
packages.
- You'll need `pip 19 and setuptools 40.8
<https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support>`_
with fully functional :pep:`518` and :pep:`517` support to install
gevent from source.
- You'll need a working C compiler toolchain that can build Python
extensions. On some platforms, you may need to install Python
development packages. You'll also need the ability to compile `cffi
<https://pypi.org/project/cffi/>`_ modules, which may require
installing FFI development packages. Installing `make
<https://en.wikipedia.org/wiki/Make_(software)>`_ and other common
utilities such as `file
<https://en.wikipedia.org/wiki/File_(command)>`_ may also be
required.
For example, on Alpine Linux, one might need to do this::
apk add --virtual build-deps file make gcc musl-dev libffi-dev
See :issue:`1567`, :issue:`1559`, and :issue:`1566`.
.. note::
The exact set of external dependencies isn't necessarily fixed
and depends on the configure scripts of the bundled C libraries
such as libev, libuv and c-ares. Disabling :ref:`embed-lib` and
using system libraries can reduce these dependencies, although
this isn't encouraged.
- Installing from source requires ``setuptools``. This is installed
automatically in virtual environments and by buildout. However,
......@@ -31,9 +57,9 @@ General Notes
`have issues installing gevent for this reason
<https://github.com/pypa/pipenv/issues/2113>`_.
- gevent 1.5 comes with a ``pyproject.toml`` file that installs the
build dependencies, including CFFI (needed for libuv support). pip
18 or above is required for this support.
- gevent 1.5 and newer come with a ``pyproject.toml`` file that
installs the build dependencies, including CFFI (needed for libuv
support). pip 18 or above is required for this support.
- You can use pip's `VCS support
<https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support>`_
......@@ -109,17 +135,18 @@ yes/no.
In general, setting ``CPPFLAGS`` is more general and can contain
other macros recognized by libev.
.. _embed-lib:
Embedding Libraries
-------------------
By default, gevent builds and embeds tested versions of its
C dependencies libev, libuv, and c-ares. This is the
recommended configuration as the specific versions used are tested by
gevent, and sometimes require patches to be applied. Moreover,
embedding, especially in the case of libev, can be more efficient as
features not needed by gevent can be disabled, resulting in smaller or
faster libraries or runtimes.
By default, gevent builds and embeds tested versions of its C
dependencies libev, libuv, and c-ares. This is the recommended
configuration as the specific versions used are tested by gevent, and
sometimes require patches to be applied. Moreover, embedding,
especially in the case of libev, can be more efficient as features not
needed by gevent can be disabled, resulting in smaller or faster
libraries or runtimes.
However, this can be disabled, either for all libraries at once or for
individual libraries.
......
......@@ -30,3 +30,35 @@ period, in a gevent release. However, we are aware that the documentation
isn’t always complete - PRs that document existing behavior with the
intention of covering that behavior with the above deprecation process
are always acceptable, and will be considered on their merits.
Releasing gevent
================
.. note:: This is a semi-organized collection of notes for gevent
maintainers.
gevent is released using `zest.releaser
<https://pypi.org/project/zest.releaser/>`_. The general flow is
something like this:
1. Push all relevant changes to master.
2. From the gevent working copy, run ``prerelease``. Fix any issues it
brings up. Let it bump the version number (or enter the correct
one) and commit.
3. Run ``release``. Let it create the tag and commit it; let it create
an sdist, but **do not** let it upload it.
4. Push the tag and master to github.
5. Let appveyor build the tag. Download all the built wheels from that
release. The easiest way to do that is with Ned Batchelder's
`appveyor-download.py script
<https://bitbucket.org/ned/coveragepy/src/tip/ci/download_appveyor.py>`_.
6. Meanwhile, spin up docker and from the root of the gevent checkout
run ``scripts/releases/make-manylinux``. This creates wheels in
``wheelhouse/``.
7. If on a mac, ``cd scripts/releases && ./geventreleases.sh``. This
creates wheels in ``/tmp/gevent/``.
8. Upload the Appveyor, manylinux, and mac wheels to pypi using
``twine``. Also be sure to upload the sdist!
9. Run ``postrelease``, let it bump the version and push the changes
to github.
===============
Running Tests
===============
.. Things to include:
- Writing tests and the gevent test framework:
- Avoiding hard test dependencies.
- Resource usage.
- test files must be executable
- Maybe these things belong in a README in the gevent.tests directory?
gevent has an extensive regression test suite, implemented using the
standard :mod:`unittest` module. It uses a :mod:`custom testrunner
<gevent.testing.testrunner>` that provides enhanced test isolation
(important for monkey-patching), runs tests in parallel, and takes
care of other gevent-specific quirks.
.. note::
The gevent test process runs Python standard library tests with
gevent's monkey-patches applied to ensure that gevent behaves
correctly (matches the standard library). The standard library
:mod:`test` must be available in order to do this.
This is usually the case automatically, but some distributions
remove this module. Notably, on Debian, you will probably need
``libpythonX.Y-testsuite`` installed to run all the tests.
The test runner has a number of options:
.. command-output:: python -mgevent.tests --help
The simplest way to run all the tests is just to invoke the test
runner, typically from the root of the source checkout::
(gevent-env) $ python -mgevent.tests
Running tests in parallel with concurrency 7
...
Ran 3107 tests (skipped=333) in 132 files in 01:52
You can also run an individual gevent test file using the test runner::
(gevent-env) $ python -m gevent.tests test__util.py
Running tests in parallel with concurrency 1
+ /.../python -u -mgevent.tests.test__util
- /.../python -u -mgevent.tests.test__util [Ran 9 tests in 1.1s]
Longest-running tests:
1.1 seconds: /.../python -u -mgevent.tests.test__util
Ran 9 tests in 1 files in 1.1s
Or you can run a monkey-patched standard library test::
(gevent-env) $ python -m gevent.tests.test___monkey_patching test_socket.py
Running tests in parallel with concurrency 1
+ /.../python -u -W ignore -m gevent.testing.monkey_test test_socket.py
Running with patch_all(Event=False): test_socket.py
Added imports 1
Skipped testEmptyFileSend (1)
...
Ran 555 tests in 23.042s
OK (skipped=172)
- /.../python -u -W ignore -m gevent.testing.monkey_test test_socket.py [took 26.7s]
Longest-running tests:
26.7 seconds: /.../python -u -W ignore -m gevent.testing.monkey_test test_socket.py
Ran 0 tests in 1 files in 00:27
Environment Variables
=====================
Some testrunner options have equivalent environment variables.
Notably, ``--quiet`` is ``GEVENTTEST_QUIET`` and ``-u`` is
``GEVENTTEST_USE_RESOURCES``.
Using tox
=========
Before submitting a pull request, it's a good idea to run the tests
across all supported versions of Python, and to check the code quality
using prospector. This is what is done on Travis CI. Locally it
can be done using tox::
pip install tox
tox
Measuring Code Coverage
=======================
This is done on CI so it's not often necessary to do locally.
The testrunner accepts a ``--coverage`` argument to enable code
coverage metrics through the `coverage.py`_ package. That would go
something like this::
python -m gevent.tests --coverage
coverage combine
coverage html -i
<open htmlcov/index.html>
.. _limiting-test-resource-usage:
Limiting Resource Usage
=======================
gevent supports the standard library test suite's resources. All
resources are enabled by default. Disabling resources disables the
tests that use those resources. For example, to disable tests that
access the external network (the Internet), disable the ``network``
resource. There's an option for this::
$ python -m gevent.tests -u-network
And an environment variable::
$ GEVENTTEST_USE_RESOURCES=-network python -m gevent.tests
.. _coverage.py: https://pypi.python.org/pypi/coverage/
.. _coveralls.io: https://coveralls.io/github/gevent/gevent
......@@ -8,25 +8,38 @@
This file is included in README.rst so it is limited to plain
ReST markup, not Sphinx.
.. note::
If you are reading this document on the `Python Package Index`_
(PyPI, https://pypi.org/), it is specific to the version of gevent that
you are viewing. If you are viewing this document on gevent.org, it
refers to the current state of gevent in source control (git
master).
Supported Platforms
===================
`gevent 1.5`_ runs on Python 2.7.9 and up, and Python 3.5, 3.6, 3.7 and
This version of gevent runs on Python 2.7.9 and up, and Python 3.5, 3.6, 3.7 and
3.8. gevent requires the `greenlet <https://greenlet.readthedocs.io>`_
library and will install the `cffi`_ library by default on Windows.
The cffi library will become the default on all platforms in a future
release of gevent.
gevent 1.5 also runs on PyPy 7.0 or above. On PyPy, there are no
external dependencies.
This version of gevent also runs on PyPy 7.0 or above. On PyPy, there
are no external dependencies.
gevent is tested on Windows, macOS, and Linux, and should run on most
other Unix-like operating systems (e.g., FreeBSD, Solaris, etc.)
.. note:: On Windows using the deprecated libev backend, gevent is
limited to a maximum of 1024 open sockets due to
`limitations in libev`_. This limitation should not exist
with the default libuv backend.
.. note::
Windows is supported as a tier 2, "best effort," platform. It is
suitable for development, but not recommended for production.
On Windows using the deprecated libev backend, gevent is
limited to a maximum of 1024 open sockets due to
`limitations in libev`_. This limitation should not exist
with the default libuv backend.
Older Versions of Python
------------------------
......@@ -85,6 +98,12 @@ distributed as binary `wheels`_.
<https://github.com/pypa/pip/pull/5008>`_ to install the
manylinux2010 wheels.
.. tip::
Binary wheels cannot be installed on non-manylinux2010 compatible
Linux systems, such as those that use `musl
<https://musl.libc.org>`_, including `Alpine Linux
<https://alpinelinux.org>`_. Those systems must install from source.
Installing From Source
----------------------
......@@ -95,7 +114,6 @@ you can build gevent from source. A normal ``pip install`` will
fall back to doing this if no binary wheel is available. See
`Installing From Source`_ for more, including common installation issues.
Extra Dependencies
==================
......
......@@ -32,6 +32,8 @@ gevent 1.5 drops support for Python 3.4, and drops support for PyPy
gevent is tested with CPython 2.7.17, 3.5.9, 3.6.10, 3.7.7, 3.8.2,
PyPy 2 7.3.0 and PyPy3 7.3.0.
.. caution:: Older releases, such as RHEL 5, are no longer supported.
Packaging Changes
=================
......@@ -41,17 +43,27 @@ instead of the older ``manylinux1`` standard. This updated platform
tag allows gevent to distribute libuv support by default. CentOS 6 is
the baseline for this tag.
.. note:: Older releases, such as RHEL 5, are no longer supported.
gevent bundles a ``pyproject.toml`` now. This is useful for building
from source.
.. caution::
The requirements for building from source may have changed,
especially in minimal container environments (e.g., Alpine Linux).
See :doc:`development/installing_from_source` for more information.
The legacy ``Makefile`` has been removed in favor of built-in setup.py
commands.
Certain environment variables used at build time have been deprecated
and renamed.
Generated ``.c`` and ``.h`` files are no longer included in the
distribution. Neither are Cython ``.pxd`` files. This is because
linking to internal C optimizations is not supported and likely to
crash if used against a different version of gevent than exactly what
it was compiled for. See :issue:`1568` for more details.
Library Updates
===============
......
......@@ -330,16 +330,21 @@ def run_setup(ext_modules):
},
package_dir={'': 'src'},
packages=find_packages('src'),
# Using include_package_data causes our generated .c and .h
# files to be included in the installation. Those aren't needed
# at runtime (the .html files generated by Cython's annotation are
# much nicer to browse anyway, but we don't want to include those either), and downstream
# distributors have complained about them, so we don't want to include them.
# Nor do we want to include .pyx or .pxd files that aren't considered public;
# the only .pxd files that ever offered the required Cython annotations to produce
# stable APIs where in the libev cext backend. For ease of packaging, though, let's not
# include those either unless we get requests.
# include_package_data=True,
# Using ``include_package_data`` causes our generated ``.c``
# and ``.h`` files to be included in the installation. Those
# aren't needed at runtime (the ``.html`` files generated by
# Cython's annotation are much nicer to browse anyway, but we
# don't want to include those either), and downstream
# distributors have complained about them, so we don't want to
# include them. Nor do we want to include ``.pyx`` or ``.pxd``
# files that aren't considered public; the only ``.pxd`` files
# that ever offered the required Cython annotations to produce
# stable APIs weere in the libev cext backend; all of the
# internal optimizations provided by Cython compiling existing
# ``.py`` files using a matching ``.pxd`` do not. Furthermore,
# there are ABI issues that make distributing those extremely
# fragile. So do not use ``include_package_data``, explicitly
# spell out what we need. See https://github.com/gevent/gevent/issues/1568.
package_data={
# For any package
'': [
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment