Instead of running python setup.py develop directly. This will allow
using zc.buildout.easy_install.develop on recent projects that have
only a pyproject.toml. It also fixes develop leaving build artifacts
in the source directory that caused later runs to do nothing, e.g.
preventing develop-eggs to be rebuilt when a build dependency passed
in setup-eggs option of zc.recipe.egg:develop changed.

A verbosity parameter to tune verbosity of pip is added, with adjusted
values for the case of buildout:develop and of zc.recipe.egg:develop,
so as to remain close to the previous behavior with regards to logs.

Technical details:

For packages using PEP-660-style editable installs, supported by more
recent versions of pip, pip will not delegate to `setup.py develop` -
enabling editable installs for pure pyproject.toml projects - and will
instead generate a .dist-info metadata folder but not a .egg-link.

Since buildout currently requires a .egg-link, as it does not support
PEP 660's mechanism that relies on having a sites-packages directory,
we need to create this .egg-link after the fact. The tricky part is
finding out where the .egg-link should point: the path containing the
pyproject.toml, or a subdirectory?

For this we make use of *.dist-info/top_level.txt to first determine
the list of top-level packages, and then importlib to extract info
from the PEP-660-style install.

If top_level.txt does not exist, is empty, or otherwise does not list
any package that resolves to an import path, fallback to the path of
the folder that contains the pyproject.toml as the .egg-link target.
If it lists multiple packages that resolve to different import paths,
arbitrarily use the first one and emit a warning.

Support namespace packages where `spec.submodule_search_locations` is
a `_NamespacePath` object instead of a simple `list` and also support
cases where the layout of the source project does not follow the same
structure as the package tree - meaning some custom magic might be
involved in making editable imports work as intended.

Trigger uninstall + install of eggs installed with zc.recipe.egg:custom
or :develop when pinned versions of setup-eggs have changed. To achieve
this the versions of setup-eggs are included in the section: this makes
them part of its signature so that when they change, buildout will call
`uninstall` and `install` for this section instead of just `update`.

Unlike other zc.recipe.egg entry points, :custom stores the path of the
installed egg; thus `uninstall` will remove it fully, leaving `install`
to reinstall it cleanly from scrach.

In the case of :develop, `uninstall` matters little as only the path of
the installed `.egg-link` is stored. Instead `install` must be fixed to
actually rebuild the egg in-place in the source directory and `develop`
should do nothing.

The main issue lies in `zc.buildout.easy_install.develop`: depending on
the build process, it may leave build artifacts in the source directory
that cause future runs to do nothing.

See merge request !31

Note: all changes with +slapos001 are tests-related.

- Fixup "[test] Add tests for build dependencies"
- Fixup "[feat] zc.recipe.egg: Improve on the fly patches."
- Fixup "[fix] Fix working set sorting".
- Fixup "[feat] Add dependencies in __buildout_signature__"

Pinning zc.buildout = >.1 is now invalid, so use >0.1 instead.

Adapt filter in tests for pip emitting Python2.7 deprecation warnings
to pip >= 20.2.2 and < 21.0 - these versions seem to accidentally add
a redundant bit of message to the warning.

The testing framework, some individual tests and 'buildout:develop'
call python setup.py <command> manually, which is deprecated by
setuptools. Depending on the setuptools version, suppress the
warning for now to avoid polluting the logs.

INFO logs from setuptools which were previously not emitted because
setuptool's ad-hoc legacy logger defaulted to WARNING and above may
now be emitted because setuptools now logs to the root logger, thus
the global root logger's level configuration applied.

This caused undesired 'root: <some setuptools info>' messages to be
emitted and caused many tests to fail due to unexpected outputs.

See merge request !30

When specifying an alternate URL as fallback, the main URL is always
used for both downloading & uploading from/to networkcache.

If slapos.libnetworkcache is importable, install it in bootstrap and
in buildout upgrade - the places where bin/buildout is (re)generated
- as though it were a dependency of zc.buildout.

This is a hack to propagate libnetworkcache as a soft dependency.

Like for URL that contain credentials, we still skip auth challenge
because it's faster and:
- we only support one auth scheme (basic)
- netrc provides no way to specify realms, which seem anyway to be
  less and less used (https://stackoverflow.com/q/69303610 reports
  that recent browsers don't display them anymore)

See merge request !25

This retries with the alternate URL in case of HTTPError with the main
one.

Used by slapos.recipe.build:download* and slapos.recipe.cmmi recipes.

- Py3: stop using legacy API of urllib.request and
       fix download of http(s) URLs containing user:passwd@
- Py2: avoid OOM when downloading huge files

This is implemented as a method in case we want to make it configurable
via [buildout].

An optimization is to avoid temporary file when possible: a rename
(or hard link) is not always possible (different mount points).

Another one is to not check md5sum twice when using cache file.

Fall-back mode is ignored if an MD5 checksum is given.

In case of checksum mismatch for a cached path, remove it and
download again, mainly to cover the following cases:
- the url content changes and the user updates the checksum
- buildout killed while downloading directly to cache
  (see above optimization)
- shutil.copyfile is interrupted

We want two identical sections at different URL to be able to produce
same signature. This feature is useful for slapos.recipe.cmmi's
shared parts.

This commit may fixup "Support ${:_profile_base_location_}." but is
purposefuly kept separate because it concerns parts signatures which
are an orthogonal feature.

When there is no pinned version for zc.buildout itself, buildout
adds a ">=<current-version>" requirement to prevent accidental
downgrading. If the current version has a local version label,
this produced an invalid version specifier. To fix this, only
the public part of the current version is used.

Use a special .pydistutils.cfg in a temporary HOME directory for
the duration of the pip wheel run to prevent build dependencies
specified in a setup_requires from being installed on the fly
without respecting pinned versions.

By default pip installs build dependencies (e.g. setuptools, poetry)
in a temporary folder and temporarily adds it to sys.path in order
to proceed to build the distribution. But we want all distributions
to be installed with buildout and respect pinned versions, so we aim
to prevent pip from installing build dependencies.

Instead, we will install the build dependencies first and pass them
explicitly to zc.recipe.egg via the setup-eggs option.

This commit prevents pip from installing the build dependencies listed
by the `build-system.requires` key in the pyproject.toml file.

It may prevent pip from installing PEP 517 dynamic build dependencies
or setuptools' setup_requires dependencies.
See https://peps.python.org/pep-0517/#get-requires-for-build-wheel

While invoke setup.py certain eggs (like scikit-learn) launch
cetain custom builds (for cython) using subprocess and sys.executable.

This commit aims to preserve the sys.path over the runs, even if an
egg is using subprocess with the same python to build a component of
the egg.

- Support on the fly patches in zc.recipe.egg by ``EGGNAME-patches``,
  ``EGGNAME-patch-options``, ``EGGNAME-patch-binary`` (or
  ``patch-binary``) and ``EGGNAME-patch-revision`` options.

- Support on the fly patches in zc.recipe.egg:custom by ``patches``,
  ``patch-options``, ``patch-binary`` and ``patch-revision`` options.
  (options ``EGGNAME-*`` are also supported as well).

Specified patches are automatically applied on required eggs as well.

This fixes cache of patches.

Clean-up + fix issue found at slapos!1674

Currently only zc.recipe.egg:custom supports setting environment
variables, and zc.recipe.egg:develop does not.

My motivation for allowing setting environment in :develop is
wendelin.core

    https://lab.nexedi.cn/nexedi/slapos/blob/b5faab3b/component/wendelin.core/buildout.cfg

There we have [wendelin.core] part which installs released egg from
pypi, and [wendelin.core-dev] part which installs wendelin.core from
its latest git version via zc.recipe.egg:develop .

The problem is, wendelin.core for setup.py to work, needs git available,
and with slapos we usually don't have git available on base system, so
we build it by our own and do something like

    [wendelin.core-dev]
    recipe = zc.recipe.egg:develop
    environment = wendelin.core-dev-env

    [wendelin.core-dev-env]
    # wendelin.core-dev needs git to build
    PATH = ${git:location}/bin:%(PATH)s

and the problem is environment does not currently work for
zc.recipe.egg:develop, and thus git is not found -> build fails.

~~~~

In order to support environment in :develop, we just move environment
setting/restoring bits from Custom to Base, and provide Base.install() which
uses this bits. Custom & Develop .install() becomes ._install() which gets
hooked into Base.install() .

I've tested the patch only manually, because currently automated tests are
broken in a lot of places for slapos.buildout and zc.recipe.egg .

/cc @kazuhiko, @Tyagov

Before this, sys.exit(1) was called to terminate the program directly.

To be dropped once all buildout in the wild are able to upgrade to a
version that supports new names in expression of conditional sections
(see previous commit).

Adding new names for expression is currently not possible because
buildout aborts before it tries to upgrade (in-place or bootstrap).

For example, existing values were not enough to distinguish
'arm-linux-gnueabi' from 'arm-linux-gnueabihf'.

The 'multiarch' value is the output of

  $CC -dumpmachine

where CC defaults to 'gcc'.

See also https://wiki.debian.org/Multiarch/Tuples

(changed option or missing path)

Sometimes, most parts are reinstalled for a reason that the user
didn't think about and it can take time to understand why.
Explaining for all parts would be too verbose and useless because
many are reinstalled just because their dependencies changed.

Egg is installed from wheel when the requested version ends with :whl

Reuse more code from slapos.core to support case of parent directory
not writable.

When recipes mutate the options, we should not allow invalid syntax
otherwise it gets written in .installed.cfg and generate invalid
configparser syntax.