Add option `isolate-from-buildout-and-setuptools-path` to prevent
buildout from scanning the directories where zc.buildout's or its
dependencies's currently running packages are located.

In apparently intended design, `buildout bootstrap` should already
provide isolation by either installing pinned versions of zc.buildout
and dependencies in local ./eggs directory, or copying them from the
currently running egg into ./eggs if the running versions match.

However, when the currently running versions are not eggs - which is
the case when they are installed by pip in site-packages, e.g. as is
the case in a virtual environment - then buildout merely creates
`.egg-link` references to the locations of the running packages, which
is often site-packages.

The presence of .egg-link referencing site-packages can cause packages
available in site-packages to interfere when considering which eggs to
install and what path items to insert in sys.path when creating
scripts.

This is in some way a workaround for the fact that buildout only knows
how to handle eggs and not .dist-info packages such as produced by pip.

With `isolate-from-buildout-and-setuptools-path`, buildout bootstrap
will allways install a fresh version of zc.buildout & co in ./eggs

In bootstrap we potentially copy eggs from the working set to ./eggs.
We then reconstruct the same working set using the moved locations.

This commits ensures we keep a correct working set order throughout
and that we avoid activating unintended dists.

Replace `pkg_resources.Environment(ws.entries).best_match(req, ws)`
with `ws.find(req)`.

The first already starts by calling `ws.find(req)` to attempt to find
an already activated dist in the working set, but if none is found it
then proceeds to scan through the entries of the environment - i.e.
the entries of the working set, `ws.entries` - to activate a dist at
these locations if one matches.

This is problematic if a dist from site-packages is in the working set
at this stage: this will consider site-packages over eggs installed in
.eggs, seemingly randomly depending on what dists are requested and
whether they are found in site-packages.

Also when calling `install(['zc.buildout'])` to resolve the dists of
buildout and its dependencies currently running there are no version
pins yet, so if `zc.buildout` comes from site-packages and specific
versions of its dependencies come from `.eggs` but other versions
exist in site-packages, then the dists from site-packages would be
found instead of the actually running dists from `.eggs`.

If a dist in the computed working set is at a location shared with
other dists - such as site-packages - then when generating scripts
these other packages may overshadow the next items on the sys.path
and result in importing a different version than the one installed
and intended by buildout.

To avert this, a sort of the working set was introduced at various
points just before generating a script.

However, that sort put the paths referenced from an `.egg-link` in
./develop-eggs first. This is truly problematic because dists from
site-packages which are not eggs - e.g. dists installed with pip -
can become referenced as `.egg-link` during buildout bootstrap and
the sort then causes site-packages to be one of the first items in
sys.path.

In particular when running buildout bootstrap from a venv in which
zc.buildout was installed by pip, if any one of zc.buildout or its
dependencies from the venv meets the version requirements, then it
can cause the generated bin/buildout to import the dists only from
the venv's site-packages, even when some do not meet requirements.

To fix this, the sort now puts the dists from `./eggs` first as we
know their locations contain only a single dist, and then puts the
dists from ./develop-eggs which have locations inside the buildout
directory before the others.

The previous sort was also activating all the dists from the paths
of the already activated dists.

Note that this also means that any working set must be manipulated
with care in general to avoid activating unintended dists from the
locations of the already activated dists.

This is passed when calling `install(['zc.buildout'])` to resolve the
dists of the currently running buildout and its dependencies, as they
may not actually correspond the version of the running interpreter.

This is needed to support new slapos.rebootstrap: after compiling the
new Python interpreter, we use it to run buildout bootstrap using the
same paths to zc.buildout and dependencies as in the initial Python.
This installs the needed dists for the new Python version and creates
a new bin/buildout for the new Python interpreter.

When specifying an alternate URL as fallback, the main URL is always
used for both downloading & uploading from/to networkcache.

If slapos.libnetworkcache is importable, install it in bootstrap and
in buildout upgrade - the places where bin/buildout is (re)generated
- as though it were a dependency of zc.buildout.

This is a hack to propagate libnetworkcache as a soft dependency.

Like for URL that contain credentials, we still skip auth challenge
because it's faster and:
- we only support one auth scheme (basic)
- netrc provides no way to specify realms, which seem anyway to be
  less and less used (https://stackoverflow.com/q/69303610 reports
  that recent browsers don't display them anymore)

See merge request nexedi/slapos.buildout!25

This retries with the alternate URL in case of HTTPError with the main
one.

Used by slapos.recipe.build:download* and slapos.recipe.cmmi recipes.

- Py3: stop using legacy API of urllib.request and
       fix download of http(s) URLs containing user:passwd@
- Py2: avoid OOM when downloading huge files

This is implemented as a method in case we want to make it configurable
via [buildout].

An optimization is to avoid temporary file when possible: a rename
(or hard link) is not always possible (different mount points).

Another one is to not check md5sum twice when using cache file.

Fall-back mode is ignored if an MD5 checksum is given.

In case of checksum mismatch for a cached path, remove it and
download again, mainly to cover the following cases:
- the url content changes and the user updates the checksum
- buildout killed while downloading directly to cache
  (see above optimization)
- shutil.copyfile is interrupted

We want two identical sections at different URL to be able to produce
same signature. This feature is useful for slapos.recipe.cmmi's
shared parts.

This commit may fixup "Support ${:_profile_base_location_}." but is
purposefuly kept separate because it concerns parts signatures which
are an orthogonal feature.

Use a special .pydistutils.cfg in a temporary HOME directory for
the duration of the pip wheel run to prevent build dependencies
specified in a setup_requires from being installed on the fly
without respecting pinned versions.

When there is no pinned version for zc.buildout itself, buildout
adds a ">=<current-version>" requirement to prevent accidental
downgrading. If the current version has a local version label,
this produced an invalid version specifier. To fix this, only
the public part of the current version is used.

Now that we use `pip wheel` + `Wheel.install_as_egg` to produce
genuine eggs, we don't need to workaround the fact that we have
not-actually-egg `.egg` bundles anymore.

If we still do need to allow for non-eggs being installed in `~/.eggs`,
via extensions or others, we should revert this.
See https://github.com/buildout/buildout/pull/352

`pip install <package>` produces a `<package-name>` package folder
and a `<package-name>.dist-info` metadata folder, which is another
format than eggs. Then buildout bundles both folders into a parent
folder `<package.egg>` and tries to act as though it were an egg.

Instead, use `pip wheel` to produce a wheel - which `pip install`
does internally anyway - and `setuptools.Wheel.install_as_egg` to
produce a genuine egg.

This is much cleaner: it consistently produces genuine eggs instead
of sometimes true eggs, sometimes `.dist-info` bundles depending on
whether `pip install` is called or the package was installed from a
`.whl` or `.egg` archive directly.

The only downside it this requires setuptools >= 38.2.3.

Before this, += and -= worked well when applied to keys defined in the
current section or in a section of the same name inherited from a file
extended by the current file, but not to keys inherited from another
section using macro extension <=, because <= substitution happens later
and all the += and -= are already resolved.

e.g.:

```
[macro]
a = 1

[part]
<= macro
a += 2
```

was equivalent to

```
[part]
a = 1
a = 2
```

instead of

```
[part]
a = 1
a += 2
```

Now a partial and brittle support for this is enabled by postponing
+= and -= resolution until <= substitution happens when the current
section contains a <= extension. But this is not guaranteed: if the
current section is overloaded in another file extending the current
one, then += and -= resolution will happend then, before <=.

A consistent solution would be to unify the implementation of
extends= and <= and update sections by taking both into account,
instead of updating based only on extends and computing <= later,
leading to inconsistencies. This could be achieved e.g. by computing
section updates on demand during substitution.
Co-authored-by: Xavier Thompson <xavier.thompson@nexedi.com>

Before this, += and -= worked correctly when applied to a key defined
in a file extended by the current file, but not when the key was in
the same file.

e.g.:

```
[part]
a = this case is
```

```
[buildout]
extends = a.cfg

[part]
a += ok
b = this case is
b += not-ok
```
Co-authored-by: Xavier Thompson <xavier.thompson@nexedi.com>

Before this, sys.exit(1) was called to terminate the program directly.

By default pip installs build dependencies (e.g. setuptools, poetry)
in a temporary folder and temporarily adds it to sys.path in order
to proceed to build the distribution. But we want all distributions
to be installed with buildout and respect pinned versions, so we aim
to prevent pip from installing build dependencies.

Instead, we will install the build dependencies first and pass them
explicitly to zc.recipe.egg via the setup-eggs option.

This commit prevents pip from installing the build dependencies listed
by the `build-system.requires` key in the pyproject.toml file.

It may prevent pip from installing PEP 517 dynamic build dependencies
or setuptools' setup_requires dependencies.
See https://peps.python.org/pep-0517/#get-requires-for-build-wheel

While invoke setup.py certain eggs (like scikit-learn) launch
cetain custom builds (for cython) using subprocess and sys.executable.

This commit aims to preserve the sys.path over the runs, even if an
egg is using subprocess with the same python to build a component of
the egg.

Now specified patches are automatically applied on required eggs as well.

- Support on the fly patches in zc.recipe.egg by ``EGGNAME-patches``,
  ``EGGNAME-patch-options``, ``EGGNAME-patch-binary`` (or
  ``patch-binary``) and ``EGGNAME-patch-revision`` options.

- Support on the fly patches in zc.recipe.egg:custom by ``patches``,
  ``patch-options``, ``patch-binary`` and ``patch-revision`` options.
  (options ``EGGNAME-*`` are also supported as well).

Currently only zc.recipe.egg:custom supports setting environment
variables, and zc.recipe.egg:develop does not.

My motivation for allowing setting environment in :develop is
wendelin.core

    https://lab.nexedi.cn/nexedi/slapos/blob/b5faab3b/component/wendelin.core/buildout.cfg

There we have [wendelin.core] part which installs released egg from
pypi, and [wendelin.core-dev] part which installs wendelin.core from
its latest git version via zc.recipe.egg:develop .

The problem is, wendelin.core for setup.py to work, needs git available,
and with slapos we usually don't have git available on base system, so
we build it by our own and do something like

    [wendelin.core-dev]
    recipe = zc.recipe.egg:develop
    environment = wendelin.core-dev-env

    [wendelin.core-dev-env]
    # wendelin.core-dev needs git to build
    PATH = ${git:location}/bin:%(PATH)s

and the problem is environment does not currently work for
zc.recipe.egg:develop, and thus git is not found -> build fails.

~~~~

In order to support environment in :develop, we just move environment
setting/restoring bits from Custom to Base, and provide Base.install() which
uses this bits. Custom & Develop .install() becomes ._install() which gets
hooked into Base.install() .

I've tested the patch only manually, because currently automated tests are
broken in a lot of places for slapos.buildout and zc.recipe.egg .

/cc @kazuhiko, @Tyagov

To be dropped once all buildout in the wild are able to upgrade to a
version that supports new names in expression of conditional sections
(see previous commit).

Adding new names for expression is currently not possible because
buildout aborts before it tries to upgrade (in-place or bootstrap).

For example, existing values were not enough to distinguish
'arm-linux-gnueabi' from 'arm-linux-gnueabihf'.

The 'multiarch' value is the output of

  $CC -dumpmachine

where CC defaults to 'gcc'.

See also https://wiki.debian.org/Multiarch/Tuples

(changed option or missing path)

Sometimes, most parts are reinstalled for a reason that the user
didn't think about and it can take time to understand why.
Explaining for all parts would be too verbose and useless because
many are reinstalled just because their dependencies changed.