When recipes mutate the options, we should not allow invalid syntax
otherwise it gets written in .installed.cfg and generate invalid
configparser syntax.

When recipe mutate options, options values are written to
.installed.cfg without escaping buildout substitution syntax,
so if a recipe sets an option value that could be interpreted
as a buildout substitution, it is written as is in .installed.cfg.

This can be a problem if options read from _read_installed_part_options
are accessed, like it's the case with slapos patched buildout which
saves installed options after an error with part installation or after
each part installation when running in verbose mode.

python ZipFile module does not support updating an entry in place, instead make
a new zip file and copy all entries.

This avoids rebootstrap changing the signature of zc.buildout when
the package is in sites-packages for the initial Python version but
not in the sites-packages of the target Python (when it's not the
same Python)

Dists in python library (e.g. site-packages) using .dist-info format
are marked as DEVELOP_DIST by setuptools. Use their version to sign
these, and not their location like actual develop dists.

To set a default requirement for not-yet processed parts

The version of Python should not affect the behaviour of a recipe.
Anyway, it was already ignored for DEVELOP_DIST eggs.

This makes the slapos mechanism to share parts more efficient. And with
the upcoming changes in buildout & slapos.recipe.cmmi, there would be
no way for the slapos.reboostrap extension to prevent everything from
being rebuilt when reboostrapping to a different version of Python,
if bootstrap parts are shared.

The monkey-patch by slapos.reboostrap now becomes useless.

_install_and_load is slow, using cache saves time when there are
many sections.

Introduce a workaround for the shebang line length limitation
in zc.buildout.easy_install.

Useful when recipes generate non-string values to be reused by other
recipes.

Add referred parts' hash strings in __buildout_signature__, that
invokes rebuild of a part when one of its (recursive) dependencies
are modified.

Also remove duplicates and sort entries in __buildout_signature__.

Also, updating a part does not put it anymore at the end of the list of
installed parts, that was making .installed.cfg too big.

https://bugs.launchpad.net/zc.buildout/+bug/144228

Even though such configuration is wrong...

Before this, += and -= worked well when applied to keys defined in the
current section or in a section of the same name inherited from a file
extended by the current file, but not to keys inherited from another
section using macro extension <=, because <= substitution happens later
and all the += and -= are already resolved.

e.g.:

```
[macro]
a = 1

[part]
<= macro
a += 2
```

was equivalent to

```
[part]
a = 1
a = 2
```

instead of

```
[part]
a = 1
a += 2
```

Now a partial and brittle support for this is enabled by postponing
+= and -= resolution until <= substitution happens when the current
section contains a <= extension. But this is not guaranteed: if the
current section is overloaded in another file extending the current
one, then += and -= resolution will happend then, before <=.

A consistent solution would be to unify the implementation of
extends= and <= and update sections by taking both into account,
instead of updating based only on extends and computing <= later,
leading to inconsistencies. This could be achieved e.g. by computing
section updates on demand during substitution.
Co-authored-by: Xavier Thompson <xavier.thompson@nexedi.com>

Before this, += and -= worked correctly when applied to a key defined
in a file extended by the current file, but not when the key was in
the same file.

e.g.:

```
[part]
a = this case is
```

```
[buildout]
extends = a.cfg

[part]
a += ok
b = this case is
b += not-ok
```
Co-authored-by: Xavier Thompson <xavier.thompson@nexedi.com>

This test asserts buildout's behavior with regards to download options,
and this was changed by the new algorithm for extends. Tests for the
new behavior have not been written yet.

The new algorithm avoids fetching the same extended file more than once
and correctly handles overriding values and += and -=:

The new algorithm starts as if there was a buildout file containing

```
[buildout]
extends =
  user/defaults.cfg # if it exists
  buildout.cfg # if it exists
  command_line_extends.cfg # if passed on the command line
```

The files are then fetched in depth-first-search postorder and fetching
child nodes in the order given by the extends directive, ignoring files
that have already been fetched.

The buildout dicts are then collected in order, and this linearisation
is then merged at the end, overriding the first configs collected with
the later ones. The first dict in the linearisation is not from a file,
but the dict of buildout's (hardcoded) defaults. This is equivalent to
acting as though every file that does not extend anything extends these
defaults.

The first time a file must be downloaded from a url, the linearisation
is merged with the configs already collected, and the resulting options
are then used to determine the download options for this download, and
every subsequent download.

This is a break with buildout's current logic for download options.

By analogy with classes in Python, we are computing a linearisation of
the class hierarchy to determine the method resolution order (MRO).
This algorithm is not the same as Python's MRO since Python 2.3 (C3).

It could be good to switch to a C3 linearisation like Python.

This avoids unecessary copies. This is a preparatory step to
reimplementing the extends algorithm. It may be that this breaks
the extends algorithm as it is currently implemented.

This avoids unecessary deepcopies. This is a preparatory step to
reimplementing the extends algorithm. It may be that this breaks
the extends algorithm as it is currently implemented.

Depending on the test environment, pip, setuptools and wheel dists
may be installed as .egg or .egg-link and may be files or folders.

Depending on the test environment, pip, setuptools and wheel dists
may be installed as .egg or .egg-link and may be files or folders.

The general case is that buildout will pick versions for setuptools,
pip and wheel when they are not pinned: it's only when these are
installed as .egg-link - either as actual develop eggs or as a
side-effect of being found in .dist-info format in the environment
e.g. in site-packages - that buildout does not consider it picked
a version for these eggs.

This affects tests that set `allow-picked-versions = false` or
`show-picked-versions = true` or `update-versions-file = ...`.

This creates a environment for tests that is isolated from the current
environment (as long as there are no .egg-link to e.g. site-packages).

In SlapOS, bootstrap is called each time software release is invoked and
we do not want to delete develop-eggs directory.

This reverts commit 55d76b34.

With recent setuptools, the way the test package index is laid out
causes a warning log to be emitted, which needs to be filtered out
to keep matching the expected test output.

Now that we use `pip wheel` + `Wheel.install_as_egg` to produce
genuine eggs, we don't need to workaround the fact that we have
not-actually-egg `.egg` bundles anymore.

If we still do need to allow for non-eggs being installed in `~/.eggs`,
via extensions or others, we should revert this.
See https://github.com/buildout/buildout/pull/352

Needed for pip wheel + setuptools.wheel.Wheel.install_as_egg.

`pip install <package>` produces a `<package-name>` package folder
and a `<package-name>.dist-info` metadata folder, which is another
format than eggs. Then buildout bundles both folders into a parent
folder `<package.egg>` and tries to act as though it were an egg.

Instead, use `pip wheel` to produce a wheel - which `pip install`
does internally anyway - and `setuptools.Wheel.install_as_egg` to
produce a genuine egg.

This is much cleaner: it consistently produces genuine eggs instead
of sometimes true eggs, sometimes `.dist-info` bundles depending on
whether `pip install` is called or the package was installed from a
`.whl` or `.egg` archive directly.

The only downside it this requires setuptools >= 38.2.3.

Since zc.buildout uses `pip` to install dists, most .egg in ./eggs and
./develop-eggs (in case of dists installed with zc.recipe.egg:custom)
are not actually eggs, but installed wheels with .dist-info format
which have been bundled into a .egg containing .dist-info metadata
folder as well as the actual package folder directly alongside.

Unlike actual eggs, such */*.dist-info bundles are not stepped into
by package_resources when the containing directory is add to the
environment, so each .egg bundle needs to be added individually.

Buildout already does this for bundles in the target directory of
egg installation, usually ./eggs, but e.g. in offline mode there
is no target directory and then already installed .egg bundles
are not detected.

So now every dist path is added to the environment individual,
including for paths where buildout looks for already installed
eggs but will not install eggs to, such as ./develop-eggs, and
also ./eggs in offline mode or when using zc.recipe.egg:custom
and the target is ./develop-eggs and ./eggs is such a scan-from
but don't-install-to path.