See merge request !5

Filter occurences of 'root: ' in buildout output in tests, caused by
setuptools >= 65.6.0 logging on root and zc.buildout formatting logs
from other libraries with the name of the emitting logger.

When possible, use venv to create the target Python in the tests.
This creates an isolated Python, and quite importantly maintains
isolation from system packages when the initial Python is itself
from a virtual environment.

Drop special-case hack when the initial and the target Python have
the same version. This used the initial Python to generate the
bin/buildout for the target Python. This could cause issues because
the initial and the target Python are unlikely to share the same
sites-packages, and this influences script generation.

The new rebootstrap mechanism also works when the Python versions
are the same, so it can be used in all cases.

Removing this hack also reduces even more the dependency on private
internal APIs of zc.buildout.

Previously, the tests for reboostrapping with a version of Python
different than the original one had an expected failure to obtain
a distribution for setuptools, because the test environment blocks
access to PyPI etc.

However the new rebootstrap method uses the existing installation
to call buildout bootstrap with the new Python. This would usually
trigger fetching zc.buildout and dependencies online because the
already installed versions are tagged for the original Python, but
in the test the Python versions are actually the same and we merely
tricked rebootstrap into temporarily believing they are different
in order to trigger the desired code path, so the already installed
eggs are just directly reused.

Raise if it's a slapos-patched zc.buildout. This replaces having
a constraint in setup.py as setting zc.buildout>=3.0.1+slapos001
is invalid.

Previously, slapos.rebootstrap switched from Python X.X to Python Y.Y
by first arranging to have source distributions for zc.buildout and its
dependencies, either already available in dowload cache, or downloading
them with Python X.X using internal APIs of zc.buildout, then launching
Python Y.Y to extract and bootstrap the installation of setuptools, and
then use setuptools.easy_install to install the other distributions,
meaning zc.buildout and any other dependency.

The versions of zc.buildout and its dependencies thus installed are
exactly the same as the one originally running with Python X.X. This
implies of course that these versions are compatible with Python X.X
and Python Y.Y - that is a hard constraint currently.

Thus instead of reinstalling them, we can actually just reuse the
already installed versions with Python Y.Y by passing the packages in
sys.path, giving us a working zc.buildout which we can use to properly
reinstall zc.buildout and its dependencies for Python Y.Y.

This allows for a much easier process, as we can merely achieve this
by calling buildout bootstrap with Python Y.Y without resorting to
using internal APIs of zc.buildout to download the source distributions
by hand, nor needing to manually bootstrap their installation by
extracting setuptools etc.

Note that the previous implementation also didn't respect :whl version
pins.

See merge request !4

zc.buildout >= 3.0.0 depends on pip and wheel as well

In addition to already supported .zip archives

Local version labels (+<local>) are no longer supported in unequality
version requirements syntax, e.g. `project == x.y.z+local` is ok but
`project >= x.y.z+local` is not ok.

SlapOS buildout does not have this method anymore.

About upstream buildout, we don't know because we don't use,
but signatures are less strict so maybe the patch was also useless.

When bootstrapping to a different version of Python, the path of
the wanted Python changes twice when using SlapOS shared parts.
This commit checks bin/buildout-rebootstrap before reusing it.

This is useful when a recipe modifies the environment during install/update.

Fix a bug occurring when setting python3.6 as ${buildout:executable} in profiles.


    ************ REBOOTSTRAP: IMPORTANT NOTICE ************
    bin/buildout is being reinstalled right now, as new python:
      /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/parts/python3.6/bin/python3.6
    is available, and buildout is using another python:
      /opt/slapgrid/4cca7f1c7db00b5e7a4348f8df5c24ab/parts/python2.7/bin/python2.7
    Buildout will be restarted automatically to have this change applied.
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************

    networkcache: Downloading pypi-index-96af89cdde216340df5aec1e9a1392e0-setuptools==40.4.3...
    networkcache: ignored unhandled exception at /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/slapos.libnetworkcache-0.20-py2.7.egg/slapos/networkcachehelper.py:104
    HTTPError: HTTP Error 404: Not Found
    networkcache: Downloading pypi-index-99e2b3234a294248d07c2bb8e6698302-setuptools==40.4.3...
    networkcache: ignored unhandled exception at /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/slapos.libnetworkcache-0.20-py2.7.egg/slapos/networkcachehelper.py:104
    HTTPError: HTTP Error 404: Not Found
    networkcache: Downloading pypi-index-a46be9bd3756ada5c303353adbb6407f-setuptools==40.4.3...
    networkcache: ignored unhandled exception at /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/slapos.libnetworkcache-0.20-py2.7.egg/slapos/networkcachehelper.py:104
    HTTPError: HTTP Error 404: Not Found
    networkcache: Downloading pypi-index-45c869bb967e16b5e30ed107541a36be-setuptools==40.4.3...
    networkcache: ignored unhandled exception at /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/slapos.libnetworkcache-0.20-py2.7.egg/slapos/networkcachehelper.py:104
    HTTPError: HTTP Error 404: Not Found
    networkcache: Downloading slapos-buildout-c4345cb200e7d70be4984e98a0864d72...
    Downloaded https://files.pythonhosted.org/packages/6e/9c/6a003320b00ef237f94aa74e4ad66c57a7618f6c79d67527136e2544b728/setuptools-40.4.3.zip#sha256=acbc5740dd63f243f46c2b4b8e2c7fd92259c2ddb55a4115b16418a2ed371b15 from network cache.
    networkcache: Downloading pypi-index-0b71f171beaaa2eb14aa9fd650059e50-zc.buildout==2.7.1+slapos001...
    Downloaded https://pypi.org/simple/zc.buildout/ from network cache.
    networkcache: Downloading slapos-buildout-fd4ac0343c559873f75e8e4be0b06366...
    Downloaded http://www.nexedi.org/static/packages/source/slapos.buildout/zc.buildout-2.7.1+slapos001.tar.gz from network cache.
    networkcache: Downloading pypi-index-f0a660f10b8bd0b2f65992c829d98404-slapos.libnetworkcache==0.20...
    Downloaded https://pypi.org/simple/slapos.libnetworkcache/ from network cache.
    networkcache: Downloading slapos-buildout-b7f445bbec09ae840c83ded70a305e7c...
    Downloaded https://files.pythonhosted.org/packages/b4/75/ae13749622d6ace9dae2442399d955d29241cfded7b48245320e2c0953b1/slapos.libnetworkcache-0.20.tar.gz#sha256=a6ceb542499fb572641abb1c7b675329bb235fda43d2f28b541906179e8db8db from network cache.
    Traceback (most recent call last):
      File "<string>", line 67, in <module>
      File "<string>", line 64, in main
      File "<string>", line 30, in setup_script
      File "/srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/zc.buildout-2.7.1+slapos001-py3.6.egg/zc/buildout/easy_install.py", line 1313, in scripts
        sname = os.path.join(dest, sname)
      File "/srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/parts/python3.6/lib/python3.6/posixpath.py", line 80, in join
        a = os.fspath(a)
    TypeError: expected str, bytes or os.PathLike object, not NoneType
    While:
      Installing.


This also fixes test suite to support python3. When running on python3.6 the problem occurred, but not with python3.5.

/reviewed-on nexedi/slapos.rebootstrap!2

Only setup.py test understands tests_require

This function is supposed to be called with script names and a directory
to store the scripts, as described in
https://github.com/buildout/buildout/blob/2.7.1/src/zc/buildout/easy_install.txt#L478

Passing an absolute path for script name and None as directory was OK on
python <= 3.5, because:
  os.path.join(None, '/absolute/path')
returns '/absolute/path', but on python >= 3.6 this is no longer the
case and failed with:

    Traceback (most recent call last):
      File "slapos.buildout/src/zc/buildout/buildout.py", line 2240, in main
        getattr(buildout, command)(args)
      File "slapos.buildout/src/zc/buildout/buildout.py", line 643, in install
        self._install_parts(install_args)
      File "slapos.buildout/src/zc/buildout/buildout.py", line 658, in _install_parts
        self._setup_directories()
      File "slapos.rebootstrap/slapos/rebootstrap/__init__.py", line 37, in wrapper
        return getattr(self, attr)(*args, **kw)
      File "slapos.rebootstrap/slapos/rebootstrap/__init__.py", line 72, in _setup_directories
        setup_script(new_bin, self.wanted_python)
      File "slapos.rebootstrap/slapos/rebootstrap/bootstrap.py", line 31, in setup_script
        python,
      File "slapos.buildout/src/zc/buildout/easy_install.py", line 1313, in scripts
        sname = os.path.join(dest, sname)
      File "python3/lib/python3.6/posixpath.py", line 80, in join
        a = os.fspath(a)
    TypeError: expected str, bytes or os.PathLike object, not NoneType

Change the parameters to respect the original signature.

> Changed in version 3.3: Added follow_symlinks argument. Now returns path
> to the newly created file.

[pylint] Anomalous backslash in string: '\S'. String constant might be
missing an r prefix.

and add the <BLANKLINES> where needed.

With shared builds, it's now possible that the [python2.7] part that is
providing executable for this buildout does not need to install any part and
makes the interpreter available since recipe initialisation.

This is causing an error during the overriden method computing part signature,
because parts are not installed yet:

    An internal error occurred due to a bug in either zc.buildout or in a
    recipe being used:
    Traceback (most recent call last):
      File "zc/buildout/buildout.py", line 2242, in main
        getattr(buildout, command)(args)
      File "zc/buildout/buildout.py", line 638, in install
        self._install_parts(install_args)
      File "zc/buildout/buildout.py", line 704, in _install_parts
        self._compute_part_signatures(install_parts)
      File "slapos/rebootstrap/__init__.py", line 37, in wrapper
        return getattr(self, attr)(*args, **kw)
      File "slapos/rebootstrap/__init__.py", line 95, in _compute_part_signatures
        installed_part_options[part]['__buildout_signature__']
    KeyError: 'already_installed_python'

To support that case, tolerate that parts are can be not installed yet at this
point.

/reviewed-on !1

- Work entirely in-place, even to switch to a different version of Python.
- Update bin/buildout to immediately use the wanted Python on subsequent
  buildout runs.

For SlapOS, the second point is required to have the instanciation done
with the built Python.