See merge request !4

zc.buildout >= 3.0.0 depends on pip and wheel as well

In addition to already supported .zip archives

Local version labels (+<local>) are no longer supported in unequality
version requirements syntax, e.g. `project == x.y.z+local` is ok but
`project >= x.y.z+local` is not ok.

SlapOS buildout does not have this method anymore.

About upstream buildout, we don't know because we don't use,
but signatures are less strict so maybe the patch was also useless.

When bootstrapping to a different version of Python, the path of
the wanted Python changes twice when using SlapOS shared parts.
This commit checks bin/buildout-rebootstrap before reusing it.

This is useful when a recipe modifies the environment during install/update.

Fix a bug occurring when setting python3.6 as ${buildout:executable} in profiles.


    ************ REBOOTSTRAP: IMPORTANT NOTICE ************
    bin/buildout is being reinstalled right now, as new python:
      /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/parts/python3.6/bin/python3.6
    is available, and buildout is using another python:
      /opt/slapgrid/4cca7f1c7db00b5e7a4348f8df5c24ab/parts/python2.7/bin/python2.7
    Buildout will be restarted automatically to have this change applied.
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************

    networkcache: Downloading pypi-index-96af89cdde216340df5aec1e9a1392e0-setuptools==40.4.3...
    networkcache: ignored unhandled exception at /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/slapos.libnetworkcache-0.20-py2.7.egg/slapos/networkcachehelper.py:104
    HTTPError: HTTP Error 404: Not Found
    networkcache: Downloading pypi-index-99e2b3234a294248d07c2bb8e6698302-setuptools==40.4.3...
    networkcache: ignored unhandled exception at /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/slapos.libnetworkcache-0.20-py2.7.egg/slapos/networkcachehelper.py:104
    HTTPError: HTTP Error 404: Not Found
    networkcache: Downloading pypi-index-a46be9bd3756ada5c303353adbb6407f-setuptools==40.4.3...
    networkcache: ignored unhandled exception at /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/slapos.libnetworkcache-0.20-py2.7.egg/slapos/networkcachehelper.py:104
    HTTPError: HTTP Error 404: Not Found
    networkcache: Downloading pypi-index-45c869bb967e16b5e30ed107541a36be-setuptools==40.4.3...
    networkcache: ignored unhandled exception at /srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/slapos.libnetworkcache-0.20-py2.7.egg/slapos/networkcachehelper.py:104
    HTTPError: HTTP Error 404: Not Found
    networkcache: Downloading slapos-buildout-c4345cb200e7d70be4984e98a0864d72...
    Downloaded https://files.pythonhosted.org/packages/6e/9c/6a003320b00ef237f94aa74e4ad66c57a7618f6c79d67527136e2544b728/setuptools-40.4.3.zip#sha256=acbc5740dd63f243f46c2b4b8e2c7fd92259c2ddb55a4115b16418a2ed371b15 from network cache.
    networkcache: Downloading pypi-index-0b71f171beaaa2eb14aa9fd650059e50-zc.buildout==2.7.1+slapos001...
    Downloaded https://pypi.org/simple/zc.buildout/ from network cache.
    networkcache: Downloading slapos-buildout-fd4ac0343c559873f75e8e4be0b06366...
    Downloaded http://www.nexedi.org/static/packages/source/slapos.buildout/zc.buildout-2.7.1+slapos001.tar.gz from network cache.
    networkcache: Downloading pypi-index-f0a660f10b8bd0b2f65992c829d98404-slapos.libnetworkcache==0.20...
    Downloaded https://pypi.org/simple/slapos.libnetworkcache/ from network cache.
    networkcache: Downloading slapos-buildout-b7f445bbec09ae840c83ded70a305e7c...
    Downloaded https://files.pythonhosted.org/packages/b4/75/ae13749622d6ace9dae2442399d955d29241cfded7b48245320e2c0953b1/slapos.libnetworkcache-0.20.tar.gz#sha256=a6ceb542499fb572641abb1c7b675329bb235fda43d2f28b541906179e8db8db from network cache.
    Traceback (most recent call last):
      File "<string>", line 67, in <module>
      File "<string>", line 64, in main
      File "<string>", line 30, in setup_script
      File "/srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/eggs/zc.buildout-2.7.1+slapos001-py3.6.egg/zc/buildout/easy_install.py", line 1313, in scripts
        sname = os.path.join(dest, sname)
      File "/srv/slapgrid/slappart14/srv/testnode/cup/soft/4281ee9e21ffccb8d017eba71a9cd9b1/parts/python3.6/lib/python3.6/posixpath.py", line 80, in join
        a = os.fspath(a)
    TypeError: expected str, bytes or os.PathLike object, not NoneType
    While:
      Installing.


This also fixes test suite to support python3. When running on python3.6 the problem occurred, but not with python3.5.

/reviewed-on nexedi/slapos.rebootstrap!2

Only setup.py test understands tests_require

This function is supposed to be called with script names and a directory
to store the scripts, as described in
https://github.com/buildout/buildout/blob/2.7.1/src/zc/buildout/easy_install.txt#L478

Passing an absolute path for script name and None as directory was OK on
python <= 3.5, because:
  os.path.join(None, '/absolute/path')
returns '/absolute/path', but on python >= 3.6 this is no longer the
case and failed with:

    Traceback (most recent call last):
      File "slapos.buildout/src/zc/buildout/buildout.py", line 2240, in main
        getattr(buildout, command)(args)
      File "slapos.buildout/src/zc/buildout/buildout.py", line 643, in install
        self._install_parts(install_args)
      File "slapos.buildout/src/zc/buildout/buildout.py", line 658, in _install_parts
        self._setup_directories()
      File "slapos.rebootstrap/slapos/rebootstrap/__init__.py", line 37, in wrapper
        return getattr(self, attr)(*args, **kw)
      File "slapos.rebootstrap/slapos/rebootstrap/__init__.py", line 72, in _setup_directories
        setup_script(new_bin, self.wanted_python)
      File "slapos.rebootstrap/slapos/rebootstrap/bootstrap.py", line 31, in setup_script
        python,
      File "slapos.buildout/src/zc/buildout/easy_install.py", line 1313, in scripts
        sname = os.path.join(dest, sname)
      File "python3/lib/python3.6/posixpath.py", line 80, in join
        a = os.fspath(a)
    TypeError: expected str, bytes or os.PathLike object, not NoneType

Change the parameters to respect the original signature.

> Changed in version 3.3: Added follow_symlinks argument. Now returns path
> to the newly created file.

[pylint] Anomalous backslash in string: '\S'. String constant might be
missing an r prefix.

and add the <BLANKLINES> where needed.

With shared builds, it's now possible that the [python2.7] part that is
providing executable for this buildout does not need to install any part and
makes the interpreter available since recipe initialisation.

This is causing an error during the overriden method computing part signature,
because parts are not installed yet:

    An internal error occurred due to a bug in either zc.buildout or in a
    recipe being used:
    Traceback (most recent call last):
      File "zc/buildout/buildout.py", line 2242, in main
        getattr(buildout, command)(args)
      File "zc/buildout/buildout.py", line 638, in install
        self._install_parts(install_args)
      File "zc/buildout/buildout.py", line 704, in _install_parts
        self._compute_part_signatures(install_parts)
      File "slapos/rebootstrap/__init__.py", line 37, in wrapper
        return getattr(self, attr)(*args, **kw)
      File "slapos/rebootstrap/__init__.py", line 95, in _compute_part_signatures
        installed_part_options[part]['__buildout_signature__']
    KeyError: 'already_installed_python'

To support that case, tolerate that parts are can be not installed yet at this
point.

/reviewed-on !1

- Work entirely in-place, even to switch to a different version of Python.
- Update bin/buildout to immediately use the wanted Python on subsequent
  buildout runs.

For SlapOS, the second point is required to have the instanciation done
with the built Python.

  Preserve bin-directory is required to while regenerate bin/buildout on
  the right location.