See merge request !1756

See merge request !1749

The previous approach was ingesting uless "Time" lines sometimes

the default is 128 lines which is sometimes not enough

After previous patch the build of this test passes further than buildout rebootstrap but fails with

    Installing software release /srv/slapgrid/slappart5/srv/project/slapos/component/ZODB/test-zodb4-wc2.cfg ...
    ...
    Installing persistent-repository.
    Warning: "branch" parameter with value "master" is ignored. Checking out to revision 4.6.4-0-g7ed95cf.
    Cloning into '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/parts/persistent'...
    HEAD is now at 7ed95cf Preparing release 4.6.4
    Updating python-cffi.
    Updating persistent.
    DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
    Installing BTrees.
    networkcache: Trying to download pypi:persistent=4.9.3 from network cache...
    Getting distribution for 'persistent==4.9.3'.
        ERROR: Command errored out with exit status 1:
         command: /srv/slapgrid/slappart5/srv/runner/shared/python2.7/474396ed0d332179d494b6d9d3f28ef5/bin/python2.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-kywY6K/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-kywY6K/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-LeIkZV
             cwd: /tmp/pip-req-build-kywY6K/
        Complete output (25 lines):
        DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
        WARNING: Url 'file:///dev/null/cffi/' is ignored: it is neither a file nor a directory.
        ERROR: Could not find a version that satisfies the requirement cffi (from versions: none)
        ERROR: No matching distribution found for cffi
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-req-build-kywY6K/setup.py", line 149, in <module>
            entry_points={})
          File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/__init__.py", line 161, in setup
            _install_setup_requires(attrs)
          File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/__init__.py", line 156, in _install_setup_requires
            dist.fetch_build_eggs(dist.setup_requires)
          File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/dist.py", line 721, in fetch_build_eggs
            replace_conflicting=True,
          File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/pkg_resources/__init__.py", line 782, in resolve
            replace_conflicting=replace_conflicting
          File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/pkg_resources/__init__.py", line 1065, in best_match
            return self.obtain(req, installer)
          File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/pkg_resources/__init__.py", line 1077, in obtain
            return installer(requirement)
          File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/dist.py", line 777, in fetch_build_egg
            return fetch_build_egg(self, req)
          File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/installer.py", line 130, in fetch_build_egg
            raise DistutilsError(str(e))
        distutils.errors.DistutilsError: Command '['/srv/slapgrid/slappart5/srv/runner/shared/python2.7/474396ed0d332179d494b6d9d3f28ef5/bin/python2.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmptaQnzb', '--quiet', '--index-url', 'file:///dev/null', 'cffi']' returned non-zero exit status 1
        ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    While:
      Installing BTrees.
      Base installation request: 'persistent'
      Getting distribution for 'persistent==4.9.3'.

    An internal error occurred due to a bug in either zc.buildout or in a
    recipe being used:
    Traceback (most recent call last):
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/buildout.py", line 2664, in main
        getattr(buildout, command)(args)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/buildout.py", line 855, in install
        self._install_parts(install_args)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/buildout.py", line 1026, in _install_parts
        installed_files = self[part]._call(recipe.install)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/buildout.py", line 1948, in _call
        return f()
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.recipe.egg-2.0.8.dev0+slapos009-py2.7.egg/zc/recipe/egg/custom.py", line 79, in install
        self._install_setup_eggs()
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.recipe.egg-2.0.8.dev0+slapos009-py2.7.egg/zc/recipe/egg/custom.py", line 119, in _install_setup_eggs
        newest=self.newest,
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 1178, in install
        return installer.install(specs, working_set, patch_dict=patch_dict)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 835, in install
        for dist in self._get_dist(requirement, ws):
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 711, in _get_dist
        dists = [_move_to_eggs_dir_and_compile(dist, self._dest, self)]
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 2205, in _move_to_eggs_dir_and_compile
        [tmp_loc] = call_pip_wheel(dist.location, tmp_dest, options)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 2100, in call_pip_wheel
        call_pip_command(['wheel', '-w', dest], [spec], options)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 2038, in call_pip_command
        subprocess.check_call(args, env=env)
      File "/srv/slapgrid/slappart5/srv/runner/shared/python2.7/474396ed0d332179d494b6d9d3f28ef5/lib/python2.7/subprocess.py", line 190, in check_call
        raise CalledProcessError(retcode, cmd)
    CalledProcessError: Command '['/srv/slapgrid/slappart5/srv/runner/shared/python2.7/474396ed0d332179d494b6d9d3f28ef5/bin/python2.7', '-m', 'pip', 'wheel', '-w', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/tmpmfO6r5', '--no-deps', '-q', '--no-index', '--no-build-isolation', '--no-python-version-warning', '--no-cache-dir', '/tmp/tmpbuYU1zget_dist/persistent-4.9.3.tar.gz']' returned non-zero exit status 1
    2025-03-03 06:25:33 slapos[1517575] ERROR Failed to run buildout profile in directory '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066'
    2025-03-03 06:25:33 slapos[1517575] INFO Finished software releases.

What happens here is that test-zodb4-wc2.cfg installs persistent via git clone
but at 4.6.4 version, while the version of wanted egg in ZODB/buildout.cfg
[versions] is 4.9.3. This way when buildout sees that BTrees needs persistent as
the dependency, it looks for "whether we already have persistent==4.9.3
installed", determines the answer as "no" (as we have persistent 4.6.4), and
tries to install persistent by itself, and fails somewhere around cffi which we
also provide via zc.recipe.egg:custom.

The cffi error is only a consequence of original problem of mismatched
persistent versions.

-> Fix BTrees/persistent build by keeping used persistent versions in sync in between buildout.cfg and test-zodb4-wc2.cfg

As test-zodb4-wc2.cfg explains

    we need persistent to be a git checkout because persistent tests want to
    discover in-tree files that are not present in persistent egg when it is
    installed in non-development mode:

      https://erp5.nexedi.net/test_result_module/20201123-3F859E35/7
      (look for "AssertionError: could not find my setup.py")

      https://github.com/zopefoundation/persistent/blob/4.6.4-0-g7ed95cf/persistent/tests/test_docs.py#L37-L43

and on the other hand we do not use git checkout way uniformly because people
wanted to use plain eggs when component/ZODB was bootstrapped.

This way we have to use both egg and git checkout and keep their versions in sync.

After this patch the build of test-zodb4-wc2 is recovered.

/cc @xavier_thompson
/reviewed-by @kazuhiko, @levin.zimmermann
/helped-and-reviewed-by @jerome
/reviewed-on !1752

Since c170fe3e (ZODB: Provide ZODB4-wc2 which backports MVCC approach from
ZODB5) we have a custom version of ZODB4 with backports from ZODB5 so that
wendelin.core can work in environments where we still use ZODB4.

In 2022 this test got broken after SlapOS transitioned to use python3 by
default and fixed in 5cb11e9e (component/ZODB: keep running the test on
python3). Then fixed in 2024 again in e742508b (! component/openssl: add
OpenSSL-3.0.13, still using OpenSSL-1.1 for python 2.).

However as of today building this test fails with

    slapos.rebootstrap:
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************
    bin/buildout is being reinstalled right now, as new python:
      /srv/slapgrid/slappart5/srv/runner/shared/python2.7/b8880e7ab7f4124a466ecb817efb447c/bin/python2.7
    is available, and buildout is using another python:
      /opt/slapos/parts/python3/bin/python3
    Buildout will be restarted automatically to have this change applied.
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg/zc/__init__.py", line 1, in <module>
        __import__('pkg_resources').declare_namespace(__name__)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-67.8.0-py3.7.egg/pkg_resources/__init__.py", line 1423
        local = f"sanitized.{_safe_segment(rest)}".strip(".")
                                                 ^
    SyntaxError: invalid syntax
    While:
      Installing.

    An internal error occurred due to a bug in either zc.buildout or in a
    recipe being used:
    Traceback (most recent call last):
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg/zc/buildout/buildout.py", line 2664, in main
        getattr(buildout, command)(args)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg/zc/buildout/buildout.py", line 855, in install
        self._install_parts(install_args)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg/zc/buildout/buildout.py", line 870, in _install_parts
        self._setup_directories()
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/slapos.rebootstrap-4.7-py3.7.egg/slapos/rebootstrap/__init__.py", line 58, in wrapper
        return getattr(self, attr)(*args, **kw)
      File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/slapos.rebootstrap-4.7-py3.7.egg/slapos/rebootstrap/__init__.py", line 108, in _setup_directories
        "zc.buildout.buildout.main()" % (paths, args + ['bootstrap'])])
      File "/opt/slapos/parts/python3/lib/python3.7/subprocess.py", line 363, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/srv/slapgrid/slappart5/srv/runner/shared/python2.7/b8880e7ab7f4124a466ecb817efb447c/bin/python2.7', '-c', "import sys ; sys.path[0:0]=['/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/slapos.libnetworkcache-0.25-py3.7.egg', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-67.8.0-py3.7.egg', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/wheel-0.41.2-py3.7.egg', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/pip-23.2.1-py3.7.egg'] ; import zc.buildout.buildout ; sys.argv[1:1]=['buildout:extends-cache=/tmp/tmpdx2upamb', 'buildout:directory=/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066', 'bootstrap'] ; zc.buildout.buildout.main()"]' returned non-zero exit status 1.

The problem here is that even though we use py2 for the software, the used
version of setuptools is py3 only and so it breaks.

This problem was already cared in f5b80301 (Downgrade pip, wheel and setuptools
in python2 SRs), but that commit missed to update test-zodb4-wc2.cfg .

-> Fix that.

After this patch the build of component/ZODB/test-zodb4-wc2.cfg becomes
recovered in the sense that it passes further than rebootstrap, but breaks at
the build of persistent / BTrees eggs. We will fix this in the next patch.

/cc @xavier_thompson
/reviewed-by @jerome, @kazuhiko, @levin.zimmermann
/reviewed-on !1752

This reverts commit 552a9854.

This reverts commit 69d7b336.

Fixed version.

slapos.core and slapos.cookbook with fixed package release.

Including changes:

 * librecipe: stop calling getInstanceParameterDict by default
 * slapos.cookbook: Fix publish_failsafe

There is no reason to have so many empty lines, as other tools can have
troubles with processing it.

Do not return installed files in order to avoid endless Uninstall/Install of
the part and instead use the uninstall hook to handle the file when part is
uinstalled.

Also re-install during update if there was a problem with publishing during
install; note that checking for file presence on each update is very fast, so
the speed gain is preserved.

Call it only on demand, to get rid of useless http queries

A new version of zodburi has been released [1] that
removes Python2 compatibility shims [2] and is therefore
needed to successfully run 'wendelin.core' tests with python3 [3].

Suggested by @kirr.

[1] https://pypi.org/project/zodburi/3.0.0/
[2] https://github.com/Pylons/zodburi/commit/a00fea755
[3] wendelin.core!35

/reviewed-by @kirr @jerome
/reviewed-on !1745

When there is not a nobody user (for example inside Docker), the
tests TestAmbientCaps and TestAmbientCapsUserns should be skipped
instead of failing.

This has been patched upstream in
https://github.com/golang/go/commit/1959703dddffef61760d5d4fdedc4df8d2d73438
and we use here that patch.

See merge request !1741

Starting from pygolang@95fd2889
pygolang needs zodbpickle for tests. So without pinning zodbpickle
testnodes starrted to error with

    Installing gpython-interpreter.
    While:
      Installing gpython-interpreter.
      Base installation request: 'pygolang[all_test]'
        Requirement of pygolang[all_test]: py; python_version >= "3" and extra == "all_test"
        Requirement of pygolang[all_test]: zodbpickle; extra == "all-test"
        Requirement of pygolang[all_test]: wheel; extra == "all-test"
        Requirement of pygolang[all_test]: setuptools-dso>=2.8; extra == "all-test"
        Requirement of pygolang[all_test]: setuptools; extra == "all-test"
        Requirement of pygolang[all_test]: pytest; extra == "all-test"
        Requirement of pygolang[all_test]: numpy; extra == "all-test"
        Requirement of pygolang[all_test]: ipython; extra == "all-test"
        Requirement of pygolang[all_test]: cython<3; extra == "all-test"
        Requirement of pygolang[all_test]: geventmp; python_version >= "3" and platform_system != "Windows"
        Requirement of pygolang[all_test]: setuptools>=60.2; python_version >= "3"
        Requirement of pygolang[all_test]: setuptools-dso>=2.8
        Requirement of pygolang[all_test]: decorator
        Requirement of pygolang[all_test]: six
        Requirement of pygolang[all_test]: gevent
      Getting distribution for 'zodbpickle; extra == "all-test"'.
    Error: Picked: zodbpickle = 4.2

    The `zodbpickle` egg does not have a version pin and `allow-picked-versions = false`.

    To resolve this, add

        zodbpickle = 4.2

    to the [versions] section,

    OR set `allow-picked-versions = true`.

/reviewed-by @jerome
/reviewed-on !1744

See merge request nexedi/slapos!1738

This reverts commit 95b1d7df as
testnodes were upgraded.