This is similar to d67f382f (wendelin.core: Fix test build 2)

Always reference to in-tree eggs via ${<name>:egg} instead of just name. This fixes e.g. the following build failure:

    Installing wendelin.core-python.
    While:
      Installing wendelin.core-python.
      Base installation request: 'wendelin.core[test]', 'pygolang[pyx.build]', 'neoppod[tests]', 'ZEO[test]'
      Getting distribution for 'ZEO[test]'.
    Error: Picked: ZEO = 5.2.3

Not sure why it worked before.

Neoppod is a bit special - leave it as is for now.

This teaches nxdtest to detect leaked processes and to clean them up
automatically instead of letting tests stuck because of them. See
nexedi/nxdtest@0ad45a9c and
nexedi/nxdtest!9 for details.

New dependencies of nxdtest are:

- psutil (already covered by stack/slapos.cfg),
- pytest-timeout, and
- setproctitle

/reviewed-by @jerome
/reviewed-on nexedi/slapos!1035

These comments are not updated automatically and very hard to update manually,
so it's better not to have them.

Users are often providing wrong values which are rejected later. So give the
clear information to the user.

This also adds a headless-chromium component and enables the http_sub
module in the Nginx component. Some basic tests are included. See merge
request nexedi/slapos!1014 for more details and discussion.

This change includes a rewrite of the previous
component/headless-chromium, which used an outdated version of Chromium.
It differs from component/chromium in that:

- It only provides a headless shell, rather than the full Chromium
  browser (though this can be changed).
- It compiles from source rather than downloading a binary build.

There is also a new software release software/headless-chromium which
makes use of the updated component/headless-chromium to expose a remote
debugging port.

See also nexedi/slapos!1014.

Wendelin.core2[test] depends on scipy[1]. Before slapos.buildout
2.7.1+slapos013 the build was succeeding because scipy was, likely, installed
via binary wheel. But after that buildout upgrade the build fails:

    WARNING: The easy_install command is deprecated and will be removed in a future version.
    /tmp/easy_install-vYdhd1/scipy-0.19.0/setup.py:323: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
      warnings.warn("Unrecognized setuptools command, proceeding with "
    Traceback (most recent call last):
      File "<string>", line 10, in <module>
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 2321, in main
        **kw
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/__init__.py", line 162, in setup
        return distutils.core.setup(**attrs)
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 424, in run
        self.easy_install(spec, not self.no_deps)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 666, in easy_install
        return self.install_item(None, spec, tmpdir, deps, True)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 711, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 896, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 1164, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 1150, in run_setup
        run_setup(setup_script, args)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-vYdhd1/scipy-0.19.0/setup.py", line 416, in <module>
      File "/tmp/easy_install-vYdhd1/scipy-0.19.0/setup.py", line 396, in setup_package
    ImportError: No module named numpy.distutils.core
    An error occurred when trying to install scipy 0.19.0. Look above this message for any errors that were output by easy_install.
    While:
      Installing wendelin.core-python.
      Base installation request: 'wendelin.core[test]', 'pygolang[pyx.build]', 'neoppod[tests]', 'ZEO[test]'
        Requirement of ZEO[test]: random2
        Requirement of ZEO[test]: manuel
        Requirement of ZEO[test]: zope.testing
        Requirement of ZEO[test]: zope.interface
        Requirement of ZEO[test]: zdaemon
        Requirement of ZEO[test]: ZConfig
        Requirement of ZEO[test]: zc.lockfile
        Requirement of ZEO[test]: persistent>=4.1.0
        Requirement of ZEO[test]: transaction<2.0.0
        Requirement of ZEO[test]: six
        Requirement of ZEO[test]: ZODB<4.999,>=4.2.0b1
        Requirement of neoppod[tests]==1.12.0: neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]
        Requirement of neoppod[tests]==1.12.0: psutil>=2
        Requirement of neoppod[tests]==1.12.0: zope.testing
        Requirement of neoppod[tests]==1.12.0: coverage
        Requirement of neoppod[tests]==1.12.0: python-dateutil
        Requirement of pygolang[pyx.build]==0.0.8: wheel
        Requirement of pygolang[pyx.build]==0.0.8: setuptools_dso>=1.7
        Requirement of pygolang[pyx.build]==0.0.8: setuptools
        Requirement of pygolang[pyx.build]==0.0.8: cython
        Requirement of pygolang[pyx.build]==0.0.8: Importing
        Requirement of pygolang[pyx.build]==0.0.8: decorator
        Requirement of pygolang[pyx.build]==0.0.8: six
        Requirement of pygolang[pyx.build]==0.0.8: gevent
        Requirement of wendelin.core[test]: scipy
        Requirement of wendelin.core[test]: pytest
        Requirement of wendelin.core[test]: psutil
        Requirement of wendelin.core[test]: six
        Requirement of wendelin.core[test]: pygolang>=0.0.8
        Requirement of wendelin.core[test]: zodbtools>=0.0.0.dev8
        Requirement of wendelin.core[test]: ZODB>=4
        Requirement of wendelin.core[test]: numpy
        Requirement of manuel: six
        Requirement of manuel: setuptools
        Requirement of zope.testing: zope.interface
        Requirement of zope.testing: zope.exceptions
        Requirement of zope.testing: setuptools
        Requirement of zope.interface: setuptools
        Requirement of zdaemon: setuptools
        Requirement of zdaemon: ZConfig
        Requirement of zc.lockfile: setuptools
        Requirement of persistent>=4.1.0: cffi
        Requirement of persistent>=4.1.0: zope.interface
        Requirement of transaction<2.0.0: zope.interface
        Requirement of ZODB<4.999,>=4.2.0b1: zodbpickle>=0.6.0
        Requirement of ZODB<4.999,>=4.2.0b1: zope.interface
        Requirement of ZODB<4.999,>=4.2.0b1: zc.lockfile
        Requirement of ZODB<4.999,>=4.2.0b1: six
        Requirement of ZODB<4.999,>=4.2.0b1: transaction>=1.5.0
        Requirement of ZODB<4.999,>=4.2.0b1: ZConfig
        Requirement of ZODB<4.999,>=4.2.0b1: BTrees>=4.2.0
        Requirement of ZODB<4.999,>=4.2.0b1: persistent>=4.2.0
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: ZODB3>=3.10dev
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: mysqlclient
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: setproctitle
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: msgpack>=0.5.6
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: ZODB3>=3.10dev
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: python-dateutil
        Requirement of python-dateutil: six>=1.5
        Requirement of gevent: greenlet>=0.4.17
        Requirement of gevent: setuptools
        Requirement of gevent: zope.interface
        Requirement of gevent: zope.event
      Getting distribution for 'scipy==0.19.0'.
    Error: Couldn't install: scipy 0.19.0

-> Fix it by explicitly installing SciPy via our component/scipy/ which lists numpy in its setup_eggs.

[1] https://lab.nexedi.com/kirr/wendelin.core/blob/5e647b59/setup.py#L384-386

If `only-sr` is not provided, all the SR tests should run.

Fixes merge request nexedi/slapos!1011

ZEO 5.2.3 fixes several client concurrency bugs that could result in
data corruption. See changelog in https://pypi.org/project/ZEO/5.2.3/
for details.

trollius 2.2.post1 -> 2.2.1 update fixes "bad file descriptor" problem for ZEO5.
See https://github.com/zopefoundation/ZEO/issues/176 for details.

futures 3.2.0 -> 3.3.0 update brings in more efficient use of threads in
ThreadPoolExecutor. See https://github.com/agronholm/pythonfutures/blob/master/CHANGES.rst#330
for details.

Test was sometimes failing with:

======================================================================
FAIL: test_http_get (test.TestRepman)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "slapos-repository/software/repman/test/test.py", line 81, in test_http_get
    self.assertTrue(cluster['isProvision'])
AssertionError: False is not true

It can take a little time before everything is ready and we don't have a
promise for this, because it's really application level.

See merge request nexedi/slapos!1031

This propery has a default, which indicates that it is not required.

https://rjsf-team.github.io/react-jsonschema-form/ can handle this
correctly, but needs a default along with the const.

Most tools work without this, but it's cleaner to define this anyway

This reverts commit f34aeeab.

this had the side effect of installing `promise-check-free-disk-space` which
is still used - at least by caddy tests.

In the test output, there are lots of tracebacks like this one:

    "./software/caddy-frontend/test/test.py", line 534, in do_GET
        for header in config.pop('X-Drop-Header', '').split():
    AttributeError: 'NoneType' object has no attribute 'split'

this does not seem to impact the test result, but it makes the result hard to
read.

Now that slapos-node-software service does not use --all we have to use another
approach for this test which re-installs an already installed software.

Use the slapos-node-software-all services which was made for similar use cases.

This was changed as part of b049c057 (upgrade to slapos.toolbox version
0.123, 2021-07-19), but this is not a slapos.toolbox promise plugin, it's
still an old style promise.

when running on test nodes some promise fail because testnode itself has
monitoring and the promise connects to testnode's own monitoring.

This reveals a real problem, but we need to fix testnode first to be able
to use this.

With gcc 6.3.0 we see build failures:

    error: no matching function for call to 'tesseract::STRING::STRING()'

According to https://github.com/tesseract-ocr/tesseract/issues/3210#issuecomment-774599019
minimal gcc version is version 7.

This probably started to break since d6dd75df (component/ghostscript:
version up 9.54.0, 2021-05-25) and 5f616dd7 (component/tesseract: version
up 4.1.1, 2021-05-20).

This gcc is needed for both tesseract and ghostscript, because ghostscript
build again its embedded copy of tesseract.

In b049c057 (upgrade to slapos.toolbox version 0.123, 2021-07-19), we
translated check-secure = 1 to config-http-code = 401, but check-secure = 1
also accepted responses with status code 200.

Thoses promises where checking servers without authentication enabled, they
reply with 200, so update the promises accordingly to expect 200.

This fixes a username/password confusion from c536eb7c (stack/monitor:
change frontend url promise to use username/password, 2021-07-26)

This change has no effect on stand-alone setups (which is what is
automatically deployed).
On setups configured to replicate from another database, it allows them to
takeover the role of the primary server, which allows rebuilding a
redundant setup. On such setup, the cost of this option is the extra I/O
load of binlogs being written to (in addition to the relay log and
database), and the corresponding disk space use.

Commit 7d1ea024 (version up: zc.buildout 2.7.1+slapos013) switched buildout to
not use egg wheels, even for wheels that are source-only without any binaries.
As the result some eggs stopped to install because they setup_require other
eggs and neither buildout handles setup_requires automatically, nor there is
explicit setup_require declaration in corresponding components.

An example of such egg is python-dateutil: it setup_requires setuptools-scm,
and without switching python-dateutil to zc.recipe.egg:custom with
setup_requires the build fails.

The way 7d1ea024 used to workaround this is to mark all such eggs with :whl
suffix marking them that "it is ok to use wheel for this particular egg". The
complete list of eggs for which such markings was done in 7d1ea024 is:

    ipykernel
    plantuml
    mock
    PasteScript
    backports.functools-lru-cache
    cliff
    stevedore
    CacheControl
    pytest-runner
    jsonschema
    lockfile
    python-dateutil

However this list is not really complete because after 7d1ea024 the build
started to fail for e.g. zodbtools test:

        Traceback (most recent call last):
          File "<string>", line 10, in <module>
          File "/srv/slapgrid/slappart16/srv/testnode/dfp/soft/c670f74c8de08964f63c4b9f7d48c3d1/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 2321, in main
          ...
          File "/srv/slapgrid/slappart16/srv/testnode/dfp/soft/c670f74c8de08964f63c4b9f7d48c3d1/eggs/setuptools-44.1.1-py2.7.egg/pkg_resources/__init__.py", line 786, in resolve
            raise DistributionNotFound(req, requirers)
        pkg_resources.DistributionNotFound: The 'setuptools-scm' distribution was not found and is required by the application
        An error occurred when trying to install pytest 4.6.11. Look above this message for any errors that were output by easy_install.
        While:
          Installing zodbtools-python.
          Base installation request: 'zodbtools[test]'
            Requirement of zodbtools[test]: mock
            Requirement of zodbtools[test]: pytz
            Requirement of zodbtools[test]: freezegun
            Requirement of zodbtools[test]: pytest
            Requirement of zodbtools[test]: dateparser
            Requirement of zodbtools[test]: six
            Requirement of zodbtools[test]: pygolang>=0.0.0.dev6
            Requirement of zodbtools[test]: zope.interface
            Requirement of zodbtools[test]: zodburi
            Requirement of zodbtools[test]: ZODB
            Requirement of mock: funcsigs>=1
            Requirement of mock: six
            Requirement of freezegun: python-dateutil>=2.7
          Getting distribution for 'pytest==4.6.11'.
        Error: Couldn't install: pytest 4.6.11

and similarly for pygolang and nxdtest tests.

I would be more comfortable if the fix would be to teach buildout to either
handle setup_requires automatically, or to automatically install wheels if the
wheel is source-only. However, given the way - that 7d1ea024 started to use
to handle such situations - let's fix this via explicitly marking pytest, and
other related eggs that setup_require something and are source-only, as :whl.

NOTE we did not notice the problem earlier because on testnodes, after the
switch to new buildout, pytest egg was already installed and incremental
rebuild did not need to reinstall pytest egg. So most of pygolang and zodbtools
test results remained in "ok" state, and only zodbtools-kirr, that was recently
reconfigured or changed testnode, indicated this issue. That's why the problem
remained unnoticed for a long time.

/reviewed-by @jerome
/reviewed-on nexedi/slapos!1026

See merge request nexedi/slapos!1022

This introduces some breaking changes to the check_url_available
promise:

- http_code has been renamed to http-code
- check-secure = 1 has been deprecated; http-code = 401 should be used
  instead

See merge request nexedi/slapos!1011

also apply the change in software/theia/test/dummy/software.cfg for consistency

  This is a partial port, just to have more tests that
  weren't afected by the patches for keep apache as entry-point
  of the backend.