See merge request nexedi/slapos!931

The resiliency pull-backup-server status feed file `pbs-status-rss`
is generated by a cron running every 5 minutes. But there is a promise
that requires it to check the notifier status, meaning instanciation
fails until this file is available.

To allow the promise to succeed immediately, we generate the
feed a first time directly from buildout.

The resiliency notifier status feed file `notifier-status-rss`
is generated by a cron running every 5 minutes. But there is a
promise that requires it to check the notifier status, meaning
instanciation fails until this file is available.

To allow the promise to succeed immediately, we generate the
feed a first time directly from buildout.

See merge request nexedi/slapos!1041

we need to be explicit for extras.
Ping egg.

With multiple security fixes

See merge request nexedi/slapos!992

Follow-up for commit 98e7a29a (component/trafficserver: Version up to
9.0.2).

Drop patch from pull-request 7577, as it has since been merged upstream.

Zodburi 2.5.0 added support for demo: URI scheme:
https://github.com/Pylons/zodburi/blob/master/CHANGES.rst#250-2021-05-12

- due to a thinko in c170fe3e (ZODB: Provide ZODB4-wc2 which backports MVCC approach from ZODB5)
  4-wc2 revision was set on a ZEO(egg) section instead of ZEO(repository)
  section. As the result it was ZEO4 - not ZEO4-wc2 - that was installed.

  This was leading to hangs in tests with wcfs waiting for ZODB server to
  update it after a commit, but no notification was coming from ZEO at all.

  For the refrence: the reason ZEO4-wc2 exists at all is the backport patches
  that teach ZEO to send invalidations even on empty transactions
  (nexedi/ZEO@bf80d23d), and to include
  just-created objects into invalidations as well (nexedi/ZEO@f2fae122).
  Those changes are needed for WCFS to be always notificated on every
  transaction and with full set of changed objects.

- there was a typo: it was `revisions` instead of `revision`.

-> Fix both issues. Now it is ZEO4-wc2 that is installed for real when `ZODB.major = 4-wc2`.

The promise is commented out because it fails inside a test suite
due to port conflicts with the testnode of the test suite.

`srv/testnode/abc/inst/test0-0`  -> `t/abc/i/0`

Replace `softwaretype` recipe with `switch-softwaretype`

This is similar to d67f382f (wendelin.core: Fix test build 2)

Always reference to in-tree eggs via ${<name>:egg} instead of just name. This fixes e.g. the following build failure:

    Installing wendelin.core-python.
    While:
      Installing wendelin.core-python.
      Base installation request: 'wendelin.core[test]', 'pygolang[pyx.build]', 'neoppod[tests]', 'ZEO[test]'
      Getting distribution for 'ZEO[test]'.
    Error: Picked: ZEO = 5.2.3

Not sure why it worked before.

Neoppod is a bit special - leave it as is for now.

This teaches nxdtest to detect leaked processes and to clean them up
automatically instead of letting tests stuck because of them. See
nexedi/nxdtest@0ad45a9c and
nexedi/nxdtest!9 for details.

New dependencies of nxdtest are:

- psutil (already covered by stack/slapos.cfg),
- pytest-timeout, and
- setproctitle

/reviewed-by @jerome
/reviewed-on nexedi/slapos!1035

These comments are not updated automatically and very hard to update manually,
so it's better not to have them.

Users are often providing wrong values which are rejected later. So give the
clear information to the user.

This also adds a headless-chromium component and enables the http_sub
module in the Nginx component. Some basic tests are included. See merge
request nexedi/slapos!1014 for more details and discussion.

This change includes a rewrite of the previous
component/headless-chromium, which used an outdated version of Chromium.
It differs from component/chromium in that:

- It only provides a headless shell, rather than the full Chromium
  browser (though this can be changed).
- It compiles from source rather than downloading a binary build.

There is also a new software release software/headless-chromium which
makes use of the updated component/headless-chromium to expose a remote
debugging port.

See also !1014.

Wendelin.core2[test] depends on scipy[1]. Before slapos.buildout
2.7.1+slapos013 the build was succeeding because scipy was, likely, installed
via binary wheel. But after that buildout upgrade the build fails:

    WARNING: The easy_install command is deprecated and will be removed in a future version.
    /tmp/easy_install-vYdhd1/scipy-0.19.0/setup.py:323: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
      warnings.warn("Unrecognized setuptools command, proceeding with "
    Traceback (most recent call last):
      File "<string>", line 10, in <module>
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 2321, in main
        **kw
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/__init__.py", line 162, in setup
        return distutils.core.setup(**attrs)
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 424, in run
        self.easy_install(spec, not self.no_deps)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 666, in easy_install
        return self.install_item(None, spec, tmpdir, deps, True)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 711, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 896, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 1164, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 1150, in run_setup
        run_setup(setup_script, args)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "<string>", line 7, in setup_context
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-vYdhd1/scipy-0.19.0/setup.py", line 416, in <module>
      File "/tmp/easy_install-vYdhd1/scipy-0.19.0/setup.py", line 396, in setup_package
    ImportError: No module named numpy.distutils.core
    An error occurred when trying to install scipy 0.19.0. Look above this message for any errors that were output by easy_install.
    While:
      Installing wendelin.core-python.
      Base installation request: 'wendelin.core[test]', 'pygolang[pyx.build]', 'neoppod[tests]', 'ZEO[test]'
        Requirement of ZEO[test]: random2
        Requirement of ZEO[test]: manuel
        Requirement of ZEO[test]: zope.testing
        Requirement of ZEO[test]: zope.interface
        Requirement of ZEO[test]: zdaemon
        Requirement of ZEO[test]: ZConfig
        Requirement of ZEO[test]: zc.lockfile
        Requirement of ZEO[test]: persistent>=4.1.0
        Requirement of ZEO[test]: transaction<2.0.0
        Requirement of ZEO[test]: six
        Requirement of ZEO[test]: ZODB<4.999,>=4.2.0b1
        Requirement of neoppod[tests]==1.12.0: neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]
        Requirement of neoppod[tests]==1.12.0: psutil>=2
        Requirement of neoppod[tests]==1.12.0: zope.testing
        Requirement of neoppod[tests]==1.12.0: coverage
        Requirement of neoppod[tests]==1.12.0: python-dateutil
        Requirement of pygolang[pyx.build]==0.0.8: wheel
        Requirement of pygolang[pyx.build]==0.0.8: setuptools_dso>=1.7
        Requirement of pygolang[pyx.build]==0.0.8: setuptools
        Requirement of pygolang[pyx.build]==0.0.8: cython
        Requirement of pygolang[pyx.build]==0.0.8: Importing
        Requirement of pygolang[pyx.build]==0.0.8: decorator
        Requirement of pygolang[pyx.build]==0.0.8: six
        Requirement of pygolang[pyx.build]==0.0.8: gevent
        Requirement of wendelin.core[test]: scipy
        Requirement of wendelin.core[test]: pytest
        Requirement of wendelin.core[test]: psutil
        Requirement of wendelin.core[test]: six
        Requirement of wendelin.core[test]: pygolang>=0.0.8
        Requirement of wendelin.core[test]: zodbtools>=0.0.0.dev8
        Requirement of wendelin.core[test]: ZODB>=4
        Requirement of wendelin.core[test]: numpy
        Requirement of manuel: six
        Requirement of manuel: setuptools
        Requirement of zope.testing: zope.interface
        Requirement of zope.testing: zope.exceptions
        Requirement of zope.testing: setuptools
        Requirement of zope.interface: setuptools
        Requirement of zdaemon: setuptools
        Requirement of zdaemon: ZConfig
        Requirement of zc.lockfile: setuptools
        Requirement of persistent>=4.1.0: cffi
        Requirement of persistent>=4.1.0: zope.interface
        Requirement of transaction<2.0.0: zope.interface
        Requirement of ZODB<4.999,>=4.2.0b1: zodbpickle>=0.6.0
        Requirement of ZODB<4.999,>=4.2.0b1: zope.interface
        Requirement of ZODB<4.999,>=4.2.0b1: zc.lockfile
        Requirement of ZODB<4.999,>=4.2.0b1: six
        Requirement of ZODB<4.999,>=4.2.0b1: transaction>=1.5.0
        Requirement of ZODB<4.999,>=4.2.0b1: ZConfig
        Requirement of ZODB<4.999,>=4.2.0b1: BTrees>=4.2.0
        Requirement of ZODB<4.999,>=4.2.0b1: persistent>=4.2.0
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: ZODB3>=3.10dev
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: mysqlclient
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: setproctitle
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: msgpack>=0.5.6
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: ZODB3>=3.10dev
        Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: python-dateutil
        Requirement of python-dateutil: six>=1.5
        Requirement of gevent: greenlet>=0.4.17
        Requirement of gevent: setuptools
        Requirement of gevent: zope.interface
        Requirement of gevent: zope.event
      Getting distribution for 'scipy==0.19.0'.
    Error: Couldn't install: scipy 0.19.0

-> Fix it by explicitly installing SciPy via our component/scipy/ which lists numpy in its setup_eggs.

[1] https://lab.nexedi.com/kirr/wendelin.core/blob/5e647b59/setup.py#L384-386