Our software using sshd were sometimes failing in tests, because the way they publish key fingerprint was racy.

It is based on `collective.recipe.shelloutput`, which as we can see in the [recipe code](https://github.com/collective/collective.recipe.shelloutput/blob/78e15c19/collective/recipe/shelloutput/__init__.py) operates on `__init__`.

We are using `collective.recipe.shelloutput` to capture the output of `ssh-keygen -lf $KEY` and this must run after the file `$KEY` is generated ( it is generated by another `plone.recipe.command` version). We were trying to run the `collective.recipe.shelloutput` after the `plone.recipe.command`, but that was incorrect anyway, because `collective.recipe.shelloutput` reads the file at `__init__` step, where `plone.recipe.command` creates the file at `install` step.
As we could see in test suite, it was sometimes working, when `slapos node instance` ran only once, but it sometimes  working, when `slapos node instance` ran more than once, for example because a promise failed and `slapos node instance` was retried.

Since `collective.recipe.shelloutput` does not take into account the exit code of the command but simply capture with `"Error ..."` whatever the command might output on stderr, we add another step checking that the captured output  is not `"Error ..."` and if it is cause a buildout error so that `slapos node instance` is retried and then succeed.

What should happen now is:
 1. `collective.recipe.shelloutput` reads the key fingerprint, the file is not present so it captures `"Error ..."``
 2. a `plone.recipe.command` creates the key
 3. another `plone.recipe.command` checks that the captured fingerprint is not `"Error ..."` it fails
 4. buildout restarts
 5. `collective.recipe.shelloutput` reads key fingerprint correctly.

Slaprunner has been heavily modified, because it was using a `sshkeys_authority` which was incompatible with this as it uses symlinks for keys. Since we don't know what is the purpose of `sshkeys_authority`, we rewrote that software to use simple commands instead of that "ssh keys authority".

/reviewed-on nexedi/slapos!681

slapos.core to 1.5.6 and slapos.toolbox to 0.104:

 * cleanup stale monitor files and stale promises
 * support download-from-binary-cache-force-url-list option

/reviewed-on nexedi/slapos!679

These keys are not managed by trust of a certificate authority, just by
"trust of first use" so it does not make sense to use a key authority.

This also cause difficulties to publish the key fingerprint as a
parameter, because we can't get the key fingerprint until the authority
service is started.

Also enable ecdsa key.

This fixes random failures with slaprunner tests, because the published
fingerprint was never correct on first buildout run.

Existing webrunners will have a new ssh host key after this.

Adjust `self._cleanup` to new signature from nexedi/slapos.core!156

/reviewed-on nexedi/slapos!677

Now that component/glib/buildout.cfg is no longer included by pkgconfig,
qemu no longer build with:

    While:
	Installing.
	Getting section template.
	Initializing section template.
	Getting section template-nbd.
	Initializing section template-nbd.
	Getting section kvm.
	Initializing section kvm.
	Getting option kvm:configure-options.
	Getting section glib.
	Error: The referenced section, 'glib', was not defined.

Also explictly add pcre to prevent same problem later.

The OS may not have Python (slapos-node package does not depend
on any version of Python). Another possible issue was found on Suse:

  make[3]: Entering directory '.../parts/glib__compile__/glib-2.56.4/gio'
    GEN      gdbus-daemon-generated.c
  Traceback (most recent call last):
    File "./gdbus-2.0/codegen/gdbus-codegen.in", line 53, in <module>
      from codegen import codegen_main
    File ".../parts/glib__compile__/glib-2.56.4/gio/gdbus-2.0/codegen/codegen_main.py", line 30, in <module>
      from . import parser
    File ".../parts/glib__compile__/glib-2.56.4/gio/gdbus-2.0/codegen/parser.py", line 23, in <module>
      import xml.parsers.expat
  ImportError: No module named xml.parsers.expat

Building normal glib is a nightmare, at least as long as we
haven't stopped using Python 2, so let's try to avoid it.

This makes bin/pkg-config bigger (GCC 6.3.0):
 old size:  47880
 new size: 573992

With Link Time Optimization, GCC 8.3.0 produces a binary of 154656 bytes.
However, GCC 6.3.0 fails to link so we have to wait.

Improvements in check_surykatka_promise:

 * SSL certificate checked for expiration (15 days before expiration)
 * messages from the promise are more readable

/reviewed-on nexedi/slapos!675

Supports SSL checks in check_surykatka_json.

This version exposes ssl_certificate information for hosts.

Caddy by itself does not raise soft limit of open files, so it has to be set
by the wrapper.

As slapos.cookbook:wrapper can't be used for such case, the
slapos.recipe.template:jinja2 recipe inline style is used to have full
control over the created wrapper.

/reviewed-on nexedi/slapos!678

Version 3.0.0 requires requests>=2.22.0

Errors were raised in ERP5 with classes inheriting from ExtensionClass.

This promise check if nextcloud app was correctly deployed, it ensure that slapos instance will deploy until nextcloud install succeed.

/reviewed-on nexedi/slapos!676

The called python program is python3 program, and provided
PYTHONPATH is from python2, resulting in bad behaviour.

On testnodes, python can be too long for linux shebang (#!) line.

We had for example:
    
    /srv/slapgrid/slappart15/srv/testnode/cqc/inst/test0-0/tmp/shared/python2.7/338ccc0e387d64185920266f8a97c87b/bin/python2.7
  

that is 123 characters long.

With this, the generated glib-genmarshall script will have:
    
    #!/usr/bin/env  /srv/slapgrid/slappart15/srv/testnode/cqc/inst/test0-0/tmp/shared/python2.7/338ccc0e387d64185920266f8a97c87b/bin/python2.7

which exceeds the linux kernel limit.

Instead, we generate scripts that will use
    
    #!/usr/bin/env python

and it will be the caller responsibility to set $PATH to use the expected python.


This fixes the issues we had on SlapOS.SoftwareReleases.IntegrationTest tests since we updated glib to 2.56.4 in a7a86db9

/reviewed-on !670

Provides a way to monitor edge PoP by using surykatka tool with
monitoring promises added in slapos.toolbox 1.101

Adds edgetest and edgebot software types, where edgetest is a software type
to request by user and requests by itself edgebot, which does the monitoring
work.

Based on Rafael Monnerat <rafael@nexedi.com>

Provides check_surykatka_json promise plugin.

python3 with venv to provide pip, which uses requirements file to install
stabilised set of distributions.

This reverts merge request !660

/reviewed-on nexedi/slapos!671

# To review

 - [x] [master parameters](https://lab.nexedi.com/luke/slapos/blob/feature/monitor-edge-test/software/monitor/instance-edgetest-input-schema.json)
 - [x] [slave parameters](https://lab.nexedi.com/luke/slapos/blob/feature/monitor-edge-test/software/monitor/instance-edgetest-slave-input-schema.json)

# Tasks

  - [x] assure that slaves are correctly destroyed (using SlapOS Master)


/reviewed-on nexedi/slapos!660

Adds edgetest and edgebot software types, where edgetest is a software type
to request by user and requests by itself edgebot, which does the monitoring
work.

Based on Rafael Monnerat <rafael@nexedi.com>

Provides check_surykatka_json promise plugin.

python3 with venv to provide pip, which uses requirements file to install
stabilised set of distributions.