The case of firefox 60 described in the commit messages showed that we should not use `$TMPDIR` instead of `/tmp` in our patch to isolate `$DISPLAY`s.

This introduce a new environment variable `XORG_LOCK_DIR` with clearer semantics.

Softwares relying on this have been updated or removed when not used.

The following softwares where already OK:
 * ERP5 and erp5testnode did not set `$TMPDIR` and were using `/tmp/`
 * cloudooo runs in `--headless` mode

/reviewed-on !394

current version failed to build in gcc6
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=811741

Slaves' hostnames (custom_domain or server-alias) can clash, resulting
with stopping the whole frontend.

In order to avoid so assure hostname unicity across whole frontend.

Even if the master partition owner will authorise given slave for custom
configuration reject this slave in case if it does not pass validation for
snippet.

Treat custom configuration as normal condition to reject/accept slave.

This will allow further anaylsis.

If server alias duplicates name of the server or another alias, simply
ignore it.

Default behavior of runUnitTest was to listen on `127.0.0.1`/`localhost`, which is not
allowed for SlapOS instance and caused issues when the used DNS server does not resolve localhost.

Instead, we decided to run the zserver on the partition IP. Because this partition's IP is not seen as a secure origin (as defined in https://goo.gl/Y0ZkNV ) from firefox when running zelenium tests, we also add some entries in apache from balancer partition to act as an https proxy on this zserver and change the testrunners to access the https URL.

See also erp5!729 for the erp5 part.

/reviewed-on !374

We don't want runUnitTest is to listen on 127.0.0.1, as it is not
allowed for SlapOS instances, runUnitTest wrapper is extended to include
`--zserver` with a pre-assigned ip:port.

runTestSuite starts several test in parrallel (controlled by
`--node_quantity` argument, which is passed by erp5testnode), so we need
to make sure that we provide it with enough ip:port. For this, we
extended runTestSuite with a `--zserver_address_list` argument and we
generate a wrapper with a list of `testrunner.node-count` pre-assigned
ip:ports.

Because zelenium tests needs to access this zserver over a secure origin
(otherwise modern browser features such as service worker are not
available), use an https proxy in the apache from the balancer
partition.

runUnitTest and runTestSuite have been extended with resp.
`--zserver_frontend_url` and `--zserver_frontend_url_list` arguments and
the URLs published by the balancer paritions are set in the wrappers.
For compatibility reasons, runTestSuite pass parameters as environment
variables.

Implementation notes:

This introduces a circular depencency, balancer partition needs to know
the address of the testrunners and zope partitions needs to know the
URLs of the corresponding http proxies on the apache. This is is handled
by `slapos.recipe:publish-early`:

 1. request zope family with an empty `test-runner-apache-url-list`.
    zope is instanciated a first time.
    zope returns `test-runner-address-list` ( a list of (host, port)
    tuples )
 2. request balancer with `test-runner-address-list`
    balancer is instanciated.
    balancer returns `{{ family_name }}-test-runner-url-list` ( a list
    of apache URLs ), which are published in the root partition.
 3. zope family is re-requested with
    updated`test-runner-apache-url-list` information

instance-erp5.cfg.in template was also reorganised to move
`[publish-early]` next to `[publish]` at the bottom of the file because
these sections are semantically related.
Also test runner generation is moved after zope generation, because we
want to allocate test runners ports after zopes, otherwise existing
zopes would get new ports when existing instances are upgraded.

`enable_authentication` parameter of `backend-list`
was not documented.

We have new parameters "how many parallel tests this instance can run"
( test-runner.node-count ) and "how many supplementary databases each
test needs" ( test-runner.extra-database-count ).

Instead of configuring "I need a total of 15 databases", we configure "I
need to be able to run 3 tests in parallel, each test requiring 4 extra
databases" which generates 3 * (4 +1) databases.

`mariadb.test-database-amount` becomes deprecated in favour of these new
arguments, but compatibility is preserved.

Set cache.direct=on when using native aio (needed by qemu)

/reviewed-on nexedi/slapos!399

We use jstestnode

Using slapos.cookbook:wrappper is easier

was dead code

For now, we are just using this software to install firefox and xvfb,
but the instance is never used, we use jstestnode instead.

replace slapos.recipe:xvfb by a simple wrapper

and stop using xvfb recipe in favour of a simple wrapper

Update the patch not to use $TMPDIR, but to use an environment variable
dedicated for this purpose, because $TMPDIR has a different meaning.

With firefox 60, running `TMPDIR=/same/tmpdir/as/xvfb firefox --marionette` no
longer work and complains:

 Gtk-WARNING **: cannot open display: :0

If we investigate with strace we can see that marionette restarts with a
different $TMPDIR ( to store temporary files in another directory ) and we can see:

 connect(15, {sa_family=AF_UNIX, sun_path=@"/srv/slapgrid/slappart8/tmp/inst/TestRemoteDriver-0/tmp/Temp-0336e727-3485-4774-a816-75fda1baca12/.X11-unix"}, 113) = -1 EINVAL (Invalid argument)
 write(2, "\n(/srv/slapgrid/slappart8/tmp/soft/c5be5b0096cd286c70df5156590cf4f6/parts/firefox-60/firefox:29812): Gtk-\33[1;33mWARNING\33[0m **: cannot open display: :0\n", 152) = 152

To prevent this kind of problems, instead of using TMPDIR, use
a new dedicated environment variable XORG_LOCK_DIR.

No backward compatibility is provided, the profiles will be updated. In the
worst cases, non updated profiles will use /tmp/ and will work unless $DISPLAY
conflicts.

Instead of relying on slapos.cookbook:certificate_authority recipe, which
stops buildout processing, extract the minimal implementation to runtime
key/certificate validator and reject slaves, which does not pass this test.

This commits results in TODO item being done.

As slave requester is able to enter any string in server-alias validate it
against being correct domain name and in case if validation fails reject that
slave.

Also use a trick to have access to global slave state, see
https://fabianlee.org/2016/10/18/saltstack-setting-a-jinja2-variable-from-an-inner-block-scope/

Install validators dependency, which is a way to easily check if email is an
email or domain is correct.

As slave requester is able to enter any string in custom domain validate it
against being correct domain name and in case if validation fails reject that
slave.

Create caddyprofiledeps egg with dummy noop recipe.

Thanks to setting dependencies of this egg and enabling it on the instance
profile, buildout will install eggs during software run and activate them
during instance run.

No existing egg (like slapos.cookbook) is used, as this technique is to allow
profile/software release developer to choose required eggs used during
instantiation.

Another apporach would be to add dependency for validators in
slapos.recipe.template (in install_requires).

Thank to escaping whole command-line it is possible to process buildout
dangerous strings, like ${section:option}, pass them to the wrapper,
instead of killing the whole profile processing.

If the value does not contain "," drop its contents instead of stopping
processing of the whole profile.

Instead of needlessly storing information in configuration section, pass it
via jinja2 parameter. This is safe, in case if extra_slave_instance_list
would contain value like ${section:option}.