This essentially aims to backport

nexedi/slapos!1435 and
nexedi/slapos!1504

to old slapos tag 1.0.249.

It also adds some more patches (that hopefully don't affect the
SR very much) to reduce merge conflicts.

Some patches are deliberately skipped:

1. nexedi/slapos@5b3fc1f2

We don't need this Zope backport fix, because this fix
aims to fix a patch that isn't even present in the Zope
we are using. See here for more context:

https://github.com/zopefoundation/Zope/issues/1191

2. nexedi/slapos@d49914a6

This seems to be a complex change that we may not need for
our purposes.

---

Patches added:

33089cc2 software/erp5/test: Run tests with ZEO & NEO
81abed4e ERP5: rework frontend instance parameter
9abbfa08 software/erp5: describe the allowed formats for family and frontend names
2fe6f860 stack/erp5: remove not used "backend-path"
27a04553 stack/erp5: use slapos.recipe.build to manage haproxy parameters
cdadb774 stack/erp5: implement Zope's rewrite rules in ERP5 balancer partition

The strategy for compatibility is that:
 - haproxy still listen on the same port as before, without rewrite rule.
   This is called "legacy" port.
 - for each frontend from request parameters, we introduce an haproxy
   frontend with a rewrite for the corresponding `internal-path`
   parameter.
 - the shared frontend instance is updated to use this new frontend
   entry from haproxy. This will cause a small downtime until the shared
   frontend is updated to the new URL on ERP5, but since this feature
   was not used, it's OK.

Technical details are that we:
 - split haproxy config to have frontends and backends.
 - introduce one frontend in haproxy for each frontend from request
   parameters.
 - routing-rule-list argument is still honored the same way, globally
   and after path from frontend.
 - change the shared frontend requests to use "" type, no longer "zope"
   type.
 - we don't do automatic detection of /VirtualHostRoot in URL but always
   add it, because it could be used to trick zope into thinking it
   serves requests for an arbitrary host and do open redirects
 - before using the request's host header in virtualhost path, we check
   that it does not contain /, to prevent injection of virutalhost path
   elements through the host header.
 - we don't use the "path" parameter from shared frontend, because we
   want the frontend to be simple, so we don't want it to rewrite the
   request path (which is also the reason why we deprecated "zope" type)
 - the tests have changed a lot, because they were using what's now the
   "legacy" URL types, so we updated it to use the new URL types with
   all the /VirtualHostRoot/../ in path and also because they use IPv6
   URL, no longer IPv4

and save the already allocated ports in a state file, so that requesting
new families does not change already allocated ports.

This is not documented in schema and has no effect in erp5 (but this is
still used for slapos-master)

This change the format or the (mostly) unused frontend parameter to
support requesting more than one frontend and also enable the request of
a frontend by default, so that requesting a frontend separately is no
longer needed.

The `frontend` parameter now also supports requesting frontends for
specific paths on the ERP5 backend, the example below requests a
frontend serving directly a web site, with the necessary rewrite rules:

```js
{
  "frontend": {
    "default": {
      "internal-path": "/erp5/web_site_module/renderjs_runner/"
    }
  }
}
```

The example below requests a default frontend to the erp5 root, to
access the ZMI or erp5_xhtml_style interface and two web sites:

```js
{
  "frontend": {
    "default": {},
    "erp5js": {
      "internal-path": "/erp5/web_site_module/renderjs_runner/"
    },
    "crm": {
      "internal-path": "/erp5/web_site_module/erp5_officejs_support_request_ui/"
    }
  }
}
```

The example below has an explicit definition of the zope families using
`zope-partition-dict` parameter, because there is more than one zope
family, no frontend is requested by default:

```js
{
  "zope-partition-dict": {
    "backoffice": {
      "family": "backoffice"
    },
    "web": {
      "family": "web"
    },
    "activities": {
      "family": "activities"
    }
  }
}
```

Continuing this example, to have frontends for backoffice and web
families, the frontend request can specify the families, like it is
demonstrated in the example below. In this example, we don't specify an
entry for "activities" family, so no frontend will be requested for
this family.

```js
{
  "frontend": {
    "backoffice": {
      "zope-family": "backoffice"
    },
    "web": {
      "zope-family": "web",
      "internal-path": "/erp5/web_site_module/web_site/"
    }
  }
  "zope-partition-dict": {
    "backoffice": {
      "family": "backoffice"
    },
    "web": {
      "family": "web"
    },
    "activities": {
      "family": "activities"
    }
  }
}
```

We configure haproxy with "verify optional", which makes haproxy request
a client certificate, but accept the case where client does not present
a certificate, but as described in [1], if client present a certificate
and this certificate can not be verified, handshake is aborted. This is
not what we want, we want to treat the case of a non verified
certificate same as the case of the absence of certificate.

This configures haproxy accordingly, using "crt-ignore-err all" to allow
handshake anyway.

Once this was fixed, there was a remaining problem with
client_cert_verified acl, haproxy acl are OR, but this rule was supposed
to be a AND (client present a certificate AND it is verified), this was
rewritten to use inline condition which are AND.

[1]: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.1-verify

Also adjust test_x_forwarded_for_stripped_when_no_certificate to assert
that there is no X-Forwarded-For header at all when no client
certificate.

Before this patch all ERP5 SlapOS Integration tests only run with ZEO
storage. We should also run them against NEO, because we are using
ERP5 with NEO in SlapOS.

In order to do so we implemented parameterized test classes for our ERP5
integration tests. Each test case can be configured via its __test_matrix__
attribute. A test matrix is a dict which maps the flavoured class name suffix to
a tuple of parameters. A parameter is a function which receives the
instance_parameter_dict and modifies it in place. You can use the
'matrix' helper function to construct a test matrix. If .__test_matrix__
is 'None' the test case is ignored.

/reviewed-by @kirr & @jerome
/reviewed-on nexedi/slapos!1306

This reverts commit 054386d6.

It proven to be insufficient to fix issues with PYTHON make variable
surcharging from top make.

also update apr and apt-util.

This was added in 48bcc52a (software/theia: some "better" defaults,
2019-08-30) because the default keybinding to create another cursor is
alt-click, but on ChromeOS alt-click does a right click, but:
 - not everybody use ChromeOS
 - this breaks the ctrl+click shortcut to go to definition/follow link
 - it makes SlapOS' Theia behaves differently from default Theia or vscode
 - ChromeOS users can use chrome://flags/#use-search-click-for-right-click
  (I'm using this flag and this seems perfect)

See merge request nexedi/slapos!1193

Since c53d0f43 (software/erp5: Remove create-erp5-site software-type
from json schema, 2018-01-08) instance is always created.

The default family was also processing activities, so it happened
sometimes that the zope does not respond in time because it's processing
activities. Create a dedicated activity family so that the default zope
is available to serve requests.

This makes slapos node instance slower, especially when there are many
kvm partitions on the node.

We don't really need to sleep because if qemu is not ready, promise will
fail and succeed on next run.

 - enable publisher-timeout by default, after 300s like haproxy used to do
 - make haproxy use similar timeout as publisher-timeout
 - small fixes to json schemas

See merge request nexedi/slapos!1192

When parameters are built by instance-erp5, we don't need to use .get,
because the key is supposed to be here - and if it's not here it would
be because of a bug so it's better to let the error propagate.

Since https://github.com/mroonga/mroonga/commit/35f593037c2d8f5a497349ff98f3a3ce31ee95de
install.sql has been split in install.sql and update.sql, adjust the
script to source both.

install.sql only contains INSTALL PLUGIN, it's safe to run it on every
update (that's what we always did).

This fixes a regression that mroonga user functions were no longer
available after d0589a41 (version up: MariaDB 10.3.35/10.4.25, groonga
12.0.4, mroonga 12.04, groonga-normalizer-mysql 1.1.8, 2022-06-10)

Instead of having an hardcoded timeout that users will hit anyway even
if they increase publisher timeout, set this timeout value to be
slightly higher than publisher timeout. This way publisher-timeout can
be used to allow longer requests and it's generally more consistent.

We already had haproxy configured to stop serving requests longer than
this duration, but the requests was still being processed by zope.

By adding such default in publisher-timeout, zope will stop processing
the request after the timeout.

This is slightly different behavior, because before this change the
request was still being processed, even if the client did not get the
response it may make change to databases.
Users who want to keep the previous behavior can request with
publisher-timeout parameter null.

When objects are not supposed to accept extra properties, setting
additionalProperties to false allows schema validator to detect typos
in key names, or usage obsolete keys.

See merge request nexedi/slapos!1190

Thanks to migration to json-in-xml serialisation some lower level bugs has
been discovered with value types in the profiles, so simply fix it.

default and kvm-cluster software types are now json-in-xml, so they accept
and return json-in-xml.

Update tests to show that xml and json-in-xml serialization works.

Note: There was invective to publish with pure XML serialisation, but it is
wrong idea because:

 1) serialisation in the software.cfg.json is provided for both ways
 2) requesting instances with given serialisation by profiles expect same
    serialisation

Request resilient instances with serialised, as they reply with json-in-xml.