This reverts commit 6c399134.

 - use caucase for balancer certificate
 - move virtual host logic on the backend
 - change "frontend" parameter to request "" type (and no longer "zope")

See merge request nexedi/slapos!1504

The strategy for compatibility is that:
 - haproxy still listen on the same port as before, without rewrite rule.
   This is called "legacy" port.
 - for each frontend from request parameters, we introduce an haproxy
   frontend with a rewrite for the corresponding `internal-path`
   parameter.
 - the shared frontend instance is updated to use this new frontend
   entry from haproxy. This will cause a small downtime until the shared
   frontend is updated to the new URL on ERP5, but since this feature
   was not used, it's OK.

Technical details are that we:
 - split haproxy config to have frontends and backends.
 - introduce one frontend in haproxy for each frontend from request
   parameters.
 - routing-rule-list argument is still honored the same way, globally
   and after path from frontend.
 - change the shared frontend requests to use "" type, no longer "zope"
   type.
 - we don't do automatic detection of /VirtualHostRoot in URL but always
   add it, because it could be used to trick zope into thinking it
   serves requests for an arbitrary host and do open redirects
 - before using the request's host header in virtualhost path, we check
   that it does not contain /, to prevent injection of virutalhost path
   elements through the host header.
 - we don't use the "path" parameter from shared frontend, because we
   want the frontend to be simple, so we don't want it to rewrite the
   request path (which is also the reason why we deprecated "zope" type)
 - the tests have changed a lot, because they were using what's now the
   "legacy" URL types, so we updated it to use the new URL types with
   all the /VirtualHostRoot/../ in path and also because they use IPv6
   URL, no longer IPv4

and save the already allocated ports in a state file, so that requesting
new families does not change already allocated ports.

This reverts commit 620c9332 (stack/erp5: stop using caucase managed
certificate for balancer, 2020-11-10) with an updated design. We add a
caucase service for balancer in the balancer partition. The caucase
service from the root partition (that was not used) is removed.

The underlying idea is that the default configuration should use multiple
caucases with limited scope, here we have one caucase to manage the
certificate used by haproxy server in the balancer partition, so we put
one caucase to manage this certificate and the caucase is configured to
auto-accept one certificate only. The plan is that when we will add a
certificate for mariadb server, we'll add another caucase inside this
mariadb server.

For more advanced usage and also to support the cases where a new
certificate needs to be re-emitted for some reason, users can request
with an existing caucase URL. In that case, they will have to accept
the certificate requests.

Notable changes:

balancer/ssl/caucase-url is no longer documented in parameters, this is
an internal parameter, users can pass one global caucase service to
manage all partition

CAUCASE environment variable is no longer set when running zope. There
was no identified use case and with this new approach of multiple
caucases, the term "caucase" alone became ambiguous.

This is not documented in schema and has no effect in erp5 (but this is
still used for slapos-master)

This change the format or the (mostly) unused frontend parameter to
support requesting more than one frontend and also enable the request of
a frontend by default, so that requesting a frontend separately is no
longer needed.

The `frontend` parameter now also supports requesting frontends for
specific paths on the ERP5 backend, the example below requests a
frontend serving directly a web site, with the necessary rewrite rules:

```js
{
  "frontend": {
    "default": {
      "internal-path": "/erp5/web_site_module/renderjs_runner/"
    }
  }
}
```

The example below requests a default frontend to the erp5 root, to
access the ZMI or erp5_xhtml_style interface and two web sites:

```js
{
  "frontend": {
    "default": {},
    "erp5js": {
      "internal-path": "/erp5/web_site_module/renderjs_runner/"
    },
    "crm": {
      "internal-path": "/erp5/web_site_module/erp5_officejs_support_request_ui/"
    }
  }
}
```

The example below has an explicit definition of the zope families using
`zope-partition-dict` parameter, because there is more than one zope
family, no frontend is requested by default:

```js
{
  "zope-partition-dict": {
    "backoffice": {
      "family": "backoffice"
    },
    "web": {
      "family": "web"
    },
    "activities": {
      "family": "activities"
    }
  }
}
```

Continuing this example, to have frontends for backoffice and web
families, the frontend request can specify the families, like it is
demonstrated in the example below. In this example, we don't specify an
entry for "activities" family, so no frontend will be requested for
this family.

```js
{
  "frontend": {
    "backoffice": {
      "zope-family": "backoffice"
    },
    "web": {
      "zope-family": "web",
      "internal-path": "/erp5/web_site_module/web_site/"
    }
  }
  "zope-partition-dict": {
    "backoffice": {
      "family": "backoffice"
    },
    "web": {
      "family": "web"
    },
    "activities": {
      "family": "activities"
    }
  }
}
```

Going 0.140 -> 0.142 introduces the following changes:

    nexedi/slapos.toolbox@0.140...0.142

Test results: https://erp5js.nexedi.net/#/test_result_module/20240215-D4F3CE96 (all ok)

(irrelevant for the coupler application).

(cherry picked from commit d1ec306a575f4dd0702aec64873794b4f822cfee)

See merge request nexedi/slapos!1531

See merge request nexedi/slapos!1529

These are already defined in stack/slapos.cfg

jupyter-py2 used an old version not py3 compatible
pygolang/test used the same version as in stack/slapos.cfg

Don't repeat slapparameter_dict['ncell_list'][k] - we can introduce a name for
current neighbour cell, and use that name in the loop. And align entries to
that they read more clearly.

No non-whitespace changes in rendered enb.cfg and gnb.cfg .

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
/proposed-for-review-on nexedi/slapos!1526
/reviewed-by TrustMe

We currently have LTE-specific handover configuration in under cell_list and
NR-specific handover configuration in under nr_cell_list. Those configuration
are different.

However in upcoming MultiRU we will need to handle LTE->NR handover, NR->LTE
handover, and also Intra-ENB handover in addition to Inter-ENB handover we
currently do.

-> Move handover code into common function as a preparatory step for that.

In the future the handover code for LTE and NR cells will be the same, so it
makes sense to move that code to common place to avoid duplication.

For rendered enb.cfg this unification introduces only space and trivial
changes as shown in the appendix.

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
/proposed-for-review-on nexedi/slapos!1528
/reviewed-by TrustMe

Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch

```
$ git diff -w --no-index config/{old,out}
```

```diff
diff --git a/config/old/enb.cfg b/config/out/enb.cfg
index 43301ee13..9dcca16c7 100644
--- a/config/old/enb.cfg
+++ b/config/out/enb.cfg
@@ -45,16 +45,18 @@
       root_sequence_index: 204,
       dl_earfcn: 36100,
       inactivity_timer: 10000,
+
       // Handover
       ncell_list: [
         // Inter-ENB HO
         {
+          rat: "eutra",
           n_id_cell: 35,
           dl_earfcn: 700,
           cell_id: 0x12345,
           tac: 123,
-        }],
-
+        },
+      ],

       // Carrier Aggregation
       scell_list: [
diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
index 2127a2f6b..23b07d6e1 100644
--- a/config/old/gnb.cfg
+++ b/config/out/gnb.cfg
@@ -57,6 +57,7 @@
     ssb_pos_bitmap: "10000000",

     inactivity_timer: 10000,
+
       // Handover
       ncell_list: [
         // Inter-ENB HO
@@ -74,8 +75,8 @@
           ssb_period: 20,
           ssb_offset: 0,
           ssb_duration: 1,
-      }],
-
+        },
+      ],

     // tune NR parameters for the cell
     manual_ref_signal_power: true,
```

Useful to have while doing handover-related changes.

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
/proposed-for-review-on !1528
/reviewed-by TrustMe

Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch

```
$ git diff --no-index config/{old,out}
```

```diff
diff --git a/config/old/enb.cfg b/config/out/enb.cfg
index 1843e0f24..43301ee13 100644
--- a/config/old/enb.cfg
+++ b/config/out/enb.cfg
@@ -45,6 +45,16 @@
       root_sequence_index: 204,
       dl_earfcn: 36100,
       inactivity_timer: 10000,
+      // Handover
+      ncell_list: [
+        // Inter-ENB HO
+        {
+          n_id_cell: 35,
+          dl_earfcn: 700,
+          cell_id: 0x12345,
+          tac: 123,
+        }],
+

       // Carrier Aggregation
       scell_list: [
diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
index d76b45d3c..2127a2f6b 100644
--- a/config/old/gnb.cfg
+++ b/config/out/gnb.cfg
@@ -57,6 +57,25 @@
     ssb_pos_bitmap: "10000000",

     inactivity_timer: 10000,
+    // Handover
+    ncell_list: [
+      // Inter-ENB HO
+      {
+        rat: "nr",
+        dl_nr_arfcn: 520000,
+        ssb_nr_arfcn: ,
+        ul_nr_arfcn: 520000,
+        n_id_cell: 75,
+        gnb_id_bits: 22,
+        nr_cell_id: 0x77712,
+        tac: 321,
+        band: 38,
+        ssb_subcarrier_spacing: 30,
+        ssb_period: 20,
+        ssb_offset: 0,
+        ssb_duration: 1,
+      }],
+
```

1. Currently we have separate log_options for LTE and NR cases with listing
different subsystems in each. But in MultiRU one enb will be driving both LTE
and NR cells at the same time, so we will need to define both LTE- and
NR-related levels.

-> Merge all log settings into one log_options as a preparatory step

For current state it does not hurt for an LTE if we set e.g. ngap.level, and it
does not hurt for NR if we set e.g. s1ap.level - since those layers will be
unused. This way merging log settings for both LTE and NR subsystems is ok.

--------

2. Factorize log_phy_debug handling: instead of duplicating whole
log_options line and changing only phy.level settings there, construct the
log_options line programmatically and handle phy.level on its own.

--------

3. Use log/enb.log log_filename for both LTE and NR cases. In the upcoming
MultiRU there might be several cells activated at the same time and in general
it will be not possible to say are we doing "enb" or "gnb" now - for example if
there will be two cells - one LTE and one NR.

-> Use enb.log for log filename uniformly similarly to how the software is
named (lteenb) even though it can work as both enb and gnb.

For the reference we do the same with enb.xlog in nexedi/slapos!1522 .

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
/proposed-for-review-on nexedi/slapos!1527
/reviewed-by TrustMe

Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch

```
$ git diff --no-index config/{old,out}
```

```diff
diff --git a/config/old/enb.cfg b/config/out/enb.cfg
index 467bb6364..1843e0f24 100644
--- a/config/old/enb.cfg
+++ b/config/out/enb.cfg
@@ -3,9 +3,7 @@

 {
-
-  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
-
+  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
   log_filename: "log/enb.log",

diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
index 18523818a..d76b45d3c 100644
--- a/config/old/gnb.cfg
+++ b/config/out/gnb.cfg
@@ -3,10 +3,8 @@

 {
-
-  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
-
-  log_filename: "log/gnb.log",
+  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
+  log_filename: "log/enb.log",

   rf_driver: {
```

Current enb config is already quite complex and with MultiRU it will be growing
more - both with added features and with more sections emitted because there
will be multiple radio units, multiple cells and cross cell interactions. So
for clarity we will want to annotate with a comment to which cell or ru object
a section belongs, or to which cell-cell pair a particular interaction belongs.

Amarisoft supports C-style comments and preprocessor directives out of the box,
but if we use them in the configuration files, yaml.load, that we use in the
test to load generated configs, will break, because // and /* ... */ is not
valid YAML. It looks like Amarisoft does preprocessing as a separate step
before further loading given configuration via yaml.

So to be able to use the comments and still have tests working we need to do
the same - in the tests preprocess the files before feeding them to yaml loader.

-> Do that with the help of https://pypi.org/project/pcpp/

In my view that library has good quality and in my experience it worked
flawlessly. Anyway we need it to only handle comments, not sophisticated CPP
features, and for that it works just ok.

Add some comments to existing enb.cfg and ue.cfg to make sure it really works.
Those are simple comments, and in current state it they might seem as not 100%
necessary, but with more upcoming config changes it would be good to have those
descriptionary anchors present in the generated configs, so I suggest we add them.
Anyway they should not do any harm at all.

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
/proposed-for-review-on nexedi/slapos!1526
/reviewed-by TrustMe

Currently on every test yaml.load issues a warning that using it without
explicitly specifying loader is deprecated, for example:

    test_enb_conf (testTDD.TestENBParameters) .../slapos/software/ors-amarisoft/test/test.py:29: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

-> Fix the warning by using appropriate loader explicitly.

Use FullLoader since msg.pyyaml.org/load describes it as

   Loads the full YAML language. Avoids arbitrary code execution.
   This is currently (PyYAML 5.1+) the default loader called by yaml.load(input) (after issuing the warning).

i.e. we get support for full YAML feature set, do not get code execution and
make sure that the behaviour stays exactly the same as before, but without the
warning.

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
/proposed-for-review-on nexedi/slapos!1526
/reviewed-by TrustMe

We are going to do fixes and enhancements to YAML loading processes. Move
corresponding code to one place before that as the preparatory step.

Place new utility in new test/test.py file - with MultiRU that will be the
place for tests of generic mode and test.jinja2.py, which will be renamed to
test_ors.py, will be using utilities from there. So place the utility into new
place from the start.

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
/proposed-for-review-on nexedi/slapos!1526
/reviewed-by TrustMe