See merge request nexedi/slapos!1500

Directly expose all passlib.hash supported hashes, using a `passwd-`
prefix. For example, to access `sha256_crypt`, use `passwd-sha256-crypt`
option name.

  [secret]
  recipe = slapos.cookbook:generate.password

  [config-file]
  hashed-password = ${secret:passwd-sha256-crypt}

This changes the format of storage-path, it used to be the password in
plain text, it is now a mapping also containing hashed passwords, to
have the same hashed passwords for each buildout run.

This needs collaboration from publish_early recipe, because .pop(k) does
raised a KeyError with the dict.__missing__  approach.

In MultiRU there will be only one ENB which supports all TDD, FDD, LTE and NR
cells and different types of radio units - all at the same time. This patch is
preparatory step for that: it merges gnb configuration template into enb
configuration template, so that enb.jinja2.cfg now serves both enb and gnb
instances.

In this patch for now we only move code from gnb.jinja2.cfg without changing
it(*) and wrap parts with `if do_lte` and `if do_nr` correspondingly.

The end result of rendered enb.cfg and gnb.cfg stays the same modulo space
changes and added innoccent `#define TDD` as Appendix shows.

(*) the only exception is set up of gtp_addr which has practically the same
code, was wrapped with `if mme_list` in enb and `if amf_list` in gnb, and is now
wrapped with `if mme_list or amf_list`.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on !1512

--------

Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch.

```
$ ./pythonwitheggs slapos-render-config.py && (git diff -w --no-index config/enb.cfg.old config/enb.cfg ; git diff -w --no-index config/gnb.cfg.old config/gnb.cfg)
```

```diff
diff --git a/config/enb.cfg.old b/config/enb.cfg
index fdf3ab24d..cb46697ea 100644
--- a/config/enb.cfg.old
+++ b/config/enb.cfg
@@ -1,11 +1,15 @@

+
 #define TDD                 1

 #define N_RB_DL             50
+
 #define N_ANTENNA_DL        2
+
 #define N_ANTENNA_UL        2

+
 {

   log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rota>
@@ -24,7 +28,6 @@
   rx_gain: 43,

   com_addr: "127.0.1.2:9001",
-
   mme_list: [

     {
@@ -33,9 +36,9 @@

   ],

+
   gtp_addr: "127.0.1.1",

-
   enb_id: 0x1A2D0,

   cell_list: [{
@@ -49,7 +52,6 @@
       ],
     }
   ],
-
   cell_default: {
     plmn_list: [
       "00101",
@@ -226,4 +228,6 @@
   meas_gap_config: "gp0",
   ho_from_meas: true,
   },
+
+
 }
\ No newline at end of file
diff --git a/config/gnb.cfg.old b/config/gnb.cfg
index e3d671e09..4e47a2094 100644
--- a/config/gnb.cfg.old
+++ b/config/gnb.cfg
@@ -1,15 +1,21 @@

+
+#define TDD                 1
+
+
 #define N_ANTENNA_DL        2

 #define N_ANTENNA_UL        2

+
 {

   log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rota>

   log_filename: "log/gnb.log",

+
   rf_driver: {
       name: "sdr",
       args: "dev0=/dev/sdr0",
@@ -30,15 +36,17 @@
   ],

-
-
   gtp_addr: "127.0.1.1",

   gnb_id_bits: 28,
   gnb_id: 0x12345,
   en_dc_support: true,
-  cell_list: [],
+
+  cell_list: [
+  ],
+
+
   nr_cell_list: [
   {
     rf_port: 0,
```

It stopped to be used after 49ce8ef5 (software/ors-amarisoft: Provide dedicated
TAP interface for each Radio Unit).

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1512

Because

1) those services are needed and used only by ru/ promises like check_cpri_lock
   and check_rx_saturated.
2) in general we will need to initialize and setup radio units not only in eNB -
   for example UEsim will use the same code library to initialize radio units.
   Thus the proper place to keep everything required for RU to be operational have to
   be located inside ru/ and activated by that radio-units library.

Push corresponding code from instance-enb to ru/ and do only minor adjustments
to instance-gnb trying not to break it, since gnb does not currently use rulib,
and because in the future gnb will be replaced by enb which will be serving
both lte and nr cells in the same service.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1511

Because:

- ssh server is needed for and used by ru/lopcomm/ only
- in general we will need to initialize and setup radio units not only in eNB -
  for example UEsim will use the same code library to initialize radio units.
  Thus the proper place to keep everything required for RU to be operational have to
  be located inside ru/ and activated by that radio-units library.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1510

Dnsmasq insists on dhcp-range's prefixlen to be at most 64, which
triggers the following error if original slaptap is wider than that:

    dnsmasq: prefix length must be at least 64 at line 5 of /srv/slapgrid/slappart6/etc/dnsmasq.cfg

-> Fix it by capping provided range to /64

/cc @tomo, @xavier_thompson, @Daetalus
/reported-by @lu.xu
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1509

our new nexedi.org currently no longer serve static pages with links,
using absolute links should allow us to have test passing until this is
fixed.

also expose MB_UNAGGREGATED_QUERY_ROW_LIMIT and MB_AGGREGATED_QUERY_ROW_LIMIT

These partition references should be kept short, they are a mechanism
to use a short path for unix sockets, because unix socket paths can not
exceed 108 characters.

When running the test in theia, this was causing errors:

   # [ALERT]    (100453) : config : parsing [/srv/slapgrid/slappart15/srv/runner/instance/slappart7/tmp/inst/with-max-rlimit-nofile6/etc/haproxy.cfg:37] : log : socket path '/srv/slapgrid/slappart15/srv/runner/instance/slappart7/tmp/inst/with-max-rlimit-nofile6/var/run/log.sock' too long (max 97

Currently, due to ensure_ascii=True default of json.dumps, we are
insisting on our JSON schemas to be ascii-only and all other characters
to be represented by \uxxxx escapes. So far this was not problematic as
all our schemas contains only ASCII characters, but upcoming
ors-amarisoft changes want to use e.g. "→" symbol:

    https://lab.nexedi.com/kirr/slapos/blob/b51f5523/software/ors-amarisoft/software.cfg.json#L15

which currently results in failure of json-schema test:

    FAIL: test_ors-amarisoft_software_cfg_json_format (slapos.test.test_json_schema.TestJSONSchemaValidation)
    ...

    First differing element 14:
    '      "title": "\\u2192  eNB/gNB | Radio Unit",'
    '      "title": "→  eNB/gNB | Radio Unit",'

And in general, in 2023 I think there is no reason to insist on our schemas to
be ASCII-only: say if one wants to describe something about "α" parameter. It
would be good to use that α character directly and seeing it in the editor,
instead of using escapes all the time.

As indicated by below stackoverflow answer "JSON spec requires UTF-8 support by
decoders": https://stackoverflow.com/a/594881/9456786 , and indeed checking
JSON specification also confirms that by default JSON decoders shall use UTF-8:

    https://datatracker.ietf.org/doc/html/rfc7159#section-8.1

This way, I think, we can switch to UTF-8 safely.

/reviewed-by @jerome, @lu.xu
/reviewed-on nexedi/slapos!1498

See merge request nexedi/slapos!1499

/reviewed-by @kirr and @jerome
/reviewed-on nexedi/slapos!1494

wendelin.core 2.0.alpha3.post9 supports golang 1.21: with this new
version we can set golang 1.21 as the new default.

/reviewed-by @kirr and @jerome
/reviewed-on nexedi/slapos!1494

Go1.21 already has the fifth minor revision (released 2023-12-05) and
should therefore already be sufficiently stable. Furthermore we need it
to fix a bug in a NEO/go dependency [1].

Please find all details in the official release note:

    https://go.dev/doc/go1.21

It was released on 2023-08-08 [2]. Due to the go promise of compatibility
most software should still compile without any problems.

In golang < 1.21 we needed to patch golang to fix
https://github.com/golang/go/issues/42525. In golang 1.21 we still need
to apply a fix, but can't apply the old patch because the code changed.
In golang > 1.21 this problem is already fixed with https://go-review.googlesource.com/c/go/+/520055.

Because of https://github.com/golang/go/commit/092671423cd95eaa6df93eb29442fef41504d097
'TestUnshareMountNameSpace' fails on golang 1.21 [3]. In golang 1.20 this test
was skipped [4]. To fix this failure the additional patch
'skip-unshare-mount-test.patch' has been added.

---

[1] nexedi/wendelin.core!22 (comment 195769)]
[2] https://go.dev/doc/devel/release
[3] --- FAIL: TestUnshareMountNameSpace (0.18s)
    exec_linux_test.go:243: unshare failed: exit status 2
        unshare: mount /tmp/TestUnshareMountNameSpace2210137852/001 failed: 0x1
[4] === RUN TestUnshareMountNameSpace exec_linux_test.go:333: kernel prohibits unshare in unprivileged process, unless using user namespace — SKIP: TestUnshareMountNameSpace (0.00s)

/reviewed-by @kirr and @jerome
/reviewed-on nexedi/slapos!1494

This version is compatible with gcc 12

using https://github.com/ilevkivskyi/com2ann

drop versions not running on debian 12 and add new versions

This was disabled to "keep compatibility with current font selection
problems", but I don't think we need to care about compatibility for
this. This was anyway causing a problem that system fontconfig will be
used if it's available, so it's better to define things explictly to
prevent falling back to system configuration.

This caused cloudooo to select different fonts, because LibreOffice-bin
comes with some Noto fonts and fontconfig now prefers Noto font

Testing conversion of HTML with font-family:"FontFamily FontVariant"
CSS rule does not really make sense. Remove a few cases for which the
behavior is not stable.

Using RegExp to validate hostnames is a bad practice, and has a lot of reasons to be wrong.
On top of that, the JSON Schema specification allows, since draft 7, to validate hostnames
against an IDN hostname, by using the `idn-hostname` format.

With these changes, IDN are now supported (.рф and .中國 for instance), and long TLD
should not be a problem anymore.

With backported fixes so that old software supports debian 12

In 9636d79c (Nextcloud Upgrade fixes, 2023-12-13) parameter changed:
 - instance.trusted-domain-* parameters are replaced by instance.trusted-domain-list
 - the frontend is part of trusted domain, so with "bypassed" frontends
from slapos proxy the backend appears twice (because the backend URL is
returned as frontend URL)