- remove no longer used ru/lopcomm/supervision.jinja2.py
- restore removed `+ keep on touching RU watchdog` comment.
  Now it relates to ru/lopcomm/stats which became responsible for this function.
- move code of RU-netconf-connection promise to be located nearby stats service
  because this promise interacts with that service, and grouping code by
  interaction makes it easier to understand for humans. Previously
  RU-netconf-connection was interacting with supervision service whcsh is gone
  after the patch in question.
- use double vertical space to delimit logical sections from each other.

open questions:

- before the patch in question stats code was reconnecting after each error
  with 10 seconds pause. Now it logs it wants to reconnect but does _not_
  because top-level `while True` was removed. Was that intended?

  If it was not the intent then new stats code will fail after any single error
  and won't reconnect to RU by itself.

/reviewed-by @lu.xu
/reviewed-on nexedi/slapos!1483

Similarly to sdr-busy, if we have several radio units, we want each of them to
have its own rx-saturated promise.

See nexedi/slapos.toolbox!126 for
corresponding part in slapos.toolbox that makes check_rx_saturated per-RU
configurable.

Not touching gnb.jinja2.cfg as currently gnb does not support multicell at all,
and in the future it will be sole enb.cfg to handle both LTE and NR
simultaneously.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1485

If we have several radio units, we want each of them to have its own sdr-busy promise.

See nexedi/slapos.toolbox!125 for
corresponding part in slapos.toolbox that makes check_sdr_busy per-RU
configurable.

Not touching gnb.jinja2.cfg as currently gnb does not support multicell at all,
and in the future it will be sole enb.cfg to handle both LTE and NR
simultaneously.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1485

This promise applies to any CPRI-based Radio Unit not only to Lopcomm RU.

-> Move its code to generic part, so that the promise is generated for any CPRI-based RU.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1485

Because for-RU DHCP-server should be provided to all CPRI-based radio unit, not
only to Lopcomm.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1485

Also cleanup duplicated section

instead of requiring users to explicitly activate it via additional `dnsmasq` parameter.

The reason here is simple: CPRI-based radio units, for example Lopcomm ORAN RU,
need to be configured to go to radio, and for that configuration to work we
need to be able to talk via TCP/IP to the RU.

/cc @jhuge, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on nexedi/slapos!1479

We already pushed dnsmasq part, that serves radio units into enb in 9f2b9db5.
However we need to push those bits further to ru/ for the following reasons:

- DHCP server is used to provide Radio Units with IP address only.
- without IP address assigned those RUs cannot be initialized and do not go to radio at all.
- in general we will need to initialize and setup radio units not only in eNB -
  for example UEsim will use the same code library to initialize radio units.
  Thus the proper place to keep everything required for RU to be operational have to
  be located inside ru/ and activated by that radio-units library.

/cc @jhuge, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on nexedi/slapos!1479

The code had check-rx-saturated under `if not ru == "lopcomm"`, but checking RX
signal for saturation is implemented on eNB side, not by RU itself, and is
RU-independent because eNB only receives raw samples from RU and decides on its
own whether those samples are higher than saturation margin or not.

As an extra proof that checking for RX saturation is meaningful for Lopcomm too here is how samples.rx data can look like in the case of 2 Lopcomm Radio Units:

    root@callbox-005:/srv/slapgrid/slappart16/var/log# cat amarisoft-stats.json.log |jq .data.samples.rx
    [
      {
        "rms": -382.3080749511719,
        "max": -382.3080749511719,
        "sat": 0,
        "count": 1221181440,
        "rms_dbm": -382.3080749511719
      },
      {
        "rms": -382.3080749511719,
        "max": -382.3080749511719,
        "sat": 0,
        "count": 1221179392,
        "rms_dbm": -382.3080749511719
      }
    ]
    ...

i.e. all the data, that check-rx-saturated promise works on, is there.

/cc @jhuge, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on nexedi/slapos!1478

On Fedora, the flag -lm is not correctly added so add it manually.

Those warnings are breaking compilation on Fedora:

[ 1280s] libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -I. -I. -I.. -I.. -Iglib -DIN_LIBTEXTSTYLE -DLIBXML_STATIC -I./libcroco -DDEPENDS_ON_LIBICONV=1 -I/home/abuild/rpmbuild/BUILD/sn_1.10.5+1.0.346+1/slapos/build/opt/slapos/parts/libxml2/include -I/home/abuild/rpmbuild/BUILD/sn_1.10.5+1.0.346+1/slapos/build/opt/slapos/parts/zlib/include -I/home/abuild/rpmbuild/BUILD/sn_1.10.5+1.0.346+1/slapos/build/opt/slapos/parts/ncurses/include -fanalyzer -Wall -Warith-conversion -Wcast-align=strict -Wdate-time -Wdisabled-optimization -Wdouble-promotion -Wduplicated-branches -Wduplicated-cond -Wextra -Wformat-signedness -Winit-self -Winvalid-pch -Wlogical-op -Wmissing-include-dirs -Wnested-externs -Wnull-dereference -Wopenmp-simd -Woverlength-strings -Wpacked -Wpointer-arith -Wstack-protector -Wstrict-overflow -Wsuggest-attribute=format -Wsuggest-final-methods -Wsuggest-final-types -Wsync-nand -Wsystem-headers -Wtrampolines -Wuninitialized -Wunknown-pragmas -Wunsafe-loop-optimizations -Wvariadic-macros -Wvector-operation-performance -Wwrite-strings -Warray-bounds=2 -Wattribute-alias=2 -Wbidi-chars=any,ucn -Wformat-overflow=2 -Wformat=2 -Wformat-truncation=2 -Wimplicit-fallthrough=5 -Wshift-overflow=2 -Wuse-after-free=3 -Wunused-const-variable=2 -Wvla-larger-than=4031 -Wno-analyzer-malloc-leak -Wno-empty-body -Wno-analyzer-double-fclose -Wno-analyzer-double-free -Wno-analyzer-null-argument -Wno-analyzer-null-dereference -Wno-analyzer-use-of-uninitialized-value -Wno-cast-align -Wno-format-nonliteral -Wno-sign-compare -Wno-type-limits -Wno-unused-parameter -Wno-clobbered -Wshadow=local -Wno-cast-function-type -Wno-duplicated-cond -Wno-format -Wno-implicit-fallthrough -Wno-null-dereference -Wno-suggest-attribute=format -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -c libcroco/cr-additional-sel.c  -fPIC -DPIC -o libcroco/.libs/rpl_la-cr-additional-sel.o
[ 1280s] cc1: error: '-Wformat-security' ignored without '-Wformat' [-Werror=format-security]
[ 1280s] cc1: some warnings being treated as errors

Some fixes for gcc12 and for ldd problems when some dev packages are installed

See merge request nexedi/slapos!1482

to prevent using system package when ncurses-dev is installed

to prevent from using system packages when libcurses-dev is installed

This uses rbnacl-sodium which bundle an old version of libsodium that
can not be compiled with gcc 12. This is an optional dependency to
support ed25519 ssh keys, but we the slapos version of gitlab does not
use ssh, only https, so we don't actually need to install this.

To prevent linking with system libtirpc if the corresponding system
-dev package is installed

We were using --without-ncurses but this does not prevent the build
from using system package when ncurses-dev system package is installed.

if only bison exists in PATH, we will get an error:
/path/to/bison: m4 subprocess failed: No such file or directory

also copy libperfschema.a in lib directory to build mroonga with MariaDB >= 10.5.

This was building against system library when ncurses-dev system
package is installed

This concerns a case of resiliency in Theia: when a resilient Theia
requests multiple import clones (by default only one is requested),
e.g. when theia0 (export) has two import clones theia1 and theia2.

Each clone requests a frontend for their theia service, which allows
the Theia editor of the clone to be accessed without doing a takeover.

Before this, all the import clones would request a frontend with the
same name. This meant that only one frontend was allocated and might
randomly redirect to one of the backup clones.

See merge request nexedi/slapos!1481

accidentally removed
[slaplte.jinja2]
_update_hash_filename_ = slaplte.jinja2
md5sum =
from buildout.hash.cfg

The Netconf subscriptions were added twice, causing the RU to reboot
because there was no supervision check on the redundant subscription.

The Ncclient does not have a timeout for taking notifications, causing it to wait indefinitely. This can result in the connection being lost while the waiting process is still ongoing.

software/ors-amarisoft: Start of slaplte package by way of fixing standalone build of enb.cfg and gnb.cfg

Please see individual patches for details.

/cc @lu.xu, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1480

After cell_list problem was fixed in the previous patch, now building enb.cfg
in standalone mode fails on mme_addr:

    slapuser35@vifibcloud-rapidspace-hosting-018:~/srv/project/slapos/software/ors-amarisoft$ ./pythonwitheggs slapos-render-config.py
    Traceback (most recent call last):
      File "/srv/slapgrid/slappart35/srv/project/slapos/software/ors-amarisoft/./pythonwitheggs", line 47, in <module>
        exec(compile(__file__f.read(), __file__, "exec"))
      File "slapos-render-config.py", line 92, in <module>
        do('enb', {"tdd_ul_dl_config": "[Configuration 6] 5ms 5UL 3DL (maximum uplink)"})
      File "slapos-render-config.py", line 90, in do
        j2render(cfg, json_params % locals())
      File "slapos-render-config.py", line 38, in j2render
        f.write(r._render().decode())
      File "/srv/slapgrid/slappart35/srv/runner/software/0b402c7d1e7c38a4324c836766690bb6/eggs/slapos.recipe.template-5.1-py3.9.egg/slapos/recipe/template/jinja2_template.py", line 215, in _render
        return template_object.render(**self.context).encode(self.encoding)
      File "/srv/slapgrid/slappart35/srv/runner/software/0b402c7d1e7c38a4324c836766690bb6/eggs/Jinja2-2.11.3-py3.9.egg/jinja2/environment.py", line 1090, in render
        self.environment.handle_exception()
      File "/srv/slapgrid/slappart35/srv/runner/software/0b402c7d1e7c38a4324c836766690bb6/eggs/Jinja2-2.11.3-py3.9.egg/jinja2/environment.py", line 832, in handle_exception
        reraise(*rewrite_traceback_stack(source=source))
      File "/srv/slapgrid/slappart35/srv/runner/software/0b402c7d1e7c38a4324c836766690bb6/eggs/Jinja2-2.11.3-py3.9.egg/jinja2/_compat.py", line 28, in reraise
        raise value.with_traceback(tb)
      File "config/enb.jinja2.cfg", line 139, in top-level template code
        mme_addr: "{{ slap_configuration['configuration.mme_addr'] }}",
    jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'configuration.mme_addr'

-> Fix it by providing both mme_addr in addition to amf_addr.

Now both enb.cfg and gnb.cfg render in standalone mode well.

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus

In b0c37a4a (software/ors-amarisoft: Start to generalize existing
lopcomm/multicell into multiRU slowly) I deduplicated cell_list initialization
and moved it into single place, but overlooked that enb.cfg build became broken
in standalone mode:

    .../software/ors-amarisoft$ ./pythonwitheggs slapos-render-config.py
    Traceback (most recent call last):
      File "/srv/slapgrid/slappart35/srv/project/slapos/software/ors-amarisoft/./pythonwitheggs", line 47, in <module>
        exec(compile(__file__f.read(), __file__, "exec"))
      File "slapos-render-config.py", line 88, in <module>
        do('enb', {"tdd_ul_dl_config": "[Configuration 6] 5ms 5UL 3DL (maximum uplink)"})
      File "slapos-render-config.py", line 86, in do
        j2render(cfg, json_params % locals())
      File "slapos-render-config.py", line 34, in j2render
        f.write(r._render().decode())
      File ".../eggs/slapos.recipe.template-5.1-py3.9.egg/slapos/recipe/template/jinja2_template.py", line 215, in _render
        return template_object.render(**self.context).encode(self.encoding)
      File ".../eggs/Jinja2-2.11.3-py3.9.egg/jinja2/environment.py", line 1090, in render
        self.environment.handle_exception()
      File ".../eggs/Jinja2-2.11.3-py3.9.egg/jinja2/environment.py", line 832, in handle_exception
        reraise(*rewrite_traceback_stack(source=source))
      File ".../eggs/Jinja2-2.11.3-py3.9.egg/jinja2/_compat.py", line 28, in reraise
        raise value.with_traceback(tb)
      File "config/enb.jinja2.cfg", line 1, in top-level template code
        {%- set cell_count = cell_list|length %}
    jinja2.exceptions.UndefinedError: 'cell_list' is undefined

-> Fix it by teaching enb.jinja2.cfg to also load cell list in standalone by itself.

Not touching gnb.jinja2.cfg as currently gnb does not support multicell at all,
and in the future it will be sole enb.cfg to handle both LTE and NR
simultaneously.

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus

This currently has severl problems which we'll be fixing up one by one in the followup patches.

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus

Currently which config to render is hardcoded in this program and only of the
is rendered - currently gnb.  This leads to situation where it is easy to miss
breakage of unselected config.

-> Factor one config rendering into common routine as the preparatory step to
render both enb.cfg and gnb.cfg

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus

software/ors-amarisoft: Start to put common code to load cells and radio units and handle them into slaplte package

We will soon need to use that shared code not only from radio library under ru/ ,
but also from enb.jinja2.cfg and to fix slapos-render-config.

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus

After 8c841ce6 slapos-render-config.py got broken:

    (py3.venv) kirr@deca:~/src/wendelin/slapos/slapos/software/ors-amarisoft$ python slapos-render-config.py
    Traceback (most recent call last):
      File "/home/kirr/src/wendelin/slapos/slapos/software/ors-amarisoft/slapos-render-config.py", line 87, in <module>
        j2render(config, json_params)
      File "/home/kirr/src/wendelin/slapos/slapos/software/ors-amarisoft/slapos-render-config.py", line 34, in j2render
        f.write(r._render().decode())
                ^^^^^^^^^^^
      File "/home/kirr/src/wendelin/slapos/slapos.recipe.template/slapos/recipe/template/jinja2_template.py", line 215, in _render
        return template_object.render(**self.context).encode(self.encoding)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/kirr/src/wendelin/venv/py3.venv/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
        self.environment.handle_exception()
      File "/home/kirr/src/wendelin/venv/py3.venv/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
        raise rewrite_traceback_stack(source=source)
      File "config/gnb.jinja2.cfg", line 586, in top-level template code
        drb_config: "{{ drb_file }}",
    ^^^^^^^^^^^^^^^^^^^^^^^^^
    jinja2.exceptions.UndefinedError: 'drb_file' is undefined

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus

We had to introduce a web page to bootstrap the web version of devtools
because since https://bugs.chromium.org/p/chromium/issues/detail?id=1232509
chrome debugger port no longer serve such a page via HTTP.

The URL also changed, /serve_file/@{version_hash}. pattern is no longer
used, both the devtools and the websocket endpoint are in /devtools

The test was made a bit more complete by actually making requests and
trying to connect to websocket endpoints.

Some problems were found with incognito and block-new-web-contents
options:
 - they are boolean type, but the software parameter serialisation is
 XML, which as of today does not support boolean types. This is left
 TODO for now
 - When both --incognito and --block-new-web-contents are true, the
 command line flag was --incognito--block-new-web-contents, which is
 unknown and was ignored. Some minmal changes are included to fix this.