software/ors-amarisoft: Start to put common code to load cells and radio units and handle them into slaplte package

We will soon need to use that shared code not only from radio library under ru/ ,
but also from enb.jinja2.cfg and to fix slapos-render-config.

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus

After 8c841ce6 slapos-render-config.py got broken:

    (py3.venv) kirr@deca:~/src/wendelin/slapos/slapos/software/ors-amarisoft$ python slapos-render-config.py
    Traceback (most recent call last):
      File "/home/kirr/src/wendelin/slapos/slapos/software/ors-amarisoft/slapos-render-config.py", line 87, in <module>
        j2render(config, json_params)
      File "/home/kirr/src/wendelin/slapos/slapos/software/ors-amarisoft/slapos-render-config.py", line 34, in j2render
        f.write(r._render().decode())
                ^^^^^^^^^^^
      File "/home/kirr/src/wendelin/slapos/slapos.recipe.template/slapos/recipe/template/jinja2_template.py", line 215, in _render
        return template_object.render(**self.context).encode(self.encoding)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/kirr/src/wendelin/venv/py3.venv/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
        self.environment.handle_exception()
      File "/home/kirr/src/wendelin/venv/py3.venv/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
        raise rewrite_traceback_stack(source=source)
      File "config/gnb.jinja2.cfg", line 586, in top-level template code
        drb_config: "{{ drb_file }}",
    ^^^^^^^^^^^^^^^^^^^^^^^^^
    jinja2.exceptions.UndefinedError: 'drb_file' is undefined

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus

By reusing recently added "split TAP" infrastructure we can adjust dnsmasq
configuration to provide unique IPv6 to each RU.

- ru_mac_addr becomes per-RU setting and without default. We talked with Lu, and
  since now all Lopcomm units are shipped from the factory with unique MAC, it
  both does not make sense to provide the default, and we can rely on all units
  to have different MACs and configure DHCP replies based on that.

- No need to provide /64 network to every RU. We cannot actually do that anyway
  because normally SlapOS provides /71 address range for its slaptap. In the
  new configuration everything works with smaller networks just ok.

/cc @jhuge, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on nexedi/slapos!1472

Those options are not needed, because we need to only provide IPv6 address to
RU, and also they are not very meaningful: in the current form we tell RU that
DNS addresses sit at RU.addr+1 and RU.addr+2, i.e. in the IP range we give to
RU and also at the addresses where no DNS server is actually running.

It was probably a thinko to add those options initially.

/cc @jhuge, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on nexedi/slapos!1472

Core Network and ENB use dnsmasq for completely different purposes:

- core network uses it to provide DNS server, while
- enb uses dnsmasq to provide DHCP server for Radio Units to be able to access
  Control & Management channel on the CPRI link.

-> Even though both those services are handled via same dnsmasq program, it
   makes sense to split dnsmasq config for clarity and as preparation for
   further adjustments of enb part.

   We also push config rendering down to -core and -enb instances also for
   clarity, and because in enb case rendering will need to know set of
   configured Radio Units - information that will become loaded only at
   instance-enb.

/cc @jhuge, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on nexedi/slapos!1472

SlapOS provides to each partition dedicated TAP interface, so that an instance
could organize internal networking. In practice this is used by KVM software
release and here in ors-amarisoft, where we feed to eNB such TAP interface for
CPRI-based radio unit so that eNB, in turn, could provide access to CPRI
Control and Management channel via provided TAP.

However there is a problem: SlapOS provides only one TAP interface per
instance, while we need to have one TAP for each Radio Unit.

-> Solve this problem by creating TAP "subinterfaces" per each RU ourselves.

The interfaces we create are full TAP interfaces, just we name them with prefix
based on main interface, and we allocate only part of address space of sole
SlapOS TAP to each subtap. For example if SlapOS provided us slaptap16 with
2401:5180:0:66:a200::/71 IPv6 range and we want to split it for 2 radio units,
we'll be splitting it into 3 regions as follows:

    slaptap16: split 2401:5180:0:66:a200::/71 by 2
    preserve         2401:5180:0:66:a200::/73
    -> slaptap16-1   2401:5180:0:66:a280::/73
    -> slaptap16-2   2401:5180:0:66:a300::/73

Here we preserve 2401:5180:0:66:a200::/73 for usage on original slaptap16, and
we create slaptap16-1 and slaptap16-2 with correspondingly allocated address
range subparts for the RUs.

The splitting is done easily but depends on having networking administration
capability to be able to work. We solve this with employing
/opt/amarisoft/setcap, which we already use for dnsmasq, and with compiled
trampoline program because setcap does not really work directly on scripts.

To avoid hard dependency on having network capability rights, we fallback to
using regular SlapOS slaptap in case there is only one RU.

ru/lopcomm/* and enb.cfg are adapted straightforwardly, but dnsmasq adaptation
is left to a later step not to mix everything into one pile.

NOTE that relying on setcap is not a good in the long term and should be
     reworked once SlapOS is improved to provide ability for instances to
     request several TAP interfaces. Please see discussion at
     nexedi/slapos!1471 (comment 194356)
     for details.

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus
/reviewed-on nexedi/slapos!1471

A preparatory step for multiRU for the same reason as for LTE and also to be
able to support Carrier Aggregation in between NR and LTE cells in the future
(called Dual Connectivity by 3GPP).

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1473

When there will be multiple cells and so multiple DRB files referenced from enb.cfg,
eNB will complain with an error that "there are multiple #define" for T_REORDERING.

-> Use jinja2 templating instead to handle FDD/TDD conditions.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1473

This is preparatory step for multiRU: when there will be several LTE cells,
each possibly having different RF mode, we'll need to configure DRB per-cell.

-> Move DRB configuration to separate jinja2 template to prepare to handle that.

This is 99% movement only, without changing the code for DRB profile. We'll
adjust the DRB profile a bit as another preparatory step in the next patch.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1473

Lopcomm part of the SR already has some partial support for cell_list: if
multiple cells are defined there enb.cfg will have multiple CPRI radio units
and multiple cells configured. But so far all promises, except cpri-link, were
done only for one RU.

-> Fix that by starting to generalize RUlib code to handle multiple radio
units, invoking model-specific RU driver for each RU, and adjusting ru/lopcomm
instance code to correctly generate and activate different promises for
different radio units.

After the patch multiRU support is still very incomplete, but it is a step forward.

/cc @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/partly-reviewed-by @jhuge
/reviewed-at nexedi/slapos!1467

- use firefox 115 ( nexedi/slapos!1470 )
- version up erp5 util ( nexedi/erp5!1832 )

See merge request nexedi/slapos!1475

The / is needed, in order to put the sbin inside DESTDIR instead of wrongly
creating DESTDIRsbin.

This fixes a crash happening after `setFile` selenium command under
some conditions (it does not always happen, but I did not investigate
in which conditions it happens exactly).

[Parent 94283, Main Thread] ###!!! ASSERTION: No GSettings schemas are installed on the system: 'glib assertion', file /builds/worker/checkouts/gecko/toolkit/xre/nsSigHandlers.cpp:164

See merge request nexedi/slapos!1470

Since yesterday, software/gitlab can no longer be installed with an
error while installing gitlab_npm:

    ERR! Invalid dependency type requested: alias

This is because we are using npm install to install a repository which
uses yarn.lock to pin versions, so the install was done without having
the dependencies pinned. Using an old yarn for this repository does not
seem so easy, so we just have made a commit to convert the yarn.lock to
a package-lock.json, so that we don't have to update the tooling here.

Once we update, we'll rework this part of the software to use yarn, it
seems gitlab still uses yarn in more recent versions.

`git describe` also produced different (more correct) output, because
some tags were missing in our mirror of gitlab-ce.

this applies to software/seleniumrunner and testnodes

this was just not tested

the default bytes is just fine

We can use mathematical relation in between ports, channels and tx/rx endpoints
to bring structure and clarity in RU setup.

Also, while on it, document how tx/rx data flow are organized as it was not
clear just from reading ORAN specs and I had to deduce it.

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on nexedi/slapos!1468

Our python interpreter uses ncclient which depends on lxml egg, but does not
explicitly specify to use that lxml from slapos component.

Until now we were lucky because slapos-cookbook depends on the correct lxml and
we have slapos-cookbook as the second entry in the part list with only
`template` preceding it.

However I needed to use pythonwitheggs inside that template and then got the
build failure, because now pythonwitheggs was built before slapos-cookbook and
tried to use lxml without slapos component:

    INFO Building without Cython.
    INFO Error: Please make sure the libxml2 and libxslt development packages are installed.
    INFO An error occurred when trying to install lxml 4.9.1. Look above this message for any errors that were output by easy_install.
    INFO While:
    INFO   Installing pythonwitheggs.
    INFO   Base installation request: 'websocket-client', 'pynacl', 'bcrypt', 'xmltodict', 'ncclient'
    INFO     Requirement of ncclient==0.6.13: six
    INFO     Requirement of ncclient==0.6.13: lxml>=3.3.0
    INFO     Requirement of ncclient==0.6.13: paramiko>=1.15.0
    INFO     Requirement of ncclient==0.6.13: setuptools>0.6
    INFO     Requirement of bcrypt==3.1.4: six>=1.4.1
    INFO     Requirement of bcrypt==3.1.4: cffi>=1.1
    INFO     Requirement of pynacl==1.3.0: cffi>=1.4.1
    INFO     Requirement of pynacl==1.3.0: six
    INFO   Getting distribution for 'lxml==4.9.1'.
    INFO Error: Couldn't install: lxml 4.9.1

-> Fix it by explicitly specifying in-slapos lxml egg to be used.

/cc @jhuge, @Daetalus
/reviewed-by @lu.xu, @xavier_thompson
/reviewed-on nexedi/slapos!1469

also build with rawmidi enabled

Bring more structure to RU-specific code as a preparatory step for multiRU support:

- move RU-specific files under ru/<RU-type>/ . This mostly moves Lopcomm programs and configuration files there.
- move RU-specific instance code there as well. This also mostly moves Lopcomm specific services and promises there.
- bring more structure in naming. As buildout has global namespace use ru_<RU-type>_ prefix to avoid collision in names.

This should be a preparatory patch with practically no semantic change, but
preparing ground for further multiRU landing.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1466

`render-templates -d` is used to remove all files that it previously
generated, so that it is easy to look around during development.

But, probably due to oversight, it was only *tdd files, that were
removed, with *fdd generated files still left in place.

-> Fix it.

The list of generated files, that is now additionally removed after this patch is:

        instance-fdd-enb-input-schema.json
        instance-fdd-gnb-input-schema.json
        instance-fdd-lopcomm-enb-input-schema.json
        instance-fdd-lopcomm-gnb-input-schema.json
        instance-fdd-lopcomm-ue-lte-input-schema.json
        instance-fdd-lopcomm-ue-nr-input-schema.json
        instance-fdd-ors-enb-input-schema.json
        instance-fdd-ors-gnb-input-schema.json
        instance-fdd-ors-ue-lte-input-schema.json
        instance-fdd-ors-ue-nr-input-schema.json
        instance-fdd-ue-lte-input-schema.json
        instance-fdd-ue-nr-input-schema.json
        software-fdd-lopcomm.cfg
        software-fdd-lopcomm.cfg.json
        software-fdd-ors.cfg
        software-fdd-ors.cfg.json
        software-fdd.cfg
        software-fdd.cfg.json
        test/testFDD-LOPCOMM.py
        test/testFDD-ORS.py
        test/testFDD.py

/cc @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1463

For its operation slapos-render-config mimics jinja2 templates rendering
done for real in slapos. But to do so it currently duplicates the code
from slapos.recipe.template.

We can do the same by reusing slapos.recipe.template instead of
duplicating code from there.

-> Do it.

The output of generated gnb.cfg is the same before and after this patch.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu, @tomo
/reviewed-on nexedi/slapos!1462

When I was first starting to look around inside ors-amarisoft software
release for me it was not clear off-hand what this program is for. It
should be more clear about what's the intent if there is explicit
comment.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu, @tomo
/reviewed-on !1462

a6eaad9a (software/ors-amarisoft: add network_name parameter) updated
enb.jinja2.cfg and gnb.jinja2.cfg to require slap_configuration['configuration.com_addr']
and other parameters to be present, but did not updated
slapos-render-confg, which got broken as the result:

    (py3.venv) kirr@deca:~/src/wendelin/slapos/slapos/software/ors-amarisoft$ python slapos-render-config.py
    ...
    Traceback (most recent call last):
      File "/home/kirr/src/wendelin/slapos/slapos/software/ors-amarisoft/slapos-render-config.py", line 232, in <module>
        f.write(r._render().decode())
                ^^^^^^^^^^^
      File "/home/kirr/src/wendelin/slapos/slapos/software/ors-amarisoft/slapos-render-config.py", line 206, in _render
        return template_object.render(**self.context).encode(self.encoding)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/kirr/src/wendelin/venv/py3.venv/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
        self.environment.handle_exception()
      File "/home/kirr/src/wendelin/venv/py3.venv/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
        raise rewrite_traceback_stack(source=source)
      File "config/gnb.jinja2.cfg", line 62, in top-level template code
        com_addr: "{{ slap_configuration['configuration.com_addr'] }}:{{ slap_configuration['configuration.com_ws_port'] }}",
      ^^^^^^^^^^^^^^^^^^^^^^^^^
    jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'configuration.com_addr'

-> Fix it.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu, @tomo
/reviewed-on nexedi/slapos!1462