See merge request nexedi/slapos!1470

Since yesterday, software/gitlab can no longer be installed with an
error while installing gitlab_npm:

    ERR! Invalid dependency type requested: alias

This is because we are using npm install to install a repository which
uses yarn.lock to pin versions, so the install was done without having
the dependencies pinned. Using an old yarn for this repository does not
seem so easy, so we just have made a commit to convert the yarn.lock to
a package-lock.json, so that we don't have to update the tooling here.

Once we update, we'll rework this part of the software to use yarn, it
seems gitlab still uses yarn in more recent versions.

`git describe` also produced different (more correct) output, because
some tags were missing in our mirror of gitlab-ce.

this applies to software/seleniumrunner and testnodes

this was just not tested

the default bytes is just fine

We can use mathematical relation in between ports, channels and tx/rx endpoints
to bring structure and clarity in RU setup.

Also, while on it, document how tx/rx data flow are organized as it was not
clear just from reading ORAN specs and I had to deduce it.

/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on nexedi/slapos!1468

Our python interpreter uses ncclient which depends on lxml egg, but does not
explicitly specify to use that lxml from slapos component.

Until now we were lucky because slapos-cookbook depends on the correct lxml and
we have slapos-cookbook as the second entry in the part list with only
`template` preceding it.

However I needed to use pythonwitheggs inside that template and then got the
build failure, because now pythonwitheggs was built before slapos-cookbook and
tried to use lxml without slapos component:

    INFO Building without Cython.
    INFO Error: Please make sure the libxml2 and libxslt development packages are installed.
    INFO An error occurred when trying to install lxml 4.9.1. Look above this message for any errors that were output by easy_install.
    INFO While:
    INFO   Installing pythonwitheggs.
    INFO   Base installation request: 'websocket-client', 'pynacl', 'bcrypt', 'xmltodict', 'ncclient'
    INFO     Requirement of ncclient==0.6.13: six
    INFO     Requirement of ncclient==0.6.13: lxml>=3.3.0
    INFO     Requirement of ncclient==0.6.13: paramiko>=1.15.0
    INFO     Requirement of ncclient==0.6.13: setuptools>0.6
    INFO     Requirement of bcrypt==3.1.4: six>=1.4.1
    INFO     Requirement of bcrypt==3.1.4: cffi>=1.1
    INFO     Requirement of pynacl==1.3.0: cffi>=1.4.1
    INFO     Requirement of pynacl==1.3.0: six
    INFO   Getting distribution for 'lxml==4.9.1'.
    INFO Error: Couldn't install: lxml 4.9.1

-> Fix it by explicitly specifying in-slapos lxml egg to be used.

/cc @jhuge, @Daetalus
/reviewed-by @lu.xu, @xavier_thompson
/reviewed-on !1469

also build with rawmidi enabled

Bring more structure to RU-specific code as a preparatory step for multiRU support:

- move RU-specific files under ru/<RU-type>/ . This mostly moves Lopcomm programs and configuration files there.
- move RU-specific instance code there as well. This also mostly moves Lopcomm specific services and promises there.
- bring more structure in naming. As buildout has global namespace use ru_<RU-type>_ prefix to avoid collision in names.

This should be a preparatory patch with practically no semantic change, but
preparing ground for further multiRU landing.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1466

`render-templates -d` is used to remove all files that it previously
generated, so that it is easy to look around during development.

But, probably due to oversight, it was only *tdd files, that were
removed, with *fdd generated files still left in place.

-> Fix it.

The list of generated files, that is now additionally removed after this patch is:

        instance-fdd-enb-input-schema.json
        instance-fdd-gnb-input-schema.json
        instance-fdd-lopcomm-enb-input-schema.json
        instance-fdd-lopcomm-gnb-input-schema.json
        instance-fdd-lopcomm-ue-lte-input-schema.json
        instance-fdd-lopcomm-ue-nr-input-schema.json
        instance-fdd-ors-enb-input-schema.json
        instance-fdd-ors-gnb-input-schema.json
        instance-fdd-ors-ue-lte-input-schema.json
        instance-fdd-ors-ue-nr-input-schema.json
        instance-fdd-ue-lte-input-schema.json
        instance-fdd-ue-nr-input-schema.json
        software-fdd-lopcomm.cfg
        software-fdd-lopcomm.cfg.json
        software-fdd-ors.cfg
        software-fdd-ors.cfg.json
        software-fdd.cfg
        software-fdd.cfg.json
        test/testFDD-LOPCOMM.py
        test/testFDD-ORS.py
        test/testFDD.py

/cc @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1463

For its operation slapos-render-config mimics jinja2 templates rendering
done for real in slapos. But to do so it currently duplicates the code
from slapos.recipe.template.

We can do the same by reusing slapos.recipe.template instead of
duplicating code from there.

-> Do it.

The output of generated gnb.cfg is the same before and after this patch.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu, @tomo
/reviewed-on nexedi/slapos!1462

When I was first starting to look around inside ors-amarisoft software
release for me it was not clear off-hand what this program is for. It
should be more clear about what's the intent if there is explicit
comment.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu, @tomo
/reviewed-on nexedi/slapos!1462

a6eaad9a (software/ors-amarisoft: add network_name parameter) updated
enb.jinja2.cfg and gnb.jinja2.cfg to require slap_configuration['configuration.com_addr']
and other parameters to be present, but did not updated
slapos-render-confg, which got broken as the result:

    (py3.venv) kirr@deca:~/src/wendelin/slapos/slapos/software/ors-amarisoft$ python slapos-render-config.py
    ...
    Traceback (most recent call last):
      File "/home/kirr/src/wendelin/slapos/slapos/software/ors-amarisoft/slapos-render-config.py", line 232, in <module>
        f.write(r._render().decode())
                ^^^^^^^^^^^
      File "/home/kirr/src/wendelin/slapos/slapos/software/ors-amarisoft/slapos-render-config.py", line 206, in _render
        return template_object.render(**self.context).encode(self.encoding)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/kirr/src/wendelin/venv/py3.venv/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
        self.environment.handle_exception()
      File "/home/kirr/src/wendelin/venv/py3.venv/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
        raise rewrite_traceback_stack(source=source)
      File "config/gnb.jinja2.cfg", line 62, in top-level template code
        com_addr: "{{ slap_configuration['configuration.com_addr'] }}:{{ slap_configuration['configuration.com_ws_port'] }}",
      ^^^^^^^^^^^^^^^^^^^^^^^^^
    jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'configuration.com_addr'

-> Fix it.

/cc @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu, @tomo
/reviewed-on nexedi/slapos!1462

In multiRU we will need to be able to check multiple CPRI boards and multiple
SFP ports on them, not only SFP ports on CPRI board 0 that was implicitly used
until now.

-> As a preparatory step the SR to explicitly specify which CPRI resources are being verified.

This patch is necessary because in nexedi/slapos.toolbox!127
we adjust check_cpri_lock plugin to require CPRI device + SFP port to be
explicitly specified.

/cc @tomo, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu, @jhuge
/reviewed-on nexedi/slapos!1461

In multiRU we will need to be able to check multiple radio units for RX saturation separately.

-> As a preparatory step adjust the SR to explicitly specify the list of RX antennas to be verified.

This patch is necessary because in nexedi/slapos.toolbox!126
we adjust check_rx_saturated plugin to require list of RX channels to check to be
explicitly specified.

/cc @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge, @lu.xu
/reviewed-on nexedi/slapos!1459

In multiRU we will need to be able to check multiple devices and DMA channels,
not only /dev/sdr0@0 that was implicitly used until now.

-> As a preparatory step adjust the SR to explicitly specify which SDR resource is being verified.

This patch is necessary because in nexedi/slapos.toolbox!125
we adjust check_sdr_busy plugin to require SDR device/DMA channel to be
explicitly specified.

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on nexedi/slapos!1458

also stop enabling pip user installation, this breaks slapos isolation too much

See merge request nexedi/slapos!1464

This reverts commit 6f167c36.

This causes issues that unexpectly slapos python processes use user
installed packages. This was to accomodate python extension prompting
user to install packages for them, but current version of the extension
no longer seem to do this.

See merge request nexedi/slapos!1460

Cover a case of not adding needless trailing / while accessing a backend from
top level of the URL.

As / are stripped while parsing the url, expose the situation that even if
url ends with a /, it won't be sent for top level access.

Here is the mapping of configured URL, path accessed on the CDN side and path
sent to the backend:

 * backend/index.html   frontend/     index.html
 * backend/index.html   frontend/a    index.html/a
 * backend/index.html   frontend/a/   index.html/a/
 * backend/index.html/  frontend/     index.html
 * backend/index.html/  frontend/a    index.html/a
 * backend/index.html/  frontend/a/   index.html/a/