See merge request nexedi/slapos!1103

On Debian 9, the given rpath seem to be ignored when compiling
the `keytar` dependency of theia, for unknown reasons.

- Move [theia] section and dependencies into component/theia
- Make [theia] section shared

This was missing in this software and in 03c5b311 (Stop using
hexagonit.recipe.download, 2021-12-03) the hash got out of sync

also mention automatic installation method

Since https://github.com/eclipse-theia/theia/pull/10343 theia depend on
nodejs >= 12 and they test on 12 and 14. Use 14 to have an officially
tested version.

This removes the strong requirement to have python2.7 in theia, so we
just use the software python, which might be python3

https://github.com/metabase/metabase/releases/tag/v0.41.4

We observed some test failures like for example 1 where this assertion
was failing because message is `Error reading SSH protocol banner`.

Because the purpose of this specific test is to check that log rotation
of ban log work as expected and because we test more thoroughly the
ban itself in TestBan.test_client_are_banned_after_5_wrong_passwords, we
can simplify this test by just asserting that connection was refused, by
expecting a general exception. We don't care about the details of the
exception here.

After a ban, the first connection attemps seem to always be refused with
"Connection reset by peer" and that's why we did not observed failures
with TestBan.test_client_are_banned_after_5_wrong_passwords

[1]: https://nexedijs.erp5.net/#/test_result_module/20211208-1520AC26C/21

Fix the following build failure in nxdtest's own test:

    [2021-12-10 15:57:15,142] INFO     While:
    [2021-12-10 15:57:15,142] INFO       Installing.
    [2021-12-10 15:57:15,142] INFO       Getting section python-interpreter.
    [2021-12-10 15:57:15,142] INFO       Initializing section python-interpreter.
    [2021-12-10 15:57:15,142] INFO       Getting option python-interpreter:eggs.
    [2021-12-10 15:57:15,142] INFO       Getting section nxdtest.
    [2021-12-10 15:57:15,142] INFO       Initializing section nxdtest.
    [2021-12-10 15:57:15,142] INFO       Getting option nxdtest:eggs.
    [2021-12-10 15:57:15,142] INFO       Getting section .nxdtest.pyexe.
    [2021-12-10 15:57:15,142] INFO       Initializing section .nxdtest.pyexe.
    [2021-12-10 15:57:15,142] INFO       Getting option .nxdtest.pyexe:eggs.
    [2021-12-10 15:57:15,142] INFO       Getting option nxdtest:eggs.
    [2021-12-10 15:57:15,142] INFO       Getting option .nxdtest.pyexe:eggs.
    [2021-12-10 15:57:15,142] INFO     Error: Circular reference in substitutions.

This is hot, not a proper, fix to recover nxdtest.UnitTest-Master status.
I will think more calmly what to properly do.

If we enter user namespace via regular unshare without help from SUID
newuidmap/newgidmap, all supplementary groups are mapped to -1. As the result
when Go test tries to chown to a supplementary group, it gets EINVAL:

https://github.com/golang/go/issues/42525

-> work it around with patch to skip this chown tests.

A more proper, longer-term fix would be to fix Linux kernel to allow writes to
/proc/self/gid_map to setup mapping not only to original gid, but to all
original supplementary groups as well here:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/user_namespace.c?id=v5.16-rc4-0-g0fcfb00b28c0#n1143

this fix, even if accepted by upstream, would be long to be waited for to
propagate to distribution kernels that we currently use. So we go with this
workaround for now.

--------

Another patch is to fix the following TestSCMCredentials failure:

    === RUN   TestSCMCredentials
        creds_test.go:81: WriteMsgUnix failed with invalid argument, want EPERM
    --- FAIL: TestSCMCredentials (0.00s)

There the code tries to send uid0/gid0 credentials from non-zero uid and
expects EPERM reject from kernel. However under `unshare -Umc` uid0/gid0 are
not mapped to anywhere and so implicitly map to -1 and are rejected with EINVAL
by the kernel.

/reviewed-by @jerome
/reviewed-on nexedi/slapos!1095

4) Hook in python-prctl, as it becomes nxdtest dependency: see
   nexedi/nxdtest!13 (79d13eff)

/reviewed-by @jerome
/reviewed-on nexedi/slapos!1095

Prepare for upcoming nxdtest changes to run each testcase with its own
/tmp and /dev/shm:

1) put unshare from SlapOS component into $PATH, so that our version is
   used even if OS provides /bin/unshare. As @jerome explains we need
   features that were added relatively recently and are missing in unshare
   on Debian 10: nexedi/nxdtest!13 (comment 146752)

   It is anyway better to "isolate" from OS by using our own component
   instead of system-provided one.

   Correspondingly adjust util-linux to enable unshare in its build.

2) similarly to "1" adjust util-linux to enable mount so that our version
   is used instead of /bin/mount. For example on Debian 9, even if we
   successfully enter user/mount namespace with `unshare -Umc`

       /bin/mount -t tmpfs none /tmp

   complains that

       mount: only root can use "--types" option

   -> Fix it the same way as with unshare by forcing usage of
   SlapOS-provided mount.

3) rework how nxdtest script is generated and split it into .nxdtest.pyexe and
   nxdtest itself. .nxdtest.pyexe is python interpreter via which nxdtest is run.
   This interpreter has all eggs required by nxdtest in sys.path, so that
   nxdtest could spawn its trun.py via sys.executable. If we don't care to have
   properly setup sys.executable, trun.py will fail when importing any module that
   nxdtest.py could already successfully import.

   Initially I tried to workaround this issue via adjusting $PYTHONPATH <-
   sys.path in main nxdtest script, but @jerome points out that, $PYTHONPATH,
   if set, also affects processes that trun.py spawns, which is not good:

   nexedi/slapos!1095 (comment 146799)

   -> so fix this via running nxdtest via environment where sys.executable is
   properly setup python interpreter with path for all eggs that nxdtest has
   access to.

   Because we already have half-way workarounds for similar problem in several
   places, and because running a script with correctly setup sys.executable is
   generally better, I would say it should be a good idea to rework
   zc.recipe.egg:scripts to generate all scripts to work this way, but I do not
   want to fight about it.

   So let's leave this scheme nxdtest-specific for now.

/cc @tomo
/helped-and-reviewed-by @jerome
/reviewed-on nexedi/slapos!1095

When adding a route to an interface not up yet, "ip route" command fails
with:

Error: Device for nexthop is not up.

So we need to up the device before adding the routes.

This is a regression with 0.48 that prevented parts to be shared depending
on which Python is used to run Python (whether it's version 2 or 3).

The patch was useless since version 6.0:
http://ncurses.scripts.mit.edu/?p=ncurses.git;h=6a530b46563470c2ca73579d1994a0c8e275dd98

slapos.core now depends on PyYAML and jsonschema

This upgrade fixes deadlock when new context is created from
already-canceled parent[1]. It also comes with other minor enhancements
and fixes needed e.g. for `zodb restore`[2].

[1] nexedi/pygolang@58d4cbfe
[2] https://pypi.org/project/pygolang/#pygolang-change-history

json-c, memcached and xpdf are still maintained but current
buildout.cfg are not useful if we want to ship a new version.