See merge request nexedi/slapos!872

This upgrade mainly fixes gpython and pymain to be more compatible with
CPython when handling command line. In particular it moves
sys.executable setup and -O handling into pymain:

nexedi/pygolang!10
nexedi/pygolang@21756bd3
nexedi/pygolang@11b367c6
nexedi/pygolang@8564dfdd

in ceb063a4 (software/repman: tests python3 compliance, 2020-10-23), test
profile was changed to run on python3 only but this was accidentally
reverted in 3a1e5628 (software/slapos-sr-testing: use nxdtest, 2020-10-28),
by running repman also on python2.

Restore the situation where we run repman only on python3

The initial intent was to use an interpreter named `python_for_test` and not
`python`, so that software do not accidentally pick up this python during
compilation.
6033e4fa (software/slapos-sr-testing: fix python3 profile, 2020-12-01)
was a quick fix for some python3 compatibility issues discovered after merge,
but it was not correct, because by using ${python-interpreter:eggs}, it was
installing the default python-interpreter section, which has an interpreter
named `python`.

This caused issues while building proxysql, which assumes /usr/bin/env python
is python2.

The fix is to override python-interpreter directly, we don't need another
section eggs here.

This code seems not used, before nexedi/slapos@51c3773e and nexedi/slapos@b77d7bae it was not possible to import.

See merge request nexedi/slapos!869

See merge request nexedi/slapos!870

the version up is needed to support more CPU architectures. For example,
version 0.2.18 was failing with Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz

This code error when it's imported, because it imports from a non existing,
slapos.tool module.
This is an "emergency commit" to repair the tests on python 3 after
 3a1e5628 (software/slapos-sr-testing: use nxdtest, 2020-10-28)

This code error when it's imported, because `install` does not exist on
this class, so the assignment caused NamError.
This is an "emergency commit" to repair the tests on python 3 after
 3a1e5628 (software/slapos-sr-testing: use nxdtest, 2020-10-28)

It got broken in 3a1e5628 (software/slapos-sr-testing: use nxdtest, 2020-10-28)

See merge request !865

This is to match Theia file hierarchy

See merge request nexedi/slapos!866

https://github.com/eclipse-theia/theia/blob/v1.8.0/CHANGELOG.md#v180---26112020

Until now, standalone subsystem was started as a daemon first time a terminal
was openned and since it was running as daemon, stopping the theia instance
did not stop any of the services running in the embedded slapos.

Before slapos.core!265 there was
two supervisor running as daemon:
 - ~/srv/slapos/etc/supervisord.conf which runs slapos proxy etc
 - ~/srv/slapos/inst/etc/supervisord.conf with runs instances in the embedded
   slapos
After, the second one runs as a service in the first one, but the first one
was still running as daemon.

This changes so that the first supervisor runs as a service managed by the
Theia instance, so stopping Theia instance will effectively stop the services.

When upgrading, running instances should continue to run as detached. To
attach them to the new service, procedure could be something like this

Stop supervisors inside Theia instance:

  supervisorctl -c ~/srv/slapos/etc/supervisord.conf shutdown
  supervisorctl -c ~/srv/slapos/inst/etc/supervisord.conf shutdown

Restart slappartX:slapos-standalone-instance-XXX-on-watch from host slapos

We are observing some segmentation fault with python curses applications
that might be because the TERMINFO are different from the system one.
I did not debugged, but since I set this environment variable I did not
observe any segmentation fault.

By comparing strace, the invocations seems same with or without $TERMINFO.
In both cases the termcaps from the correct ncurses are selected, this
just seem to workaround for some reason.

 * switch software/slapos-testing (`SlapOS.Eggs.UnitTest-*`) to use nxdtest and update README to describe how to run tests locally with nxdtest's runTestSuite. Switch tests to `python -m unittest` (or slight variations) instead of the deprecated `python setup.py`, except a few tests which only pass when invoked with `setup.py`.
 * switch software/slapos-sr-testing (`SlapOS.SoftwareReleases.IntegrationTest-*`) to use nxdtest and update README to describe how to run tests locally with nxdtest's runTestSuite. Switch all tests to `python -m unittest discover` intead of the deprecated `python setup.py` - which was especially problematic in these tests, because it sets `$PYTHONPATH` which affect subprocesses ran by the tests. In follow up commits we'll drop the workarounds for `$PYTHONPATH`.

`EggTestSuite` becomes unused and will be removed in follow up changes.

See merge request nexedi/slapos!862

 * use python3 (newest builds requires python3)
 * correctly exclude release directories during rsync so that resiliency
works
 * use correct names for project/software/instances

We also try to switch the tests to use python -m unittest as a way of
invoking tests, instead of the deprecated python setup.py test

When use this interpreter to run buildout, buildout will pre-compile eggs with
different level of optimisations, for this it spawns sys.executable with -O
option (see [1]). Since pymain does not support -O argument and python only
support changing optimization level at startup, we can not easily handle -O
argument, execpt by re-executing ourselves with PYTHONOPTIMIZE environment
variable, which has same effect than setting -O flag

1: https://lab.nexedi.com/nexedi/slapos.buildout/blob/488b2dfd3088cba7b322827c904fe1f93fba73e6/src/zc/buildout/easy_install.py#L1716-1722

Change to run all tests with python -m unittest, which will allow us to
remove workaround for the $PYTHONPATH set by setup.py test.

For this we use `[python-interpreter]` which uses pygolang.main to behave
like a "normal" python interpreter with eggs available, but without the
usual problems of interpreters generated by zc.recipe.eggs - it supports
all command line flags and have current directory in sys.path.

Tests now have an explicit name, that we set to the software name as before,
and the special test dynamically checking all profiles json schemas (part
of slapos.cookbook egg, at the root of slapos repository),is named
json-schemas. It is also changed to run as default tests in python2 and
python3, not only in extras for python2.

fixes "component/trafficserver: Version up to 8.1.0"

Jérome reports that Pygolang build on Python3 fails because geventmp turned out to be not pinned:

nexedi/slapos!862 (comment 121359)

Geventmp is indeed used by pygolang only on Python3

nexedi/pygolang@cd67996e

and with Python2-only testing I missed to handle it.

-> Fix it and add test for pygolang/py3.

/reviewed-on nexedi/slapos!864