software/slapos-testing: Quickfix python -E errors

Since upgrading to zc.buildout = 3.0.1+slapos004, slapos.rebootstrap
and slapos.recipe.template tests are failing due to: these two tests
running buildout, buildout installing develop eggs, develop eggs now
being installed with pip install -e instead of setup.py develop, and
pip install -e launching sys.executable -E at some point. This fails
because the tests are run using gpython:main script, which currently
does not support -E option.

As a quick fix run these two tests with a barebone python executable
and use export PYTHONPATH to provide the needed eggs. This is not as
good as gpython:main because PYTHONPATH will be inherited by all sub
processes and not just those launched with sys.executable, including
potentially other versions of python.

software/slapos-testing/buildout.hash.cfg

@@ -15,4 +15,4 @@
 
 [template]
 filename = instance.cfg
-md5sum = 5f5378550470b551d280dd432878a0ba
+md5sum = 9493b2789de783c375920eec74252499

software/slapos-testing/instance.cfg

@@ -102,6 +102,17 @@ template = inline:#!/bin/sh
   python -m unittest discover -v
 rendered = $${re6stnet:location}/test-runner.sh
 
+# XXX: Quick fix for tests failing due to python -E unknown option.
+# When gpython:main supports -E, move back to using that instead.
+[real-python-with-eggs]
+recipe = slapos.recipe.template
+inline =
+  #!/bin/sh
+  export PYTHONPATH=$(python -c "import sys; print(':'.join(p for p in sys.path if p))")
+  python=$(python -c "import sys; print(sys._gpy_underlying_executable)")
+  exec $python "$@"
+output = $${buildout:bin-directory}/$${:_buildout_section_name_}
+
 [slapos-test-runner-nxdtest-environment.sh]
 recipe = slapos.recipe.template
 output = $${create-directory:etc}/$${:_buildout_section_name_}
@@ -171,9 +182,9 @@ inline =
   )
   TestCase(
       "slapos.recipe.template",
-      # ['python', '-m', 'unittest', 'slapos.recipe.template.tests.test_suite'],
+      # ['$${real-python-with-eggs:output}', '-m', 'unittest', 'slapos.recipe.template.tests.test_suite'],
       # XXX some test fail when running with unittest, so we keep using setup.py test
-      ['python', 'setup.py', 'test'],
+      ['$${real-python-with-eggs:output}', 'setup.py', 'test'],
       cwd="""$${slapos.recipe.template:location}""",
       summaryf=UnitTest.summary,
   )
@@ -205,7 +216,7 @@ inline =
   )
   TestCase(
       "slapos.rebootstrap",
-      ['python', '-m', 'unittest', '-v', 'slapos.rebootstrap.tests.test_suite'],
+      ['$${real-python-with-eggs:output}', '-m', 'unittest', '-v', 'slapos.rebootstrap.tests.test_suite'],
       cwd="""$${slapos.rebootstrap:location}""",
       summaryf=UnitTest.summary,
   )

[Xavier Thompson]

/cc @jerome

[Jérome Perrin]

Thanks !

Are you also planning to work on making pygolang support -E ? It should not be too hard to implement this in pygolang, code is https://lab.nexedi.com/nexedi/pygolang/-/blob/master/gpython/__init__.py and tests are https://lab.nexedi.com/nexedi/pygolang/-/blob/master/gpython/gpython_test.py . On tests nodes we run tests with test suites named Pygolang.UnitTest-Master.Python3 and Pygolang.UnitTest-Master.Python2.

as far as I remember we have support for re-exec'ing the interpreter in a similar way and started implementing support for -S but we were stuck on a bug preventing running the test on python2 ( https://github.com/pypa/virtualenv/issues/2025 ) we probably did not have an immediate need for -S, we waited a bit for support in virtualenv but this did not happen. All this was 4 years ago, now if implement this in pygolang with having test coverage only python3 (skipping the corresponding tests on python2 - I assume this should be easy but I might be wrong here), that seems better.

/cc @kirr

[Kirill Smelkov]

@jerome, thanks for feedback. We talked with Xavier about this issue over jabber yesterday and similarly to you I offered Xavier to work on -E support, but in the end we agreed that it will be me to implement it. Thanks for bringing -S memories and context. Do you maybe have references where we did something for -S already? It would be a bit pity to redo that from scratch but offhand I could not find our draft changes.

[Kirill Smelkov]

(I restored all context by myself; will provide an update hopefully soon)

[Kirill Smelkov]

@xavier_thompson, @jerome, please see pygolang!30 (closed) for -E support.