WIP: Testnode: Use shared parts when building softwares
I still need to test this a bit more, but this should be enough to have testnode:
- share parts between selenium-runner and tested softwares when installing softwares
- share parts also with tests running in SLAPOS-SR-TEST
This covers only unit tests. I have not tried to cover scalability tests.
This also need a small change to profiles , slapos!627
( also I'm not sure selenium-server supports building as shared, I also need to check this )
Seems fine for me.
I'm not sure to fully get what is shared. Do you mean that we can share between selenium-runner installed by testnode and test suite software (like ERP5-MASTER) ? But if you have two test suite like ERP5-TestRUNNER1 and ERP5-TESTRUNNER2 running both on the same testnode, are we going to share parts of the the building ?
marked as a Work In ProgressToggle commit list
The idea is that testnode first installs selenium-runner using a folder for shared parts, then testnode will install each tested software (eg. ERP5-MASTER, ERP5-TESTRUNNER1) using selenium-runner's shared parts as "read-only" and another folder for share parts of each software (
When installing tested software, for each part:
- if it's not marked as shared it installs in
- if part is marked as shared (a slapos.recipe.cmmi part with shared=true, like this one)
- if that shared part was already installed as a shared part in selenium runner, then it will be used directly from selenium runner.
- if not installed a shared part in selenium runner, then it will be installed in
- ERP5-MASTER will not use parts from ERP5-TESTRUNNER1 (so it's always safe to remove software)
- because "shared" takes into account all options, even if selenium-runner already have install a part let's say for curl from http://curl.haxx.se/download/curl-7.60.0.tar.xz
and ERP5-MASTER uses another version of curl, then url option will be different and it will be different hash, so it will be installed again in
- if both ERP5-MASTER and ERP5-TESTRUNER1 use exactly same version of curl than selenium-runner, then it will not be reinstalled again.
- if both ERP5-MASTER and ERP5-TESTRUNER1 use exactly same version of curl but a different one that selenium-runner, then it will be duplicated in each
Also it's good to know that:
- all "basic" parts should be shared already
- some heavy parts like mysql are not shared yet
- eggs are not shared yet (some eggs like pandas or scipy really takes time to install)
shared parts are not garbage collected automatically, which means that:
- seleniumrunner shared parts increase everytime we change the version of selenium runner used here.
- tested software (ERP5-MASTER, ERP5-TESTRUNNER1) shared parts increase every time their signatures change. If this become a too big problem, until something like slapos.core!139 is finished, a workaround to reset a big software could be to change test suite reference, which would trigger a "remove everything and rebuild in another folder".
- for slapos SR tests, everything is in instance folder, which is removed between each run, so SR tests should not be a problem.
- I posted a quick summary on vifib forum last week ("News about SR tests and shared builds"). I'll try to post another more "user oriented" summary on nexedi forum once this gets ready to use.
I'm installing a testnode to try a bit these patches. I just have set WIP because I forgot to set earlier, this was not yet ready to production.
- if it's not marked as shared it installs in
@jerome , thank you for detailed summary.
Regarding scipy / numpy / openblas : I do not recall why default ERP5 needs them (I recall there was a decision back in the days) ?
I think they should be added only for Wendelin. This of course is unrelated to "shared" improvement but can help reduce ERP5's compilation time
I think long time ago (at the time of erp5 jupyter kernel), Jean Paul said it would be nice to include a scientific python stack directly in ERP5 and not only in wendelin, but I don't remember clearly the reasons, maybe something like "so that it's already here when we will need it".
I remember we also consider sharing eggs. Buildout supports shared directory for eggs and develop eggs.( http://www.buildout.org/en/latest/topics/optimizing.html#optimizing-buildouts-with-shared-eggs-and-download-caches ), but :
- there's only one directory in which all buildout will try to write
- the signature for develop eggs does not seem to include "enough", for example if we have a develop egg for pycurl, it will be same egg name whatever options where used to provide curl.
added 118 commits
Toggle commit list
274e7689...2f25a327 - 114 commits from branch
- b53ef5e5 - testnode: python3 support
- 82e3cb34 - testnode: SlapOS shared parts support
- 8236b6f0 - testnode: pass shared_part_list to runTestSuite who understand it
- 4bca2ca8 - Eggtest: support --shared_part_list
- 274e7689...2f25a327 - 114 commits from branch