GitLab

(see also slapos repository's shared-egg branch)

This merge request contains changes for both zc.buildout and zc.recipe.egg.

what can be shared ?

• buildout extension eggs
• eggs installed for recipes
• eggs installed with zc.recipe.egg and zc.recipe.egg:custom

what cannot be shared ?

• buildout bootstrap eggs, and their possible updates
• eggs installed with zc.recipe.egg:develop

performance impact ?

With a 'hot' enough shared directory, I could build EPR5 software release in another directory in 9 minutes, instead of 1 hour.

directories / files

/(shared)/__eggs__/AccessControl-4.4-py2.7-linux-x86_64.egg
/(shared)/__eggs__/Acquisition-4.13+slapospatched001-py2.7-linux-x86_64.egg
/(shared)/__eggs__/aiohappyeyeballs-2.4.4-py3.9.egg
/(shared)/__eggs__/aiohttp-3.11.11-py3.9-linux-x86_64.egg
...
/(shared)/python-cffi/(hash)/cffi-1.15.0-py2.7-linux-x86_64.egg
/(shared)/python-cffi/(hash)/cffi-1.15.0-py3.9-linux-x86_64.egg

normal eggs are used directly from shared __eggs__ directory and we have symlinks of develop-eggs under (SR)/develop-eggs.

how they are protected from slapos node prune ?

Inside (SR)/.installed.cfg,

• [buildout] part contais the information of eggs for recipes like __eggs__ = slapos.recipe.cmmi-0.22-py3.9.egg plone.recipe.command-1.1-py3.9.egg.
• the part where egg is installed with zc.recipe.egg:custom contains like location = /(shared)/python-cffi/(hash) like slapos.recipe.cmmi shared ones.
• the part where eggs are installed with zc.recipe.egg contains the information of used eggs in the part like __eggs__ = six-1.16.0-py3.9.egg pycparser-2.20-py3.9.egg.

design questions

• copy or symlink ?
  • symlink is faster and takes less storage (totally 1GB eggs + develop-eggs for ERP5 software release)
  • copy is safer, for the case shared eggs are deleted
  • if symlink, egg paths in scripts are real path of shared eggs (setuptool's pkg_resources behaviour) that can be 'old' shared ones in case of updates [FIXED]
  • my conclusion is 'copy', as software release and instances are safe even if some shared eggs are deleted by mistake