This MR only take care of upgrading numpy. Originally, it was part of !1207 (merged) but since we removed numpy in all python3 SR (see !1279 (merged)), we don't need to upgrade numpy now.

Problem description

In recent versions, numpy and scipy require Cython during their setup, and zc.recipe.egg:custom's setup-eggs option doesn't work because in their setup.py, Cython is launched in a subprocess, and the sys.path does not propagate.

Manually setting the PYTHONPATH with the environment option of zc.recipe.egg:custom is not ideal because it requires "guessing" where the eggs will be installed, and this depends on the current version of the eggs, of Python, on the system etc.

And numpy and scipy must be upgraded to support Python3.9, since older versions are incompatible.

Proposed solution

We inject the PYTHONPATH of the required eggs into the environment of zc.recipe.egg:custom during the install phase of buildout (zc.recipe.egg:custom accepts an environment option to link another buildout section whose keys/values will be the environment variables).

In the install phase, the PYTHONPATH can be computed automatically once the eggs are already installed.

Luckily, it just so happens that zc.recipe.egg:custom does not retrieve the contents of the environment section before the install phase, so there is a window of opportunity to 1) install the required eggs, 2) compute and inject the PYTHONPATH in the python dict of the environment section using slapos.recipe.build, 3) install numpy/scipy.

This precise ordering is achieved through carefully chained dependencies, and the logic to compute the PYTHONPATH and inject it is factored into a new macro macro.pythonpath.eggs in component/macros.

Note that we cannot simply expose the computed PYTHONPATH as a key/value and set it in the environment section by value substitution, as in:

[numpy-env]
PYTHONPATH = ${numpy-pythonpath:pythonpath}

Because the substitution of ${numpy-pythonpath:pythonpath} takes place during the init phase, and at that point the eggs are not installed yet and the PYTHONPATH cannot be computed yet.