Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
slapos slapos
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Labels
    • Labels
  • Merge requests 122
    • Merge requests 122
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Environments
  • Analytics
    • Analytics
    • CI/CD
    • Repository
    • Value Stream
  • Snippets
    • Snippets
  • Members
    • Members
  • Activity
  • Graph
  • Jobs
  • Commits
Collapse sidebar
  • nexedi
  • slaposslapos
  • Merge requests
  • !1280

Merged
Created Nov 03, 2022 by Thomas Gambier@tomo🚴🏼Owner

Feat/upgrade numpy

  • Overview 39
  • Commits 6
  • Changes 9

This MR only take care of upgrading numpy. Originally, it was part of !1207 (merged) but since we removed numpy in all python3 SR (see !1279 (merged)), we don't need to upgrade numpy now.

Problem description

In recent versions, numpy and scipy require Cython during their setup, and zc.recipe.egg:custom's setup-eggs option doesn't work because in their setup.py, Cython is launched in a subprocess, and the sys.path does not propagate.

Manually setting the PYTHONPATH with the environment option of zc.recipe.egg:custom is not ideal because it requires "guessing" where the eggs will be installed, and this depends on the current version of the eggs, of Python, on the system etc.

And numpy and scipy must be upgraded to support Python3.9, since older versions are incompatible.

Proposed solution

We inject the PYTHONPATH of the required eggs into the environment of zc.recipe.egg:custom during the install phase of buildout (zc.recipe.egg:custom accepts an environment option to link another buildout section whose keys/values will be the environment variables).

In the install phase, the PYTHONPATH can be computed automatically once the eggs are already installed.

Luckily, it just so happens that zc.recipe.egg:custom does not retrieve the contents of the environment section before the install phase, so there is a window of opportunity to 1) install the required eggs, 2) compute and inject the PYTHONPATH in the python dict of the environment section using slapos.recipe.build, 3) install numpy/scipy.

This precise ordering is achieved through carefully chained dependencies, and the logic to compute the PYTHONPATH and inject it is factored into a new macro macro.pythonpath.eggs in component/macros.

Note that we cannot simply expose the computed PYTHONPATH as a key/value and set it in the environment section by value substitution, as in:

[numpy-env]
PYTHONPATH = ${numpy-pythonpath:pythonpath}

Because the substitution of ${numpy-pythonpath:pythonpath} takes place during the init phase, and at that point the eggs are not installed yet and the PYTHONPATH cannot be computed yet.

Edited Dec 13, 2022 by Xavier Thompson
Assignee
Assign to
Reviewer
Request review from
None
Milestone
None
Assign milestone
Time tracking
Source branch: feat/upgrade_numpy
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7