1. 13 Nov, 2024 36 commits
    • Xavier Thompson's avatar
      [fix] Sign develop eggs with directory hash · 281dc5d1
      Xavier Thompson authored
      Mark .dist-info eggs installed by buildout as develop to distinguish
      them from .dist-info eggs from the system and decide how to sign the
      egg accordingly (with directory hash or with version).
      281dc5d1
    • Xavier Thompson's avatar
      [test] Fix tests after PEP 625 · 06ee62e8
      Xavier Thompson authored
      Adapt tests to ignore warning generated by setuptools not finding
      zc.recipe.egg on first index lookup, because it's indexed as
      zc-recipe-egg, because since PEP 625 the sdist filename is
      zc_recipe_egg-xyz.tar.gz.
      06ee62e8
    • Xavier Thompson's avatar
      [wkrd] Workaround package index after PEP 625 · 310d1e28
      Xavier Thompson authored
      Since PEP 625, sdist filenames replace . and - in the project name by
      _, such as e.g. zc.buildout==3.0.1 yielding zc_buildout-3.0.1.tar.gz.
      
      However, when looking up available dists in a package index, when an
      sdist with such a normalized filename is found, setuptools interprets
      the filename by replacing _ with - and uses that as the dist's name;
      e.g. zc_buildout-3.0.1.tar.gz yields zc-buildout. This causes lookups
      for the name with . (e.g.: zc.buildout) to fail.
      
      Workaround by also looking up the name with . replaced by _, and then
      fixing the names of found dists back to expected name.
      310d1e28
    • Xavier Thompson's avatar
      85add0a0
    • Xavier Thompson's avatar
      0bcb1ed8
    • Xavier Thompson's avatar
      [wkrd] Use pip install --editable --user · 4868bf4e
      Xavier Thompson authored
      Prior to pip 21.1, pip install --editable --target fails because it
      results in wrong parameters being passed to setup.py develop by pip.
      
      Prior to setuptools 45.2.0, both pip install --editable --target and
      pip install --editable --prefix fail because the temporary install
      directory used internally by pip is not added to PYTHONPATH prior
      to pip calling setup.py develop. In later version setuptools emits a
      warning instead of an error.
      
      Temporarily override PYTHONUSERBASE to point to the target directory,
      so as to emulate --prefix=<dir> with PYTHONUSERBASE=<dir> and --user.
      
      This is needed for Python2 because pip 21.1 and setuptools 45.2.0 are
      both Python3 only.
      4868bf4e
    • Xavier Thompson's avatar
      [tool] Gitignore *.dist-info · b6993b79
      Xavier Thompson authored
      b6993b79
    • Xavier Thompson's avatar
      [feat] Use pip install --editable in easy_install.develop · ef843325
      Xavier Thompson authored
      Instead of running python setup.py develop directly. This will allow
      using zc.buildout.easy_install.develop on recent projects that have
      only a pyproject.toml. It also fixes develop leaving build artifacts
      in the source directory that caused later runs to do nothing, e.g.
      preventing develop-eggs to be rebuilt when a build dependency passed
      in setup-eggs option of zc.recipe.egg:develop changed.
      
      A verbosity parameter to tune verbosity of pip is added, with adjusted
      values for the case of buildout:develop and of zc.recipe.egg:develop,
      so as to remain close to the previous behavior with regards to logs.
      
      Technical details:
      
      For packages using PEP-660-style editable installs, supported by more
      recent versions of pip, pip will not delegate to `setup.py develop` -
      enabling editable installs for pure pyproject.toml projects - and will
      instead generate a .dist-info metadata folder but not a .egg-link.
      
      Since buildout currently requires a .egg-link, as it does not support
      PEP 660's mechanism that relies on having a sites-packages directory,
      we need to create this .egg-link after the fact. The tricky part is
      finding out where the .egg-link should point: the path containing the
      pyproject.toml, or a subdirectory?
      
      For this we make use of *.dist-info/top_level.txt to first determine
      the list of top-level packages, and then importlib to extract info
      from the PEP-660-style install.
      
      If top_level.txt does not exist, is empty, or otherwise does not list
      any package that resolves to an import path, fallback to the path of
      the folder that contains the pyproject.toml as the .egg-link target.
      If it lists multiple packages that resolve to different import paths,
      arbitrarily use the first one and emit a warning.
      
      Support namespace packages where `spec.submodule_search_locations` is
      a `_NamespacePath` object instead of a simple `list` and also support
      cases where the layout of the source project does not follow the same
      structure as the package tree - meaning some custom magic might be
      involved in making editable imports work as intended.
      ef843325
    • Xavier Thompson's avatar
      [feat] zc.recipe.egg: Reinstall when setup-eggs versions change · 0681728b
      Xavier Thompson authored
      Trigger uninstall + install of eggs installed with zc.recipe.egg:custom
      or :develop when pinned versions of setup-eggs have changed. To achieve
      this the versions of setup-eggs are included in the section: this makes
      them part of its signature so that when they change, buildout will call
      `uninstall` and `install` for this section instead of just `update`.
      
      Unlike other zc.recipe.egg entry points, :custom stores the path of the
      installed egg; thus `uninstall` will remove it fully, leaving `install`
      to reinstall it cleanly from scrach.
      
      In the case of :develop, `uninstall` matters little as only the path of
      the installed `.egg-link` is stored. Instead `install` must be fixed to
      actually rebuild the egg in-place in the source directory and `develop`
      should do nothing.
      
      The main issue lies in `zc.buildout.easy_install.develop`: depending on
      the build process, it may leave build artifacts in the source directory
      that cause future runs to do nothing.
      0681728b
    • Xavier Thompson's avatar
      [dev] Change local versions to +slapos002 · b2a0e91c
      Xavier Thompson authored
      See merge request !31
      
      Note: all changes with +slapos001 are tests-related.
      
      - Fixup "[test] Add tests for build dependencies"
      - Fixup "[feat] zc.recipe.egg: Improve on the fly patches."
      - Fixup "[fix] Fix working set sorting".
      - Fixup "[feat] Add dependencies in __buildout_signature__"
      b2a0e91c
    • Xavier Thompson's avatar
      [fix] Fix invalid specifier in test · 61a6d7bc
      Xavier Thompson authored
      Pinning zc.buildout = >.1 is now invalid, so use >0.1 instead.
      61a6d7bc
    • Xavier Thompson's avatar
      [fix] Fix pip Py2 deprecation filter in tests · eedda83d
      Xavier Thompson authored
      Adapt filter in tests for pip emitting Python2.7 deprecation warnings
      to pip >= 20.2.2 and < 21.0 - these versions seem to accidentally add
      a redundant bit of message to the warning.
      eedda83d
    • Xavier Thompson's avatar
      [fix] Ignore setuptools deprecation warnings · 74dbc10c
      Xavier Thompson authored
      The testing framework, some individual tests and 'buildout:develop'
      call python setup.py <command> manually, which is deprecated by
      setuptools. Depending on the setuptools version, suppress the
      warning for now to avoid polluting the logs.
      74dbc10c
    • Xavier Thompson's avatar
      [fix] Fix logging filters for Python2 · 638f1985
      Xavier Thompson authored
      638f1985
    • Xavier Thompson's avatar
      35b7a793
    • Xavier Thompson's avatar
      [fix] Adapt to setuptools>=65.6.0 logging on root · d92d890f
      Xavier Thompson authored
      INFO logs from setuptools which were previously not emitted because
      setuptool's ad-hoc legacy logger defaulted to WARNING and above may
      now be emitted because setuptools now logs to the root logger, thus
      the global root logger's level configuration applied.
      
      This caused undesired 'root: <some setuptools info>' messages to be
      emitted and caused many tests to fail due to unexpected outputs.
      d92d890f
    • Xavier Thompson's avatar
      [dev] Change local versions to +slapos001 · 56a946c6
      Xavier Thompson authored
      See merge request !30
      56a946c6
    • Julien Muchembled's avatar
    • Julien Muchembled's avatar
      [feat] download: add support for slapos.libnetworkcache · c1320b90
      Julien Muchembled authored
      When specifying an alternate URL as fallback, the main URL is always
      used for both downloading & uploading from/to networkcache.
      c1320b90
    • Xavier Thompson's avatar
      [feat] Propagate libnetworkcache installation · 97c76d6c
      Xavier Thompson authored
      If slapos.libnetworkcache is importable, install it in bootstrap and
      in buildout upgrade - the places where bin/buildout is (re)generated
      - as though it were a dependency of zc.buildout.
      
      This is a hack to propagate libnetworkcache as a soft dependency.
      97c76d6c
    • Julien Muchembled's avatar
      [feat] download: add netrc file support · 461e4ae2
      Julien Muchembled authored
      Like for URL that contain credentials, we still skip auth challenge
      because it's faster and:
      - we only support one auth scheme (basic)
      - netrc provides no way to specify realms, which seem anyway to be
        less and less used (https://stackoverflow.com/q/69303610 reports
        that recent browsers don't display them anymore)
      
      See merge request !25
      461e4ae2
    • Julien Muchembled's avatar
      [feat] Extend Download API to use an alternate URL as fallback · 0c9cd8a3
      Julien Muchembled authored
      This retries with the alternate URL in case of HTTPError with the main
      one.
      
      Used by slapos.recipe.build:download* and slapos.recipe.cmmi recipes.
      0c9cd8a3
    • Julien Muchembled's avatar
      [fix] Rewrite 'urlretrieve' helper to fix various download-related issues · 5887c78c
      Julien Muchembled authored
      - Py3: stop using legacy API of urllib.request and
             fix download of http(s) URLs containing user:passwd@
      - Py2: avoid OOM when downloading huge files
      
      This is implemented as a method in case we want to make it configurable
      via [buildout].
      5887c78c
    • Julien Muchembled's avatar
      [fix/opti] download: clean-up, fix, optimization · 04cb6305
      Julien Muchembled authored
      An optimization is to avoid temporary file when possible: a rename
      (or hard link) is not always possible (different mount points).
      
      Another one is to not check md5sum twice when using cache file.
      
      Fall-back mode is ignored if an MD5 checksum is given.
      
      In case of checksum mismatch for a cached path, remove it and
      download again, mainly to cover the following cases:
      - the url content changes and the user updates the checksum
      - buildout killed while downloading directly to cache
        (see above optimization)
      - shutil.copyfile is interrupted
      04cb6305
    • Jérome Perrin's avatar
      [fixup] Ignore _profile_base_location_ when computing signatures · 706971fb
      Jérome Perrin authored
      We want two identical sections at different URL to be able to produce
      same signature. This feature is useful for slapos.recipe.cmmi's
      shared parts.
      
      This commit may fixup "Support ${:_profile_base_location_}." but is
      purposefuly kept separate because it concerns parts signatures which
      are an orthogonal feature.
      706971fb
    • Kazuhiko Shiozaki's avatar
      1094e25e
    • Kazuhiko Shiozaki's avatar
      150fafe2
    • Xavier Thompson's avatar
      bebbac46
    • Xavier Thompson's avatar
      5f76700f
    • Xavier Thompson's avatar
      [fix] Support local version label for zc.buildout · 31b82814
      Xavier Thompson authored
      When there is no pinned version for zc.buildout itself, buildout
      adds a ">=<current-version>" requirement to prevent accidental
      downgrading. If the current version has a local version label,
      this produced an invalid version specifier. To fix this, only
      the public part of the current version is used.
      31b82814
    • Xavier Thompson's avatar
      [test] Add tests for build dependencies · 7583d73a
      Xavier Thompson authored
      7583d73a
    • Xavier Thompson's avatar
      [feat] Prevent pip installing setup_requires · 920359bc
      Xavier Thompson authored
      Use a special .pydistutils.cfg in a temporary HOME directory for
      the duration of the pip wheel run to prevent build dependencies
      specified in a setup_requires from being installed on the fly
      without respecting pinned versions.
      920359bc
    • Xavier Thompson's avatar
      [feat] Prevent pip installing build dependencies · ea7886ca
      Xavier Thompson authored
      By default pip installs build dependencies (e.g. setuptools, poetry)
      in a temporary folder and temporarily adds it to sys.path in order
      to proceed to build the distribution. But we want all distributions
      to be installed with buildout and respect pinned versions, so we aim
      to prevent pip from installing build dependencies.
      
      Instead, we will install the build dependencies first and pass them
      explicitly to zc.recipe.egg via the setup-eggs option.
      
      This commit prevents pip from installing the build dependencies listed
      by the `build-system.requires` key in the pyproject.toml file.
      
      It may prevent pip from installing PEP 517 dynamic build dependencies
      or setuptools' setup_requires dependencies.
      See https://peps.python.org/pep-0517/#get-requires-for-build-wheel
      ea7886ca
    • Rafael Monnerat's avatar
      [feat] Propagate sys.path as PYTHONPATH while develop · 5fb40747
      Rafael Monnerat authored
      While invoke setup.py certain eggs (like scikit-learn) launch
      cetain custom builds (for cython) using subprocess and sys.executable.
      
      This commit aims to preserve the sys.path over the runs, even if an
      egg is using subprocess with the same python to build a component of
      the egg.
      5fb40747
    • Kazuhiko Shiozaki's avatar
    • Kazuhiko Shiozaki's avatar
      [feat] zc.recipe.egg: Support on the fly patches. · 23421106
      Kazuhiko Shiozaki authored
      - Support on the fly patches in zc.recipe.egg by ``EGGNAME-patches``,
        ``EGGNAME-patch-options``, ``EGGNAME-patch-binary`` (or
        ``patch-binary``) and ``EGGNAME-patch-revision`` options.
      
      - Support on the fly patches in zc.recipe.egg:custom by ``patches``,
        ``patch-options``, ``patch-binary`` and ``patch-revision`` options.
        (options ``EGGNAME-*`` are also supported as well).
      
      Specified patches are automatically applied on required eggs as well.
      
      This fixes cache of patches.
      
      Clean-up + fix issue found at slapos!1674
      23421106
  2. 20 May, 2024 4 commits
    • Kirill Smelkov's avatar
      [feat] zc.recipe.egg: Support environment in :develop · 8a0eed18
      Kirill Smelkov authored
      Currently only zc.recipe.egg:custom supports setting environment
      variables, and zc.recipe.egg:develop does not.
      
      My motivation for allowing setting environment in :develop is
      wendelin.core
      
          https://lab.nexedi.cn/nexedi/slapos/blob/b5faab3b/component/wendelin.core/buildout.cfg
      
      There we have [wendelin.core] part which installs released egg from
      pypi, and [wendelin.core-dev] part which installs wendelin.core from
      its latest git version via zc.recipe.egg:develop .
      
      The problem is, wendelin.core for setup.py to work, needs git available,
      and with slapos we usually don't have git available on base system, so
      we build it by our own and do something like
      
          [wendelin.core-dev]
          recipe = zc.recipe.egg:develop
          environment = wendelin.core-dev-env
      
          [wendelin.core-dev-env]
          # wendelin.core-dev needs git to build
          PATH = ${git:location}/bin:%(PATH)s
      
      and the problem is environment does not currently work for
      zc.recipe.egg:develop, and thus git is not found -> build fails.
      
      ~~~~
      
      In order to support environment in :develop, we just move environment
      setting/restoring bits from Custom to Base, and provide Base.install() which
      uses this bits. Custom & Develop .install() becomes ._install() which gets
      hooked into Base.install() .
      
      I've tested the patch only manually, because currently automated tests are
      broken in a lot of places for slapos.buildout and zc.recipe.egg .
      
      /cc @kazuhiko, @Tyagov
      8a0eed18
    • Kazuhiko Shiozaki's avatar
    • Xavier Thompson's avatar
      [feat] Raise an exception when pip fails · 07595d60
      Xavier Thompson authored
      Before this, sys.exit(1) was called to terminate the program directly.
      07595d60
    • Julien Muchembled's avatar
      [feat] Conditional sections: expose new 'multiarch' value in sys · 6109e381
      Julien Muchembled authored
      To be dropped once all buildout in the wild are able to upgrade to a
      version that supports new names in expression of conditional sections
      (see previous commit).
      6109e381