Commit eb5c6047 authored by Arnaud Fontaine's avatar Arnaud Fontaine

RFC: Make maintenance of Zope eggs versions by pulling them from upstream.

I don't see any reason why we don't use upstream zope *versions.cfg files. This
just duplicates work and makes maintenance tedious especially now that we have
zope2, zope4py2 and zope4py3.

Also:
  * Remove pinned versions on eggs with newer version or already defined in
    Zope versions.cfg files.
  * Make sure that pinned versions in stack/slapos.cfg do not override Zope versions
    (for example ZConfig versions set in stack/slapos.cfg was creating a version
    conflict while installing another egg).
parent e2b53e1f
This commit is part of merge request !1122. Comments created here will be created in the context of that merge request.
[buildout]
extends =
# Exact version of Zope
ztk-versions.cfg
zope-versions.cfg
buildout.hash.cfg
# This has to be before Zope versions.cfg otherwise eggs such as ZConfig have
# their versions overridden when slapos.cfg is extends by some component...
../../stack/slapos.cfg
https://raw.githubusercontent.com/zopefoundation/Zope/4.5.3/versions.cfg
  • This isn't a good idea, since this url on extends isn't cacheable on shacache (or anywhere else). So, if we do have problems to access it, the software release won't work anymore.

    /cc @tomo @Daetalus @romain

  • ah, this was also my first impression ( there's also discussion below BTW ), but then I thought that because it uses zc.buildout.download.Download it would automatically use shacache.

    extends are downloaded here: https://lab.nexedi.com/nexedi/slapos.buildout/blob/78ced4136882b38355642a2b017c9c7de993d4b9/src/zc/buildout/buildout.py#L1929-1932

    Even if it uses shacache, what I am not so comfortable with is that we cannot use md5sum mechanism here, I tried this:

    [buildout]
    extends = https://raw.githubusercontent.com/zopefoundation/Zope/4.5.3/versions.cfg#91f3d70f6ffd114f23f1e71f170a81dx

    91f3d70f6ffd114f23f1e71f170a81dx is not the md5sum of this profile, but there's no error, which means this is ignored.

Please register or sign in to reply
../../component/fonts/buildout.cfg
../../component/git/buildout.cfg
../../component/ghostscript/buildout.cfg
......@@ -676,9 +677,7 @@ Zope = 4.5.3+SlapOSPatched002
pylint = 1.4.4
# use newer version than specified in ZTK
PasteDeploy = 1.5.2
argparse = 1.4.0
zope.dottedname = 4.1.0
# modified version that works fine for buildout installation
SOAPpy = 0.12.0nxd001
......@@ -719,7 +718,6 @@ WSGIUtils = 0.7
astroid = 1.3.8
erp5diff = 0.8.1.7
five.formlib = 1.0.4
five.localsitemanager = 2.0.5
google-api-python-client = 1.6.1
httplib2 = 0.10.3
huBarcode = 1.0.0
......@@ -761,8 +759,6 @@ zbarlight = 2.3
cloudpickle = 0.5.3
dask = 0.18.1
toolz = 0.9.0
zope.globalrequest = 1.5
waitress = 1.4.4
xlrd = 1.1.0
# Re-add for as it is required to be there for uninstallation
......@@ -791,20 +787,12 @@ jedi = 0.15.1
parso = 0.5.1
yapf = 0.28.0
z3c.etestbrowser = 3.0.1
zope.testbrowser = 5.5.1
WSGIProxy2 = 0.4.6
WebTest = 2.0.33
beautifulsoup4 = 4.8.2
WebOb = 1.8.5
soupsieve = 1.9.5
eggtestinfo = 0.3
oic = 0.15.1
Beaker = 1.11.0
Mako = 1.1.4
pyjwkest = 1.4.2
alabaster = 0.7.12
future = 0.18.2
pycryptodomex = 3.10.1
strict-rfc3339 = 0.7
......@@ -814,8 +802,6 @@ jsonpointer = 2.2
# WIP Zope 4 ⚠
zope.interface = 5.2.0
ZConfig = 3.5.0
Products.CMFCore = 2.4.0
Products.StandardCacheManagers = 4.1.0
Products.ZSQLMethods = 3.14
......
[versions]
#Zope = 4.5.3
Zope2 = 4.0
# AccessControl 5+ no longer supports Zope 4.
AccessControl = 4.2
Acquisition = 4.7
AuthEncoding = 4.2
BTrees = 4.7.2
Chameleon = 3.8.1
DateTime = 4.3
# DocumentTemplate 4+ no longer supports Zope 4.
DocumentTemplate = 3.4
ExtensionClass = 4.5.0
Missing = 4.1
MultiMapping = 4.1
Paste = 3.5.0
PasteDeploy = 2.1.1
Persistence = 3.0
Products.BTreeFolder2 = 4.2
# ZCatalog 6+ no longer supports Zope 4.
Products.ZCatalog = 5.2
Record = 3.5
RestrictedPython = 5.1
WSGIProxy2 = 0.4.6
WebOb = 1.8.6
WebTest = 2.0.35
ZConfig = 3.5.0
ZEO = 5.2.2
ZODB = 5.6.0
ZServer = 4.0.2
five.globalrequest = 99.1
five.localsitemanager = 3.2.2
funcsigs = 1.0.2
future = 0.18.2
ipaddress = 1.0.23
# mock 4.0 and up requires Python 3
mock = 3.0.5
pbr = 5.5.1
persistent = 4.6.4
pytz = 2020.4
roman = 3.3
shutilwhich = 1.1.0
six = 1.15.0
transaction = 3.0.0
waitress = 1.4.4
z3c.pt = 3.3.0
zExceptions = 4.1
zc.lockfile = 2.0
zdaemon = 4.3
zodbpickle = 2.0.0
zope.annotation = 4.7.0
zope.browser = 2.3
zope.browsermenu = 4.4
zope.browserpage = 4.4.0
zope.browserresource = 4.4
zope.cachedescriptors = 4.3.1
zope.component = 4.6.2
zope.componentvocabulary = 2.2.0
zope.configuration = 4.4.0
zope.container = 4.4.0
zope.contentprovider = 4.2.1
zope.contenttype = 4.5.0
zope.datetime = 4.2.0
zope.deferredimport = 4.3.1
zope.deprecation = 4.4.0
zope.dottedname = 4.3
zope.event = 4.5.0
zope.exceptions = 4.4
zope.filerepresentation = 5.0.0
zope.formlib = 4.7.1
zope.globalrequest = 1.5
zope.hookable = 5.0.1
zope.i18n = 4.7.0
zope.i18nmessageid = 5.0.1
zope.interface = 5.2.0
zope.lifecycleevent = 4.3.0
zope.location = 4.2
zope.pagetemplate = 4.5.0
zope.processlifetime = 2.3.0
zope.proxy = 4.3.5
zope.ptresource = 4.2.0
zope.publisher = 5.2.1
zope.ramcache = 2.3
zope.schema = 6.0.0
zope.security = 5.1.1
zope.sendmail = 5.1
zope.sequencesort = 4.1.2
zope.site = 4.4.0
zope.size = 4.3
zope.structuredtext = 4.3
zope.tal = 4.4
zope.tales = 5.1
zope.testbrowser = 5.5.1
# Version 4.8+ dropped support for Python 3.5
zope.testing = 4.7
zope.testrunner = 5.2
zope.traversing = 4.4.1
zope.viewlet = 4.2.1
[buildout]
extends =
zope-versions.cfg
versions = versions
[versions]
# Version pins for development and optional dependencies.
Babel = 2.8.1
Jinja2 = 2.11.2
MarkupSafe = 1.1.1
# Pygments 2.6.0 and up require Python 3
Pygments = 2.5.2
# Version 2.0+ needs Python 3.x
Sphinx = 1.8.5
alabaster = 0.7.12
appdirs = 1.4.4
attrs = 20.3.0
backports.functools-lru-cache = 1.6.1
beautifulsoup4 = 4.9.3
bleach = 3.2.1
buildout.wheel = 0.2.0
# Version 2020.4.5.2 and up claim no Python 2 support
certifi = 2020.4.5.1
cffi = 1.14.3
chardet = 3.0.4
cmarkgfm = 0.4.2
collective.recipe.cmd = 0.11
collective.recipe.sphinxbuilder = 1.1
collective.recipe.template = 2.1
colorama = 0.4.4
# configparser 5 and up require Python 3
configparser = 4.0.2
contextlib2 = 0.6.0.post1
coverage = 5.3
distlib = 0.3.1
docutils = 0.16
filelock = 3.0.12
idna = 2.10
imagesize = 1.2.0
# tox and pluggy require importlib-metadata <1
importlib-metadata = 0.23
lxml = 4.6.1
manuel = 1.10.1
# Version 6+ needs Python 3.x
more-itertools = 5.0.0
mr.developer = 2.0.1
nose = 1.3.7
packaging = 20.4
pathlib2 = 2.3.5
pip = 20.2.4
pkginfo = 1.6.1
plone.recipe.command = 1.1
pluggy = 0.13.1
py = 1.9.0
pycparser = 2.20
pyparsing = 2.4.7
python-gettext = 4.0
readme-renderer = 28.0
repoze.sphinx.autointerface = 0.8
requests = 2.25.0
requests-toolbelt = 0.9.1
scandir = 1.10.0
snowballstemmer = 2.0.0
# soupsieve 2 needs Python 3
soupsieve = 1.9.6
sphinx-rtd-theme = 0.5.0
sphinxcontrib-serializinghtml = 1.1.4
sphinxcontrib-websupport = 1.2.4
# tempstorage is only needed because the Sphinx documentation parses
# ZConfig xml configurations, which still contain references to it
tempstorage = 5.1
toml = 0.10.2
tox = 3.20.1
tqdm = 4.51.0
# Version 2+ needs Python 3.x
twine = 1.15.0
typing = 3.7.4.3
urllib3 = 1.26.1
virtualenv = 20.1.0
webencodings = 0.5.1
wheel = 0.35.1
z3c.checkversions = 1.2
zc.recipe.egg = 2.0.7
zc.recipe.testrunner = 2.1
zest.releaser = 6.22.1
# Version 2 requires Python 3
zipp = 1.1.1
zodbupdate = 1.5
  • We use to extends from github directly and we stopped in 8f222b41 (without explanation on commit message, but maybe @kazuhiko remembers ?).

    By having the profiles fetched from an external source, there's a risk that one day they are not available or that they became different. I don't think networkcache applies for buildout extends and buildout extends are not verified (there's no md5sum).

    By having the profiles copied in the repository, it's a little bit easier to look at the history (that point is not really important)

    My preference would be to continue copying these profiles in slapos repository. If it helps, maybe we can use different filenames ( something like Zope-4.5.3-versions.cfg )

    BTW, this seem to break the build because we use a different version of zc.buildout

  • Thanks for the review ! I don't have a strong preference between an URL or a copy, except that it was not obvious to me that this was a copy from Zope buildout and I started to do modifications... So let's wait for @kazuhiko to reply then (@kazuhiko: it would have been nice to write an useful commit message BTW).

  • The reason why I stopped direct extend in 8f222b41 is simply because versions.cfg was no longer included in repository in 2.13.23 or later in 2.13 branch, by https://github.com/zopefoundation/Zope/commit/56b6790d1fd3a6e0105925adcc7cca29e9ab21cf

    Edited by Kazuhiko Shiozaki
  • For reference: @arnau and I discussed a bit and there were not clearly a "best way" of handling zope versions.

    Using zope versions like ( https://raw.githubusercontent.com/zopefoundation/Zope/4.5.3/versions.cfg ) is good because we use the "canonical source of information", so it's easy to look for new version. One problem is that we don't use exactly these versions.

    Sometimes slapos.cfg uses a different version ( zc.buildout of course, but also ZConfig = 2.9.3 or zope.interface = 5.2.0 ) sometimes ERP5 uses a different version from Zope: for security reasons like 571e78f2 or because we need to patch, like with Acquisition.

    We could use versions from github and have explicit overrides in stack/erp5/buildout.cfg, but there are two questions:

    I hope I summarized correctly, I wanted to update also because my first comment sounds like I'm against this, but I'm not against.

  • mentioned in merge request !1122 (merged)

    Toggle commit list
  • mentioned in commit 4fa33dfc

    Toggle commit list
  • In a7e370de we updated Zope, I have used some curl commands to download zope profiles https://lab.nexedi.com/nexedi/slapos/blob/a7e370de393797dbb1f373622acfacdf51b36ed1/stack/erp5/buildout.cfg#L4-6

    # versions pins from zope, vendored with
    # curl https://zopefoundation.github.io/Zope/releases/4.8.2/versions.cfg | sed -e s/versions-prod.cfg/zope-versions.cfg/g > ztk-versions.cfg
    # curl https://zopefoundation.github.io/Zope/releases/4.8.2/versions-prod.cfg > zope-versions.cfg

    there are still many open questions, but I have noted there the procedure used

  • mentioned in merge request !1332 (merged)

    Toggle commit list
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment