Commits · t · Kirill Smelkov / nxd-bom

07 Jun, 2023 3 commits

Kirill Smelkov authored Jun 07, 2023

* master: (43 commits)
  Also detect eggs that are used indirectly
  Fix reporting of multiple detected eggs
  namever: Treat rfcXXX as non-version
  fmt: Sort emitted entries ignoring their case
  Mark generic_testrunner_init as synthethic setction
  Fix namever for findutils url
  Fix support for HDF5 URL
  Fix support for libstemmer_c
  Fix support for onlyoffice_core URL
  Cloudooo uses both py2 and py3 simultaneously
  Fix support for icu4c url
  Assert that ver != ø in addbom
  Cloudooo also uses libpng1.2 and libpng1.6 simultaneously
  Fix handling of CUPS url
  Fix for LibreOffice
  Refer to nexedi.org/static if egg version has slaposXXX
  Add support for collective.recipe.template
  Fix handling of non-py2 eggs
  isconf += .cfg.jinja2
  Fix handling GeoLite2 URLs
  ...

f4d899e9

Improvements to support ERP5 · 89548b01

Kirill Smelkov authored Jun 07, 2023

Turn nxdbom into full egg with testsuite and incrementally enhance it step-by-step so that it could work ok on built ERP5 SR.

Example output is attached bellow.

Note: information about dependent Go modules, Ruby gems and JS NPM
packages are currently _not_ emitted(*)

/reviewed-by @jerome
/reviewed-on kirr/nxd-bom!1

(*) see kirr/nxd-bom!1 (comment 185412)
for details.

Appendix. BOM(ERP5)
-------------------

>>> gits:
erp5 d496e223d23b611c3835561de044047cab2d53d8 https://lab.nexedi.com/nexedi/erp5.git
erp5-bin 13d3ed05f0f07a53786fdf1ec0855a94ba1acdfc https://lab.nexedi.com/nexedi/erp5-bin.git
erp5-doc 3996ae1f832ca438ead9fbcf0ab8082dd1d22021 https://lab.nexedi.com/nexedi/erp5-doc.git
neoppod fd87e153d3906c9120bbb087827ee4666dc388e7 https://lab.nexedi.com/nexedi/neoppod.git
wendelin.core 2.0.alpha3-0-g6315384 https://lab.nexedi.com/nexedi/wendelin.core.git

89548b01

Also detect eggs that are used indirectly · 950b65ce

Kirill Smelkov authored Jun 07, 2023

@jerome reports that BOM(ERP5) does not contain `certifi` even though it
is installed and used:

kirr/nxd-bom!1 (comment 185323)

The problem is: we were emitting information only about eggs that are
explicitly listed in the recipes, but dependent eggs were not reported
at all.

-> Fix this by using scripts, that buildout installs, as another root to
scan for eggs from. This works because zc.recipe.eggs puts all installed
eggs into sys.path adjustement inside those scripts.

On kirr/nxd-bom!1 (comment 185331) I
made wrong statement that .installed.cfg does not contain information
about which scripts were installed. This is not true: zc.recipe.egg puts
generated scripts into __buildout_installed__. This makes those scripts
to be reliable source of truth.

The diff for BOM(ERP5) before and after this patch is as follows:

--- d	2023-06-06 09:51:35.800234166 +0200
+++ e	2023-06-07 14:11:54.646892623 +0200
@@ -189,42 +189,91 @@
 zstd                         1.4.9      https://github.com/facebook/zstd/releases/download/v1.4.9/zstd-1.4.9.tar.gz

 >>> eggs:
+AccessControl                4.3        https://pypi.org/project/AccessControl/4.3/
 Acquisition                  4.7+slapospatched001 https://pypi.org/project/Acquisition/4.7/
+alabaster                    0.7.12     https://pypi.org/project/alabaster/0.7.12/
 APacheDEX                    1.8        https://pypi.org/project/APacheDEX/1.8/
 astor                        0.5        https://pypi.org/project/astor/0.5/
 astroid                      1.3.8+slapospatched001 https://pypi.org/project/astroid/1.3.8/
+atomize                      0.2.0      https://pypi.org/project/atomize/0.2.0/
+attrs                        18.2.0     https://pypi.org/project/attrs/18.2.0/
+AuthEncoding                 4.3        https://pypi.org/project/AuthEncoding/4.3/
+backports.functools_lru_cache 1.6.1      https://pypi.org/project/backports.functools_lru_cache/1.6.1/
 backports.lzma               0.0.14     https://pypi.org/project/backports.lzma/0.0.14/
+backports.shutil_get_terminal_size 1.0.0      https://pypi.org/project/backports.shutil_get_terminal_size/1.0.0/
+backports_abc                0.5        https://pypi.org/project/backports_abc/0.5/
 bcrypt                       3.1.4      https://pypi.org/project/bcrypt/3.1.4/
+Beaker                       1.11.0     https://pypi.org/project/Beaker/1.11.0/
+beautifulsoup4               4.8.2      https://pypi.org/project/beautifulsoup4/4.8.2/
 BTrees                       4.11.3     https://pypi.org/project/BTrees/4.11.3/
+CacheControl                 0.12.6     https://pypi.org/project/CacheControl/0.12.6/
 caucase                      0.9.14     https://pypi.org/project/caucase/0.9.14/
+certifi                      2020.4.5.1 https://pypi.org/project/certifi/2020.4.5.1/
 cffi                         1.15.0     https://pypi.org/project/cffi/1.15.0/
+Chameleon                    3.9.1      https://pypi.org/project/Chameleon/3.9.1/
 chardet                      3.0.4      https://pypi.org/project/chardet/3.0.4/
+click                        6.7        https://pypi.org/project/click/6.7/
+cliff                        2.8.3      https://pypi.org/project/cliff/2.8.3/
+cloudpickle                  0.5.3      https://pypi.org/project/cloudpickle/0.5.3/
+cmd2                         0.7.0      https://pypi.org/project/cmd2/0.7.0/
 cns.recipe.symlink           0.2.3      https://pypi.org/project/cns.recipe.symlink/0.2.3/
 collective.recipe.template   2.0        https://pypi.org/project/collective.recipe.template/2.0/
+cookies                      2.2.1      https://pypi.org/project/cookies/2.2.1/
 coverage                     5.5        https://pypi.org/project/coverage/5.5/
+croniter                     0.3.25     https://pypi.org/project/croniter/0.3.25/
 cryptography                 3.3.2      https://pypi.org/project/cryptography/3.3.2/
+cycler                       0.10.0     https://pypi.org/project/cycler/0.10.0/
 Cython                       0.29.24    https://pypi.org/project/Cython/0.29.24/
 cython_zstd                  0.2        https://pypi.org/project/cython_zstd/0.2/
+dask                         0.18.1     https://pypi.org/project/dask/0.18.1/
+dateparser                   0.7.6      https://pypi.org/project/dateparser/0.7.6/
+DateTime                     4.9        https://pypi.org/project/DateTime/4.9/
 decorator                    4.3.0      https://pypi.org/project/decorator/4.3.0/
 deepdiff                     3.3.0      https://pypi.org/project/deepdiff/3.3.0/
+distro                       1.6.0      https://pypi.org/project/distro/1.6.0/
+dnspython                    1.16.0     https://pypi.org/project/dnspython/1.16.0/
+DocumentTemplate             3.4        https://pypi.org/project/DocumentTemplate/3.4/
 docutils                     0.17.1     https://pypi.org/project/docutils/0.17.1/
+enum34                       1.1.10     https://pypi.org/project/enum34/1.1.10/
 erp5_coverage_plugin         0.0.1      https://pypi.org/project/erp5_coverage_plugin/0.0.1/
 erp5diff                     0.8.1.8    https://pypi.org/project/erp5diff/0.8.1.8/
+et_xmlfile                   1.0.1      https://pypi.org/project/et_xmlfile/1.0.1/
+ExtensionClass               4.9        https://pypi.org/project/ExtensionClass/4.9/
 facebook_sdk                 2.0.0      https://pypi.org/project/facebook_sdk/2.0.0/
 feedparser                   5.2.1      https://pypi.org/project/feedparser/5.2.1/
+five.localsitemanager        3.4        https://pypi.org/project/five.localsitemanager/3.4/
+Flask                        1.1.2      https://pypi.org/project/Flask/1.1.2/
+fpconst                      0.7.2      https://pypi.org/project/fpconst/0.7.2/
+funcsigs                     1.0.2      https://pypi.org/project/funcsigs/1.0.2/
+functools32                  3.2.3.post2 https://pypi.org/project/functools32/3.2.3.post2/
+future                       0.18.2     https://pypi.org/project/future/0.18.2/
+futures                      3.3.0      https://pypi.org/project/futures/3.3.0/
+gevent                       20.9.0     https://pypi.org/project/gevent/20.9.0/
+gitdb2                       2.0.5      https://pypi.org/project/gitdb2/2.0.5/
+GitPython                    2.1.11     https://pypi.org/project/GitPython/2.1.11/
 google_api_python_client     1.6.1      https://pypi.org/project/google_api_python_client/1.6.1/
+graphviz                     0.5.2      https://pypi.org/project/graphviz/0.5.2/
+greenlet                     0.4.17     https://pypi.org/project/greenlet/0.4.17/
 h5py                         2.7.1      https://pypi.org/project/h5py/2.7.1/
 haufe.requestmonitoring      0.6.0      https://pypi.org/project/haufe.requestmonitoring/0.6.0/
+html5lib                     1.1        https://pypi.org/project/html5lib/1.1/
 httplib2                     0.10.3     https://pypi.org/project/httplib2/0.10.3/
 huBarcode                    1.0.0      https://pypi.org/project/huBarcode/1.0.0/
+idna                         2.9        https://pypi.org/project/idna/2.9/
+Importing                    1.10       https://pypi.org/project/Importing/1.10/
 interval                     1.0.0      https://pypi.org/project/interval/1.0.0/
+ipaddress                    1.0.23     https://pypi.org/project/ipaddress/1.0.23/
 ipdb                         0.10.2     https://pypi.org/project/ipdb/0.10.2/
 ipykernel                    4.5.2      https://pypi.org/project/ipykernel/4.5.2/
 ipython                      5.3.0      https://pypi.org/project/ipython/5.3.0/
 ipython_genutils             0.1.0      https://pypi.org/project/ipython_genutils/0.1.0/
 ipywidgets                   6.0.0      https://pypi.org/project/ipywidgets/6.0.0/
+itsdangerous                 0.24       https://pypi.org/project/itsdangerous/0.24/
+jdcal                        1.3        https://pypi.org/project/jdcal/1.3/
 jedi                         0.15.1     https://pypi.org/project/jedi/0.15.1/
 Jinja2                       2.11.3     https://pypi.org/project/Jinja2/2.11.3/
+jsonpickle                   0.9.6      https://pypi.org/project/jsonpickle/0.9.6/
+jsonpointer                  2.2        https://pypi.org/project/jsonpointer/2.2/
 jsonschema                   3.0.2      https://pypi.org/project/jsonschema/3.0.2/
 jupyter                      1.0.0      https://pypi.org/project/jupyter/1.0.0/
 jupyter_client               5.0.0      https://pypi.org/project/jupyter_client/5.0.0/
@@ -233,19 +282,31 @@
 jupyterlab                   0.26.3     https://pypi.org/project/jupyterlab/0.26.3/
 jupyterlab_launcher          0.3.1      https://pypi.org/project/jupyterlab_launcher/0.3.1/
 lock_file                    2.0        https://pypi.org/project/lock_file/2.0/
+lockfile                     0.12.2     https://pypi.org/project/lockfile/0.12.2/
+logilab_common               1.3.0      https://pypi.org/project/logilab_common/1.3.0/
 lxml                         4.9.1      https://pypi.org/project/lxml/4.9.1/
+Mako                         1.1.4      https://pypi.org/project/Mako/1.1.4/
+MarkupSafe                   1.0        https://pypi.org/project/MarkupSafe/1.0/
 matplotlib                   2.1.2      https://pypi.org/project/matplotlib/2.1.2/
 mechanize                    0.4.8      https://pypi.org/project/mechanize/0.4.8/
+Missing                      4.2        https://pypi.org/project/Missing/4.2/
+mistune                      0.7.3      https://pypi.org/project/mistune/0.7.3/
 mock                         3.0.5      https://pypi.org/project/mock/3.0.5/
 more_itertools               5.0.0      https://pypi.org/project/more_itertools/5.0.0/
+mpmath                       0.19       https://pypi.org/project/mpmath/0.19/
 msgpack                      0.6.2      https://pypi.org/project/msgpack/0.6.2/
+MultiMapping                 4.1        https://pypi.org/project/MultiMapping/4.1/
+multipart                    0.1.1      https://pypi.org/project/multipart/0.1.1/
 munnel                       0.3        https://pypi.org/project/munnel/0.3/
 mysqlclient                  1.3.12     https://pypi.org/project/mysqlclient/1.3.12/
 nbconvert                    4.1.0      https://pypi.org/project/nbconvert/4.1.0/
 nbformat                     4.3.0      https://pypi.org/project/nbformat/4.3.0/
+netaddr                      0.7.19     https://pypi.org/project/netaddr/0.7.19/
+netifaces                    0.10.7     https://pypi.org/project/netifaces/0.10.7/
 networkx                     2.1        https://pypi.org/project/networkx/2.1/
 notebook                     4.4.1      https://pypi.org/project/notebook/4.4.1/
 numpy                        1.13.1     https://pypi.org/project/numpy/1.13.1/
+oauth2client                 4.0.0      https://pypi.org/project/oauth2client/4.0.0/
 oauthlib                     3.1.0      https://pypi.org/project/oauthlib/3.1.0/
 objgraph                     3.1.0      https://pypi.org/project/objgraph/3.1.0/
 ocropy                       1.0+slapospatched001 https://pypi.org/project/ocropy/1.0/
@@ -253,40 +314,72 @@
 openpyxl                     2.4.8      https://pypi.org/project/openpyxl/2.4.8/
 pandas                       0.24.2     https://pypi.org/project/pandas/0.24.2/
 paramiko                     2.11.0     https://pypi.org/project/paramiko/2.11.0/
+parso                        0.5.1      https://pypi.org/project/parso/0.5.1/
+passlib                      1.7.4      https://pypi.org/project/passlib/1.7.4/
+PasteDeploy                  2.1.1      https://pypi.org/project/PasteDeploy/2.1.1/
+pathlib2                     2.2.1      https://pypi.org/project/pathlib2/2.2.1/
 patsy                        0.4.1      https://pypi.org/project/patsy/0.4.1/
+pbr                          5.9.0      https://pypi.org/project/pbr/5.9.0/
+pem                          21.1.0     https://pypi.org/project/pem/21.1.0/
+Persistence                  3.6        https://pypi.org/project/Persistence/3.6/
 persistent                   4.9.3      https://pypi.org/project/persistent/4.9.3/
+pexpect                      4.2.1      https://pypi.org/project/pexpect/4.2.1/
+pickleshare                  0.7.4      https://pypi.org/project/pickleshare/0.7.4/
 Pillow                       6.2.2      https://pypi.org/project/Pillow/6.2.2/
 plone.recipe.command         1.1        https://pypi.org/project/plone.recipe.command/1.1/
 ply                          3.11       https://pypi.org/project/ply/3.11/
 polib                        1.0.8      https://pypi.org/project/polib/1.0.8/
 pprofile                     2.0.4      https://pypi.org/project/pprofile/2.0.4/
+prettytable                  0.7.2      https://pypi.org/project/prettytable/0.7.2/
+Products.BTreeFolder2        4.4+slapospatched001 https://pypi.org/project/Products.BTreeFolder2/4.4/
+Products.CMFCore             2.7.0+slapospatched001 https://pypi.org/project/Products.CMFCore/2.7.0/
 Products.DCWorkflow          2.4.1+slapospatched001 https://pypi.org/project/Products.DCWorkflow/2.4.1/
 Products.ExternalMethod      4.7        https://pypi.org/project/Products.ExternalMethod/4.7/
+Products.GenericSetup        2.3.0      https://pypi.org/project/Products.GenericSetup/2.3.0/
+Products.MailHost            4.13       https://pypi.org/project/Products.MailHost/4.13/
 Products.MimetypesRegistry   2.1.8      https://pypi.org/project/Products.MimetypesRegistry/2.1.8/
 Products.PluggableAuthService 2.8.1      https://pypi.org/project/Products.PluggableAuthService/2.8.1/
+Products.PluginRegistry      1.6        https://pypi.org/project/Products.PluginRegistry/1.6/
+Products.PythonScripts       4.15       https://pypi.org/project/Products.PythonScripts/4.15/
 Products.Sessions            4.15       https://pypi.org/project/Products.Sessions/4.15/
 Products.SiteErrorLog        5.7        https://pypi.org/project/Products.SiteErrorLog/5.7/
+Products.StandardCacheManagers 4.2        https://pypi.org/project/Products.StandardCacheManagers/4.2/
 Products.TIDStorage          5.5.0      https://pypi.org/project/Products.TIDStorage/5.5.0/
+Products.ZCatalog            5.4        https://pypi.org/project/Products.ZCatalog/5.4/
 Products.ZODBMountPoint      1.3        https://pypi.org/project/Products.ZODBMountPoint/1.3/
 Products.ZSQLMethods         3.16       https://pypi.org/project/Products.ZSQLMethods/3.16/
+prompt_toolkit               1.0.13     https://pypi.org/project/prompt_toolkit/1.0.13/
 psutil                       5.8.0      https://pypi.org/project/psutil/5.8.0/
+ptyprocess                   0.5.1      https://pypi.org/project/ptyprocess/0.5.1/
+pyasn1                       0.4.5      https://pypi.org/project/pyasn1/0.4.5/
+pyasn1_modules               0.0.8      https://pypi.org/project/pyasn1_modules/0.0.8/
 pycountry                    17.1.8     https://pypi.org/project/pycountry/17.1.8/
+pycparser                    2.20       https://pypi.org/project/pycparser/2.20/
 pycrypto                     2.6.1      https://pypi.org/project/pycrypto/2.6.1/
+pycryptodomex                3.10.1     https://pypi.org/project/pycryptodomex/3.10.1/
 pycurl                       7.43.0     https://pypi.org/project/pycurl/7.43.0/
 pydot                        1.4.2      https://pypi.org/project/pydot/1.4.2/
 pyflakes                     1.5.0      https://pypi.org/project/pyflakes/1.5.0/
+Pygments                     2.2.0      https://pypi.org/project/Pygments/2.2.0/
 pygolang                     0.1        https://pypi.org/project/pygolang/0.1/
+pyjwkest                     1.4.2      https://pypi.org/project/pyjwkest/1.4.2/
+PyJWT                        1.7.1      https://pypi.org/project/PyJWT/1.7.1/
 pylint                       1.4.4+slapospatched002 https://pypi.org/project/pylint/1.4.4/
 Pympler                      0.4.3      https://pypi.org/project/Pympler/0.4.3/
 PyMySQL                      0.10.1     https://pypi.org/project/PyMySQL/0.10.1/
 PyNaCl                       1.3.0      https://pypi.org/project/PyNaCl/1.3.0/
+pyOpenSSL                    19.1.0     https://pypi.org/project/pyOpenSSL/19.1.0/
+pyparsing                    2.2.0      https://pypi.org/project/pyparsing/2.2.0/
 PyPDF2                       1.26.0+slapospatched001 https://pypi.org/project/PyPDF2/1.26.0/
 pyrsistent                   0.16.1     https://pypi.org/project/pyrsistent/0.16.1/
 PyRSS2Gen                    1.1        https://pypi.org/project/PyRSS2Gen/1.1/
 PyStemmer                    1.3.0      https://pypi.org/project/PyStemmer/1.3.0/
 pysvn                        1.9.15+slapospatched001 https://pypi.org/project/pysvn/1.9.15/
 pytesseract                  0.2.2      https://pypi.org/project/pytesseract/0.2.2/
+python_dateutil              2.8.2      https://pypi.org/project/python_dateutil/2.8.2/
+python_gettext               4.1        https://pypi.org/project/python_gettext/4.1/
 python_ldap                  2.4.32+slapospatched001 https://pypi.org/project/python_ldap/2.4.32/
+python_libmilter             1.0.3      https://pypi.org/project/python_libmilter/1.0.3/
 python_magic                 0.4.12+slapospatched001 https://pypi.org/project/python_magic/0.4.12/
 python_memcached             1.58       https://pypi.org/project/python_memcached/1.58/
 pytracemalloc                1.2        https://pypi.org/project/pytracemalloc/1.2/
@@ -295,32 +388,68 @@
 PyYAML                       5.4.1      https://pypi.org/project/PyYAML/5.4.1/
 pyzmq                        16.0.2     https://pypi.org/project/pyzmq/16.0.2/
 qrcode                       5.3        https://pypi.org/project/qrcode/5.3/
+qtconsole                    4.3.0      https://pypi.org/project/qtconsole/4.3.0/
 Record                       3.6        https://pypi.org/project/Record/3.6/
+regex                        2020.9.27  https://pypi.org/project/regex/2020.9.27/
 requests                     2.27.1     https://pypi.org/project/requests/2.27.1/
 responses                    0.10.6     https://pypi.org/project/responses/0.10.6/
+RestrictedPython             5.2        https://pypi.org/project/RestrictedPython/5.2/
+rfc3987                      1.3.8      https://pypi.org/project/rfc3987/1.3.8/
+roman                        3.3        https://pypi.org/project/roman/3.3/
+rsa                          3.4.2      https://pypi.org/project/rsa/3.4.2/
+scandir                      1.5        https://pypi.org/project/scandir/1.5/
 scikit_image                 0.14.0     https://pypi.org/project/scikit_image/0.14.0/
 scikit_learn                 0.18.1     https://pypi.org/project/scikit_learn/0.18.1/
 scipy                        0.19.0     https://pypi.org/project/scipy/0.19.0/
 seaborn                      0.7.1      https://pypi.org/project/seaborn/0.7.1/
+selectors34                  1.2        https://pypi.org/project/selectors34/1.2/
 selenium                     3.141.0+slapospatched001 https://pypi.org/project/selenium/3.141.0/
 setproctitle                 1.1.10     https://pypi.org/project/setproctitle/1.1.10/
+setuptools                   44.1.1     https://pypi.org/project/setuptools/44.1.1/
+simplegeneric                0.8.1      https://pypi.org/project/simplegeneric/0.8.1/
+singledispatch               3.4.0.3    https://pypi.org/project/singledispatch/3.4.0.3/
+six                          1.16.0     https://pypi.org/project/six/1.16.0/
 slapos.cookbook              1.0.305    https://pypi.org/project/slapos.cookbook/1.0.305/
+slapos.core                  1.9.3      https://pypi.org/project/slapos.core/1.9.3/
 slapos.recipe.template       5.1        https://pypi.org/project/slapos.recipe.template/5.1/
 slapos.toolbox               0.128      https://pypi.org/project/slapos.toolbox/0.128/
+smmap2                       2.0.5      https://pypi.org/project/smmap2/2.0.5/
 SOAPpy                       0.12.0nxd001 http://www.nexedi.org/static/packages/source/SOAPpy-0.12.0nxd001.tar.gz
+soupsieve                    1.9.5      https://pypi.org/project/soupsieve/1.9.5/
 spyne                        2.12.14    https://pypi.org/project/spyne/2.12.14/
 statsmodels                  0.8.0      https://pypi.org/project/statsmodels/0.8.0/
+stevedore                    1.21.0     https://pypi.org/project/stevedore/1.21.0/
 strict_rfc3339               0.7        https://pypi.org/project/strict_rfc3339/0.7/
 StructuredText               2.11.1     https://pypi.org/project/StructuredText/2.11.1/
+subprocess32                 3.5.4      https://pypi.org/project/subprocess32/3.5.4/
 suds                         0.4        https://pypi.org/project/suds/0.4/
+supervisor                   4.1.0      https://pypi.org/project/supervisor/4.1.0/
 sympy                        1.1.1      https://pypi.org/project/sympy/1.1.1/
 tempstorage                  5.2        https://pypi.org/project/tempstorage/5.2/
+terminado                    0.6        https://pypi.org/project/terminado/0.6/
+toolz                        0.9.0      https://pypi.org/project/toolz/0.9.0/
+tornado                      4.4.2      https://pypi.org/project/tornado/4.4.2/
+traitlets                    4.3.3      https://pypi.org/project/traitlets/4.3.3/
+transaction                  3.0.1      https://pypi.org/project/transaction/3.0.1/
+trollius                     2.2.1      https://pypi.org/project/trollius/2.2.1/
 typing                       3.10.0.0   https://pypi.org/project/typing/3.10.0.0/
+tzlocal                      1.5.1      https://pypi.org/project/tzlocal/1.5.1/
+unicodecsv                   0.14.1     https://pypi.org/project/unicodecsv/0.14.1/
 unidiff                      0.5.5      https://pypi.org/project/unidiff/0.5.5/
+uritemplate                  3.0.0      https://pypi.org/project/uritemplate/3.0.0/
+urllib3                      1.26.12    https://pypi.org/project/urllib3/1.26.12/
 urlnorm                      1.1.4      https://pypi.org/project/urlnorm/1.1.4/
 uuid                         1.30       https://pypi.org/project/uuid/1.30/
 validictory                  1.1.0      https://pypi.org/project/validictory/1.1.0/
 waitress                     1.4.4+slapospatched006 https://pypi.org/project/waitress/1.4.4/
+wcwidth                      0.1.7      https://pypi.org/project/wcwidth/0.1.7/
+webcolors                    1.10       https://pypi.org/project/webcolors/1.10/
+webencodings                 0.5.1      https://pypi.org/project/webencodings/0.5.1/
+WebOb                        1.8.5      https://pypi.org/project/WebOb/1.8.5/
+WebTest                      2.0.33     https://pypi.org/project/WebTest/2.0.33/
+Werkzeug                     1.0.1      https://pypi.org/project/Werkzeug/1.0.1/
+widgetsnbextension           2.0.0      https://pypi.org/project/widgetsnbextension/2.0.0/
+WSGIProxy2                   0.4.6      https://pypi.org/project/WSGIProxy2/0.4.6/
 xfw                          0.10       https://pypi.org/project/xfw/0.10/
 xlrd                         1.1.0      https://pypi.org/project/xlrd/1.1.0/
 xml_marshaller               1.0.2      https://pypi.org/project/xml_marshaller/1.0.2/
@@ -328,16 +457,70 @@
 xupdate_processor            0.5        https://pypi.org/project/xupdate_processor/0.5/
 yapf                         0.28.0     https://pypi.org/project/yapf/0.28.0/
 z3c.etestbrowser             3.0.1      https://pypi.org/project/z3c.etestbrowser/3.0.1/
+z3c.pt                       3.3.1      https://pypi.org/project/z3c.pt/3.3.1/
 zbarlight                    2.3        https://pypi.org/project/zbarlight/2.3/
+zc.buildout                  2.7.1+slapos019 http://www.nexedi.org/static/packages/source/slapos.buildout/zc.buildout-2.7.1+slapos019.tar.gz
+zc.lockfile                  1.4        https://pypi.org/project/zc.lockfile/1.4/
+ZConfig                      3.6.1      https://pypi.org/project/ZConfig/3.6.1/
+zdaemon                      4.2.0      https://pypi.org/project/zdaemon/4.2.0/
 ZEO                          5.4.0      https://pypi.org/project/ZEO/5.4.0/
+zExceptions                  4.3        https://pypi.org/project/zExceptions/4.3/
 zLOG                         3.1        https://pypi.org/project/zLOG/3.1/
 ZODB                         5.8.0      https://pypi.org/project/ZODB/5.8.0/
+ZODB3                        3.11.0     https://pypi.org/project/ZODB3/3.11.0/
+zodbpickle                   2.6        https://pypi.org/project/zodbpickle/2.6/
 zodbtools                    0.0.0.dev8 https://pypi.org/project/zodbtools/0.0.0.dev8/
+zodburi                      2.5.0      https://pypi.org/project/zodburi/2.5.0/
 Zope                         4.8.7      https://pypi.org/project/Zope/4.8.7/
+zope.annotation              4.8        https://pypi.org/project/zope.annotation/4.8/
+zope.app.appsetup            4.2.0      https://pypi.org/project/zope.app.appsetup/4.2.0/
+zope.app.publication         4.5        https://pypi.org/project/zope.app.publication/4.5/
+zope.authentication          5.0        https://pypi.org/project/zope.authentication/5.0/
+zope.browser                 2.4        https://pypi.org/project/zope.browser/2.4/
+zope.browsermenu             4.4        https://pypi.org/project/zope.browsermenu/4.4/
+zope.browserpage             4.4.0      https://pypi.org/project/zope.browserpage/4.4.0/
+zope.browserresource         4.4        https://pypi.org/project/zope.browserresource/4.4/
+zope.cachedescriptors        4.4        https://pypi.org/project/zope.cachedescriptors/4.4/
+zope.component               5.0.1      https://pypi.org/project/zope.component/5.0.1/
+zope.configuration           4.4.1      https://pypi.org/project/zope.configuration/4.4.1/
 zope.container               4.10       https://pypi.org/project/zope.container/4.10/
+zope.contentprovider         4.2.1      https://pypi.org/project/zope.contentprovider/4.2.1/
+zope.contenttype             4.6        https://pypi.org/project/zope.contenttype/4.6/
+zope.datetime                4.3.0      https://pypi.org/project/zope.datetime/4.3.0/
+zope.deferredimport          4.4        https://pypi.org/project/zope.deferredimport/4.4/
+zope.deprecation             4.4.0      https://pypi.org/project/zope.deprecation/4.4.0/
+zope.dottedname              4.3        https://pypi.org/project/zope.dottedname/4.3/
+zope.error                   4.6        https://pypi.org/project/zope.error/4.6/
+zope.event                   4.6        https://pypi.org/project/zope.event/4.6/
+zope.exceptions              4.6        https://pypi.org/project/zope.exceptions/4.6/
+zope.filerepresentation      5.0.0      https://pypi.org/project/zope.filerepresentation/5.0.0/
 zope.globalrequest           1.6        https://pypi.org/project/zope.globalrequest/1.6/
+zope.hookable                5.4        https://pypi.org/project/zope.hookable/5.4/
+zope.i18n                    4.9.0      https://pypi.org/project/zope.i18n/4.9.0/
+zope.i18nmessageid           5.1.1      https://pypi.org/project/zope.i18nmessageid/5.1.1/
+zope.interface               5.4.0      https://pypi.org/project/zope.interface/5.4.0/
+zope.lifecycleevent          4.4        https://pypi.org/project/zope.lifecycleevent/4.4/
+zope.location                4.3        https://pypi.org/project/zope.location/4.3/
+zope.minmax                  2.3        https://pypi.org/project/zope.minmax/2.3/
+zope.pagetemplate            4.6.0      https://pypi.org/project/zope.pagetemplate/4.6.0/
+zope.processlifetime         2.4        https://pypi.org/project/zope.processlifetime/2.4/
+zope.proxy                   4.6.1      https://pypi.org/project/zope.proxy/4.6.1/
+zope.ptresource              4.3.0      https://pypi.org/project/zope.ptresource/4.3.0/
+zope.publisher               6.1.0      https://pypi.org/project/zope.publisher/6.1.0/
+zope.schema                  6.2.1      https://pypi.org/project/zope.schema/6.2.1/
 zope.security                5.8        https://pypi.org/project/zope.security/5.8/
+zope.sendmail                5.3        https://pypi.org/project/zope.sendmail/5.3/
+zope.sequencesort            4.2        https://pypi.org/project/zope.sequencesort/4.2/
+zope.session                 4.5        https://pypi.org/project/zope.session/4.5/
+zope.site                    4.6.1      https://pypi.org/project/zope.site/4.6.1/
+zope.size                    4.4        https://pypi.org/project/zope.size/4.4/
+zope.structuredtext          4.4        https://pypi.org/project/zope.structuredtext/4.4/
+zope.tal                     4.5        https://pypi.org/project/zope.tal/4.5/
+zope.tales                   5.2        https://pypi.org/project/zope.tales/5.2/
+zope.testbrowser             5.5.1      https://pypi.org/project/zope.testbrowser/5.5.1/
 zope.testing                 4.7        https://pypi.org/project/zope.testing/4.7/
+zope.traversing              4.4.1      https://pypi.org/project/zope.traversing/4.4.1/
+zope.viewlet                 4.3        https://pypi.org/project/zope.viewlet/4.3/
 Zope2                        4.0        https://pypi.org/project/Zope2/4.0/
 ZServer                      4.0.2      https://pypi.org/project/ZServer/4.0.2/

950b65ce

06 Jun, 2023 3 commits

Fix reporting of multiple detected eggs · 6bcdd408

Kirill Smelkov authored Jun 06, 2023

It was failing with

                        if len(eggv) > 1:
    >                       raise ValueError('egg %s is present multiple times: %s' % (egg, eggv))
    E                       NameError: name 'egg' is not defined

I hit this for real when trying to build multiple SRs on the same Theia.

6bcdd408

namever: Treat rfcXXX as non-version · 4b90d0a4

Kirill Smelkov authored Jun 06, 2023

This fixes version detection for strict_rfc3339-0.7-py2.7.egg :

    def test_namever(url, nameok, verok):
>       assert nxdbom.namever(url) == (nameok, verok)
E       AssertionError: assert ('strict', 'rfc3339-0.7') == ('strict_rfc3339', '0.7')
E         At index 0 diff: 'strict' != 'strict_rfc3339'
E         Full diff:
E         - ('strict_rfc3339', '0.7')
E         ?         ^       ^^^^
E         + ('strict', 'rfc3339-0.7')
E         ?         ^^^^       ^

4b90d0a4

fmt: Sort emitted entries ignoring their case · ab9720a0

Jérome Perrin authored Jun 06, 2023

@jerome says at kirr/nxd-bom!1 (comment 185299) :

    I feel this would be more natural sorting, in the current output we have
    all the upper case first and then all the lower case.

ab9720a0

05 Jun, 2023 34 commits

Mark generic_testrunner_init as synthethic setction · 53be4c2e

Kirill Smelkov authored Jun 05, 2023

else:

    NotImplementedError: generic_testrunner_init uses slapos.recipe.build without url

Occurs in Wendelin.

53be4c2e

Fix namever for findutils url · 49fc9a26
Kirill Smelkov authored Jun 05, 2023

49fc9a26

Fix support for HDF5 URL · aef17caf

Kirill Smelkov authored Jun 05, 2023

Because in 'patch1' the fraction of digits is only 1/6 it was failed to
be recognized as version. Rework version detection heuristic to
recpognize 'patch' and similar words as versions and increase ndigit
threshold - since we now do not need to keep it so low as 1/5 due to
e.g. slapospatched012 and similar cases.

Occured in Wendelin.

aef17caf

Fix support for libstemmer_c · f46be314
Kirill Smelkov authored Jun 05, 2023
```
It comes without a version.
Occurs in Wendelin.
```
f46be314

Fix support for onlyoffice_core URL · ed3cf0a3

Kirill Smelkov authored Jun 05, 2023

BOM(CloudOOO) was failing as

    RuntimeError: Unsupported url: 'https://lab.nexedi.com/bk/onlyoffice_core/repository/archive.tar.bz2?ref=8a40eb47bd80a40ecde14c223525b21852d2fc9f'

ed3cf0a3

Cloudooo uses both py2 and py3 simultaneously · b6efa70a
Kirill Smelkov authored Jun 05, 2023

b6efa70a

Fix support for icu4c url · 589ca02a

Kirill Smelkov authored Jun 05, 2023

See previous patch for how it was crashing.

Needed to improve version detection heuristic because with only trimming
-src suffix the version was detected as "2" and name as "icu4c-58".

589ca02a

Assert that ver != ø in addbom · 49514a19

Kirill Smelkov authored Jun 05, 2023

This emits useful information instead of further crash if an url could
not be parsef successfully as e.g. in below.

Before the patch:

Traceback (most recent call last):
File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 434, in <module>
main()
File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 423, in main
bom = bom_software(arg)
File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 108, in bom_software
addbom(geturl(part), '') # XXX detect kind?
File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 56, in addbom
ver = removeprefix(ver, name+'-') # wendelin.core-2.0.alpha3-0-g6315384 -> 2.0.alpha3-0-g6315384
File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 384, in removeprefix
if s.startswith(prefix):
AttributeError: 'NoneType' object has no attribute 'startswith'

After the patch:

Traceback (most recent call last):
File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 435, in <module>
main()
File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 424, in main
bom = bom_software(arg)
File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 109, in bom_software
addbom(geturl(part), '') # XXX detect kind?
File "/srv/slapgrid/slappart5/srv/project/nxd-bom/nxdbom/__init__.py", line 56, in addbom
assert ver is not None, urlpath
AssertionError: https://github.com/unicode-org/icu/releases/download/release-58-2/icu4c-58_2-src.tgz

49514a19

Cloudooo also uses libpng1.2 and libpng1.6 simultaneously · 7586c2ae
Kirill Smelkov authored Jun 05, 2023
```
Fix it similarly to how readline is handled in ERP5 SR.
```
7586c2ae
Fix handling of CUPS url · 488dbbc2
Kirill Smelkov authored Jun 05, 2023
```
Without the fix cups version was not detected.

Occurs in Cloudooo.
```
488dbbc2

Fix for LibreOffice · 3218660b

Kirill Smelkov authored Jun 05, 2023

Withthout this patch LibreOffice version was not detected and instead
was put into name.

Noticed while doing BOM(Cloudooo).

3218660b

Refer to nexedi.org/static if egg version has slaposXXX · a0d7df61

Kirill Smelkov authored Jun 05, 2023

This covers Nexedi versions of e.g. zc.buildout and zc.recipe.egg.
Without the fix those eggs were reported as e.g.

    zc.buildout                  2.7.1+slapos019 https://pypi.org/project/zc.buildout/2.7.1+slapos019/

which was not correct.

a0d7df61

Add support for collective.recipe.template · 44274bf8
Kirill Smelkov authored Jun 05, 2023
```
Occurs in SlapOS-node.
```
44274bf8

Fix handling of non-py2 eggs · 8c2c7dee

Kirill Smelkov authored Jun 05, 2023

namever restructuring caugh a thinko/bug in a2107a92 (Handle
%(__buildout_space_n__)s & co in .installed.cfg) - there test eggs were
wrongly emitted into BOM with .egg suffix in version.

8c2c7dee

isconf += .cfg.jinja2 · 3c24faaa
Kirill Smelkov authored Jun 05, 2023
```
Occured during Caucase.
```
3c24faaa

Fix handling GeoLite2 URLs · 79c38fb3

Kirill Smelkov authored Jun 05, 2023

- handle hash in url of slapos.recipe.build:download
  A hash there is instruction to verify checksum of the tarball, not
  part of the real URL.

- fix namever to detect that '-Country' is not version suffix.

Before this patch BOM entry for GeoLite2-Country was emittted as:

    GeoLite2                     Country.tar.gz#dc6224c648350d90f344a0c5c3ca5474 http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz#dc6224c648350d90f344a0c5c3ca5474

Noticed while computing BOM(Re6stnet).

79c38fb3

Detect if .installed.cfg could not be loaded · 9b39f658
Kirill Smelkov authored Jun 05, 2023
```
We need special care because configparser ignores nonexisting arguments
by default.
```
9b39f658

Name(egg) != Directory(egg) on the filesystem · 64514152

Kirill Smelkov authored Jun 05, 2023

For example an egg might be refered to as 'cython' and placed on the
filesystem as 'Cython' (titled). And for cython-zstd it is placed on the
filesytem as cython_zstd ('-' -> '_'). Try to account for that.

Without the fix nxdbom breaks as

	ValueError: egg cython-zstd not found

64514152

Handle both readline-8 and readline-5 simultaneously · da338904

Kirill Smelkov authored Jun 05, 2023

ERP5 SR uses both of them, and without dedicated handling nxdbom breaks as

AssertionError: (PkgInfo(name='readline', version='8.1', kind='', url='http://ftp.gnu.org/gnu/readline/readline-8.1.tar.gz'), PkgInfo(name='readline', version='5.2', kind='', url='http://ftp.gnu.org/gnu/readline/readline-5.2.tar.gz'))

readline is the only item that ERP5 SR uses with multiple versions.

da338904

Add support for plone.recipe.command · c7a8c41b
Kirill Smelkov authored Jun 05, 2023
```
Simply ignore it from the BOM point of view.
```
c7a8c41b

Point eggs with nxd in version to http://www.nexedi.org/static/ · fea4635a

Kirill Smelkov authored Jun 05, 2023

For example SOAPpy-0.12.0nxd001 is available only there, and

	https://pypi.org/project/SOAPpy/0.12.0nxd001/

is 404.

fea4635a

Remove +slapospatchedXXX from URL of patched eggs · 3ef92f4b

Kirill Smelkov authored Jun 05, 2023

Even if slapos.buildout indicates that an egg was locally patched via
+slapospatchedXXX suffix, the URL of the egg should not have it.

3ef92f4b

Handle %(__buildout_space_n__)s & co in .installed.cfg · a2107a92

Kirill Smelkov authored Jun 05, 2023

For example here is how it occurs in .installed.cfg for ERP5 SR:

    [jupyter]
    eggs = jupyter
            jupyter_client
            jupyter_console
            ...
            jupyterlab
            jupyterlab-launcher%(__buildout_space_n__)s

and without the fix it breaks as

    configparser.InterpolationMissingOptionError: Bad value substitution: option 'eggs' in section 'zzz' contains an interpolation key '__buildout_space_n__' which is not a valid option name. Raw value: '\naaa\nbbb%(__buildout_space_n__)s'

a2107a92

One name might refer to both an egg and a C library · 98687ba0

Kirill Smelkov authored Jun 05, 2023

This happens for real with msgpack egg / C-library.

-> Fix handling of such situation via using (name, kind) instead of only
name as BOM key. Without the fix added test breaks as

    def addbom(urlpath, kind, version=None):
        name, ver = namever(urlpath)
        if version is not None:
            assert ver is None
            ver = version
        ver = removeprefix(ver, name+'-')   # wendelin.core-2.0.alpha3-0-g6315384  -> 2.0.alpha3-0-g6315384
        if '//' in urlpath:
            url = urlpath
        else:
            if kind == 'egg':
                # XXX not strictly correct -> better retrieve the actual URL, but buildout does not save it in installed.cfg
                url = 'https://pypi.org/project/%s/%s/' % (name, ver)
            else:
                raise NotImplementedError('TODO url for kind %r  (urlpath: %r)' % (kind, urlpath))

        info = PkgInfo(name, ver, kind, url)
        if name in bom:
>           assert bom[name] == info,  (bom[name], info)
E           AssertionError: (PkgInfo(name='msgpack', version='0.5.4', kind='', url='http://downloads.sourceforge.net/project/msgpack/msgpack/cpp/msgpack-0.5.4.tar.gz'), PkgInfo(name='msgpack', version='0.6.2', kind='egg', url='https://pypi.org/project/msgpack/0.6.2/'))

98687ba0

zc.recipe.egg: `:egg` might be missing · 22de517e
Kirill Smelkov authored Jun 05, 2023
```
-> in such a case the recipe uses section name as the default name for installed egg.
```
22de517e

Add support for zc.recipe.egg:eggs and zc.recipe.egg:scripts · 34c4526c

Kirill Smelkov authored Jun 05, 2023

Notes:

- egg can come with options, e.g. 'neoppod[admin, ctl, master]'
- eggs are separated by \n, not space
- we ignore *.egg-link because those softwares are reported due to their
  original downloading sections

34c4526c

Add support for zc.recipe.egg:develop · 59db6ac4
Kirill Smelkov authored Jun 05, 2023

59db6ac4

slapos.recipe.build:download* : Ignore urls that point to file coming from SR itself · b1742031

Kirill Smelkov authored Jun 05, 2023

Without this fix SR files fetched via slapos.recipe.build:download and
slapos.recipe.build:download-unpacked produce leat to error e.g. like

urlpath = '/SR-BASE/stack/erp5/instance-mariadb-resiliency-after-import-script.sh.in', kind = '', version = None

    def addbom(urlpath, kind, version=None):
        name, ver = namever(urlpath)
        if version is not None:
            assert ver is None
            ver = version
        ver = removeprefix(ver, name+'-')   # wendelin.core-2.0.alpha3-0-g6315384  -> 2.0.alpha3-0-g6315384
        if '//' in urlpath:
            url = urlpath
        else:
            if kind == 'egg':
                # XXX not strictly correct -> better retrieve the actual URL, but buildout does not save it in installed.cfg
                url = 'https://pypi.org/project/%s/%s/' % (name, ver)
            else:
>               raise NotImplementedError('TODO url for kind %r  (urlpath: %r)' % (kind, urlpath))
E               NotImplementedError: TODO url for kind ''  (urlpath: '/SR-BASE/stack/erp5/instance-mariadb-resiliency-after-import-script.sh.in')

nxdbom/__init__.py:64: NotImplementedError

b1742031

slapos.recipe.build:download* : Recognize supported archives and don't complain about them · 673189bd

Kirill Smelkov authored Jun 05, 2023

e.g.

	https://raw.githubusercontent.com/zuphilip/ocropy-models/master/en-default.pyrnn.gz

is fetched via slapos.recipe.build:download in ERP5 SR.

673189bd

Add support for slapos.recipe.build:gitclone · 43319739
Kirill Smelkov authored Jun 05, 2023

43319739
Add support for more kind of tarball URLs · 61fd444f
Kirill Smelkov authored Jun 05, 2023
```
Hit issues with those urls on ERP5 SR because they were not recognized previously.
```
61fd444f
Add more isconf patterns · 6d3814ef
Kirill Smelkov authored Jun 05, 2023
```
Hit complaints about those files being non-configuration on ERP5 SR.
```
6d3814ef

Don't crash on URLs with %20 inside · f67f71ed

Kirill Smelkov authored Jun 05, 2023

Without the fix configparser complains about bad interpolation:

    configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%20Latest%20Stable/2.0.10/zabbix-2.0.10.tar.gz'

f67f71ed

ZODB, ZEO and tempstorage are also synthetic sections · dbba5cde
Kirill Smelkov authored Jun 05, 2023
```
Ignore them instead of complaining.
```
dbba5cde