- 05 Jun, 2023 22 commits
-
-
Kirill Smelkov authored
- handle hash in url of slapos.recipe.build:download A hash there is instruction to verify checksum of the tarball, not part of the real URL. - fix namever to detect that '-Country' is not version suffix. Before this patch BOM entry for GeoLite2-Country was emittted as: GeoLite2 Country.tar.gz#dc6224c648350d90f344a0c5c3ca5474 http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz#dc6224c648350d90f344a0c5c3ca5474 Noticed while computing BOM(Re6stnet).
-
Kirill Smelkov authored
We need special care because configparser ignores nonexisting arguments by default.
-
Kirill Smelkov authored
For example an egg might be refered to as 'cython' and placed on the filesystem as 'Cython' (titled). And for cython-zstd it is placed on the filesytem as cython_zstd ('-' -> '_'). Try to account for that. Without the fix nxdbom breaks as ValueError: egg cython-zstd not found
-
Kirill Smelkov authored
ERP5 SR uses both of them, and without dedicated handling nxdbom breaks as AssertionError: (PkgInfo(name='readline', version='8.1', kind='', url='http://ftp.gnu.org/gnu/readline/readline-8.1.tar.gz'), PkgInfo(name='readline', version='5.2', kind='', url='http://ftp.gnu.org/gnu/readline/readline-5.2.tar.gz')) readline is the only item that ERP5 SR uses with multiple versions.
-
Kirill Smelkov authored
Simply ignore it from the BOM point of view.
-
http://www.nexedi.org/static/Kirill Smelkov authored
For example SOAPpy-0.12.0nxd001 is available only there, and https://pypi.org/project/SOAPpy/0.12.0nxd001/ is 404.
-
Kirill Smelkov authored
Even if slapos.buildout indicates that an egg was locally patched via +slapospatchedXXX suffix, the URL of the egg should not have it.
-
Kirill Smelkov authored
For example here is how it occurs in .installed.cfg for ERP5 SR: [jupyter] eggs = jupyter jupyter_client jupyter_console ... jupyterlab jupyterlab-launcher%(__buildout_space_n__)s and without the fix it breaks as configparser.InterpolationMissingOptionError: Bad value substitution: option 'eggs' in section 'zzz' contains an interpolation key '__buildout_space_n__' which is not a valid option name. Raw value: '\naaa\nbbb%(__buildout_space_n__)s'
-
Kirill Smelkov authored
This happens for real with msgpack egg / C-library. -> Fix handling of such situation via using (name, kind) instead of only name as BOM key. Without the fix added test breaks as def addbom(urlpath, kind, version=None): name, ver = namever(urlpath) if version is not None: assert ver is None ver = version ver = removeprefix(ver, name+'-') # wendelin.core-2.0.alpha3-0-g6315384 -> 2.0.alpha3-0-g6315384 if '//' in urlpath: url = urlpath else: if kind == 'egg': # XXX not strictly correct -> better retrieve the actual URL, but buildout does not save it in installed.cfg url = 'https://pypi.org/project/%s/%s/' % (name, ver) else: raise NotImplementedError('TODO url for kind %r (urlpath: %r)' % (kind, urlpath)) info = PkgInfo(name, ver, kind, url) if name in bom: > assert bom[name] == info, (bom[name], info) E AssertionError: (PkgInfo(name='msgpack', version='0.5.4', kind='', url='http://downloads.sourceforge.net/project/msgpack/msgpack/cpp/msgpack-0.5.4.tar.gz'), PkgInfo(name='msgpack', version='0.6.2', kind='egg', url='https://pypi.org/project/msgpack/0.6.2/'))
-
Kirill Smelkov authored
-> in such a case the recipe uses section name as the default name for installed egg.
-
Kirill Smelkov authored
Notes: - egg can come with options, e.g. 'neoppod[admin, ctl, master]' - eggs are separated by \n, not space - we ignore *.egg-link because those softwares are reported due to their original downloading sections
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Without this fix SR files fetched via slapos.recipe.build:download and slapos.recipe.build:download-unpacked produce leat to error e.g. like urlpath = '/SR-BASE/stack/erp5/instance-mariadb-resiliency-after-import-script.sh.in', kind = '', version = None def addbom(urlpath, kind, version=None): name, ver = namever(urlpath) if version is not None: assert ver is None ver = version ver = removeprefix(ver, name+'-') # wendelin.core-2.0.alpha3-0-g6315384 -> 2.0.alpha3-0-g6315384 if '//' in urlpath: url = urlpath else: if kind == 'egg': # XXX not strictly correct -> better retrieve the actual URL, but buildout does not save it in installed.cfg url = 'https://pypi.org/project/%s/%s/' % (name, ver) else: > raise NotImplementedError('TODO url for kind %r (urlpath: %r)' % (kind, urlpath)) E NotImplementedError: TODO url for kind '' (urlpath: '/SR-BASE/stack/erp5/instance-mariadb-resiliency-after-import-script.sh.in') nxdbom/__init__.py:64: NotImplementedError
-
Kirill Smelkov authored
e.g. https://raw.githubusercontent.com/zuphilip/ocropy-models/master/en-default.pyrnn.gz is fetched via slapos.recipe.build:download in ERP5 SR.
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Hit issues with those urls on ERP5 SR because they were not recognized previously.
-
Kirill Smelkov authored
Hit complaints about those files being non-configuration on ERP5 SR.
-
Kirill Smelkov authored
Without the fix configparser complains about bad interpolation: configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%20Latest%20Stable/2.0.10/zabbix-2.0.10.tar.gz'
-
Kirill Smelkov authored
Ignore them instead of complaining.
-
Kirill Smelkov authored
We will be adding more changes to improve nxdbom. Going without testsuite is not practical because there will be no easy way to catch regressions.
-
Kirill Smelkov authored
We will need to add tests and, maybe in the future, helper packages.
-
Kirill Smelkov authored
nxd-bom file refers to it as See COPYING file for full licensing terms. See https://www.nexedi.com/licensing for rationale and options.
-
- 12 Oct, 2022 1 commit
-
-
Jérome Perrin authored
-
- 16 Sep, 2022 1 commit
-
-
Kirill Smelkov authored
Jérome writes at https://www.erp5.com/group_section/forum/nxd-bom-XopAhODA1A/view?list_start=5&reset=1#2059332074 : I have a few suggestions for this tool: - we could also list if a component had patches applied ( for example in https://lab.nexedi.com/nexedi/nxd-bom/blob/9273cfb92e1163e786397b7665c62d61830fbb9b/example/ors-bom.txt#L36 slapos does not really install http://www.python.org/ftp/python/2.7.18/Python-2.7.18.tar.xz it also applies a series of patches ( https://lab.nexedi.com/nexedi/slapos/blob/035f3382a4d768ea9c702842f3c3c2a1c20d6a21/component/python-2.7/buildout.cfg#L40-45 ). - output a machine readable format such as bom.json -> Make corresponding TODO notes.
-
- 14 Sep, 2022 2 commits
-
-
Kirill Smelkov authored
Program nxd-bom is intended to help generating bill of material for a software-release or slapos node. Generation of the BOM for a software-release is already draftly implemented. Generation of the BOM for slapos node is thought-out, but not currently implemented. In provided example/ors-bom.txt it could be seen how both parts should look like. And nxd-bom documents in corresponding TODO section how BOM(slapos-node) should be computed. I originally used this draft script to build ors-bom.txt as requested by a customer, and I no longer actively work on it. However I suggest we incrementally improve it as needed and maybe also integrate it to be run automatically in _all_ our builds, so that whenever an SR is built, the BOM.txt is also built automatically for it.
-
Kirill Smelkov authored
An utility to help preparing bill of materials for Nexedi software.
-