slapos.buildout:master commitshttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commits/master2022-02-12T22:10:27+01:00https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/00e86b85566f27a1a6d12788fcbba17972d4f18cfixup! fixup! download: clean-up, fix, optimization2022-02-12T22:10:27+01:00Julien Muchembledjm@nexedi.com
Commit <a href="/nexedi/slapos.buildout/-/commit/c3903c7e25db2f2883fb96a915c7762f66f05f05" data-original="c3903c7e25db2f2883fb96a915c7762f66f05f05" data-link="false" data-link-reference="false" data-project="55" data-commit="c3903c7e25db2f2883fb96a915c7762f66f05f05" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="fixup! download: clean-up, fix, optimization" class="gfm gfm-commit has-tooltip">c3903c7e</a> was error-prone
because (lambda f1, f2: False) may be wrong in other cases than where
it's currently used.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/e8a56bd0519306d9d33c64a9cee45376855a7275Release 2.7.1+slapos0162022-02-11T19:44:08+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/c3903c7e25db2f2883fb96a915c7762f66f05f05fixup! download: clean-up, fix, optimization2022-02-11T19:44:07+01:00Julien Muchembledjm@nexedi.com
To be squashed with commit <a href="/tleymone/slapos.buildout/-/commit/4ccb79e9e6985c0802f36fc7a5de335f5850bc18" data-original="4ccb79e9e6985c0802f36fc7a5de335f5850bc18" data-link="false" data-link-reference="false" data-project="1456" data-commit="4ccb79e9e6985c0802f36fc7a5de335f5850bc18" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="download: clean-up, fix, optimization" class="gfm gfm-commit has-tooltip">4ccb79e9</a>.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/09701a6f0d5c6ce5e0816e96e63a3760a8ca2bfcRelease 2.7.1+slapos0152022-01-07T19:49:47+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/bcc011911217c172ab279f6542d64a7385dba8b2download: add netrc file support2022-01-07T19:24:43+01:00Julien Muchembledjm@nexedi.com
Like for URL that contain credentials, we still skip auth challenge
because it's faster and:
- we only support one auth scheme (basic)
- netrc provides no way to specify realms, which seem anyway to be
less and less used (<a href="https://stackoverflow.com/q/69303610" rel="nofollow noreferrer noopener" target="_blank">https://stackoverflow.com/q/69303610</a> reports
that recent browsers don't display them anymore)
See merge request <a href="/nexedi/slapos.buildout/-/merge_requests/25" data-original="nexedi/slapos.buildout!25" data-link="false" data-link-reference="false" data-project="55" data-merge-request="5333" data-project-path="nexedi/slapos.buildout" data-iid="25" data-mr-title="netrc support" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">!25</a>https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/171c245942102f64b2acf56e8186183cfcdcf3a2Rewrite 'urlretrieve' helper to fix various download-related issues2022-01-07T19:24:01+01:00Julien Muchembledjm@nexedi.com
- Py3: stop using legacy API of urllib.request and
fix download of http(s) URLs containing user:passwd@
- Py2: avoid OOM when downloading huge files
This is implemented as a method in case we want to make it configurable
via [buildout].https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/c9e9b267872e9412d03a7399242a003c0fb96019Conditional configuration sections: expose new 'multiarch' value in sys2021-12-29T11:41:42+01:00Julien Muchembledjm@nexedi.com
To be dropped once all buildout in the wild are able to upgrade to a
version that supports new names in expression of conditional sections
(see previous commit).https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/1b3a3e1ea3546966acbdf8270f3e355cb45841d4Ignore NameError in unused conditional configuration sections2021-12-27T14:25:13+01:00Julien Muchembledjm@nexedi.com
Adding new names for expression is currently not possible because
buildout aborts before it tries to upgrade (in-place or bootstrap).https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/1ffd9e2fd52bcc9a6e52cf9f8649a6464ac912c4The monkey patch for urllib.request._urlopener does nothing on Python 3.2021-12-27T14:25:13+01:00Jason Maddenjamadden@gmail.com
(cherry picked from commit <a href="/nexedi/slapos.buildout/-/commit/f15893781e607771b997b04238921b3f2770157b" data-original="f15893781e607771b997b04238921b3f2770157b" data-link="false" data-link-reference="false" data-project="55" data-commit="f15893781e607771b997b04238921b3f2770157b" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="The monkey patch for urllib.request._urlopener does nothing on Python 3." class="gfm gfm-commit has-tooltip">f1589378</a>)https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/5f1f7d23cfa25e5eb3039ffe3e9259ebd1dee4b0Conditional configuration sections: compute values at most once, add 'multiarch'2021-12-14T15:29:20+01:00Julien Muchembledjm@nexedi.com
For example, existing values were not enough to distinguish
'arm-linux-gnueabi' from 'arm-linux-gnueabihf'.
The 'multiarch' value is the output of
$CC -dumpmachine
where CC defaults to 'gcc'.
See also <a href="https://wiki.debian.org/Multiarch/Tuples" rel="nofollow noreferrer noopener" target="_blank">https://wiki.debian.org/Multiarch/Tuples</a>https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/be00bb12a7ca8c6fead9928cccbbb77cb3f65273Release 2.7.1+slapos0142021-09-25T03:37:07+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/9e5dcbc6ff4318abdc5e869f0908a238a5f24b41Be more verbose for the first part to reinstall (changed option or missing path)2021-09-25T03:37:02+02:00Julien Muchembledjm@nexedi.com
Sometimes, most parts are reinstalled for a reason that the user
didn't think about and it can take time to understand why.
Explaining for all parts would be too verbose and useless because
many are reinstalled just because their dependencies changed.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/a2bcbf2d6721200480d5c1fb7004b1c000c9021aExtend Download API with a new parameter to use an alternate URL as fallback2021-09-24T12:17:04+02:00Julien Muchembledjm@nexedi.com
This alternate URL is used in case of HTTPError with the main one.
In any case, the main URL is used for both downloading & uploading
from/to networkcache.
This will be exposed by slapos.recipe.build:download* and
slapos.recipe.cmmi recipes.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/4ccb79e9e6985c0802f36fc7a5de335f5850bc18download: clean-up, fix, optimization2021-09-24T12:17:04+02:00Julien Muchembledjm@nexedi.com
An optimization is to avoid temporary file when possible: a rename
(or hard link) is not always possible (different mount points).
Another one is to not check md5sum twice when using cache file.
Fall-back mode is ignored if an MD5 checksum is given.
In case of checksum mismatch for a cached path, remove it and
download again, mainly to cover the following cases:
- the url content changes and the user updates the checksum
- buildout killed while downloading directly to cache
(see above optimization)
- shutil.copyfile is interruptedhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/6cec641f48f52fe2365b47e9b046e3ee58e51040Release 2.7.1+slapos0132021-05-15T18:17:35+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/b3885eac55013ff1939749f2090131a47f4f7e34Do not allow setuptools to obtain eggs, unless picked versions are allowed2021-05-15T17:54:54+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/f29dac7a9f55af6bd893c08f9e377e95f1e8a60eExplain egg dependencies when an egg fails to install2021-05-15T17:54:54+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/0abf873d4a226e3b1afe5a23bb52789985fc9db1Install egg from wheel only if explictly required2021-05-14T18:30:11+02:00Julien Muchembledjm@nexedi.com
Egg is install from wheel the requested version ends with :whl
This replaces:
<a href="/nexedi/slapos.buildout/-/commit/6d9b748391b20820de07d51c4aa211258b8de9dd" data-original="6d9b7483" data-link="false" data-link-reference="false" data-project="55" data-commit="6d9b748391b20820de07d51c4aa211258b8de9dd" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Avoid installing eggs from wheels when possible" class="gfm gfm-commit has-tooltip">6d9b7483</a> Avoid installing eggs from wheels when possible
<a href="/nexedi/slapos.buildout/-/commit/ff2ee2186fc7f006a0eb57ad73f6aae31cbb2433" data-original="ff2ee218" data-link="false" data-link-reference="false" data-project="55" data-commit="ff2ee2186fc7f006a0eb57ad73f6aae31cbb2433" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Do never install from wheels" class="gfm gfm-commit has-tooltip">ff2ee218</a> Do never install from wheelshttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/33ae258319f9c55e36570f8ba7f1420bc96d94f5Release 2.7.1+slapos0122021-05-12T10:58:25+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/6d9b748391b20820de07d51c4aa211258b8de9ddAvoid installing eggs from wheels when possible2021-05-12T10:21:57+02:00Julien Muchembledjm@nexedi.com
This relaxes commit <a href="/nexedi/slapos.buildout/-/commit/ff2ee2186fc7f006a0eb57ad73f6aae31cbb2433" data-original="ff2ee2186fc7f006a0eb57ad73f6aae31cbb2433" data-link="false" data-link-reference="false" data-project="55" data-commit="ff2ee2186fc7f006a0eb57ad73f6aae31cbb2433" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Do never install from wheels" class="gfm gfm-commit has-tooltip">ff2ee218</a>
("Do never install from wheels") because for some eggs like plantuml,
there's no tarball.
About the implementation, no need to sort by version (as it was done)
because the list is already filtered to keep the best one.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/1aeb4ae08ef586dd15667d0aa280cd5903e6be01Release 2.7.1+slapos0112021-05-11T16:33:36+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/ff2ee2186fc7f006a0eb57ad73f6aae31cbb2433Do never install from wheels2021-05-11T07:53:19+02:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/25ac00a3b5c8a5ddc65ed795511a13114219701dRelease 2.7.1+slapos0102020-12-23T09:38:29+01:00Thomas Gambierthomas.gambier@gmail.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/508c6a3f7132206d60e9f40e2a6de2df8ae07a93rmtree: import more code from slapos.core to support case of parent directory...2020-12-23T09:34:13+01:00Thomas Gambierthomas.gambier@nexedi.com
See merge request <a href="/nexedi/slapos.buildout/-/merge_requests/22" data-original="nexedi/slapos.buildout!22" data-link="false" data-link-reference="false" data-project="55" data-merge-request="4448" data-project-path="nexedi/slapos.buildout" data-iid="22" data-mr-title="rmtree: import more code from slapos.core to support case of parent directory not writable" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">!22</a>https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/6dcf6b6ed8a2228f9331af91395d0ea1c17e7374rmtree: import more code from slapos.core to support case of parent directory...2020-12-22T10:43:36+01:00Thomas Gambierthomas.gambier@gmail.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/44b3c0766c4874bf392ab3af7f490465fd35f5e7fixup! fixup! zc.recipe.egg: Improve on the fly pathces.2020-09-11T03:18:24+02:00Jérome Perrinjerome@nexedi.com
other tests have been disabled by mistake.
test was randomly failing because of glob.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/9aeda0ea037f0d685ac1fb232a4b90383da7adc0Release 2.7.1+slapos0092020-09-11T02:42:48+02:00Jérome Perrinjerome@nexedi.comhttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/6ee519d2225a01fb8fc859251e7b39937046af94Reject invalid options set by recipes2020-09-10T15:54:06+02:00Jérome Perrinjerome@nexedi.com
When recipes mutate the options, we should not allow invalid syntax otherwise
it gets written in .installed.cfg and generate invalid configparser syntax.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/d42b790b4ba40a0807c432f39d6271d0c4b95e75Escape option values set by recipes2020-09-10T15:54:06+02:00Jérome Perrinjerome@nexedi.com
When recipe mutate options, options values are written to .installed.cfg
without escaping buildout substitution syntax, so if a recipe sets an option
value that could be interpreted as a buildout substitution, it is written as is
in .installed.cfg.
This can be a problem if options read from _read_installed_part_options are
accessed, like it's the case with slapos patched buildout which saves
installed options after an error with part installation or after each part
installation when running in verbose mode.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/32facc788ad4e1c096d1b400226c1838c237c66ctests: use _ instead of - in environment variables2020-09-10T15:47:07+02:00Jérome Perrinjerome@nexedi.com
Contrary to bash, dash filters out environment variables containing
'-' characters. And for example, /bin/sh is dash on Debian.
Rebase instructions:
- squash with <a href="/nexedi/slapos.buildout/-/commit/93369b047ff558f94dcca2dc3578a9780d779774" data-original="93369b047ff558f94dcca2dc3578a9780d779774" data-link="false" data-link-reference="false" data-project="55" data-commit="93369b047ff558f94dcca2dc3578a9780d779774" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Fix testing index when /bin/sh is dash (like on Debian)" class="gfm gfm-commit has-tooltip">93369b04</a>
and keep the above commit message
fixup! Fix testing index when /bin/sh is dash (like on Debian)https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/121b182f72638dea34e2fdb92d9c322ee6f44f44tests: Fix makeNewRelease when installed as a zipped egg2020-09-10T15:47:07+02:00Jérome Perrinjerome@nexedi.com
python ZipFile module does not support updating an entry in place, instead make
a new zip file and copy all entries.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/64ac43b11e9c95381dc7685a0bf75d0fd10f795ashut up easy_install deprecation2020-09-10T15:47:07+02:00Godefroid Chapellegotcha@bubblenet.behttps://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/5985f2425486aa19466c4b22d8985ddc1335e7c9Fix most Deprecation and Resource warnings.2020-09-10T15:47:07+02:00Jason Maddenjamadden@gmail.com
On Python 3, these made the build output very hard to read (e.g., <a href="https://travis-ci.org/buildout/buildout/jobs/394026829" rel="nofollow noreferrer noopener" target="_blank">https://travis-ci.org/buildout/buildout/jobs/394026829</a>)
On Python 3.7, these could actually break the doctests.
Closing the files should get us closer to being able to pass the tests with PyPy.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/8a4fcd213f68c62164d7e097cbf0df43458a880bfixup! Write .installed.cfg only once, in safe way and only if there's any ch...2020-09-10T15:47:07+02:00Jérome Perrinjerome@nexedi.com
Because updating a section no longer add the section t the end of parts in
.installed, uninstallation happens in this order.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/1960030f9f91038ff4d61d755059ce834e7b83c4fixup! Add referred parts' hash strings in __buildout_signature__, that...2020-09-10T15:47:07+02:00Jérome Perrinjerome@nexedi.comfixup! Add referred parts' hash strings in __buildout_signature__, that invokes rebuild of a part when one of its (recursive) dependencies are modified.
egg signatures are always {egg_name}-{egg_verson}, there should not be a "egg" prefix for setuptools
https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/31fb43378ee9fb74a97110b2dcba7f5aa02431affixup! Add referred parts' hash strings in __buildout_signature__, that...2020-09-10T15:37:29+02:00Jérome Perrinjerome@nexedi.comfixup! Add referred parts' hash strings in __buildout_signature__, that invokes rebuild of a part when one of its (recursive) dependencies are modified.
Comment by Julien Muchembled:
Maybe we again have the original behaviour because of
commit <a href="/nexedi/slapos.buildout/-/commit/dffdeffc34a1bc01c4c0f9728365f897827b162d" data-original="dffdeffc34a1bc01c4c0f9728365f897827b162d" data-link="false" data-link-reference="false" data-project="55" data-commit="dffdeffc34a1bc01c4c0f9728365f897827b162d" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="fixup! Add referred parts' hash strings in __buildout_signature__, that..." class="gfm gfm-commit has-tooltip">dffdeffc</a>.
Anyway, the behaviour of buildout at this point is meaningless.
Buildout was previously invoked with install d3 d4 command-line
arguments and I consider that d2 should have been reinstalled,
because a section shall not access the option of another section
if the latter is not properly installed.
https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/45eb64b4d378e73b8d0f02ff0ecefe98e814100cfixup! zc.recipe.egg: Improve on the fly pathces.2020-09-08T11:56:41+02:00Jérome Perrinjerome@nexedi.com
Only build egg at this point if we have some patches to apply otherwise
buildout would install dependent egg without logging this operation, which
would cause many tests to fail.
Also cover the whole scenario of egg patching in a new doctest.https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/7488bf75e2b897825fcf2e029c3a1956bf9cc357fixup! Add referred parts' hash strings in __buildout_signature__, that...2020-05-22T16:39:09+02:00Julien Muchembledjm@nexedi.comfixup! Add referred parts' hash strings in __buildout_signature__, that invokes rebuild of a part when one of its (recursive) dependencies are modified.
https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/2f1d705adfbe9ee124936c9d584f7df3b2e3686bFix tests2020-05-21T01:56:49+02:00Marius Gedminasmarius@gedmin.as
Add a RENormalizer so the easy_install.txt will be happy with scripts
generated on Python 2 (using open(..., 'U')) and on Python 3 (using just
open(...)).
This was suggested by @sallner.
(cherry picked from commit <a href="/nexedi/slapos.buildout/-/commit/2cf42066fc8238ffe72ce6aeaef0e66b5df13f79" data-original="2cf42066fc8238ffe72ce6aeaef0e66b5df13f79" data-link="false" data-link-reference="false" data-project="55" data-commit="2cf42066fc8238ffe72ce6aeaef0e66b5df13f79" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Fix tests" class="gfm gfm-commit has-tooltip">2cf42066</a>)https://lab.nexedi.com/Daetalus/slapos.buildout/-/commit/20adbdb698ca485e6c419539f4204ea9141050a0Avoid DeprecationWarning: 'U' mode is deprecated2020-05-21T01:56:49+02:00Laurence Rowelaurence@lrowe.co.uk
Python 3.7 began warning about open's 'U' mode. Universal newline mode (newline=None) is default in Python 3 so avoid specifying it on Python 3.
(cherry picked from commit <a href="/nexedi/slapos.buildout/-/commit/978a6af9e78db44b9db25cb08c7f9d98c24bbc03" data-original="978a6af9e78db44b9db25cb08c7f9d98c24bbc03" data-link="false" data-link-reference="false" data-project="55" data-commit="978a6af9e78db44b9db25cb08c7f9d98c24bbc03" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="Avoid DeprecationWarning: 'U' mode is deprecated" class="gfm gfm-commit has-tooltip">978a6af9</a>)