Commit deda4108 authored by Julien Muchembled's avatar Julien Muchembled

Switch to slapos.recipe.build implementation of environment & shared options

'environment-section' option is dropped.
parent 3b128565
Pipeline #18545 passed with stage
in 0 seconds
...@@ -43,7 +43,7 @@ setup( ...@@ -43,7 +43,7 @@ setup(
install_requires=[ install_requires=[
'zc.buildout>=2', 'zc.buildout>=2',
'setuptools', 'setuptools',
'slapos.recipe.build>=0.40', 'slapos.recipe.build>=0.49',
], ],
extras_require={ extras_require={
'test': ['zope.testing', 'manuel'], 'test': ['zope.testing', 'manuel'],
......
This document describe the usages of ``slapos.recipe.cmmi`` in SlapOS. This document describes some usages of ``slapos.recipe.cmmi`` in SlapOS.
Nothing here is specific to ``slapos.recipe.cmmi``:
- ``${:_profile_base_location_}`` support comes from SlapOS buildout
- the mechanism to share parts is inherited from ``slapos.recipe.build``
SlapOS ``${:_profile_base_location_}`` support SlapOS ``${:_profile_base_location_}`` support
============================================== ==============================================
...@@ -52,7 +56,7 @@ We have profiles for this, they are located at URL ``URL1``. ...@@ -52,7 +56,7 @@ We have profiles for this, they are located at URL ``URL1``.
... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
... url = file://%s ... url = file://%s
... patches = ${:_profile_base_location_}/../dummy.patch ... patches = ${:_profile_base_location_}/../dummy.patch
... shared = True ... shared = true
... """ % package_path) ... """ % package_path)
>>> write('URL1/haproxy.cfg', >>> write('URL1/haproxy.cfg',
...@@ -65,7 +69,7 @@ We have profiles for this, they are located at URL ``URL1``. ...@@ -65,7 +69,7 @@ We have profiles for this, they are located at URL ``URL1``.
... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
... url = file://%s ... url = file://%s
... patches = ${:_profile_base_location_}/../dummy.patch ... patches = ${:_profile_base_location_}/../dummy.patch
... shared = True ... shared = true
... environment= ... environment=
... CFLAGS=-I${package:location}/include/ ... CFLAGS=-I${package:location}/include/
... """ % package_path) ... """ % package_path)
...@@ -87,16 +91,14 @@ We have a buildout using these profiles from ``URL1``: ...@@ -87,16 +91,14 @@ We have a buildout using these profiles from ``URL1``:
package: shared at /shared/package/<MD5SUM:0> package: shared at /shared/package/<MD5SUM:0>
haproxy: shared at /shared/haproxy/<MD5SUM:1> haproxy: shared at /shared/haproxy/<MD5SUM:1>
Installing package. Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: Applying patches package: Applying patches
patching file dummy.txt patching file dummy.txt
configure --prefix=/shared/package/<MD5SUM:0> configure --prefix=/shared/package/<MD5SUM:0>
building package building package
installing package installing package
Installing haproxy. Installing haproxy.
haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:1>
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:0>/include/
haproxy: Applying patches haproxy: Applying patches
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:0>/include/
patching file dummy.txt patching file dummy.txt
configure --prefix=/shared/haproxy/<MD5SUM:1> configure --prefix=/shared/haproxy/<MD5SUM:1>
building package building package
...@@ -139,11 +141,9 @@ and not installed again. ...@@ -139,11 +141,9 @@ and not installed again.
Uninstalling haproxy. Uninstalling haproxy.
Uninstalling package. Uninstalling package.
Installing package. Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0> package: shared part is already installed
package: This shared package has been installed by other package
Installing haproxy. Installing haproxy.
haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:1> haproxy: shared part is already installed
haproxy: This shared package has been installed by other package
On the other hand, if the ``package`` becomes different, then it will be re-installed at another On the other hand, if the ``package`` becomes different, then it will be re-installed at another
...@@ -155,7 +155,7 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins ...@@ -155,7 +155,7 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins
... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
... url = file://%s ... url = file://%s
... # no patch this time ... # no patch this time
... shared = True ... shared = true
... """ % package_path) ... """ % package_path)
>>> print(system(buildout)) >>> print(system(buildout))
...@@ -164,14 +164,12 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins ...@@ -164,14 +164,12 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins
Uninstalling haproxy. Uninstalling haproxy.
Uninstalling package. Uninstalling package.
Installing package. Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:2>
configure --prefix=/shared/package/<MD5SUM:2> configure --prefix=/shared/package/<MD5SUM:2>
building package building package
installing package installing package
Installing haproxy. Installing haproxy.
haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:3>
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:2>/include/
haproxy: Applying patches haproxy: Applying patches
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:2>/include/
patching file dummy.txt patching file dummy.txt
configure --prefix=/shared/haproxy/<MD5SUM:3> configure --prefix=/shared/haproxy/<MD5SUM:3>
building package building package
...@@ -213,15 +211,17 @@ tool will be able to delete this previous version of ``haproxy``. ...@@ -213,15 +211,17 @@ tool will be able to delete this previous version of ``haproxy``.
False False
We also needed something to know the dependences between shared parts. For this, we are using We also needed something to know the dependences between shared parts. For this, we are using
``.slapos.recipe.cmmi.signature`` files in folders where shared parts are installed. Because ``.buildout-shared.json`` files in folders where shared parts are installed. Because
``haproxy`` depends on ``package`` version ``<MD5SUM:2>``, we can see ``<MD5SUM:2>`` in its ``haproxy`` depends on ``package`` version ``<MD5SUM:2>``, we can see ``<MD5SUM:2>`` in its
signature file. signature file.
>>> haproxy_location = read_installed()['haproxy']['location'] >>> haproxy_location = read_installed()['haproxy']['location']
>>> haproxy_location >>> haproxy_location
'/shared/haproxy/<MD5SUM:3>' '/shared/haproxy/<MD5SUM:3>'
>>> cat(join(haproxy_location, '.slapos.recipe.cmmi.signature')) # doctest: +ELLIPSIS >>> cat(join(haproxy_location, '.buildout-shared.json')) # doctest: +ELLIPSIS
'__buildout_signature__': ... {
"__buildout_signature__": ...
...<MD5SUM:2>... ...<MD5SUM:2>...
}
This is also useful during testing, we check that signatures do not have references to non-shared parts. This is also useful during testing, we check that signatures do not have references to non-shared parts.
...@@ -33,11 +33,7 @@ Supported options ...@@ -33,11 +33,7 @@ Supported options
``shared`` ``shared``
Specify the path in which this package is shared by many other See documentation of slapos.recipe.build's default recipe.
packages.
``shared-part-list`` should be defined in ``[buildout]`` section
Shared option is True or False
The package will be installed on path/name/hash of options.
``md5sum`` ``md5sum``
...@@ -179,40 +175,9 @@ Supported options ...@@ -179,40 +175,9 @@ Supported options
If any item doesn't exist, the recipe shows a warning message. The If any item doesn't exist, the recipe shows a warning message. The
default value is empty. default value is empty.
``environment-section``
Name of a section that provides environment variables that will be used to
augment the variables read from ``os.environ`` before executing the
recipe.
This recipe does not modify ``os.environ`` directly. External commands
run as part of the recipe (e.g. make, configure, etc.) get an augmented
environment when they are forked. Python hook scripts are passed the
augmented as a parameter.
The values of the environment variables may contain references to other
existing environment variables (including themselves) in the form of
Python string interpolation variables using the dictionary notation. These
references will be expanded using values from ``os.environ``. This can be
used, for example, to append to the ``PATH`` variable, e.g.::
[component]
recipe = slapos.recipe.cmmi
environment-section =
environment
[environment]
PATH = %(PATH)s:${buildout:directory}/bin
``environment`` ``environment``
A sequence of ``KEY=VALUE`` pairs separated by newlines that define See documentation of slapos.recipe.build's default recipe.
additional environment variables used to update ``os.environ`` before
executing the recipe.
The semantics of this option are the same as ``environment-section``. If
both ``environment-section`` and ``environment`` are provided the values from
the former will be overridden by the latter allowing per-part customization.
Additionally, the recipe honors the ``download-cache`` option set Additionally, the recipe honors the ``download-cache`` option set
in the ``[buildout]`` section and stores the downloaded files under in the ``[buildout]`` section and stores the downloaded files under
...@@ -444,17 +409,13 @@ with a new buildout and provide more options. ...@@ -444,17 +409,13 @@ with a new buildout and provide more options.
... newest = false ... newest = false
... parts = package ... parts = package
... ...
... [build-environment]
... CFLAGS = -I/sw/include
... LDFLAGS = -I/sw/lib
...
... [package] ... [package]
... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
... url = file://%s ... url = file://%s
... md5sum = 6b94295c042a91ea3203857326bc9209 ... md5sum = 6b94295c042a91ea3203857326bc9209
... prefix = /somewhere/else ... prefix = /somewhere/else
... environment-section = build-environment
... environment = ... environment =
... CFLAGS=-I/sw/include
... LDFLAGS=-L/sw/lib -L/some/extra/lib ... LDFLAGS=-L/sw/lib -L/some/extra/lib
... configure-options = ... configure-options =
... --with-threads ... --with-threads
...@@ -467,16 +428,16 @@ with a new buildout and provide more options. ...@@ -467,16 +428,16 @@ with a new buildout and provide more options.
... patches/Makefile.dist.patch ... patches/Makefile.dist.patch
... """ % package_path) ... """ % package_path)
This configuration uses custom configure options, an environment section, This configuration uses custom configure options, environment variables,
per-part customization to the environment, custom prefix, multiple make custom prefix, multiple make targets and also patches the source code
targets and also patches the source code before the scripts are run. before the scripts are run.
>>> print(system(buildout)) >>> print(system(buildout))
Uninstalling package. Uninstalling package.
Installing package. Installing package.
package: Applying patches
package: [ENV] CFLAGS = -I/sw/include package: [ENV] CFLAGS = -I/sw/include
package: [ENV] LDFLAGS = -L/sw/lib -L/some/extra/lib package: [ENV] LDFLAGS = -L/sw/lib -L/some/extra/lib
package: Applying patches
patching file configure patching file configure
patching file Makefile.dist patching file Makefile.dist
patched-configure --prefix=/somewhere/else --with-threads --without-foobar patched-configure --prefix=/somewhere/else --with-threads --without-foobar
...@@ -847,166 +808,6 @@ replaced with the recipe final prefix. ...@@ -847,166 +808,6 @@ replaced with the recipe final prefix.
package: Executing pre-install package: Executing pre-install
installing package at /sample_buildout/parts/package -lib installing package at /sample_buildout/parts/package -lib
Install shared package
======================
Use option ``shared`` to install a shared package.
>>> import subprocess
>>> shared_dir = tmpdir('shared')
>>> another_shared_dir = tmpdir('another_shared_dir')
>>> __tear_downs.insert(0, lambda: subprocess.call(
... ('chmod', '-R', 'u+w', shared_dir, another_shared_dir)))
If no ``shared-part-list`` is set, and ``shared`` is True, ``shared`` feature
is not used.
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared = True
... """ % package_path)
>>> print(system(buildout)) #doctest:+ELLIPSIS
Uninstalling package.
Installing package.
configure --prefix=/sample_buildout/parts/package
building package
installing package
If ``shared-part-list`` is set, shared is True, and build package fails, the
part location is left for debugging.
Also a shell script with the environment variable is created, so that
developer can try same build process as the recipe tried.
>>> os.rename(package_path, package_path + '.bak')
>>> import tarfile
>>> from io import BytesIO
>>> import sys
>>> with tarfile.open(package_path, 'w:gz') as tar:
... configure = b'invalid'
... info = tarfile.TarInfo('configure.off')
... info.size = len(configure)
... info.mode = 0o755
... tar.addfile(info, BytesIO(configure))
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list = %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared = True
... environment =
... FOO=bar
... """ % (shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
>>> import glob
>>> cat(glob.glob(join(shared_dir, 'package/*/.build/slapos.recipe.build.env.sh'))[0])
export FOO=bar
...
Next time buildout runs, it detects that the build failed, remove the compile dir and retry.
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
If ``shared-part-list`` is set as an option in buildout section and
``shared`` is True, package will be installed in shared_part/package
and a hash of the recipe's configuration options.
There can be multiple path listed in ``shared-part-list``, the recipe
will look in each of these paths if package was already installed and
if not, it will install the package in the last entry.
>>> os.rename(package_path + '.bak', package_path)
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list =
... %s
... not/exists
... %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared = True
... environment =
... FOO=bar
... """ % (shared_dir, another_shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>
configure --prefix=/shared/package/<MD5SUM:0>
building package
installing package
If options change, reinstall in different location:
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list =
... %s
... not/exists
... %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared =True
... change = True
... """ % (shared_dir, another_shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /another_shared_dir/package/<MD5SUM:1>
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /another_shared_dir/package/<MD5SUM:1>
configure /another_shared_dir/package/<MD5SUM:1>
building package
installing package
For even more specific needs you can write your own recipe that uses For even more specific needs you can write your own recipe that uses
``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``. ``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``.
You can then continue from where this recipe finished by reading the location You can then continue from where this recipe finished by reading the location
......
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment