Skip to content

S
slapos.recipe.cmmi
Commit deda4108 authored by Julien Muchembled's avatar Julien Muchembled
Browse files
Options

Switch to slapos.recipe.build implementation of environment & shared options 

'environment-section' option is dropped.
parent 3b128565
Expand all Show whitespace changes
Inline Side-by-side
Showing with 102 additions and 363 deletions
setup.py
View file @ deda4108
......@@ -43,7 +43,7 @@ setup(
install_requires=[
'zc.buildout>=2',
'setuptools',
'slapos.recipe.build>=0.40',
'slapos.recipe.build>=0.49',
],
extras_require={
'test': ['zope.testing', 'manuel'],
......
slapos/recipe/cmmi/README-SlapOS.rst
View file @ deda4108
This document describe the usages of ``slapos.recipe.cmmi`` in SlapOS.
This document describes some usages of ``slapos.recipe.cmmi`` in SlapOS.
Nothing here is specific to ``slapos.recipe.cmmi``:
- ``${:_profile_base_location_}`` support comes from SlapOS buildout
- the mechanism to share parts is inherited from ``slapos.recipe.build``
SlapOS ``${:_profile_base_location_}`` support
==============================================
......@@ -52,7 +56,7 @@ We have profiles for this, they are located at URL ``URL1``.
... recipe = slapos.recipe.cmmi
... url = file://%s
... patches = ${:_profile_base_location_}/../dummy.patch
... shared = True
... shared = true
... """ % package_path)
>>> write('URL1/haproxy.cfg',
......@@ -65,7 +69,7 @@ We have profiles for this, they are located at URL ``URL1``.
... recipe = slapos.recipe.cmmi
... url = file://%s
... patches = ${:_profile_base_location_}/../dummy.patch
... shared = True
... shared = true
... environment=
... CFLAGS=-I${package:location}/include/
... """ % package_path)
......@@ -87,16 +91,14 @@ We have a buildout using these profiles from ``URL1``:
package: shared at /shared/package/<MD5SUM:0>
haproxy: shared at /shared/haproxy/<MD5SUM:1>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: Applying patches
patching file dummy.txt
configure --prefix=/shared/package/<MD5SUM:0>
building package
installing package
Installing haproxy.
haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:1>
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:0>/include/
haproxy: Applying patches
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:0>/include/
patching file dummy.txt
configure --prefix=/shared/haproxy/<MD5SUM:1>
building package
......@@ -139,11 +141,9 @@ and not installed again.
Uninstalling haproxy.
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: This shared package has been installed by other package
package: shared part is already installed
Installing haproxy.
haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:1>
haproxy: This shared package has been installed by other package
haproxy: shared part is already installed
On the other hand, if the ``package`` becomes different, then it will be re-installed at another
......@@ -155,7 +155,7 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins
... recipe = slapos.recipe.cmmi
... url = file://%s
... # no patch this time
... shared = True
... shared = true
... """ % package_path)
>>> print(system(buildout))
......@@ -164,14 +164,12 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins
Uninstalling haproxy.
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:2>
configure --prefix=/shared/package/<MD5SUM:2>
building package
installing package
Installing haproxy.
haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:3>
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:2>/include/
haproxy: Applying patches
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:2>/include/
patching file dummy.txt
configure --prefix=/shared/haproxy/<MD5SUM:3>
building package
......@@ -213,15 +211,17 @@ tool will be able to delete this previous version of ``haproxy``.
False
We also needed something to know the dependences between shared parts. For this, we are using
``.slapos.recipe.cmmi.signature`` files in folders where shared parts are installed. Because
``.buildout-shared.json`` files in folders where shared parts are installed. Because
``haproxy`` depends on ``package`` version ``<MD5SUM:2>``, we can see ``<MD5SUM:2>`` in its
signature file.
>>> haproxy_location = read_installed()['haproxy']['location']
>>> haproxy_location
'/shared/haproxy/<MD5SUM:3>'
>>> cat(join(haproxy_location, '.slapos.recipe.cmmi.signature')) # doctest: +ELLIPSIS
'__buildout_signature__': ...
>>> cat(join(haproxy_location, '.buildout-shared.json')) # doctest: +ELLIPSIS
{
"__buildout_signature__": ...
...<MD5SUM:2>...
}
This is also useful during testing, we check that signatures do not have references to non-shared parts.
slapos/recipe/cmmi/README.rst
View file @ deda4108
......@@ -33,11 +33,7 @@ Supported options
``shared``
Specify the path in which this package is shared by many other
packages.
``shared-part-list`` should be defined in ``[buildout]`` section
Shared option is True or False
The package will be installed on path/name/hash of options.
See documentation of slapos.recipe.build's default recipe.
``md5sum``
......@@ -179,40 +175,9 @@ Supported options
If any item doesn't exist, the recipe shows a warning message. The
default value is empty.
``environment-section``
Name of a section that provides environment variables that will be used to
augment the variables read from ``os.environ`` before executing the
recipe.
This recipe does not modify ``os.environ`` directly. External commands
run as part of the recipe (e.g. make, configure, etc.) get an augmented
environment when they are forked. Python hook scripts are passed the
augmented as a parameter.
The values of the environment variables may contain references to other
existing environment variables (including themselves) in the form of
Python string interpolation variables using the dictionary notation. These
references will be expanded using values from ``os.environ``. This can be
used, for example, to append to the ``PATH`` variable, e.g.::
[component]
recipe = slapos.recipe.cmmi
environment-section =
environment
[environment]
PATH = %(PATH)s:${buildout:directory}/bin
``environment``
A sequence of ``KEY=VALUE`` pairs separated by newlines that define
additional environment variables used to update ``os.environ`` before
executing the recipe.
The semantics of this option are the same as ``environment-section``. If
both ``environment-section`` and ``environment`` are provided the values from
the former will be overridden by the latter allowing per-part customization.
See documentation of slapos.recipe.build's default recipe.
Additionally, the recipe honors the ``download-cache`` option set
in the ``[buildout]`` section and stores the downloaded files under
......@@ -444,17 +409,13 @@ with a new buildout and provide more options.
... newest = false
... parts = package
...
... [build-environment]
... CFLAGS = -I/sw/include
... LDFLAGS = -I/sw/lib
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... md5sum = 6b94295c042a91ea3203857326bc9209
... prefix = /somewhere/else
... environment-section = build-environment
... environment =
... CFLAGS=-I/sw/include
... LDFLAGS=-L/sw/lib -L/some/extra/lib
... configure-options =
... --with-threads
......@@ -467,16 +428,16 @@ with a new buildout and provide more options.
... patches/Makefile.dist.patch
... """ % package_path)
This configuration uses custom configure options, an environment section,
per-part customization to the environment, custom prefix, multiple make
targets and also patches the source code before the scripts are run.
This configuration uses custom configure options, environment variables,
custom prefix, multiple make targets and also patches the source code
before the scripts are run.
>>> print(system(buildout))
Uninstalling package.
Installing package.
package: Applying patches
package: [ENV] CFLAGS = -I/sw/include
package: [ENV] LDFLAGS = -L/sw/lib -L/some/extra/lib
package: Applying patches
patching file configure
patching file Makefile.dist
patched-configure --prefix=/somewhere/else --with-threads --without-foobar
......@@ -847,166 +808,6 @@ replaced with the recipe final prefix.
package: Executing pre-install
installing package at /sample_buildout/parts/package -lib
Install shared package
======================
Use option ``shared`` to install a shared package.
>>> import subprocess
>>> shared_dir = tmpdir('shared')
>>> another_shared_dir = tmpdir('another_shared_dir')
>>> __tear_downs.insert(0, lambda: subprocess.call(
... ('chmod', '-R', 'u+w', shared_dir, another_shared_dir)))
If no ``shared-part-list`` is set, and ``shared`` is True, ``shared`` feature
is not used.
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared = True
... """ % package_path)
>>> print(system(buildout)) #doctest:+ELLIPSIS
Uninstalling package.
Installing package.
configure --prefix=/sample_buildout/parts/package
building package
installing package
If ``shared-part-list`` is set, shared is True, and build package fails, the
part location is left for debugging.
Also a shell script with the environment variable is created, so that
developer can try same build process as the recipe tried.
>>> os.rename(package_path, package_path + '.bak')
>>> import tarfile
>>> from io import BytesIO
>>> import sys
>>> with tarfile.open(package_path, 'w:gz') as tar:
... configure = b'invalid'
... info = tarfile.TarInfo('configure.off')
... info.size = len(configure)
... info.mode = 0o755
... tar.addfile(info, BytesIO(configure))
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list = %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared = True
... environment =
... FOO=bar
... """ % (shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
>>> import glob
>>> cat(glob.glob(join(shared_dir, 'package/*/.build/slapos.recipe.build.env.sh'))[0])
export FOO=bar
...
Next time buildout runs, it detects that the build failed, remove the compile dir and retry.
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
If ``shared-part-list`` is set as an option in buildout section and
``shared`` is True, package will be installed in shared_part/package
and a hash of the recipe's configuration options.
There can be multiple path listed in ``shared-part-list``, the recipe
will look in each of these paths if package was already installed and
if not, it will install the package in the last entry.
>>> os.rename(package_path + '.bak', package_path)
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list =
... %s
... not/exists
... %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared = True
... environment =
... FOO=bar
... """ % (shared_dir, another_shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>
configure --prefix=/shared/package/<MD5SUM:0>
building package
installing package
If options change, reinstall in different location:
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list =
... %s
... not/exists
... %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared =True
... change = True
... """ % (shared_dir, another_shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /another_shared_dir/package/<MD5SUM:1>
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /another_shared_dir/package/<MD5SUM:1>
configure /another_shared_dir/package/<MD5SUM:1>
building package
installing package
For even more specific needs you can write your own recipe that uses
``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``.
You can then continue from where this recipe finished by reading the location
......
slapos/recipe/cmmi/__init__.py
View file @ deda4108
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7