Skip to content

S
slapos.recipe.cmmi
Commit deda4108 authored by Julien Muchembled's avatar Julien Muchembled
Browse files
Options

Switch to slapos.recipe.build implementation of environment & shared options 

'environment-section' option is dropped.
parent 3b128565
Hide whitespace changes
Inline Side-by-side
Showing with 102 additions and 363 deletions
setup.py
View file @ deda4108
......@@ -43,7 +43,7 @@ setup(
install_requires=[
'zc.buildout>=2',
'setuptools',
'slapos.recipe.build>=0.40',
'slapos.recipe.build>=0.49',
],
extras_require={
'test': ['zope.testing', 'manuel'],
......
slapos/recipe/cmmi/README-SlapOS.rst
View file @ deda4108
This document describe the usages of ``slapos.recipe.cmmi`` in SlapOS.
This document describes some usages of ``slapos.recipe.cmmi`` in SlapOS.
Nothing here is specific to ``slapos.recipe.cmmi``:
- ``${:_profile_base_location_}`` support comes from SlapOS buildout
- the mechanism to share parts is inherited from ``slapos.recipe.build``
SlapOS ``${:_profile_base_location_}`` support
==============================================
......@@ -52,7 +56,7 @@ We have profiles for this, they are located at URL ``URL1``.
... recipe = slapos.recipe.cmmi
... url = file://%s
... patches = ${:_profile_base_location_}/../dummy.patch
... shared = True
... shared = true
... """ % package_path)
>>> write('URL1/haproxy.cfg',
......@@ -65,7 +69,7 @@ We have profiles for this, they are located at URL ``URL1``.
... recipe = slapos.recipe.cmmi
... url = file://%s
... patches = ${:_profile_base_location_}/../dummy.patch
... shared = True
... shared = true
... environment=
... CFLAGS=-I${package:location}/include/
... """ % package_path)
......@@ -87,16 +91,14 @@ We have a buildout using these profiles from ``URL1``:
package: shared at /shared/package/<MD5SUM:0>
haproxy: shared at /shared/haproxy/<MD5SUM:1>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: Applying patches
patching file dummy.txt
configure --prefix=/shared/package/<MD5SUM:0>
building package
installing package
Installing haproxy.
haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:1>
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:0>/include/
haproxy: Applying patches
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:0>/include/
patching file dummy.txt
configure --prefix=/shared/haproxy/<MD5SUM:1>
building package
......@@ -139,11 +141,9 @@ and not installed again.
Uninstalling haproxy.
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: This shared package has been installed by other package
package: shared part is already installed
Installing haproxy.
haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:1>
haproxy: This shared package has been installed by other package
haproxy: shared part is already installed
On the other hand, if the ``package`` becomes different, then it will be re-installed at another
......@@ -155,7 +155,7 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins
... recipe = slapos.recipe.cmmi
... url = file://%s
... # no patch this time
... shared = True
... shared = true
... """ % package_path)
>>> print(system(buildout))
......@@ -164,14 +164,12 @@ shared location and ``haproxy``, which depend on ``package`` will also be re-ins
Uninstalling haproxy.
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:2>
configure --prefix=/shared/package/<MD5SUM:2>
building package
installing package
Installing haproxy.
haproxy: Checking whether package is installed at shared path: /shared/haproxy/<MD5SUM:3>
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:2>/include/
haproxy: Applying patches
haproxy: [ENV] CFLAGS = /shared/package/<MD5SUM:2>/include/
patching file dummy.txt
configure --prefix=/shared/haproxy/<MD5SUM:3>
building package
......@@ -213,15 +211,17 @@ tool will be able to delete this previous version of ``haproxy``.
False
We also needed something to know the dependences between shared parts. For this, we are using
``.slapos.recipe.cmmi.signature`` files in folders where shared parts are installed. Because
``.buildout-shared.json`` files in folders where shared parts are installed. Because
``haproxy`` depends on ``package`` version ``<MD5SUM:2>``, we can see ``<MD5SUM:2>`` in its
signature file.
>>> haproxy_location = read_installed()['haproxy']['location']
>>> haproxy_location
'/shared/haproxy/<MD5SUM:3>'
>>> cat(join(haproxy_location, '.slapos.recipe.cmmi.signature')) # doctest: +ELLIPSIS
'__buildout_signature__': ...
...<MD5SUM:2>...
>>> cat(join(haproxy_location, '.buildout-shared.json')) # doctest: +ELLIPSIS
{
"__buildout_signature__": ...
...<MD5SUM:2>...
}
This is also useful during testing, we check that signatures do not have references to non-shared parts.
slapos/recipe/cmmi/README.rst
View file @ deda4108
......@@ -33,11 +33,7 @@ Supported options
``shared``
Specify the path in which this package is shared by many other
packages.
``shared-part-list`` should be defined in ``[buildout]`` section
Shared option is True or False
The package will be installed on path/name/hash of options.
See documentation of slapos.recipe.build's default recipe.
``md5sum``
......@@ -179,40 +175,9 @@ Supported options
If any item doesn't exist, the recipe shows a warning message. The
default value is empty.
``environment-section``
Name of a section that provides environment variables that will be used to
augment the variables read from ``os.environ`` before executing the
recipe.
This recipe does not modify ``os.environ`` directly. External commands
run as part of the recipe (e.g. make, configure, etc.) get an augmented
environment when they are forked. Python hook scripts are passed the
augmented as a parameter.
The values of the environment variables may contain references to other
existing environment variables (including themselves) in the form of
Python string interpolation variables using the dictionary notation. These
references will be expanded using values from ``os.environ``. This can be
used, for example, to append to the ``PATH`` variable, e.g.::
[component]
recipe = slapos.recipe.cmmi
environment-section =
environment
[environment]
PATH = %(PATH)s:${buildout:directory}/bin
``environment``
A sequence of ``KEY=VALUE`` pairs separated by newlines that define
additional environment variables used to update ``os.environ`` before
executing the recipe.
The semantics of this option are the same as ``environment-section``. If
both ``environment-section`` and ``environment`` are provided the values from
the former will be overridden by the latter allowing per-part customization.
See documentation of slapos.recipe.build's default recipe.
Additionally, the recipe honors the ``download-cache`` option set
in the ``[buildout]`` section and stores the downloaded files under
......@@ -444,17 +409,13 @@ with a new buildout and provide more options.
... newest = false
... parts = package
...
... [build-environment]
... CFLAGS = -I/sw/include
... LDFLAGS = -I/sw/lib
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... md5sum = 6b94295c042a91ea3203857326bc9209
... prefix = /somewhere/else
... environment-section = build-environment
... environment =
... CFLAGS=-I/sw/include
... LDFLAGS=-L/sw/lib -L/some/extra/lib
... configure-options =
... --with-threads
......@@ -467,16 +428,16 @@ with a new buildout and provide more options.
... patches/Makefile.dist.patch
... """ % package_path)
This configuration uses custom configure options, an environment section,
per-part customization to the environment, custom prefix, multiple make
targets and also patches the source code before the scripts are run.
This configuration uses custom configure options, environment variables,
custom prefix, multiple make targets and also patches the source code
before the scripts are run.
>>> print(system(buildout))
Uninstalling package.
Installing package.
package: Applying patches
package: [ENV] CFLAGS = -I/sw/include
package: [ENV] LDFLAGS = -L/sw/lib -L/some/extra/lib
package: Applying patches
patching file configure
patching file Makefile.dist
patched-configure --prefix=/somewhere/else --with-threads --without-foobar
......@@ -847,166 +808,6 @@ replaced with the recipe final prefix.
package: Executing pre-install
installing package at /sample_buildout/parts/package -lib
Install shared package
======================
Use option ``shared`` to install a shared package.
>>> import subprocess
>>> shared_dir = tmpdir('shared')
>>> another_shared_dir = tmpdir('another_shared_dir')
>>> __tear_downs.insert(0, lambda: subprocess.call(
... ('chmod', '-R', 'u+w', shared_dir, another_shared_dir)))
If no ``shared-part-list`` is set, and ``shared`` is True, ``shared`` feature
is not used.
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared = True
... """ % package_path)
>>> print(system(buildout)) #doctest:+ELLIPSIS
Uninstalling package.
Installing package.
configure --prefix=/sample_buildout/parts/package
building package
installing package
If ``shared-part-list`` is set, shared is True, and build package fails, the
part location is left for debugging.
Also a shell script with the environment variable is created, so that
developer can try same build process as the recipe tried.
>>> os.rename(package_path, package_path + '.bak')
>>> import tarfile
>>> from io import BytesIO
>>> import sys
>>> with tarfile.open(package_path, 'w:gz') as tar:
... configure = b'invalid'
... info = tarfile.TarInfo('configure.off')
... info.size = len(configure)
... info.mode = 0o755
... tar.addfile(info, BytesIO(configure))
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list = %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared = True
... environment =
... FOO=bar
... """ % (shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
>>> import glob
>>> cat(glob.glob(join(shared_dir, 'package/*/.build/slapos.recipe.build.env.sh'))[0])
export FOO=bar
...
Next time buildout runs, it detects that the build failed, remove the compile dir and retry.
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
If ``shared-part-list`` is set as an option in buildout section and
``shared`` is True, package will be installed in shared_part/package
and a hash of the recipe's configuration options.
There can be multiple path listed in ``shared-part-list``, the recipe
will look in each of these paths if package was already installed and
if not, it will install the package in the last entry.
>>> os.rename(package_path + '.bak', package_path)
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list =
... %s
... not/exists
... %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared = True
... environment =
... FOO=bar
... """ % (shared_dir, another_shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>
configure --prefix=/shared/package/<MD5SUM:0>
building package
installing package
If options change, reinstall in different location:
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list =
... %s
... not/exists
... %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared =True
... change = True
... """ % (shared_dir, another_shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /another_shared_dir/package/<MD5SUM:1>
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /another_shared_dir/package/<MD5SUM:1>
configure /another_shared_dir/package/<MD5SUM:1>
building package
installing package
For even more specific needs you can write your own recipe that uses
``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``.
You can then continue from where this recipe finished by reading the location
......
slapos/recipe/cmmi/__init__.py
View file @ deda4108
......@@ -7,11 +7,13 @@ import shutil
import stat
import subprocess
import sys
from hashlib import md5
import zc.buildout
from zc.buildout import UserError
from zc.buildout.buildout import bool_option
from slapos.recipe import downloadunpacked
from .. import (
# from slapos.recipe.build
EnvironMixin, Shared, downloadunpacked, is_true, rmtree
)
startup_environ = os.environ.copy()
......@@ -30,64 +32,50 @@ def quote(s):
return "'" + s.replace("'", "'\"'\"'") + "'"
###
class Recipe(object):
class Recipe(EnvironMixin):
"""zc.buildout recipe for compiling and installing software"""
buildout_prefix = ''
_shared = None
def __init__(self, buildout, name, options):
self.options = options
self.buildout = buildout
self.name = name
self.logger = logging.getLogger(self.name)
environment_section = options.get('environment-section')
self.environ = (
buildout[environment_section].copy()
if environment_section else {})
# Trigger computation of part signature for shared signature.
# From now on, we should not pull new dependencies.
# Ignore if buildout is too old.
options.get('__buildout_signature__')
shared = ((options.get('shared', '').lower() == 'true') and
buildout['buildout'].get('shared-part-list', None))
if shared:
self._signature = downloadunpacked.Signature(
'.slapos.recipe.cmmi.signature')
buildout_directory = buildout['buildout']['directory']
profile_base_location = options.get('_profile_base_location_', '')
for k, v in sorted(options.items()):
if k != '_profile_base_location_':
# Key not vary on profile base location
if profile_base_location:
v = v.replace(profile_base_location,
'${:_profile_base_location_}')
self._signature.update(k, v)
signature_digest = self._signature.hexdigest()
for x in shared.splitlines():
x = x.strip().rstrip('/')
if x:
shared = os.path.join(os.path.join(x, self.name),
signature_digest)
if os.path.exists(shared):
break
self.logger.info('shared at %s', shared)
else:
shared = ''
options['shared'] = shared
location = options['location'] = shared or os.path.join(
buildout['buildout']['parts-directory'],
self.name)
shared = Shared(buildout, name, options)
# It was never possible to choose location. It is sometimes set by the
# user when the value is needed before __init__ is called: in such
# case, the value must be the same as what Shared computes. However,
# the user should use %(location)s when possible.
location = options.get('location')
if location is not None and shared.location != location.replace(
'@@LOCATION@@', shared.location):
raise UserError("invalid 'location' value")
location = options['location'] = shared.location
self.buildout_prefix = ''
prefix = options.get('prefix')
if not prefix:
prefix = buildout['buildout'].get('prefix')
if prefix and 'cygwin' == sys.platform:
self.buildout_prefix = prefix
options['prefix'] = prefix or location
if prefix:
prefix = prefix.replace('@@LOCATION@@', location)
if os.path.commonprefix((prefix, location)) != location:
shared.assertNotShared("'prefix' can't be outside location")
else:
prefix = buildout['buildout'].get('prefix') # XXX: buggy
if prefix:
# XXX: one issue is that a change of ${buildout:prefix}
# does not cause a reinstallation of the part
shared.assertNotShared(
"option 'prefix' must be set"
" or ${buildout:prefix} can't be set")
shared = None
if 'cygwin' == sys.platform: # XXX: why?
self.buildout_prefix = prefix
options['prefix'] = prefix
else:
options['prefix'] = location
if shared:
shared.keep_on_error = True
self._shared = shared
url = options.get('url')
path = options.get('path')
......@@ -104,28 +92,7 @@ class Recipe(object):
if '@@LOCATION@@' in v:
options[k] = v.replace('@@LOCATION@@', location)
for variable in options.get('environment', '').splitlines():
if variable.strip():
try:
key, value = variable.split('=', 1)
self.environ[key.strip()] = value
except ValueError:
raise UserError('Invalid environment variable definition: %s' % variable)
def augmented_environment(self):
"""Returns a dictionary containing the current environment variables
augmented with the part specific overrides.
The dictionary is an independent copy of ``os.environ`` and
modifications will not be reflected in back in ``os.environ``.
"""
# Note that we don't set TMPDIR or TMP here as we use to do, because
# this path might be too deep and this will cause problem with some
# software (for example golang) who runs a test suite after build and
# use this TMPDIR to create unix sockets.
env = os.environ.copy()
env.update(self.environ)
return env
EnvironMixin.__init__(self, False)
def update(self):
pass
......@@ -140,15 +107,10 @@ class Recipe(object):
# if [buildout] has option 'prefix', then return all the files
# in this path which create time is newer than ref_file.
# Exclude directory and don't follow link.
assert self.buildout_prefix
args = ['find', self.buildout_prefix, '-cnewer', ref_file, '!', '-type', 'd']
try:
files = subprocess.check_output(args,
universal_newlines=True, close_fds=True)
except Exception as e:
self.logger.error(e)
raise UserError('System error')
return files.splitlines()
return subprocess.check_output((
'find', self.buildout_prefix,
'-cnewer', ref_file, '!', '-type', 'd',
), universal_newlines=True, close_fds=True).splitlines()
def check_promises(self):
result = True
......@@ -171,8 +133,7 @@ class Recipe(object):
module = imp.load_source('script', filename)
script = getattr(module, callable.strip())
try:
script(self.options, self.buildout,
self.augmented_environment())
script(self.options, self.buildout, self.environ)
except TypeError:
# BBB: Support hook scripts that do not take the environment as
# the third parameter
......@@ -185,27 +146,24 @@ class Recipe(object):
"""Run the given ``cmd`` in a child process."""
try:
subprocess.check_call('set -e;' + cmd, shell=True,
env=self.augmented_environment(), close_fds=True)
env=self.environ, close_fds=True)
except Exception as e:
self.logger.error(e)
raise UserError('System error')
def install(self):
shared = self._shared
if shared:
return shared.install(self._install)
location = self.options['location']
rmtree(location)
os.makedirs(location)
return self._install()
def _install(self):
log = self.logger
parts = []
# In shared mode, do nothing if package has been installed.
if (not self.options['shared'] == ''):
log.info('Checking whether package is installed at shared path: %s', self.options['shared'])
if self._signature.test(self.options['shared']):
log.info('This shared package has been installed by other package')
return parts
# Extrapolate the environment variables using values from the current
# environment.
for key in self.environ:
self.environ[key] %= os.environ
# Add prefix to PATH, CPPFLAGS, CFLAGS, CXXFLAGS, LDFLAGS
if self.buildout_prefix:
self.environ['PATH'] = '%s/bin:%s' % (self.buildout_prefix, self.environ.get('PATH', '/usr/bin'))
......@@ -241,35 +199,25 @@ class Recipe(object):
patch_options = ' '.join(self.options.get('patch-options', '-p0').split())
patches = self.options.get('patches', '').split()
if self.environ:
for key in sorted(self.environ.keys()):
log.info('[ENV] %s = %s', key, self.environ[key])
current_dir = os.getcwd()
url = self.options.get('url')
compile_dir = self.options['compile-directory']
location = self.options['location']
# Clean the install directory if it already exists as it is
# a remain from a previous failed installation
if os.path.exists(location):
log.warning('Removing already existing directory %s', location)
shutil.rmtree(location)
parts = [location]
# Download the source using slapos.recipe.downloadunpacked
if url:
os.mkdir(compile_dir)
self.options.get('md5sum') # so that buildout does not complain "unused option md5sum"
opt = self.options.copy()
opt['destination'] = compile_dir
# no need to shared build for compile dir
opt['shared'] = 'false'
opt['strip-top-level-dir'] = opt.get(
'strip-top-level-dir') or 'false'
downloadunpacked.Recipe(self.buildout, self.name, opt).install()
else:
log.info('Using local source directory: %s', compile_dir)
try:
os.makedirs(location)
# Download the source using slapos.recipe.downloadunpacked
if url:
os.mkdir(compile_dir)
self.options.get('md5sum') # so that buildout does not complain "unused option md5sum"
opt = self.options.copy()
opt['destination'] = compile_dir
# no need to shared build for compile dir
opt['shared'] = 'false'
opt['strip-top-level-dir'] = opt.get(
'strip-top-level-dir') or 'false'
downloadunpacked.Recipe(self.buildout, self.name, opt).install()
else:
log.info('Using local source directory: %s', compile_dir)
os.chdir(compile_dir)
try:
# We support packages that either extract contents to the $PWD
......@@ -327,15 +275,14 @@ class Recipe(object):
if post_install_cmd != '':
log.info('Executing post-install')
self.run(post_install_cmd)
if (self.buildout_prefix != ''
and self.options['shared'] == ''
if (self.buildout_prefix
and os.path.exists(self.buildout_prefix)):
log.info('Getting installed file lists')
parts.extend(self.get_installed_files(compile_dir))
parts += self.get_installed_files(compile_dir)
except:
with open('slapos.recipe.build.env.sh', 'w') as env_script:
for key, v in sorted(self.augmented_environment().items()):
for key, v in sorted(self.environ.items()):
if v != startup_environ.get(key):
env_script.write('%sexport %s=%s\n' % (
'#'[:key in ('TEMP', 'TMP', 'TMPDIR')],
......@@ -362,24 +309,15 @@ echo %s
finally:
os.chdir(current_dir)
if url:
if (self.options.get('keep-compile-dir') or
self.buildout['buildout'].get('keep-compile-dir') or
'').lower() not in ('true', 'yes', '1', 'on'):
shutil.rmtree(compile_dir)
if self.options['shared']:
self._signature.save(self.options["shared"])
if url and not is_true(
self.options.get('keep-compile-dir') or
self.buildout['buildout'].get('keep-compile-dir')):
shutil.rmtree(compile_dir)
# Check promises
self.check_promises()
if self.options['shared'] == '':
parts.append(location)
self.fix_shebang(location)
if self.options['shared']:
downloadunpacked.make_read_only_recursively(location)
return parts
def fix_shebang(self, location):
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7