Skip to content

S
slapos.recipe.cmmi
Commit 50958d2e authored by Jondy Zhao's avatar Jondy Zhao Committed by Cédric de Saint Martin
Browse files
Options

Add support for multiple platforms.

parent f10c38e3
Hide whitespace changes
Inline Side-by-side
Showing with 813 additions and 11 deletions
slapos/recipe/cmmi/README.txt
View file @ 50958d2e
......@@ -89,10 +89,13 @@ Supported options
``configure`` script. The format of the options is::
/path/to/the/module.py:name_of_callable
url:name_of_callable
url#md5sum:name_of_callable
where the first part is a filesystem path to the python module and the
second part is the name of the callable in the module that will be called.
The callable will be passed three parameters in the following order:
where the first part is a filesystem path or url to the python
module and the second part is the name of the callable in the
module that will be called. The callable will be passed three
parameters in the following order:
1. The ``options`` dictionary from the recipe.
......@@ -121,6 +124,31 @@ Supported options
``make``. The format and semantics are the same as with the
``pre-configure-hook`` option.
.. hook shell command:
``pre-configure``
Shell command that will be executed before running ``configure``
script. It takes the same effect as ``pre-configure-hook`` option
except it's shell command.
``pre-build``
Shell command that will be executed before running ``make``. It
takes the same effect as ``pre-make-hook`` option except it's
shell command.
``pre-install``
Shell command that will be executed before running ``make``
install.
``post-install``
Shell command that will be executed after running ``make``. It
takes the same effect as ``post-make-hook`` option except it's
shell command.
``keep-compile-dir``
Switch to optionally keep the temporary directory where the
......@@ -130,6 +158,16 @@ Supported options
compile directory is stored in ``options['compile-directory']``.
Accepted values are ``true`` or ``false``, defaults to ``false``.
``dependencies``
List all the depended parts:
dependencies = part1 part2 ...
All the dependent parts will be installed before this part, besides
the changes in any dependent parts will trigger to reinstall
current part.
``environment-section``
Name of a section that provides environment variables that will be used to
......@@ -165,6 +203,29 @@ Supported options
both ``environment-section`` and ``environment`` are provided the values from
the former will be overridden by the latter allowing per-part customization.
The recipe uses separated part to support custom options in different
platforms. These platform's part has a pattern "platform-part" or
"arch-platform-part".
arch could be 'x86', 'x86_64', 'ia64' ... which equals
platform.machine().
platform could be 'linux', 'cygwin', 'macos', 'sunos', 'freebsd',
'netbsd', 'unixware' ... which equals a formatted sys.platform.
For example,
[bzip2]
recipe = slapos.recipe.cmmi
[x86-cygwin-bzip2]
patches = cygwin-bzip2-1.0.6.src.patch
All the options in the platform-part have high priority level.
The recipe first searches the exact match, if no found. Ignore arch
and search again, if still found nothing. Use no platform part.
Additionally, the recipe honors the ``download-cache`` option set
in the ``[buildout]`` section and stores the downloaded files under
it. If the value is not set a directory called ``downloads`` will be
......@@ -176,6 +237,27 @@ before downloading it from the net. Files can be shared among
different buildouts by setting the ``download-cache`` to the same
location.
The recipe honors the ``prefix`` option set in the ``[buildout]``
section either. It implicts all the parts which recipe is
slapos.recipe.cmmi in this buildout process will be installed in the
same ``prefix`` option in the ``[buildout]``. Besides, once it takes
effects, recipe will return all the installed files in the prefix
directory. The own ``prefix`` of part will disable this behaviour.
If the ``buildout`` section has a valid ``prefix`` option, the recipe
will add it to environmet variables as the following:
PATH=${buildout:prefix}/bin:$PATH
CPPFLAGS=-I${buildout:prefix} $CPPFLAGS
CFLAGS=-I${buildout:prefix} $CFFLAGS
CXXFLAGS=-I${buildout:prefix} $CXXFLAGS
LDFLAGS=-L${buildout:prefix}/lib
Besides, the recipe changes environment variable ``TMP`` when building
and installing, and make a corresponding directory 'tmp' in the
``location``. This temporary directory will be removed after
installing finished.
Example usage
=============
......@@ -210,6 +292,7 @@ default build options.
>>> print(system(buildout))
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
configure --prefix=/sample_buildout/parts/package
building package
......@@ -245,6 +328,7 @@ a custom location within the buildout::
>>> print(system(buildout))
Uninstalling package.
Installing foobar.
foobar: [ENV] TMP = /sample_buildout/parts/foobar/tmp
foobar: Extracting package to /sample_buildout/parts/foobar__compile__
building package
installing package
......@@ -284,6 +368,7 @@ Makefile and using explicit ``make`` options to control the build process.
>>> print(system(buildout))
Uninstalling foobar.
Installing haproxy.
haproxy: [ENV] TMP = /sample_buildout/parts/haproxy/tmp
haproxy: Extracting package to /sample_buildout/parts/haproxy__compile__
Building HAProxy 1.4.8 (dummy package)
TARGET: linux26
......@@ -325,6 +410,7 @@ and building that.
>>> print(system(buildout))
Uninstalling haproxy.
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Using local source directory: /checkout/package-0.0.0
configure --prefix=/sample_buildout/parts/package
building package
......@@ -384,6 +470,7 @@ targets and also patches the source code before the scripts are run.
Installing package.
package: [ENV] CFLAGS = -I/sw/include
package: [ENV] LDFLAGS = -L/sw/lib -L/some/extra/lib
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
package: Applying patches
patching file configure
......@@ -467,6 +554,7 @@ and a new buildout to try it out
>>> print(system(buildout))
Uninstalling package.
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
package: Executing pre-configure-hook
hook: This is pre-configure-hook!
......@@ -478,6 +566,464 @@ and a new buildout to try it out
package: Executing post-make-hook
hook: This is post-make-hook!
If you prefer to use shell script, then try these options:
pre-configure
pre-build
pre-install
post-install
Let's create a buildout to use these options.
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... pre-configure = echo "Configure part: ${:_buildout_section_name_}"
... pre-build = echo "OH OH OH" > a.txt
... pre-install = cat a.txt
... post-install = rm -f a.txt && echo "Finished."
... """ % src)
This will run pre-configure, pre-build, pre-install, post-install as
shell command in the corresponding stage.
>>> print system(buildout)
Uninstalling package.
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
package: Executing pre-configure
Configure part: package
configure --prefix=/sample_buildout/parts/package
package: Executing pre-build
building package
package: Executing pre-install
OH OH OH
installing package
package: Executing post-install
Finished.
Building in multi-platforms
===========================
The recipe can specify build options for each platform. For example,
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... pre-configure = echo "Configure in common platform"
... post-install = echo "Finished."
...
... [cygwin-package]
... pre-configure = echo "Configure in the CYGWIN platform"
... pre-install = echo "Installing in the CYGWIN"
... post-install = echo -n "CYGWIN " && ${package:post-install}
... """ % src)
In the linux, the recipe gets the options from part 'package', there
are only ``pre-configure`` and ``post-install``. the output will be
#>>> print system(buildout)
Uninstalling package.
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
package: Executing pre-configure
Configure part: Configure in common platform
configure --prefix=/sample_buildout/parts/package
building package
installing package
package: Executing post-install
Finished.
In the cygwin, the recipe merges the options in the parts 'package'
and 'cygwin-package'. The output will be
>>> print system(buildout)
Uninstalling package.
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
package: Executing pre-configure
Configure in the CYGWIN platform
configure --prefix=/sample_buildout/parts/package
building package
package: Executing pre-install
Installing in the CYGWIN
installing package
package: Executing post-install
CYGWIN Finished.
Arch is supported either, For example,
[x86-cygwin-package]
...
The recipe will first search part looks like "arch-platform-part",
then "platform-part".
Union prefix
============
If the recipe finds ``prefix`` option in the section buildout, it will
* First, use this ``prefix`` as configure prefix, if
``configure-command`` isn't set in the part, or ``make-binary``
equals 'make' and ``make-target`` includes pattern '\s+install.*'
* Second, return all the new installed files in the prefix when the
recipe returns after intall.
* Finally, change some environment variables(See first section).
Let's see what happens when set prefix in the buildout section:
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... prefix = ${buildout:directory}/mylocal
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... pre-configure = mkdir -p "${buildout:prefix}"
... """ % src)
>>> print system(buildout)
Uninstalling package.
Installing package.
package: [ENV] CFLAGS = /sample-buildout/mylocal/include
package: [ENV] CPPFLAGS = /sample-buildout/mylocal/include
package: [ENV] CXXFLAGS = /sample-buildout/mylocal/include
package: [ENV] LDFLAGS = /sample-buildout/mylocal/lib
package: [ENV] PATH = /sample_buildout/mylocal/bin:/usr/bin
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
package: Executing pre-configure
configure --prefix=/sample_buildout/mylocal
building package
installing package
Look these environment variables and prefix's value, you know what's
the differences.
If part has its own ``prefix``, it will disable above behavious. For
example,
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... prefix = ${buildout:directory}/mylocal
...
... [package]
... recipe = slapos.recipe.cmmi
... prefix = ${buildout:parts-directory}/package
... url = file://%s/package-0.0.0.tar.gz
... pre-configure = rm -rf "${buildout:prefix}"
... post-install = test -d "${buildout:prefix}" || echo "None"
... """ % src)
>>> print system(buildout)
Uninstalling package.
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
package: Executing pre-configure
configure --prefix=/sample_buildout/parts/package
building package
installing package
package: Executing post-install
None
Then no extra environment variables such as CFLAGS etc., and no
${buildout:prefix} directory is created.
The following example shows how to install package, package-2 in one
prefix:
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package package-2
... prefix = ${buildout:directory}/mylocal
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... pre-install = sleep 2; mkdir -p "${buildout:prefix}" ; echo x >"${buildout:prefix}/a.txt"
... [package-2]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... pre-install = sleep 2; mkdir -p "${buildout:prefix}" ; echo x >"${buildout:prefix}/b.txt"; echo
... """ % (src, src))
>>> print system(buildout)
Uninstalling package.
Installing package.
package: [ENV] CFLAGS = /sample-buildout/mylocal/include
package: [ENV] CPPFLAGS = /sample-buildout/mylocal/include
package: [ENV] CXXFLAGS = /sample-buildout/mylocal/include
package: [ENV] LDFLAGS = /sample-buildout/mylocal/lib
package: [ENV] PATH = /sample_buildout/mylocal/bin:/usr/bin
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
configure --prefix=/sample_buildout/mylocal
building package
package: Executing pre-install
installing package
Installing package-2.
package-2: [ENV] CFLAGS = /sample-buildout/mylocal/include
package-2: [ENV] CPPFLAGS = /sample-buildout/mylocal/include
package-2: [ENV] CXXFLAGS = /sample-buildout/mylocal/include
package-2: [ENV] LDFLAGS = /sample-buildout/mylocal/lib
package-2: [ENV] PATH = /sample_buildout/mylocal/bin:/usr/bin
package-2: [ENV] TMP = /sample_buildout/parts/package-2/tmp
package-2: Extracting package to /sample_buildout/parts/package-2__compile__
configure --prefix=/sample_buildout/mylocal
building package
package-2: Executing pre-install
installing package
>>> ls('mylocal')
- a.txt
- b.txt
Next we unintall package-2, it should only remove file b.txt:
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... prefix = ${buildout:directory}/mylocal
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... pre-install = sleep 2; mkdir -p "${buildout:prefix}" ; echo x >"${buildout:prefix}/a.txt"
... """ % src)
>>> print system(buildout)
Uninstalling package-2.
Updating package.
>>> ls('mylocal')
- a.txt
Magic prefix
============
If configure-command is set, the recipe wouldn't insert "--prefix"
into configure-options. Then it checks whether both of make-binary and
make-targets aren't set, if so, string "prefix=xxx" will be appended
in the make-targets. xxx is the final prefix of this recipe. We call
it Magic Prefix.
In these options magic prefix can be represented by %(prefix)s:
``onfigure-command`` ``configure-options``
``make-binary`` ``make-options`` ``make-targets``
``pre-configure`` ``pre-build`` ``pre-install`` ``post-install``
For example,
[bzip2]
post-install = rm %(prefix)s/*.h
The other part can refer to magic prefix of this part by
${part:prefix}, it will return the magic prefix, other than literal
value in the part section. For example,
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package package-2
... prefix = /mytemp
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... configure-command = true
... make-binary = true
...
... [package-2]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... configure-command = true
... make-binary = true
... post-install = echo package magic prefix is ${package:prefix}
... """ % (src, src))
>>> print system(buildout)
Uninstalling package.
Installing package.
package: [ENV] CFLAGS = -I/mytemp/include
package: [ENV] CPPFLAGS = -I/mytemp/include
package: [ENV] CXXFLAGS = -I/mytemp/include
package: [ENV] LDFLAGS = -L/mytemp/lib
package: [ENV] PATH = /mytemp/bin:/usr/bin
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
Installing package-2.
package-2: [ENV] CFLAGS = -I/mytemp/include
package-2: [ENV] CPPFLAGS = -I/mytemp/include
package-2: [ENV] CXXFLAGS = -I/mytemp/include
package-2: [ENV] LDFLAGS = -L/mytemp/lib
package-2: [ENV] PATH = /mytemp/bin:/usr/bin
package-2: [ENV] TMP = /sample_buildout/parts/package-2/tmp
package-2: Extracting package to /sample_buildout/parts/package-2__compile__
package-2: Executing post-install
package magic prefix is /mytemp
Here it's another sample, we change Makefile before installing so it
can display "prefix" value in the stdout.
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... configure-command = ./configure
... pre-install = sed -i -e "s/installing package/installing package at \$\$prefix /g" Makefile
... """ % src)
>>> print system(buildout)
Uninstalling package-2.
Uninstalling package.
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
configure
building package
package: Executing pre-install
installing package at /sample_buildout/parts/package
You even can include pattern %(prefix)s in this option, it will be
replaced with the recipe final prefix.
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... configure-command = ./configure
... make-targets = install-lib prefix=%%(prefix)s
... pre-install = sed -i -e "s/installing package/installing package at \$\$prefix /g" Makefile
... """ % src)
>>> print system(buildout)
Uninstalling package.
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
configure
building package
package: Executing pre-install
installing package at /sample_buildout/parts/package -lib
Extra part dependencies
=======================
The recipe will treat all the parts list in the option
``dependencies`` as dependent parts. zc.buildout will install all the
dependent parts before install this part. For example,
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = slapos.recipe.cmmi
... dependencies = package-2
... url = file://%s/package-0.0.0.tar.gz
...
... [package-2]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... """ % (src, src))
Here "package-2" will be installed first, because it's a denpend part
of "package":
>>> print system(buildout)
Uninstalling package.
Installing package-2.
package-2: [ENV] TMP = /sample_buildout/parts/package-2/tmp
package-2: Extracting package to /sample_buildout/parts/package-2__compile__
configure --prefix=/sample_buildout/parts/package-2
building package
installing package
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
configure --prefix=/sample_buildout/parts/package
building package
installing package
Now let's add a new option for "package-2",
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
...
... [package]
... recipe = slapos.recipe.cmmi
... dependencies = package-2
... url = file://%s/package-0.0.0.tar.gz
...
... [package-2]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... configure-command = ./configure
... """ % (src, src))
Look, "package" is reinstalled either:
>>> print system(buildout)
Uninstalling package.
Uninstalling package-2.
Installing package-2.
package-2: [ENV] TMP = /sample_buildout/parts/package-2/tmp
package-2: Extracting package to /sample_buildout/parts/package-2__compile__
configure
building package
installing package
Installing package.
package: [ENV] TMP = /sample_buildout/parts/package/tmp
package: Extracting package to /sample_buildout/parts/package__compile__
configure --prefix=/sample_buildout/parts/package
building package
installing package
For even more specific needs you can write your own recipe that uses
``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``.
You can then continue from where this recipe finished by reading the location
......
slapos/recipe/cmmi/__init__.py
View file @ 50958d2e
import errno
from hashlib import md5
import hexagonit.recipe.download
import imp
import logging
import os
import pkg_resources
from platform import machine as platform_machine
import re
import shutil
import subprocess
import sys
import zc.buildout
from zc.buildout.download import Download
class Recipe(object):
......@@ -16,13 +22,45 @@ class Recipe(object):
self.buildout = buildout
self.name = name
# Merge options if there is a matched platform section
platform_options = self.get_platform_options()
if platform_options is None:
self.original_options = options
else:
self.original_options = options.copy()
options.update(platform_options)
options['location'] = os.path.join(
buildout['buildout']['parts-directory'],
self.name)
options['prefix'] = options.get('prefix', options['location'])
prefix = options.get('prefix', '').strip()
if prefix == '':
prefix = self.buildout_prefix = buildout['buildout'].get('prefix', '').strip()
else:
self.buildout_prefix = ''
options['prefix'] = options['location'] if prefix == '' else prefix
options['url'] = options.get('url', '').strip()
options['path'] = options.get('path', '').strip()
# Check dependencies, all the dependent parts will be installed first. It
# seems once part is referenced, for example, self.buildout[part], it will
# be appended as an install part.
# dpendent_parts = options.get('dependencies', '').split()
# assert isinstance(buildout, zc.buildout.buildout.Buildout)
# buildout._compute_part_signatures(
# [part for part in dpendent_parts if part not in buildout._parts])
# Calculate md5sum of all the options for each dependent part, and save it
# as option "dependencies". So if any dependent part changes, zc.buildout
# will reinstall the part because this option is changed.
dependencies = []
for part in options.get('dependencies', '').split():
m = md5()
for (k, v) in self.buildout[part].iteritems():
m.update(v)
dependencies.append(m.hexdigest())
options['dependencies'] = ' '.join(dependencies)
if options['url'] and options['path']:
raise zc.buildout.UserError('You must use either "url" or "path", not both!')
if not (options['url'] or options['path']):
......@@ -47,10 +85,19 @@ class Recipe(object):
self.environ[key.strip()] = value
except ValueError:
raise zc.buildout.UserError('Invalid environment variable definition: %s', variable)
# Add prefix to PATH, CPPFLAGS, CFLAGS, CXXFLAGS, LDFLAGS
if self.buildout_prefix != '':
self.environ['PATH'] = '%s/bin:%s' % (self.buildout_prefix, self.environ.get('PATH', '/usr/bin'))
self.environ['CPPFLAGS'] = '-I%s/include %s' % (self.buildout_prefix, self.environ.get('CPPFLAGS', ''))
self.environ['CFLAGS'] = '-I%s/include %s' % (self.buildout_prefix, self.environ.get('CFLAGS', ''))
self.environ['CXXFLAGS'] = '-I%s/include %s' % (self.buildout_prefix, self.environ.get('CXXFLAGS', ''))
self.environ['LDFLAGS'] = '-L%s/lib %s' % (self.buildout_prefix, self.environ.get('LDFLAGS', ''))
# Extrapolate the environment variables using values from the current
# environment.
for key in self.environ:
self.environ[key] = self.environ[key] % os.environ
self.environ['TMP'] = os.path.join(options['location'], 'tmp')
def augmented_environment(self):
"""Returns a dictionary containing the current environment variables
......@@ -66,13 +113,80 @@ class Recipe(object):
def update(self):
pass
def _compute_part_signatures(self, options):
# Copy from zc.buildout.Buildout, compute recipe signature
recipe, entry = zc.buildout.buildout._recipe(options)
req = pkg_resources.Requirement.parse(recipe)
sig = zc.buildout.buildout._dists_sig(pkg_resources.working_set.resolve([req]))
return ' '.join(sig)
def get_platform(self):
# Get value of sys.platform
for platform in ['linux', 'cygwin', 'beos', 'darwin', 'atheos', 'osf1',
'netbsd', 'openbsd', 'freebsd', 'unixware', 'sunos']:
if sys.platform.startswith(platform):
return platform
return sys.platform
def get_machine(self):
arch = platform_machine()
# i?86-*-* : x86
if arch in ('i386', 'i586', 'i686'):
return 'x86'
# x86_64-*-* : amd64
elif arch == 'x86_64':
return 'amd64'
# ia64-*-* : ia64
# and others
return arch
def get_platform_options(self):
platform_part = self.get_platform() + '-' + self.name
part_list = [part for part in self.buildout if part.endswith(platform_part)]
if part_list[:1]:
arch_prefix = self.get_machine() + '-'
for part in part_list:
if part.startswith(arch_prefix):
return self.buildout[part]
else:
return self.buildout.get(platform_part)
def download_file(self, url):
download = Download(self.buildout['buildout'])
url, _s_, md5sum = url.partition('#')
return download(url, md5sum=None if md5sum == '' else md5sum)
def get_installed_files(self, ref_file):
# if [buildout] has option 'prefix', then return all the files
# in this path which create time is newer than ref_file.
# Exclude directory and don't follow link.
assert self.buildout_prefix
log = logging.getLogger(self.name)
cmd = 'find %s -cnewer %s ! -type d' % (self.buildout_prefix, ref_file)
try:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
files, _i_ = p.communicate()
retcode = p.returncode
if retcode < 0:
log.error('Command received signal %s: %s' % (-retcode, cmd))
raise zc.buildout.UserError('System error')
elif retcode > 0:
log.error('Command failed with exit code %s: %s' % (retcode, cmd))
raise zc.buildout.UserError('System error')
except OSError, e:
log.error('Command failed: %s: %s' % (e, cmd))
raise zc.buildout.UserError('System error')
return files.split()
def call_script(self, script):
"""This method is copied from z3c.recipe.runscript.
See http://pypi.python.org/pypi/z3c.recipe.runscript for details.
"""
filename, callable = script.rsplit(':', 1)
filename = os.path.abspath(filename)
url, callable = script.rsplit(':', 1)
filename, is_temp = self.download_file(url)
if not is_temp:
filename = os.path.abspath(filename)
module = imp.load_source('script', filename)
script = getattr(module, callable.strip())
......@@ -82,6 +196,9 @@ class Recipe(object):
# BBB: Support hook scripts that do not take the environment as
# the third parameter
script(self.options, self.buildout)
finally:
if is_temp:
os.remove(filename)
def run(self, cmd):
"""Run the given ``cmd`` in a child process."""
......@@ -116,6 +233,8 @@ class Recipe(object):
# Inject the --prefix parameter if not already present
if '--prefix' not in ' '.join(configure_options):
configure_options.insert(0, '--prefix=\"%s\"' % self.options['prefix'])
elif make_cmd == 'make' and make_targets == 'install':
make_targets += ' prefix=\"%s\"' % self.options['prefix']
patch_cmd = self.options.get('patch-binary', 'patch').strip()
patch_options = ' '.join(self.options.get('patch-options', '-p0').split())
......@@ -152,6 +271,9 @@ class Recipe(object):
if e.errno == errno.EEXIST:
pass
os.chdir(compile_dir)
tmp_path = self.environ['TMP']
shutil.rmtree(tmp_path, True)
os.mkdir(tmp_path)
try:
try:
......@@ -165,34 +287,62 @@ class Recipe(object):
if patches:
log.info('Applying patches')
for patch in patches:
self.run('%s %s < %s' % (patch_cmd, patch_options, patch))
patch_filename, is_temp = self.download_file(patch)
self.run('%s %s < %s' % (patch_cmd, patch_options, patch_filename))
if is_temp:
os.remove(path_filename)
if 'pre-configure-hook' in self.options and len(self.options['pre-configure-hook'].strip()) > 0:
log.info('Executing pre-configure-hook')
self.call_script(self.options['pre-configure-hook'])
self.run('%s %s' % (configure_cmd, ' '.join(configure_options)))
pre_configure_cmd = self.options.get('pre-configure', '').strip() % self.options
if pre_configure_cmd != '':
log.info('Executing pre-configure')
self.run(pre_configure_cmd)
self.run(('%s %s' % (configure_cmd, ' '.join(configure_options))) % self.options)
if 'pre-make-hook' in self.options and len(self.options['pre-make-hook'].strip()) > 0:
log.info('Executing pre-make-hook')
self.call_script(self.options['pre-make-hook'])
self.run('%s %s' % (make_cmd, make_options))
self.run('%s %s %s' % (make_cmd, make_options, make_targets))
pre_build_cmd = self.options.get('pre-build', '').strip() % self.options
if pre_build_cmd != '':
log.info('Executing pre-build')
self.run(pre_build_cmd)
self.run(('%s %s' % (make_cmd, make_options)) % self.options)
pre_install_cmd = self.options.get('pre-install', '').strip() % self.options
if pre_install_cmd != '':
log.info('Executing pre-install')
self.run(pre_install_cmd)
self.run(('%s %s %s' % (make_cmd, make_options, make_targets)) % self.options)
if 'post-make-hook' in self.options and len(self.options['post-make-hook'].strip()) > 0:
log.info('Executing post-make-hook')
self.call_script(self.options['post-make-hook'])
post_install_cmd = self.options.get('post-install', '').strip() % self.options
if post_install_cmd != '':
log.info('Executing post-install')
self.run(post_install_cmd)
if self.buildout_prefix != '' and os.path.exists(self.buildout_prefix):
log.info('Getting installed file lists')
parts.extend(self.get_installed_files(tmp_path))
except:
log.error('Compilation error. The package is left as is at %s where '
'you can inspect what went wrong' % os.getcwd())
raise
finally:
os.chdir(current_dir)
shutil.rmtree(tmp_path)
if self.options['url']:
if self.options.get('keep-compile-dir', '').lower() in ('true', 'yes', '1', 'on'):
if self.options.get('keep-compile-dir',
self.buildout['buildout'].get('keep-compile-dir', '')).lower() in ('true', 'yes', '1', 'on'):
# If we're keeping the compile directory around, add it to
# the parts so that it's also removed when this recipe is
# uninstalled.
......
slapos/recipe/cmmi/tests.py
View file @ 50958d2e
......@@ -60,6 +60,7 @@ class NonInformativeTests(unittest.TestCase):
}
}
bo.update(buildout)
bo['buildout'].update(buildout_options)
return Recipe(bo, name, options)
def test_working_directory_restored_after_failure(self):
......@@ -199,6 +200,111 @@ class NonInformativeTests(unittest.TestCase):
except ValueError as e:
self.assertEqual(str(e), 'sentinel bar')
def no_test_make_target_with_prefix(self):
recipe = self.make_recipe({}, 'test', {
'url': 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__),
'configure-command' : './configure',
'pre-install' : 'sed -i -e "s/installing package/installing package at \$\$prefix /g" Makefile',
})
os.chdir(self.dir)
recipe.install()
recipe = self.make_recipe({}, 'test', {
'url': 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__),
'pre-install' : 'sed -i -e "s/installing package/installing package at \$\$prefix /g" Makefile',
'make-targets' : 'install-lib prefix=%(prefix)s',
})
recipe.install()
def test_download_file(self):
recipe = self.make_recipe({}, 'test', {
'url': 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__),
})
url = '%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__)
file, is_temp = recipe.download_file(url)
self.assertFalse(is_temp)
self.assertEquals(file, url)
url = 'ftp://ftp.gnu.org/welcome.msg'
file, is_temp = recipe.download_file(url)
self.assertTrue(is_temp)
self.assertTrue(os.path.exists(file))
url = 'ftp://ftp.gnu.org/welcome.msg#ec7ab8024467ba3b6e173c57fd4990f6'
file, is_temp = recipe.download_file(url)
self.assertTrue(is_temp)
self.assertTrue(os.path.exists(file))
url = 'ftp://ftp.gnu.org/welcome.msg#0205'
self.assertRaises(zc.buildout.download.ChecksumError, recipe.download_file, url)
def test_buildout_prefix(self):
buildout_prefix = os.path.join(self.dir, 'test_parts/test_local')
os.makedirs(buildout_prefix)
recipe = self.make_recipe({}, 'test', {
'url': 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__),
},
prefix=buildout_prefix
)
self.assertEquals(recipe.options.get('prefix'), buildout_prefix)
recipe = self.make_recipe({}, 'test', {
'url': 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__),
'prefix' : self.dir,
},
prefix=buildout_prefix
)
self.assertEquals(recipe.options.get('prefix'), self.dir)
def test_get_installed_files(self):
prefix = os.path.join(self.dir, 'test_parts/test_local')
os.makedirs(prefix)
recipe = self.make_recipe({}, 'test', {
'url': 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__),
})
os.chdir(self.dir)
# The hook script does not return anything so we (ab)use exceptions
# as a mechanism for asserting the function behaviour.
no_installed_files = ('a.txt', 'b.txt', 'c', 'c/d.txt')
installed_files = ['e.txt', 'f.txt', 'g', 'g/h.txt']
for s in no_installed_files:
if s.endswith('.txt'):
f = open(os.path.join(prefix, s), 'w')
f.write(s)
f.close()
else:
os.makedirs(os.path.join(prefix, s))
sleep(2)
ref_path = os.path.join(self.dir, 'refs')
os.makedirs(ref_path)
sleep(2)
for s in installed_files:
if s.endswith('.txt'):
f = open(os.path.join(prefix, s), 'w')
f.write(s)
f.close()
else:
os.makedirs(os.path.join(prefix, s))
recipe.buildout_prefix = prefix
file_list = recipe.get_installed_files(ref_path)
installed_files.pop(2)
self.assertEquals([os.path.relpath(f, prefix) for f in file_list], installed_files)
def test_honor_buidlout_keep_compile_directory(self):
buildout = {'keep-compile-dir' : 'true'}
recipe = self.make_recipe({}, 'test', {
'url': 'file://%s/testdata/package-0.0.0.tar.gz' % os.path.dirname(__file__),
},
**buildout
)
os.chdir(self.dir)
recipe.install()
build_directory = os.path.join(self.dir, 'test_parts/test__compile__')
self.assertTrue(os.path.exists(build_directory))
def test_suite():
suite = unittest.TestSuite((
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7