Skip to content

S
slapos.recipe.cmmi
Commit 509ee4a0 authored by Julien Muchembled's avatar Julien Muchembled
3 Browse files
Options

For compiling, use a temporary directory that is inside the part location 

We want to keep the build directory when keep-compile-dir=true,
whereas sharing a part is reliable only if everything goes inside
the same directory.
parent 73f3f296
Show whitespace changes
Inline Side-by-side
Showing with 48 additions and 132 deletions
slapos/recipe/cmmi/README.rst
View file @ 509ee4a0
......@@ -943,9 +943,8 @@ is not used.
building package
installing package
If ``shared-part-list`` is set and shared is True, build package failed, the
build directory is removed, a build directory__compile__ is left for
debugging.
If ``shared-part-list`` is set, shared is True, and build package fails, the
part location is left for debugging.
Also a shell script with the environment variable is created, so that
developer can try same build process as the recipe tried.
......@@ -980,14 +979,14 @@ developer can try same build process as the recipe tried.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>__compile__ where you can inspect what went wrong.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
>>> import glob
>>> cat(glob.glob(join(shared_dir, 'package/**__compile__/slapos.recipe.build.env.sh'))[0])
>>> cat(glob.glob(join(shared_dir, 'package/*/.build/slapos.recipe.build.env.sh'))[0])
export FOO=bar
...
......@@ -998,9 +997,9 @@ Next time buildout runs, it detects that the build failed, remove the compile di
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>__compile__
package: Removing already existing directory /shared/package/<MD5SUM:0>
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>__compile__ where you can inspect what went wrong.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found
While:
......@@ -1008,72 +1007,15 @@ Next time buildout runs, it detects that the build failed, remove the compile di
Error: System error
But we had a bug with version v0.11 is that if build was interrupted in the middle of the build of a
shared part. The shared part was left in an inconsistent state that looked like installation succeeded.
Let's simulate a scenario where buildout is terminated in the middle of a build.
For this, we need a package that takes a lot of time to install.
>>> with tarfile.open(package_path, 'w:gz') as tar:
... configure = b'#!/bin/sh\necho configure started\nsleep 300;'
... info = tarfile.TarInfo('configure')
... info.size = len(configure)
... info.mode = 0o755
... tar.addfile(info, BytesIO(configure))
We also need a bit more complex method to run buildout so that we send a termination signal in the
middle of build process.
>>> import signal
>>> import time
>>> buildout_process = subprocess.Popen(buildout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>>> output_line = ''
>>> for _ in range(10):
... time.sleep(1)
... output_line = buildout_process.stdout.readline()
... if output_line and b'configure started' in output_line:
... print ('configure started')
... buildout_process.send_signal(signal.SIGKILL)
... print ('buildout terminated')
... break
configure started
buildout terminated
>>> _ = buildout_process.wait()
If we run buildout again, the compile dir is removed again and installation is retried. This time
installation can succeed.
This was not the case in version 0.11 of this recipe. If installation of a shared part failed, this was
not detected and the part was considered as installed.
>>> os.rename(package_path + '.bak', package_path)
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>__compile__
configure --prefix=/shared/package/<MD5SUM:0>
building package
installing package
If ``shared-part-list`` is set as an option in buildout section and
``shared`` is True, package will be installed in shared_part/package
and a hash of the recipe's configuration options.
There can be multiple path listed in ``shared-part-list``, the recipe
will look in each of these paths if package was already installed and
if not, it will install the package in the last entry the last entry
from the list of ``shared-part-list``.
if not, it will install the package in the last entry.
If package was already installed in any of the ``shared-part-list`` used, it will be
used instead of installing if one package has been installed.
>>> remove('.installed.cfg')
>>> os.rename(package_path + '.bak', package_path)
>>> write('buildout.cfg',
... """
... [buildout]
......@@ -1095,29 +1037,36 @@ used instead of installing if one package has been installed.
package: shared at /shared/package/<MD5SUM:0>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: This shared package has been installed by other package
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>
configure --prefix=/shared/package/<MD5SUM:0>
building package
installing package
If options change, reinstall in different location:
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part-list = %s
... shared-part-list =
... %s
... not/exists
... %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s
... shared =True
... change = True
... """ % (shared_dir, package_path))
... """ % (shared_dir, another_shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:1>
package: shared at /another_shared_dir/package/<MD5SUM:1>
Uninstalling package.
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:1>
configure --prefix=/shared/package/<MD5SUM:1>
package: Checking whether package is installed at shared path: /another_shared_dir/package/<MD5SUM:1>
configure /another_shared_dir/package/<MD5SUM:1>
building package
installing package
......
slapos/recipe/cmmi/__init__.py
View file @ 509ee4a0
......@@ -111,7 +111,7 @@ class Recipe(object):
raise UserError('You must provide either "url" or "path".')
if options['url']:
options['compile-directory'] = location + '__compile__'
options['compile-directory'] = os.path.join(location, '.build')
else:
options['compile-directory'] = options['path']
......@@ -297,42 +297,33 @@ class Recipe(object):
for key in sorted(self.environ.keys()):
log.info('[ENV] %s = %s', key, self.environ[key])
# Download the source using slapos.recipe.downloadunpacked
if self.options['url']:
current_dir = os.getcwd()
compile_dir = self.options['compile-directory']
if os.path.exists(compile_dir):
# leftovers from a previous failed attempt, removing it.
log.warning('Removing already existing directory %s', compile_dir)
shutil.rmtree(compile_dir)
os.makedirs(compile_dir)
location = self.options['location']
# Clean the install directory if it already exists as it is
# a remain from a previous failed installation
if os.path.exists(location):
log.warning('Removing already existing directory %s', location)
shutil.rmtree(location)
try:
os.makedirs(location)
# Download the source using slapos.recipe.downloadunpacked
if self.options['url']:
os.mkdir(compile_dir)
self.options.get('md5sum') # so that buildout does not complain "unused option md5sum"
opt = self.options.copy()
opt['destination'] = compile_dir
# no need to shared build for compile dir
opt['shared'] = 'false'
downloadunpacked.Recipe(self.buildout, self.name, opt).install()
except:
if os.path.exists(compile_dir):
shutil.rmtree(compile_dir)
raise
else:
log.info('Using local source directory: %s', self.options['path'])
compile_dir = self.options['path']
log.info('Using local source directory: %s', compile_dir)
current_dir = os.getcwd()
location = self.options['location']
# Clean the install directory if it already exists as it is
# a remain from a previous failed installation
if os.path.exists(location):
shutil.rmtree(location)
os.mkdir(location)
try:
os.chdir(compile_dir)
try:
# We support packages that either extract contents to the $PWD
# or alternatively have a single directory.
contents = os.listdir(compile_dir)
contents = os.listdir('.')
if len(contents) == 1 and os.path.isdir(contents[0]):
# Single container
os.chdir(contents[0])
......@@ -413,30 +404,22 @@ echo %s
'You can source it in your shell to reproduce build environment.',
os.getcwd())
# Delete shared directory if not correctly installed
if self.options.get('shared'):
shutil.rmtree(self.options['shared'])
raise
finally:
os.chdir(current_dir)
if self.options['url']:
if (self.options.get('keep-compile-dir') or
self.buildout['buildout'].get('keep-compile-dir') or
'').lower() not in ('true', 'yes', '1', 'on'):
shutil.rmtree(compile_dir)
if self.options['shared']:
self._signature.save(self.options["shared"])
# Check promises
self.check_promises()
if self.options['url']:
if self.options.get('keep-compile-dir',
self.buildout['buildout'].get('keep-compile-dir', '')).lower() in ('true', 'yes', '1', 'on'):
# If we're keeping the compile directory around, add it to
# the parts so that it's also removed when this recipe is
# uninstalled.
parts.append(self.options['compile-directory'])
else:
shutil.rmtree(compile_dir)
del self.options['compile-directory']
if self.options['shared'] == '':
parts.append(location)
......
slapos/recipe/cmmi/tests.py
View file @ 509ee4a0
......@@ -107,22 +107,6 @@ class NonInformativeTests(unittest.TestCase):
os.chdir(self.dir)
self.assertEqual(self.dir, os.getcwd())
def test_compile_directory_exists(self):
"""
Do not fail if the compile-directory already exists
"""
compile_directory = os.path.join(self.dir, 'test_parts/test__compile__')
os.makedirs(compile_directory)
recipe = self.make_recipe({}, 'test', dict(url="some invalid url"))
os.chdir(self.dir)
# if compile directory exists, recipe should raise an IOError because
# of the bad URL, and _not_ some OSError because test__compile__
# already exists
with self.assertRaises(IOError):
recipe.install()
def test_restart_after_failure(self):
with fake_package(
configure='#!/bin/sh\n',
......@@ -173,7 +157,7 @@ class NonInformativeTests(unittest.TestCase):
'keep-compile-dir': 'true'})
os.chdir(self.dir)
recipe.install()
self.assertTrue(os.path.exists('test_parts/test__compile__/package-0.0.0'))
self.assertTrue(os.path.exists('test_parts/test/.build/package-0.0.0'))
def test_strip_top_dir(self):
recipe = self.make_recipe({}, 'test', {
......@@ -182,7 +166,7 @@ class NonInformativeTests(unittest.TestCase):
'strip-top-level-dir': 'true'})
os.chdir(self.dir)
recipe.install()
self.assertTrue(os.path.exists('test_parts/test__compile__/configure'))
self.assertTrue(os.path.exists('test_parts/test/.build/configure'))
def test_stop_shell_on_error_midway(self):
recipe = self.make_recipe({}, 'test', {
......@@ -345,7 +329,7 @@ class NonInformativeTests(unittest.TestCase):
os.chdir(self.dir)
recipe.install()
build_directory = os.path.join(self.dir, 'test_parts/test__compile__')
build_directory = os.path.join(self.dir, 'test_parts/test/.build')
self.assertTrue(os.path.exists(build_directory))
def test_bad_symlink_in_bin(self):
......
  • Julien Muchembled @jm ·
    Owner

    In slapos/testing/check_software.py (slapos.core), I didn't notice:

          # left over of compilation failures
      '*/*__compile__/*',

    See also slapos@a526a1d8.

    /cc @jerome

  • Jérome Perrin @jerome ·
    Owner

    Why do we need this ? I see we are using keep-compile-dir=true only in "unstable" softwares

  • Jérome Perrin @jerome ·
    Owner

    I don't want to look in details, but this looks like when installation fail it leaves a folder that looks like installation succeeded (like the test that was removed "But we had a bug with version v0.11 is ...")

Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7