Skip to content

S
slapos.recipe.cmmi
Commit 509ee4a0 authored by Julien Muchembled's avatar Julien Muchembled
Browse files
Options

For compiling, use a temporary directory that is inside the part location 

We want to keep the build directory when keep-compile-dir=true,
whereas sharing a part is reliable only if everything goes inside
the same directory.
parent 73f3f296
Hide whitespace changes
Inline Side-by-side
Showing with 48 additions and 132 deletions
slapos/recipe/cmmi/README.rst
View file @ 509ee4a0
...@@ -943,9 +943,8 @@ is not used. ...@@ -943,9 +943,8 @@ is not used.
building package building package
installing package installing package
If ``shared-part-list`` is set and shared is True, build package failed, the If ``shared-part-list`` is set, shared is True, and build package fails, the
build directory is removed, a build directory__compile__ is left for part location is left for debugging.
debugging.
Also a shell script with the environment variable is created, so that Also a shell script with the environment variable is created, so that
developer can try same build process as the recipe tried. developer can try same build process as the recipe tried.
...@@ -980,14 +979,14 @@ developer can try same build process as the recipe tried. ...@@ -980,14 +979,14 @@ developer can try same build process as the recipe tried.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0> package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar package: [ENV] FOO = bar
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127. package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>__compile__ where you can inspect what went wrong. package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment. A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found /bin/sh: 1: ./configure: not found
While: While:
Installing package. Installing package.
Error: System error Error: System error
>>> import glob >>> import glob
>>> cat(glob.glob(join(shared_dir, 'package/**__compile__/slapos.recipe.build.env.sh'))[0]) >>> cat(glob.glob(join(shared_dir, 'package/*/.build/slapos.recipe.build.env.sh'))[0])
export FOO=bar export FOO=bar
... ...
...@@ -998,9 +997,9 @@ Next time buildout runs, it detects that the build failed, remove the compile di ...@@ -998,9 +997,9 @@ Next time buildout runs, it detects that the build failed, remove the compile di
Installing package. Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0> package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>__compile__ package: Removing already existing directory /shared/package/<MD5SUM:0>
package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127. package: Command 'set -e;./configure --prefix="/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>__compile__ where you can inspect what went wrong. package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>/.build where you can inspect what went wrong.
A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment. A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
/bin/sh: 1: ./configure: not found /bin/sh: 1: ./configure: not found
While: While:
...@@ -1008,72 +1007,15 @@ Next time buildout runs, it detects that the build failed, remove the compile di ...@@ -1008,72 +1007,15 @@ Next time buildout runs, it detects that the build failed, remove the compile di
Error: System error Error: System error
But we had a bug with version v0.11 is that if build was interrupted in the middle of the build of a
shared part. The shared part was left in an inconsistent state that looked like installation succeeded.
Let's simulate a scenario where buildout is terminated in the middle of a build.
For this, we need a package that takes a lot of time to install.
>>> with tarfile.open(package_path, 'w:gz') as tar:
... configure = b'#!/bin/sh\necho configure started\nsleep 300;'
... info = tarfile.TarInfo('configure')
... info.size = len(configure)
... info.mode = 0o755
... tar.addfile(info, BytesIO(configure))
We also need a bit more complex method to run buildout so that we send a termination signal in the
middle of build process.
>>> import signal
>>> import time
>>> buildout_process = subprocess.Popen(buildout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>>> output_line = ''
>>> for _ in range(10):
... time.sleep(1)
... output_line = buildout_process.stdout.readline()
... if output_line and b'configure started' in output_line:
... print ('configure started')
... buildout_process.send_signal(signal.SIGKILL)
... print ('buildout terminated')
... break
configure started
buildout terminated
>>> _ = buildout_process.wait()
If we run buildout again, the compile dir is removed again and installation is retried. This time
installation can succeed.
This was not the case in version 0.11 of this recipe. If installation of a shared part failed, this was
not detected and the part was considered as installed.
>>> os.rename(package_path + '.bak', package_path)
>>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:0>
Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>__compile__
configure --prefix=/shared/package/<MD5SUM:0>
building package
installing package
If ``shared-part-list`` is set as an option in buildout section and If ``shared-part-list`` is set as an option in buildout section and
``shared`` is True, package will be installed in shared_part/package ``shared`` is True, package will be installed in shared_part/package
and a hash of the recipe's configuration options. and a hash of the recipe's configuration options.
There can be multiple path listed in ``shared-part-list``, the recipe There can be multiple path listed in ``shared-part-list``, the recipe
will look in each of these paths if package was already installed and will look in each of these paths if package was already installed and
if not, it will install the package in the last entry the last entry if not, it will install the package in the last entry.
from the list of ``shared-part-list``.
If package was already installed in any of the ``shared-part-list`` used, it will be >>> os.rename(package_path + '.bak', package_path)
used instead of installing if one package has been installed.
>>> remove('.installed.cfg')
>>> write('buildout.cfg', >>> write('buildout.cfg',
... """ ... """
... [buildout] ... [buildout]
...@@ -1095,29 +1037,36 @@ used instead of installing if one package has been installed. ...@@ -1095,29 +1037,36 @@ used instead of installing if one package has been installed.
package: shared at /shared/package/<MD5SUM:0> package: shared at /shared/package/<MD5SUM:0>
Installing package. Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0> package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
package: This shared package has been installed by other package package: [ENV] FOO = bar
package: Removing already existing directory /shared/package/<MD5SUM:0>
configure --prefix=/shared/package/<MD5SUM:0>
building package
installing package
If options change, reinstall in different location: If options change, reinstall in different location:
>>> write('buildout.cfg', >>> write('buildout.cfg',
... """ ... """
... [buildout] ... [buildout]
... newest = false ... newest = false
... parts = package ... parts = package
... shared-part-list = %s ... shared-part-list =
... %s
... not/exists
... %s
... ...
... [package] ... [package]
... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
... url = file://%s ... url = file://%s
... shared =True ... shared =True
... change = True ... change = True
... """ % (shared_dir, package_path)) ... """ % (shared_dir, another_shared_dir, package_path))
>>> print(system(buildout)) #doctest:+ELLIPSIS >>> print(system(buildout)) #doctest:+ELLIPSIS
package: shared at /shared/package/<MD5SUM:1> package: shared at /another_shared_dir/package/<MD5SUM:1>
Uninstalling package. Uninstalling package.
Installing package. Installing package.
package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:1> package: Checking whether package is installed at shared path: /another_shared_dir/package/<MD5SUM:1>
configure --prefix=/shared/package/<MD5SUM:1> configure /another_shared_dir/package/<MD5SUM:1>
building package building package
installing package installing package
......
slapos/recipe/cmmi/__init__.py
View file @ 509ee4a0
...@@ -111,7 +111,7 @@ class Recipe(object): ...@@ -111,7 +111,7 @@ class Recipe(object):
raise UserError('You must provide either "url" or "path".') raise UserError('You must provide either "url" or "path".')
if options['url']: if options['url']:
options['compile-directory'] = location + '__compile__' options['compile-directory'] = os.path.join(location, '.build')
else: else:
options['compile-directory'] = options['path'] options['compile-directory'] = options['path']
...@@ -297,42 +297,33 @@ class Recipe(object): ...@@ -297,42 +297,33 @@ class Recipe(object):
for key in sorted(self.environ.keys()): for key in sorted(self.environ.keys()):
log.info('[ENV] %s = %s', key, self.environ[key]) log.info('[ENV] %s = %s', key, self.environ[key])
# Download the source using slapos.recipe.downloadunpacked
if self.options['url']:
compile_dir = self.options['compile-directory']
if os.path.exists(compile_dir):
# leftovers from a previous failed attempt, removing it.
log.warning('Removing already existing directory %s', compile_dir)
shutil.rmtree(compile_dir)
os.makedirs(compile_dir)
try:
self.options.get('md5sum') # so that buildout does not complain "unused option md5sum"
opt = self.options.copy()
opt['destination'] = compile_dir
# no need to shared build for compile dir
opt['shared'] = 'false'
downloadunpacked.Recipe(self.buildout, self.name, opt).install()
except:
if os.path.exists(compile_dir):
shutil.rmtree(compile_dir)
raise
else:
log.info('Using local source directory: %s', self.options['path'])
compile_dir = self.options['path']
current_dir = os.getcwd() current_dir = os.getcwd()
compile_dir = self.options['compile-directory']
location = self.options['location'] location = self.options['location']
# Clean the install directory if it already exists as it is # Clean the install directory if it already exists as it is
# a remain from a previous failed installation # a remain from a previous failed installation
if os.path.exists(location): if os.path.exists(location):
log.warning('Removing already existing directory %s', location)
shutil.rmtree(location) shutil.rmtree(location)
os.mkdir(location)
try: try:
os.makedirs(location)
# Download the source using slapos.recipe.downloadunpacked
if self.options['url']:
os.mkdir(compile_dir)
self.options.get('md5sum') # so that buildout does not complain "unused option md5sum"
opt = self.options.copy()
opt['destination'] = compile_dir
# no need to shared build for compile dir
opt['shared'] = 'false'
downloadunpacked.Recipe(self.buildout, self.name, opt).install()
else:
log.info('Using local source directory: %s', compile_dir)
os.chdir(compile_dir) os.chdir(compile_dir)
try: try:
# We support packages that either extract contents to the $PWD # We support packages that either extract contents to the $PWD
# or alternatively have a single directory. # or alternatively have a single directory.
contents = os.listdir(compile_dir) contents = os.listdir('.')
if len(contents) == 1 and os.path.isdir(contents[0]): if len(contents) == 1 and os.path.isdir(contents[0]):
# Single container # Single container
os.chdir(contents[0]) os.chdir(contents[0])
...@@ -413,30 +404,22 @@ echo %s ...@@ -413,30 +404,22 @@ echo %s
'You can source it in your shell to reproduce build environment.', 'You can source it in your shell to reproduce build environment.',
os.getcwd()) os.getcwd())
# Delete shared directory if not correctly installed
if self.options.get('shared'):
shutil.rmtree(self.options['shared'])
raise raise
finally: finally:
os.chdir(current_dir) os.chdir(current_dir)
if self.options['url']:
if (self.options.get('keep-compile-dir') or
self.buildout['buildout'].get('keep-compile-dir') or
'').lower() not in ('true', 'yes', '1', 'on'):
shutil.rmtree(compile_dir)
if self.options['shared']: if self.options['shared']:
self._signature.save(self.options["shared"]) self._signature.save(self.options["shared"])
# Check promises # Check promises
self.check_promises() self.check_promises()
if self.options['url']:
if self.options.get('keep-compile-dir',
self.buildout['buildout'].get('keep-compile-dir', '')).lower() in ('true', 'yes', '1', 'on'):
# If we're keeping the compile directory around, add it to
# the parts so that it's also removed when this recipe is
# uninstalled.
parts.append(self.options['compile-directory'])
else:
shutil.rmtree(compile_dir)
del self.options['compile-directory']
if self.options['shared'] == '': if self.options['shared'] == '':
parts.append(location) parts.append(location)
......
slapos/recipe/cmmi/tests.py
View file @ 509ee4a0
...@@ -107,22 +107,6 @@ class NonInformativeTests(unittest.TestCase): ...@@ -107,22 +107,6 @@ class NonInformativeTests(unittest.TestCase):
os.chdir(self.dir) os.chdir(self.dir)
self.assertEqual(self.dir, os.getcwd()) self.assertEqual(self.dir, os.getcwd())
def test_compile_directory_exists(self):
"""
Do not fail if the compile-directory already exists
"""
compile_directory = os.path.join(self.dir, 'test_parts/test__compile__')
os.makedirs(compile_directory)
recipe = self.make_recipe({}, 'test', dict(url="some invalid url"))
os.chdir(self.dir)
# if compile directory exists, recipe should raise an IOError because
# of the bad URL, and _not_ some OSError because test__compile__
# already exists
with self.assertRaises(IOError):
recipe.install()
def test_restart_after_failure(self): def test_restart_after_failure(self):
with fake_package( with fake_package(
configure='#!/bin/sh\n', configure='#!/bin/sh\n',
...@@ -173,7 +157,7 @@ class NonInformativeTests(unittest.TestCase): ...@@ -173,7 +157,7 @@ class NonInformativeTests(unittest.TestCase):
'keep-compile-dir': 'true'}) 'keep-compile-dir': 'true'})
os.chdir(self.dir) os.chdir(self.dir)
recipe.install() recipe.install()
self.assertTrue(os.path.exists('test_parts/test__compile__/package-0.0.0')) self.assertTrue(os.path.exists('test_parts/test/.build/package-0.0.0'))
def test_strip_top_dir(self): def test_strip_top_dir(self):
recipe = self.make_recipe({}, 'test', { recipe = self.make_recipe({}, 'test', {
...@@ -182,7 +166,7 @@ class NonInformativeTests(unittest.TestCase): ...@@ -182,7 +166,7 @@ class NonInformativeTests(unittest.TestCase):
'strip-top-level-dir': 'true'}) 'strip-top-level-dir': 'true'})
os.chdir(self.dir) os.chdir(self.dir)
recipe.install() recipe.install()
self.assertTrue(os.path.exists('test_parts/test__compile__/configure')) self.assertTrue(os.path.exists('test_parts/test/.build/configure'))
def test_stop_shell_on_error_midway(self): def test_stop_shell_on_error_midway(self):
recipe = self.make_recipe({}, 'test', { recipe = self.make_recipe({}, 'test', {
...@@ -345,7 +329,7 @@ class NonInformativeTests(unittest.TestCase): ...@@ -345,7 +329,7 @@ class NonInformativeTests(unittest.TestCase):
os.chdir(self.dir) os.chdir(self.dir)
recipe.install() recipe.install()
build_directory = os.path.join(self.dir, 'test_parts/test__compile__') build_directory = os.path.join(self.dir, 'test_parts/test/.build')
self.assertTrue(os.path.exists(build_directory)) self.assertTrue(os.path.exists(build_directory))
def test_bad_symlink_in_bin(self): def test_bad_symlink_in_bin(self):
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7