Skip to content

S
slapos.recipe.cmmi
Commit 5239a27d authored by Xiaowu Zhang's avatar Xiaowu Zhang
Browse files
Options

Reimplement shared features 

1.use shared-part in [buildout] section to define shared path
2.shared option is True or Flase
3.a magic substitution @@LOCATION@@ is used to for dynamic path
parent aa5a651b
Hide whitespace changes
Inline Side-by-side
Showing with 159 additions and 48 deletions
slapos/recipe/cmmi/README.txt
View file @ 5239a27d
...@@ -26,25 +26,13 @@ Supported options ...@@ -26,25 +26,13 @@ Supported options
``--prefix`` injection takes place. You can also set the ``--prefix`` ``--prefix`` injection takes place. You can also set the ``--prefix``
parameter explicitly in ``configure-options``. parameter explicitly in ``configure-options``.
``share`` ``shared``
Specify the path in which this package is shared by many other Specify the path in which this package is shared by many other
packages. When it's unset or blank, it means the package isn't packages.
shared. Otherwise, it shared by many packages. Recipe will return Shared-part should be defined in [buildout] section
an empty list so that buildout will not uninstall the package when Shared option is True or False
uninstalling part. The package will be installed on path/name/hash of options.
In share mode, ``promises`` should be set so that recipe can tell
whether the package is installed. Otherwise nohting to do.
If ``share`` is not empty, ``location`` will be set to value of
``share``, and remove trailing '/'. So in case ``share`` is '/',
``location`` will be set to blank. Thus any reference like
"${part:location}/bin" in other parts will get the correct value.
This option is experiment now. Recipe doesn't try to set prefix
with the valule of this opton in the configure or make command,
you should specify them accordingly by yourself.
``md5sum`` ``md5sum``
...@@ -1019,10 +1007,17 @@ Look, "package" is reinstalled either: ...@@ -1019,10 +1007,17 @@ Look, "package" is reinstalled either:
building package building package
installing package installing package
Install share package Install shared package
===================== =====================
Use option ``share`` to install a share pacakge. Use option ``shared`` to install a shared pacakge.
>>> import os
>>> _ = system('rm -rf %s' % join(os.path.dirname(__file__), 'shared'))
>>> shared_dir = join(os.path.dirname(__file__), 'shared')
>>> os.mkdir(shared_dir)
If no shared-part is set, and shared is True, an error is raised:
>>> write('buildout.cfg', >>> write('buildout.cfg',
... """ ... """
... [buildout] ... [buildout]
...@@ -1032,41 +1027,111 @@ Use option ``share`` to install a share pacakge. ...@@ -1032,41 +1027,111 @@ Use option ``share`` to install a share pacakge.
... [package] ... [package]
... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz ... url = file://%s/package-0.0.0.tar.gz
... share = /usr/local ... shared = True
... promises = ${:share}/bin/mypackage.exe ... """% src)
... """ % (src,))
>>> print system(buildout) #doctest:+ELLIPSIS
>>> print system(buildout) While:
Installing.
Getting section package.
Initializing section package.
...
ValueError: Set ${buildout:shared-part} for shared feature
If shared-part is set and shared is True, build package failed, the build directory is removed, a build directory__complile__ is left for debugging
>>> _ = system('mv %s/package-0.0.0.tar.gz %s/package-0.0.0.tar.gz.bak' % (src, src))
>>> import tarfile
>>> from io import BytesIO
>>> import sys
>>> tarpath = os.path.join(src, 'package-0.0.0.tar.gz')
>>> with tarfile.open(tarpath, 'w:gz') as tar:
... configure = 'invalid'
... info = tarfile.TarInfo('configure.off')
... info.size = len(configure)
... info.mode = 0o755
... tar.addfile(info, BytesIO(configure))
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part = %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... shared = True
... """ % (shared_dir, src))
>>> print system(buildout) #doctest:+ELLIPSIS
package: shared directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/... set for package
Uninstalling package. Uninstalling package.
Uninstalling package-2. Uninstalling package-2.
Installing package. Installing package.
package: Checking whether package is installed at share path: /usr/local package: Checking whether package is installed at shared path: .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
package: could not find promise "/usr/local/bin/mypackage.exe" package: [ENV] TMP = .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/.../tmp
package: [ENV] TMP = /sample_buildout/parts/package/tmp package: Command failed with exit code 127: ./configure --prefix=".../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/..."
configure --prefix=/usr/local package: Compilation error. The package is left as is at .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...__compile__ where you can inspect what went wrong
/bin/sh: 1: ./configure: not found
While:
Installing package.
Error: System error
If shared-part is set and shared is True, package will be installed in shared_part/package/a hash of the recipe's configuration options
>>> _ = system('mv %s/package-0.0.0.tar.gz.bak %s/package-0.0.0.tar.gz' % (src, src))
>>> print system(buildout) #doctest:+ELLIPSIS
package: shared directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/... set for package
Installing package.
package: Checking whether package is installed at shared path: .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
package: [ENV] TMP = .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/.../tmp
package: Removing already existing directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...__compile__
configure --prefix=.../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
building package building package
installing package installing package
package: could not find promise "/usr/local/bin/mypackage.exe"
Do nothing if one package has been installed. Do nothing if one package has been installed.
>>> remove('.installed.cfg')
>>> write('buildout.cfg', >>> write('buildout.cfg',
... """ ... """
... [buildout] ... [buildout]
... newest = false ... newest = false
... parts = package ... parts = package
... shared-part = %s
... ...
... [package] ... [package]
... recipe = slapos.recipe.cmmi ... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz ... url = file://%s/package-0.0.0.tar.gz
... share = /usr/local/bin ... shared = True
... promises = ... """ % (shared_dir, src))
... """ % (src,)) >>> print system(buildout) #doctest:+ELLIPSIS
package: shared directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/... set for package
Installing package.
package: Checking whether package is installed at shared path: .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
package: This shared package has been installed by other package
>>> print system(buildout) If options change, reinstall in different location:
>>> write('buildout.cfg',
... """
... [buildout]
... newest = false
... parts = package
... shared-part = %s
...
... [package]
... recipe = slapos.recipe.cmmi
... url = file://%s/package-0.0.0.tar.gz
... shared =True
... change = True
... """ % (shared_dir, src))
>>> print system(buildout) #doctest:+ELLIPSIS
package: shared directory .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/... set for package
Uninstalling package. Uninstalling package.
Installing package. Installing package.
package: Checking whether package is installed at share path: /usr/local/bin package: Checking whether package is installed at shared path: .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
package: This shared package has been installed by other package package: [ENV] TMP = .../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/.../tmp
configure --prefix=.../slapos.recipe.cmmi/slapos/recipe/cmmi/shared/package/...
building package
installing package
For even more specific needs you can write your own recipe that uses For even more specific needs you can write your own recipe that uses
``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``. ``slapos.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``.
......
slapos/recipe/cmmi/__init__.py
View file @ 5239a27d
...@@ -29,7 +29,7 @@ class Recipe(object): ...@@ -29,7 +29,7 @@ class Recipe(object):
self.options = options self.options = options
self.buildout = buildout self.buildout = buildout
self.name = name self.name = name
log = logging.getLogger(self.name)
# Merge options if there is a matched platform section # Merge options if there is a matched platform section
platform_options = self.buildout.get( platform_options = self.buildout.get(
"%s:%s:%s" % (name, sys.platform, self.get_machine()), "%s:%s:%s" % (name, sys.platform, self.get_machine()),
...@@ -40,12 +40,41 @@ class Recipe(object): ...@@ -40,12 +40,41 @@ class Recipe(object):
self.original_options = options.copy() self.original_options = options.copy()
options.update(platform_options) options.update(platform_options)
options['share'] = options.get('share', '').strip() shared = options.get('shared', '')
options['default-location'] = os.path.join( if shared:
shared_part = buildout['buildout'].get('shared-part', None)
if not shared_part:
raise ValueError(
" Set ${buildout:shared-part} for shared feature")
shared = os.path.join(shared_part.strip().rstrip('/'), self.name)
if not os.path.exists(shared):
os.makedirs(shared)
self._debug_signature_text = []
# compute hash from options
m = md5()
buildout_directory = buildout['buildout']['directory']
profile_base_location = options.get('_profile_base_location_', '')
for k, v in sorted(options.items()):
# Key not vary on profile base location
if profile_base_location:
v = v.replace(profile_base_location, '${:_profile_base_location_}')
option_signature = ('%r: %r' % (k, v)).encode()
log.debug("Updating hash with %s", option_signature)
self._debug_signature_text.append(option_signature)
# Or maybe warn and disable cache
assert buildout_directory not in option_signature
m.update(option_signature)
shared = os.path.join(shared, m.hexdigest())
log.info('shared directory %s set for %s' % (shared, self.name))
options['shared'] = shared
default_location = options['default-location'] = os.path.join(
buildout['buildout']['parts-directory'], buildout['buildout']['parts-directory'],
self.name) self.name) if shared == '' else shared
share = options['share'] options['location'] = options['default-location']
options['location'] = options['default-location'] if share == '' else share.rstrip('/')
prefix = options.get('prefix', '').strip() prefix = options.get('prefix', '').strip()
if prefix == '': if prefix == '':
...@@ -88,6 +117,11 @@ class Recipe(object): ...@@ -88,6 +117,11 @@ class Recipe(object):
else: else:
options['compile-directory'] = options['path'] options['compile-directory'] = options['path']
from copy import copy
for k, v in copy(options.items()):
if '@@LOCATION@@' in v:
options[k] = v.replace('@@LOCATION@@', default_location)
self.environ = {} self.environ = {}
self.original_environment = os.environ.copy() self.original_environment = os.environ.copy()
...@@ -250,10 +284,10 @@ class Recipe(object): ...@@ -250,10 +284,10 @@ class Recipe(object):
log = logging.getLogger(self.name) log = logging.getLogger(self.name)
parts = [] parts = []
# In share mode, do nothing if package has been installed. # In shared mode, do nothing if package has been installed.
if (not self.options['share'] == ''): if (not self.options['shared'] == ''):
log.info('Checking whether package is installed at share path: %s' % self.options['share']) log.info('Checking whether package is installed at shared path: %s' % self.options['shared'])
if self.check_promises(log): if os.path.exists(self.options['shared']):
log.info('This shared package has been installed by other package') log.info('This shared package has been installed by other package')
return parts return parts
...@@ -314,6 +348,13 @@ class Recipe(object): ...@@ -314,6 +348,13 @@ class Recipe(object):
shutil.rmtree(tmp_path, True) shutil.rmtree(tmp_path, True)
os.mkdir(tmp_path) os.mkdir(tmp_path)
# Create a debug file with the signatures.
if self.options['shared']:
with open(os.path.join(
self.options["shared"],
".slapos.recipe.cmmi.signature"), 'w') as f:
f.write('\n'.join(self._debug_signature_text))
try: try:
try: try:
# We support packages that either extract contents to the $PWD # We support packages that either extract contents to the $PWD
...@@ -369,17 +410,22 @@ class Recipe(object): ...@@ -369,17 +410,22 @@ class Recipe(object):
log.info('Executing post-install') log.info('Executing post-install')
self.run(post_install_cmd) self.run(post_install_cmd)
if (self.buildout_prefix != '' if (self.buildout_prefix != ''
and self.options['share'] == '' and self.options['shared'] == ''
and os.path.exists(self.buildout_prefix)): and os.path.exists(self.buildout_prefix)):
log.info('Getting installed file lists') log.info('Getting installed file lists')
parts.extend(self.get_installed_files(tmp_path)) parts.extend(self.get_installed_files(tmp_path))
except: except:
log.error('Compilation error. The package is left as is at %s where ' log.error('Compilation error. The package is left as is at %s where '
'you can inspect what went wrong' % os.getcwd()) 'you can inspect what went wrong' % os.getcwd())
# Delete shared directory if not correctly installed
if self.options.get('shared'):
shutil.rmtree(self.options['shared'])
raise raise
finally: finally:
os.chdir(current_dir) os.chdir(current_dir)
shutil.rmtree(tmp_path) # If in shared mode and have error during installed, tmp_path was already deleted
if os.path.isdir(tmp_path):
shutil.rmtree(tmp_path)
# Check promises # Check promises
self.check_promises(log) self.check_promises(log)
...@@ -395,7 +441,7 @@ class Recipe(object): ...@@ -395,7 +441,7 @@ class Recipe(object):
shutil.rmtree(compile_dir) shutil.rmtree(compile_dir)
del self.options['compile-directory'] del self.options['compile-directory']
if self.options['share'] == '': if self.options['shared'] == '':
parts.append(self.options['default-location']) parts.append(self.options['default-location'])
self.fix_shebang(self.options['default-location']) self.fix_shebang(self.options['default-location'])
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7