Skip to content

S
slapos.recipe.template
Commit 62f55c69 authored by Julien Muchembled's avatar Julien Muchembled
1 Browse files
Options

Update doc & tests

parent 46493e12
Hide whitespace changes
Inline Side-by-side
Showing with 294 additions and 529 deletions
README.txt
View file @ 62f55c69
......@@ -2,9 +2,9 @@
slapos.recipe.template
======================
Template recipe which supports remote resource.
.. contents::
Inspired by collective.recipe.template, with minimum set of features, but with
(hopefully) safer buildout-based templating.
Collection of recipes to generate a file from a template that can be
either inline or fetched with the buildout download API.
"jinja2" entry point allows rendering jinja2 templates.
Inspired by `collective.recipe.template`.
slapos/recipe/template/README.jinja2.txt
View file @ 62f55c69
Jinja2 usage
============
----------
``jinja2``
----------
Getting started
---------------
Similar to the default recipe but the template syntax is Jinja2 instead of
buildout. Other significant differences are:
Example buildout demonstrating some types::
- For historical reasons, the generated file is specified with ``rendered``
instead of ``output``.
- For historical reasons, the template is specified with ``template`` only
and an inline template is prefixed with ``inline:`` + an optional newline.
- Rendering, and download if requested, is done during the install phase.
- Dependencies are explicit (see ``context`` option) instead of deduced from
the template.
- Some extra features (options detailed below).
Example demonstrating some types::
>>> write('buildout.cfg',
... '''
......@@ -62,107 +72,83 @@ And the template has been rendered::
UTF-8 text: привет мир!
Unicode text: 你好世界
Parameters
----------
Mandatory:
``template``
Template url/path, as accepted by zc.buildout.download.Download.__call__ .
For very short template, it can make sense to put it directly into
buildout.cfg: the value is the template itself, prefixed by the string
"inline:" + an optional newline.
``rendered``
Where rendered template should be stored.
Optional:
``context``
Jinja2 context specification, one variable per line, with 3
whitespace-separated parts: type, name and expression. Available types are
described below. "name" is the variable name to declare. Expression semantic
varies depending on the type.
Available types:
``raw``
Immediate literal string.
``key``
Indirect literal string.
``import``
Import a python module.
Options
-------
``section``
Make a whole buildout section available to template, as a dictionary.
``md5sum``, ``mode``
~~~~~~~~~~~~~~~~~~~~
Indirection targets are specified as: [section]:key .
It is possible to use buildout's buit-in variable replacement instead instead
of ``key`` type, but keep in mind that different lines are different
variables for this recipe. It might be what you want (factorising context
chunk declarations), otherwise you should use indirect types.
Same as for the default recipe.
``md5sum``
Template's MD5, for file integrity checking. By default, no integrity check
is done.
``once`` - avoiding file re-creation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``mode``
Mode, in octal representation (no need for 0-prefix) to set output file to.
This is applied before storing anything in output file.
Path of a marker file to prevents rendering altogether.
``once``
Path of a marker file to prevents rendering altogether.
Normally, each time the section is installed/updated the file gets
re-generated. This may be undesirable in some cases.
``extensions``
Jinja2 extensions to enable when rendering the template,
whitespace-separated. By default, none is loaded.
``once`` allows specifying a marker file, which when present prevents template
rendering::
``import-delimiter``
Delimiter character for in-temlate imports.
Defaults to ``/``.
See also: import-list
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = inline:dummy
... rendered = foo_once
... once = foo_flag
... ''')
>>> run_buildout()
Uninstalling template.
Installing template.
The template install returned None. A path or iterable os paths should be returned.
``import-list``
Declares a list of import paths. Format is similar to ``context``.
"name" becomes import's base name.
Template was rendered::
Available types:
>>> cat('foo_once')
dummy
``rawfile``
Literal path of a file.
And canary exists::
``file``
Indirect path of a file.
>>> import os
>>> os.path.exists('foo_flag')
True
``rawfolder``
Literal path of a folder. Any file in such folder can be imported.
Remove rendered file and re-render::
``folder``
Indirect path of a folder. Any file in such folder can be imported.
>>> os.unlink('foo_once')
>>> with open('buildout.cfg', 'a') as f:
... f.writelines(['extra = useless'])
>>> run_buildout()
Uninstalling template.
Installing template.
The template install returned None. A path or iterable os paths should be returned.
Unused options for template: 'extra'.
``encoding``
Encoding for input template and output file.
Defaults to ``utf-8``.
Template was not rendered::
FAQ
---
>>> os.path.exists('foo_once')
False
Q: How do I generate ${foo:bar} where foo comes from a variable ?
Removing the canary allows template to be re-rendered::
A: ``{{ '${' ~ foo_var ~ ':bar}' }}``
This is required as jinja2 fails parsing "${{{ foo_var }}:bar}". Though,
jinja2 succeeds at parsing "${foo:{{ bar_var }}}" so this trick isn't
needed for that case.
>>> os.unlink('foo_flag')
>>> with open('buildout.cfg', 'a') as f:
... f.writelines(['moreextra = still useless'])
>>> run_buildout()
Uninstalling template.
Installing template.
The template install returned None. A path or iterable os paths should be returned.
Unused options for template: 'extra'.
>>> cat('foo_once')
dummy
Use jinja2 extensions
~~~~~~~~~~~~~~~~~~~~~
It's also possible to use the same file for ``rendered`` and ``once``::
>>> write('foo.in',
... '''{% set foo = ['foo'] -%}
... {% do foo.append(bar) -%}
... {{ foo | join(', ') }}''')
>>> write('buildout.cfg',
... '''
... [buildout]
......@@ -170,135 +156,201 @@ Use jinja2 extensions
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = foo.in
... rendered = foo
... context = key bar buildout:parts
... # We don't actually use all those extensions in this minimal example.
... extensions = jinja2.ext.do jinja2.ext.loopcontrols
... jinja2.ext.with_
... template = inline:initial content
... rendered = rendered
... once = ${:rendered}
... ''')
>>> run_buildout()
>>> run_buildout() # doctest: +ELLIPSIS
Uninstalling template.
Installing template.
The template install returned None. A path or iterable os paths should be returned.
>>> cat('foo')
foo, template
Template was rendered::
Check file integrity
~~~~~~~~~~~~~~~~~~~~
>>> cat('rendered')
initial content
Compute template's MD5 sum::
When buildout options are modified, the template will not be rendered again::
>>> with open('buildout.cfg', 'a') as f:
... f.writelines(['template = inline:something different'])
>>> write('foo.in', '{{bar}}')
>>> from hashlib import md5
>>> with open('foo.in', 'rb') as f:
... md5sum = md5(f.read()).hexdigest()
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = foo.in
... rendered = foo
... context = key bar buildout:parts
... md5sum = ''' + md5sum + '''
... ''')
>>> run_buildout()
Uninstalling template.
Installing template.
The template install returned None. A path or iterable os paths should be returned.
>>> cat('foo')
template
Even though we used a different template, the file still contain the first template::
If the md5sum doesn't match, the buildout fail::
>>> cat('rendered')
initial content
>>> write('buildout.cfg',
... '''
``context`` - template variables and section dependency
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jinja2 context specification, one variable per line, with 3 whitespace-separated
parts: type, name and expression. Available types are described below. "name" is
the variable name to declare. Expression semantic varies depending on the type.
Available types:
``raw``
Immediate literal string.
``key``
Indirect literal string.
``import``
Import a python module.
``section``
Make a whole buildout section available to template, as a dictionary.
Indirection targets are specified as: [section]:key .
It is possible to use buildout's buit-in variable replacement instead instead
of ``key`` type, but keep in mind that different lines are different
variables for this recipe. It might be what you want (factorising context
chunk declarations), otherwise you should use indirect types.
You can use other part of buildout in the template. This way this parts
will be installed as dependency::
>>> write('buildout.cfg', '''
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = foo.in
... template = inline:{{bar}}
... rendered = foo
... context = key bar buildout:parts
... md5sum = 0123456789abcdef0123456789abcdef
... context = key bar dependency:foobar
...
... [dependency]
... foobar = dependency content
... recipe = zc.buildout:debug
... ''')
>>> run_buildout()
Uninstalling template.
Installing dependency.
foobar='dependency content'
recipe='zc.buildout:debug'
Installing template.
While:
Installing template.
Error: MD5 checksum mismatch for local resource at 'foo.in'.
This way you can get options which are computed in the ``__init__`` of
the dependent recipe.
Let's create a sample recipe modifying its option dict::
Specify filesystem permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> write('setup.py',
... '''
... from setuptools import setup
...
... setup(name='samplerecipe',
... entry_points = {
... 'zc.buildout': [
... 'default = main:Recipe',
... ],
... }
... )
... ''')
>>> write('main.py',
... '''
... class Recipe(object):
...
... def __init__(self, buildout, name, options):
... options['data'] = 'foobar'
...
... def install(self):
... return []
... ''')
You can specify the mode for rendered file::
Let's just use ``buildout.cfg`` using this egg::
>>> write('template.in', '{{bar}}')
>>> write('buildout.cfg',
... '''
... [buildout]
... develop = .
... parts = template
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = template.in
... template = inline:
... {{bar}}
... rendered = foo
... context = key bar buildout:parts
... mode = 205
... context = key bar sample:data
...
... [sample]
... recipe = samplerecipe
... ''')
>>> run_buildout()
Develop: '/sample-buildout/.'
Uninstalling template.
Uninstalling dependency.
Installing sample.
Installing template.
>>> cat('foo')
foobar
And the generated file with have the right permissions::
>>> import os, stat
>>> print("0%o" % stat.S_IMODE(os.stat('foo').st_mode))
0205
Note that Buildout will not allow you to have write permission for others
and will silently remove it (i.e a 207 mode will become 205).
Fetching resources from URLs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``extensions`` - Jinja2 extensions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can specify the resource to fetch from an URL::
Jinja2 extensions to enable when rendering the template,
whitespace-separated. By default, none is loaded.
>>> server_data = tmpdir('server_data')
>>> server_url = start_server(server_data)
::
>>> write(server_data, 'foo.in', '{{bar}}')
>>> write('foo.in',
... '''{% set foo = ['foo'] -%}
... {% do foo.append(bar) -%}
... {{ foo | join(', ') }}''')
>>> write('buildout.cfg',
... '''
... [buildout]
... develop = .
... parts = template
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = %sfoo.in
... template = foo.in
... rendered = foo
... context = key bar buildout:parts
... md5sum = %s
... ''' % (server_url, md5sum))
... # We don't actually use all those extensions in this minimal example.
... extensions = jinja2.ext.do jinja2.ext.loopcontrols
... jinja2.ext.with_
... ''')
>>> run_buildout()
Develop: '/sample-buildout/.'
Uninstalling template.
Uninstalling sample.
Installing template.
Downloading http://localhost/foo.in
Cannot download http://localhost/foo.in from network cache.
>>> cat('foo')
template
foo, template
``import-delimiter``, ``import-list`` - template imports
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``import-delimiter`` specifies the delimiter character for in-temlate imports.
Defaults to ``/``.
``import-list`` is a list of import paths. Format is similar to ``context``.
"name" becomes import's base name. Available types:
``rawfile``
Literal path of a file.
``file``
Indirect path of a file.
Template imports
~~~~~~~~~~~~~~~~
``rawfolder``
Literal path of a folder. Any file in such folder can be imported.
Here is a simple template importing an equaly-simple library:
``folder``
Indirect path of a folder. Any file in such folder can be imported.
Here is a simple template importing an equaly-simple library::
>>> write('template.in', '''
... {%- import "library" as library -%}
......@@ -431,91 +483,26 @@ path)::
b1foo
bc1foo
Section dependency
------------------
You can use other part of buildout in the template. This way this parts
will be installed as dependency::
>>> write('buildout.cfg', '''
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = inline:{{bar}}
... rendered = foo
... context = key bar dependency:foobar
...
... [dependency]
... foobar = dependency content
... recipe = zc.buildout:debug
... ''')
>>> run_buildout()
Uninstalling template.
Installing dependency.
foobar='dependency content'
recipe='zc.buildout:debug'
Installing template.
This way you can get options which are computed in the ``__init__`` of
the dependent recipe.
``encoding``
~~~~~~~~~~~~
Let's create a sample recipe modifying its option dict::
Encoding for input template and output file.
Defaults to ``utf-8``.
>>> write('setup.py',
... '''
... from setuptools import setup
...
... setup(name='samplerecipe',
... entry_points = {
... 'zc.buildout': [
... 'default = main:Recipe',
... ],
... }
... )
... ''')
>>> write('main.py',
... '''
... class Recipe(object):
...
... def __init__(self, buildout, name, options):
... options['data'] = 'foobar'
...
... def install(self):
... return []
... ''')
FAQ
---
Let's just use ``buildout.cfg`` using this egg::
Q: How do I generate ${foo:bar} where foo comes from a variable ?
>>> write('buildout.cfg',
... '''
... [buildout]
... develop = .
... parts = template
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = inline:
... {{bar}}
... rendered = foo
... context = key bar sample:data
...
... [sample]
... recipe = samplerecipe
... ''')
>>> run_buildout()
Develop: '/sample-buildout/.'
Uninstalling template.
Uninstalling dependency.
Installing sample.
Installing template.
>>> cat('foo')
foobar
A: ``{{ '${' ~ foo_var ~ ':bar}' }}``
This is required as jinja2 fails parsing "${{{ foo_var }}:bar}". Though,
jinja2 succeeds at parsing "${foo:{{ bar_var }}}" so this trick isn't
needed for that case.
Errors in template
~~~~~~~~~~~~~~~~~~
------------------
::
>>> write('template.in', '''\
... foo
......@@ -541,106 +528,3 @@ Errors in template
File "template.in", line 3, in template
bar
...TemplateSyntaxError: Encountered unknown tag 'bar'.
Avoiding file re-creation
~~~~~~~~~~~~~~~~~~~~~~~~~
Normally, each time the section is installed/updated the file gets
re-generated. This may be undesirable in some cases.
``once`` allows specifying a marker file, which when present prevents template
rendering.
>>> import os
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = inline:dummy
... rendered = foo_once
... once = foo_flag
... ''')
>>> run_buildout()
Installing template.
The template install returned None. A path or iterable os paths should be returned.
Template was rendered::
>>> cat('foo_once')
dummy
And canary exists::
>>> os.path.exists('foo_flag')
True
Remove rendered file and re-render::
>>> import os
>>> os.unlink('foo_once')
>>> with open('buildout.cfg', 'a') as f:
... f.writelines(['extra = useless'])
>>> run_buildout()
Uninstalling template.
Installing template.
The template install returned None. A path or iterable os paths should be returned.
Unused options for template: 'extra'.
Template was not rendered::
>>> os.path.exists('foo_once')
False
Removing the canary allows template to be re-rendered::
>>> os.unlink('foo_flag')
>>> with open('buildout.cfg', 'a') as f:
... f.writelines(['moreextra = still useless'])
>>> run_buildout()
Uninstalling template.
Installing template.
The template install returned None. A path or iterable os paths should be returned.
Unused options for template: 'extra'.
>>> cat('foo_once')
dummy
It's also possible to use the same file for ``rendered`` and ``once``::
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template:jinja2
... template = inline:initial content
... rendered = rendered
... once = ${:rendered}
... ''')
>>> run_buildout() # doctest: +ELLIPSIS
Uninstalling template.
Installing template.
The template install returned None. A path or iterable os paths should be returned.
Template was rendered::
>>> cat('rendered')
initial content
When buildout options are modified, the template will not be rendered again::
>>> with open('buildout.cfg', 'a') as f:
... f.writelines(['template = inline:something different'])
>>> run_buildout()
Uninstalling template.
Installing template.
The template install returned None. A path or iterable os paths should be returned.
Even though we used a different template, the file still contain the first template::
>>> cat('rendered')
initial content
slapos/recipe/template/README.txt
View file @ 62f55c69
Usage
=====
--------------
default recipe
--------------
Getting started
---------------
The default recipe generates a file (option ``output``) from a template using
buildout expansion. The template is specified with either ``url`` (optionally
combined with ``md5sum``) or ``inline``.
You can start by a simple buildout::
Here is a simple buildout::
>>> write('buildout.cfg',
... '''
>>> base = """
... [buildout]
... parts = template
...
... [section]
... option = value
...
... [template]
... recipe = slapos.recipe.template
... url = template.in
... output = template.out
...
... [section]
... option = value
... ''')
... """
>>> write('buildout.cfg', base)
And a simple template::
A simple template::
>>> write('template.in', '${section:option}')
We run buildout::
And the output file has been parsed by buildout itself::
>>> run_buildout()
Installing template.
And the output file has been parsed by buildout itself::
>>> cat('template.out')
value
Full options
------------
There is two non required options:
``md5sum``
Check the integrity of the input file.
The recipe relies on buildout expansion to pull sections it depends on, which
implies that the rendering (including the download if requested) is done during
the initialization phase.
``mode``
Specify the filesystem permissions in octal notation.
Options
-------
Check file integrity
~~~~~~~~~~~~~~~~~~~~
``md5sum`` - check file integrity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Let's write a file template::
>>> write('template.in', '${buildout:parts}')
Compute its MD5 sum::
>>> from hashlib import md5
>>> with open('template.in', 'rb') as f:
... md5sum = md5(f.read()).hexdigest()
Write the ``buildout.cfg`` using slapos.recipe.template::
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template
... url = template.in
... output = template.out
... md5sum = ''' + md5sum + '''
... ''')
And run buildout, and see the result::
If the template is specified with the ``url`` option, an MD5 checksum can be
given to check the contents of the template::
>>> base += """
... md5sum = 1993226f57db37c4a19cb785f826a1aa
... """
>>> write(sample_buildout, 'buildout.cfg', base)
>>> run_buildout()
Uninstalling template.
Installing template.
>>> cat('template.out')
template
value
If the md5sum doesn't match, the buildout fail::
In such case, updating the part does nothing::
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template
... url = template.in
... output = template.out
... md5sum = 0123456789abcdef0123456789abcdef
... ''')
>>> write('template.out', 'altered')
>>> run_buildout()
Updating template.
>>> cat('template.out')
altered
In case of checksum mismatch::
>>> run_buildout('template:md5sum=00000000000000000000000000000000')
While:
Installing.
Getting section template.
Initializing section template.
Error: MD5 checksum mismatch for local resource at 'template.in'.
``inline``
~~~~~~~~~~
Specify filesystem permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You may prefer to inline small templates::
You can specify the mode of the written file::
>>> write('template.in', '${buildout:installed}')
>>> write('buildout.cfg',
... '''
>>> write('buildout.cfg', """
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template
... url = template.in
... output = template.out
... mode = 0627
... ''')
>>> run_buildout()
Uninstalling template.
Installing template.
And the generated file with have the right permissions::
>>> import os, stat
>>> print("0%o" % stat.S_IMODE(os.stat('template.out').st_mode))
0627
Fetching template source from an URL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can fetch resources from an URL::
>>> server_data = tmpdir('server_data')
>>> server_url = start_server(server_data)
>>> write(server_data, 'template.in', '${buildout:parts}')
>>> write('buildout.cfg', '''
... [buildout]
... parts = template
... [section]
... option = inlined
...
... [template]
... recipe = slapos.recipe.template
... url = %stemplate.in
... md5sum = %s
... inline = ${section:option}
... output = template.out
...
... ''' % (server_url, md5sum))
... """)
>>> run_buildout()
Downloading http://localhost/template.in
Cannot download http://localhost/template.in from network cache.
Uninstalling template.
Installing template.
>>> cat('template.out')
template
inlined
Section dependency
------------------
Note that in such case, the rendering is done by buildout itself:
it just creates a file with the value of ``inline``.
You can use other part of buildout in the template. This way this parts
will be installed as dependency::
``mode`` - specify filesystem permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> write('template.in', '${dependency:foobar}')
>>> write('buildout.cfg', '''
... [buildout]
... parts = template
...
... [template]
... recipe = slapos.recipe.template
... url = template.in
... output = template.out
...
... [dependency]
... foobar = dependency content
... recipe = zc.buildout:debug
... ''')
By default, executable permissions are set if the content of the output file
looks like an executable script, i.e. it has a shebang that points to an
executable file. This is done by respecting umask::
>>> run_buildout()
>>> import os, stat
>>> os.access('template.out', os.X_OK)
False
>>> run_buildout('section:option=#!/bin/sh')
Uninstalling template.
Installing dependency.
foobar='dependency content'
recipe='zc.buildout:debug'
Installing template.
>>> os.access('template.out', os.X_OK)
True
This way you can get options which are computed in the ``__init__`` of
the dependent recipe.
Let's create a sample recipe modifying its option dict::
>>> write('setup.py',
... '''
... from setuptools import setup
...
... setup(name='samplerecipe',
... entry_points = {
... 'zc.buildout': [
... 'default = main:Recipe',
... ],
... }
... )
... ''')
>>> write('main.py',
... '''
... class Recipe(object):
...
... def __init__(self, buildout, name, options):
... options['data'] = 'foobar'
...
... def install(self):
... return []
... ''')
Let's just use ``buildout.cfg`` using this egg::
File permissions can be forced using the ``mode`` option in octal representation
(no need for 0-prefix)::
>>> write('template.in', '${sample:data}')
>>> write('buildout.cfg',
... '''
... [buildout]
... develop = .
... parts = template
...
... [template]
... recipe = slapos.recipe.template
... url = template.in
... output = template.out
...
... [sample]
... recipe = samplerecipe
... ''')
>>> run_buildout()
Develop: '/sample-buildout/.'
>>> run_buildout('template:mode=627')
Uninstalling template.
Uninstalling dependency.
Installing sample.
Installing template.
>>> cat('template.out')
foobar
>>> print(oct(stat.S_IMODE(os.stat('template.out').st_mode)))
0627
slapos/recipe/template/tests.py
View file @ 62f55c69
......@@ -42,7 +42,8 @@ def setUp(test):
testing.buildoutSetUp(test)
testing.install_develop('slapos.recipe.template', test)
(lambda system, buildout, **kw: test.globs.update(
run_buildout = lambda: print(system(buildout), end='')
run_buildout = lambda *args: print(system(' '.join((buildout,) + args)),
end='')
))(**test.globs)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7