• Kirill Smelkov's avatar
    jinja2: Handle non-ascii input/output · c901c411
    Kirill Smelkov authored
    Currently slapos.recipe.template:jinja2 fails if source contains
    non-ascii input:
    
        <BLANKLINE>
        An internal error occurred due to a bug in either zc.buildout or in a
        recipe being used:
        Traceback (most recent call last):
          File ".../site-packages/zc/buildout/buildout.py", line 1992, in main
            getattr(buildout, command)(args)
          File ".../site-packages/zc/buildout/buildout.py", line 550, in install
            [self[part]['recipe'] for part in install_parts]
          File ".../site-packages/zc/buildout/buildout.py", line 1135, in __getitem__
            options._initialize()
          File ".../site-packages/zc/buildout/buildout.py", line 1239, in _initialize
            self.initialize()
          File ".../site-packages/zc/buildout/buildout.py", line 1248, in initialize
            self.recipe = recipe_class(buildout, name, self)
          File ".../slapos.recipe.template/slapos/recipe/template/jinja2_template.py", line 208, in __init__
            env.compile(source, filename=template),
          File ".../site-packages/jinja2/environment.py", line 551, in compile
            source = self._parse(source, name, filename)
          File ".../site-packages/jinja2/environment.py", line 470, in _parse
            return Parser(self, source, name, encode_filename(filename)).parse()
          File ".../site-packages/jinja2/parser.py", line 31, in __init__
            self.stream = environment._tokenize(source, name, filename, state)
          File ".../site-packages/jinja2/environment.py", line 501, in _tokenize
            source = self.preprocess(source, name, filename)
          File ".../site-packages/jinja2/environment.py", line 495, in preprocess
            self.iter_extensions(), text_type(source))
        UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 219: ordinal not in range(128)
    
    it also fails writing-out non-ascii input (which could originate even
    programmatically even if source was purely ascii):
    
        <BLANKLINE>
        An internal error occurred due to a bug in either zc.buildout or in a
        recipe being used:
        Traceback (most recent call last):
          File ".../site-packages/zc/buildout/buildout.py", line 1992, in main
            getattr(buildout, command)(args)
          File ".../site-packages/zc/buildout/buildout.py", line 666, in install
            installed_files = self[part]._call(recipe.install)
          File ".../site-packages/zc/buildout/buildout.py", line 1407, in _call
            return f()
          File ".../slapos/slapos.recipe.template/slapos/recipe/template/jinja2_template.py", line 227, in install
            out.write(self.template.render(**self.context))
        UnicodeEncodeError: 'ascii' codec can't encode characters in position 206-211: ordinal not in range(128)
    
    Fix it.
    
    @vpelletier suggested to add `encoding` option so it is flexible and
    'utf-8' is not hardcoded.
    
    /reviewed-by @vpelletier  (on nexedi/slapos.recipe.template!1)
    c901c411
README.jinja2.txt 12.2 KB