1dc49d3e (tests: ignore CryptographyDeprecationWarning on python2, 2021-08-13)
was wrong, because this warning includes the path of the software, which
also has a md5sum hash, so the normalization has to be done before the
md5sum normalization, otherwise this consumes an md5sum and they become
off by one, with errors like:

    Expected:
        package: shared at /shared/package/<MD5SUM:0>
        Uninstalling package.
        Installing package.
        package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:0>
        package: [ENV] FOO = bar
        package: Command 'set -e;./configure --prefix=/shared/package/<MD5SUM:0>"' returned non-zero exit status 127.
        package: Compilation error. The package is left as is at /shared/package/<MD5SUM:0>__compile__ where you can inspect what went wrong.
        A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
        /bin/sh: 1: ./configure: not found
        While:
          Installing package.
        Error: System error
    Got:
        package: shared at /shared/package/<MD5SUM:1>
        Uninstalling package.
        Installing package.
        package: Checking whether package is installed at shared path: /shared/package/<MD5SUM:1>
        package: [ENV] FOO = bar
        package: Command 'set -e;./configure --prefix=/shared/package/<MD5SUM:1>"' returned non-zero exit status 127.
        package: Compilation error. The package is left as is at /shared/package/<MD5SUM:1>__compile__ where you can inspect what went wrong.
        A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
        <BLANKLINE>
        /bin/sh: 1: ./configure: not found
        While:
          Installing package.
        Error: System error
        <BLANKLINE>

When we switch to using `python -m unittest discover -v` test_suite is not
used, but load_tests is used instead.

Symlinks can point anywhere outside the parts. Also symlinks can be
broken.

By using lstat instead of stat, the symlink is not considered as a
regular file.

See merge request !14

Don't force strip_top_level_dir option to be false when calling for slapos.recipe.build:downloadunpacked - let the user decide.

Effectively adds strip_top_level_dir option to the script which allows for shorter and more consistent paths.
Also it can help mitigating the variability of path length between archives downloaded for branch, commit hash and release.

Add some doctests to describe the use cases of shared parts with slapos,
including the need for our `${:_profile_base_location_}` buildout patch
and the need to save signature files.

It is redundant with the fact that slapos.buildout already rebuilds
if any dependency is modified. slapos.buildout does not yet understand
the new => syntax from upstream buildout but meanwhile a dummy option
can be set to any options of required parts.

The value was always '${:_profile_base_location_}'

In conjunction with the new Options.barrier() in buildout,
parts will be able to modify the environment during install/update.

If not set, the commands in the middle of custom configure-command,
pre-build, post-build etc can fail without resulting in SystemError
for user and without stopping cmmi process, which thus can formally
succeed despite being misconfigured or in some cases with the code
not even built.
This is because when executing a multiline command, only the exit
status of the last one actually gets caught by check_call().

Setting -e makes debugging easier by allowing shell to terminate
mid-way if errors (non-zero exit status) occur.

-e only covers 'simple commands', and does not affect 'if' tests
and more complex cases where non-zero exit status is expected to
occur normally.
Detailed description of -e effects can be seen in the POSIX
standard:

https://pubs.opengroup.org/onlinepubs/009695399/utilities/set.html

When build process is interrupted (like when the buildout process
terminated by SIGKILL), building with shared leave the system in a state
that looks like installation succeeded, so next execution will assume
the part was properly installed.


The behavior was something like this:


    if isSharedPartSignatureOK():
      useThisAlreadyInstalledPart()
    else:
      writeSignatureInSharedPart()
      try:
        build()
      except:
        removeSharedPart()
        raise


and when program is terminated during build step, the signature is
there.

The fix is to write signature later, algorithm becomes:


    if isSharedPartSignatureOK():
      useThisAlreadyInstalledPart()
    else:
      try:
        build()
      except:
        removeSharedPart()
        raise
      writeSignatureInSharedPart()

/reviewed-on nexedi/slapos.recipe.cmmi!10

Use a new option `buildout:shared-path-list` that is now a new line separated list of path, only the last part is used to install new parts, other are read-only.

Also some cleanups in the test suite.

/reviewed-on !9

on python3.6, subprocess error message is different, use a
zope.testing.renormalizing rule to normalize the message.

This will allow nested slapos where the inner slapos can re-use shared
parts from outer slapos, but not write into the outer shared parts.

For compatibility reasons, the syntax of the buildout option changed,
instead of the old ${buildout:shared-parts} that use to be one path, we
now use ${buildout:shared-part-list} as a \n separated list of paths.
All paths can be used to lookup existing parts, only the last one can be
used to install new parts.
There is no compatibility with the old option as it was not used much,
only in profiles which were not really compatible with shared options.

https://pypi.org/project/zope.testing/ features "renormalizing" that
this test was already using to make this test independent from actual
paths that can be different but use some paths that looks always the
same (like /sample_buildout/).
Also use this in our patches for shared support so that the test is a
bit more readable and also so that we can assert the actual hash being
used.

This is restructured text.