shared: Fix recovery after an interrupted build

When build process is interrupted (like when the buildout process
terminated by SIGKILL), building with shared leave the system in a state
that looks like installation succeeded, so next execution will assume
the part was properly installed.

The behavior was something like this:

    if isSharedPartSignatureOK():
      useThisAlreadyInstalledPart()
    else:
      writeSignatureInSharedPart()
      try:
        build()
      except:
        removeSharedPart()
        raise

and when program is terminated during build step, the signature is
there.

The fix is to write signature later, algorithm becomes:

    if isSharedPartSignatureOK():
      useThisAlreadyInstalledPart()
    else:
      try:
        build()
      except:
        removeSharedPart()
        raise
      writeSignatureInSharedPart()

slapos/recipe/cmmi/README.rst

@@ -1070,14 +1070,64 @@ developer can try same build process as the recipe tried.
     export FOO="bar"
     ...
 
-If ``shared-part-list`` is set as an option in buildout section and
-``shared`` is True, package will be installed in shared_part/package
-and a hash of the recipe's configuration options.
+Next time buildout runs, it detects that the build failed, remove the compile dir and retry.
+
+    >>> print(system(buildout)) #doctest:+ELLIPSIS
+    package: shared directory /shared/package/FIRST_SHARED_PACKAGE_HASH set for package
+    Installing package.
+    package: Checking whether package is installed at shared path: /shared/package/FIRST_SHARED_PACKAGE_HASH
+    package: [ENV] FOO = bar
+    package: Removing already existing directory /shared/package/FIRST_SHARED_PACKAGE_HASH__compile__
+    package: Command './configure --prefix="/shared/package/FIRST_SHARED_PACKAGE_HASH"' returned non-zero exit status 127.
+    package: Compilation error. The package is left as is at /shared/package/FIRST_SHARED_PACKAGE_HASH__compile__ where you can inspect what went wrong.
+    A shell script slapos.recipe.build.env.sh has been generated. You can source it in your shell to reproduce build environment.
+    /bin/sh: 1: ./configure: not found
+    While:
+      Installing package.
+    Error: System error
+
+
+But we had a bug with version v0.11 is that if build was interrupted in the middle of the build of a
+shared part. The shared part was left in an inconsistent state that looked like installation succeeded.
+
+Let's simulate a scenario where buildout is terminated in the middle of a build.
+For this, we need a package that takes a lot of time to install.
+
+    >>> tarpath = os.path.join(src, 'package-0.0.0.tar.gz')
+    >>> with tarfile.open(tarpath, 'w:gz') as tar:
+    ...   configure = b'#!/bin/sh\necho configure started\nsleep 300;'
+    ...   info = tarfile.TarInfo('configure')
+    ...   info.size = len(configure)
+    ...   info.mode = 0o755
+    ...   tar.addfile(info, BytesIO(configure))
+
+We also need a bit more complex method to run buildout so that we send a termination signal in the
+middle of build process.
+
+    >>> import subprocess
+    >>> import signal
+    >>> import time
+    >>> buildout_process = subprocess.Popen(buildout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+    >>> output_line = ''
+    >>> for _ in range(10):
+    ...   time.sleep(1)
+    ...   output_line = buildout_process.stdout.readline()
+    ...   if output_line and b'configure started' in output_line:
+    ...     print ('configure started')
+    ...     buildout_process.send_signal(signal.SIGKILL)
+    ...     print ('buildout terminated')
+    ...     break
+    configure started
+    buildout terminated
+    >>> _ = buildout_process.wait()
+
+
+If we run buildout again, the compile dir is removed again and installation is retried. This time
+installation can succeed.
+
+This was not the case in version 0.11 of this recipe. If installation of a shared part failed, this was
+not detected and the part was considered as installed.
 
-There can be multiple path listed in ``shared-part-list``, the recipe
-will look in each of these paths if package was already installed and
-if not, it will install the package in the last entry the last entry
-from the list of ``shared-part-list``.
 
     >>> _ = system('mv %s/package-0.0.0.tar.gz.bak  %s/package-0.0.0.tar.gz' % (src, src))
     >>> print(system(buildout)) #doctest:+ELLIPSIS
@@ -1090,6 +1140,17 @@ from the list of ``shared-part-list``.
     building package
     installing package
 
+
+
+If ``shared-part-list`` is set as an option in buildout section and
+``shared`` is True, package will be installed in shared_part/package
+and a hash of the recipe's configuration options.
+
+There can be multiple path listed in ``shared-part-list``, the recipe
+will look in each of these paths if package was already installed and
+if not, it will install the package in the last entry the last entry
+from the list of ``shared-part-list``.
+
 If package was already installed in any of the ``shared-part-list`` used, it will be
 used instead of installing if one package has been installed.
 

slapos/recipe/cmmi/__init__.py

@@ -332,9 +332,6 @@ class Recipe(object):
         os.mkdir(location)
         os.chdir(compile_dir)
 
-        if self.options['shared']:
-          self._signature.save(self.options["shared"])
-
         try:
             try:
                 # We support packages that either extract contents to the $PWD
@@ -420,6 +417,9 @@ class Recipe(object):
         finally:
             os.chdir(current_dir)
 
+        if self.options['shared']:
+          self._signature.save(self.options["shared"])
+
         # Check promises
         self.check_promises(log)