slapgrid-sr: do not rebootstrap unnecessarily

Currently, bootstrapBuildout is called unconditionally, and since
Software Releases use slapos.reboostrap, we end up with the
following behaviour in the best case:

    Processing software releases...
    Installing software release /srv/slapgrid/slappart9/srv/testnode/aai/software.cfg...
    Generated script '/srv/slapgrid/slappart9/srv/testnode/aai/soft/08f8010370c519147fe23fed3170be9e/bin/buildout'.
    slapos.rebootstrap: Make sure that the section 'python2.7' won't be reinstalled after rebootstrap.
    Unused options for buildout: 'allowed-eggs-from-site-packages' 'exec-sitecustomize' 'include-site-packages' 'unzip'.
    Updating xz-utils.
    Updating patch.
    ...
    Updating gcc.
    Updating python2.7.
    Stripping binaries ...
    Done.
    slapos.rebootstrap:
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************
    bin/buildout is being reinstalled right now, as new python:
      /srv/slapgrid/slappart9/srv/testnode/aai/soft/08f8010370c519147fe23fed3170be9e/parts/python2.7/bin/python2.7
    is available, and buildout is using another python:
      /opt/slapgrid/b9f9e0967ab8491759399f306d65239a/parts/python2.7/bin/python2.7
    Buildout will be restarted automatically to have this change applied.
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************

    Unused options for buildout: 'allowed-eggs-from-site-packages' 'exec-sitecustomize' 'include-site-packages' 'unzip'.
    Updating xz-utils.
    Updating patch.
    ...
    Updating gcc.
    Updating python2.7.
    Updating autoconf.
    ...

Which means that bin/buildout always runs twice: updating parts is usually fast
but loading extends and checking that binaries are stripped can take a while.

The idea of this commit is minimize the amount of work when bin/buildout
already exists, in particular if it is already changed by slapos.reboostrap
to use the built Python. We also hope this will avoid complete rebuilds when
building a different version of Python:

    Installing software release /srv/slapgrid/slappart1/srv/testnode/bsu/software.cfg...
    Generated script '/srv/slapgrid/slappart1/srv/testnode/bsu/soft/90adf14823b5e2bc7dd89ccfbd9388df/bin/buildout'.
    slapos.rebootstrap: Make sure that the section 'python3.5' won't be reinstalled after rebootstrap.
    Uninstalling python3.5.
    Uninstalling file.
    Uninstalling file-msooxml.
    Uninstalling gettext.
    Uninstalling lunzip.
    Uninstalling libxml2.
    Uninstalling sqlite3.
    Uninstalling openssl.
    Uninstalling ca-certificates.
    Uninstalling perl.
    Uninstalling gdbm.
    Uninstalling bzip2.
    Uninstalling libffi.
    Uninstalling libexpat.
    Uninstalling readline.
    Uninstalling ncurses.
    Uninstalling zlib.
    Uninstalling patch.
    Uninstalling xz-utils.
    Unused options for buildout: 'allowed-eggs-from-site-packages' 'exec-sitecustomize' 'include-site-packages' 'unzip'.
    Installing xz-utils.
    ...
    Stripping binaries ...
    Done.
    slapos.rebootstrap:
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************
    bin/buildout is being reinstalled right now, as new python:
      /srv/slapgrid/slappart1/srv/testnode/bsu/soft/90adf14823b5e2bc7dd89ccfbd9388df/parts/python3.5/bin/python3.5
    is available, and buildout is using another python:
      /opt/slapgrid/b9f9e0967ab8491759399f306d65239a/parts/python2.7/bin/python2.7
    Buildout will be restarted automatically to have this change applied.
    ************ REBOOTSTRAP: IMPORTANT NOTICE ************

    Uninstalling template.
    ...
    Uninstalling m4.
    Unused options for buildout: 'allowed-eggs-from-site-packages' 'exec-sitecustomize' 'include-site-packages' 'unzip'.
    Updating xz-utils.
    ...
    Updating python3.5.
    Installing m4.
    ...

About the implementation, we could merely add a `if not os.path.exists(...):`.
I fear cases where a previous run may have left bin/buildout in a non-working
state, so I suggest the use of a marker to force bootstrap if a previous run
failed whereas the existing bin/buildout was reused.

/reviewed-on nexedi/slapos.core!54

slapos/grid/SlapObject.py

@@ -28,6 +28,7 @@
 #
 ##############################################################################
 
+import errno
 import os
 import pkg_resources
 import pwd
@@ -254,6 +255,7 @@ class Software(object):
     extends_cache = tempfile.mkdtemp()
     self._set_ownership(extends_cache)
 
+    f = None
     try:
       buildout_cfg = os.path.join(self.software_path, 'buildout.cfg')
       if not os.path.exists(buildout_cfg):
@@ -262,18 +264,32 @@ class Software(object):
       additional_parameters = list(self._additional_buildout_parameters(extends_cache))
       additional_parameters.extend(['-c', buildout_cfg])
 
-      utils.bootstrapBuildout(path=self.software_path,
-                              buildout=self.buildout,
-                              logger=self.logger,
-                              additional_buildout_parameter_list=additional_parameters)
+      buildout_binary = os.path.join(self.software_path, 'bin', 'buildout')
+      buildout_marker = buildout_binary + "-bootstrap-skipped"
 
+      try:
+        os.remove(buildout_marker)
+      except OSError as e:
+        if e.errno != errno.ENOENT:
+          raise
+        if os.path.exists(buildout_binary):
+          f = open(buildout_marker, "w")
+      if f is None:
+        utils.bootstrapBuildout(path=self.software_path,
+                                buildout=self.buildout,
+                                logger=self.logger,
+                                additional_buildout_parameter_list=additional_parameters)
       utils.launchBuildout(path=self.software_path,
-                           buildout_binary=os.path.join(self.software_path, 'bin', 'buildout'),
+                           buildout_binary=buildout_binary,
                            logger=self.logger,
                            additional_buildout_parameter_list=additional_parameters,
                            debug=self.buildout_debug)
+      if f is not None:
+        os.remove(buildout_marker)
     finally:
       shutil.rmtree(extends_cache)
+      if f is not None:
+        f.close()
 
   def _create_buildout_profile(self, buildout_cfg, url):
     with open(buildout_cfg, 'wb') as fout: