Commit f9d5bde3 authored by Julien Muchembled's avatar Julien Muchembled

boost-lib: enable parallel build

parent 5a24d499
...@@ -13,9 +13,9 @@ url = http://downloads.sourceforge.net/sourceforge/boost/boost_1_67_0.tar.bz2 ...@@ -13,9 +13,9 @@ url = http://downloads.sourceforge.net/sourceforge/boost/boost_1_67_0.tar.bz2
md5sum = ced776cb19428ab8488774e1415535ab md5sum = ced776cb19428ab8488774e1415535ab
location = @@LOCATION@@ location = @@LOCATION@@
configure-command = ./bootstrap.sh --prefix=${:location} --with-python=${python2.7:location}/bin/python2.7 configure-command = ./bootstrap.sh --prefix=${:location} --with-python=${python2.7:location}/bin/python2.7
make-binary = make-binary = ./b2
make-options = # XXX: we assume that $MAKEFLAGS is only used for -j
  • We should not assume this. Testnodes uses something like MAKEFLAGS="-j10 -l10" (here).

    That's also what I use and it seem to cause a strange error like this:

    ...
    gcc.compile.c++ bin.v2/libs/wave/build/gcc-5.5.0/release/threading-multi/instantiate_re2c_lexer_str.o
    g++: internal compiler error: Killed (program cc1plus)
    Please submit a full bug report,
    with preprocessed source if appropriate.
    See <http://gcc.gnu.org/bugs.html> for instructions.
    10 second time limit exceeded
    ...

    If I change to MAKEFLAGS="-j10 -l4", it fails with:

    ...
    4 second time limit exceeded
    ...

    so it seems -l apply a time limit to commands and that must be the cause of the failures we have sometimes with boost on test nodes.

    I don't see -l option being documented in https://boostorg.github.io/build/manual/master/index.html, but I see:

    -j N

    Run up to N commands in parallel. Default number of jobs is the number of detected available CPU threads. Note: There are circumstances when that default can be larger than the allocated cpu resources, for instance in some virtualized container installs.

    I have not verified if this is true. I'm not even sure this doc applies to this ./b2 we are using here, but I guess we better not pass $MAKEFLAGS like this.

  • @jerome, might be wrong but I observed a case when cc1 was OOM killed with same message (while compiling groonga in the standalone tests inside a 2GB RAM VM). Might that be the reason (you can check kernel's logs) ?

  • The b2 documentation is wrong (at least for 1.67.0): the default is 1. I also can't find any documentation for -l. Another strange thing is that 'b2' eats 100% CPU even without parallel build.

    While debugging, I found a slapos.recipe.cmmi bug: __buildout_installed__ is empty for shared parts.

  • Thanks for feedback and for the fix in 4cd0606c

    @Tyagov yes, I should have mentioned it, I first suspected something like OOM, but this was not in dmesg.

    @jm maybe this doc does not apply for the ./b2 script. Anyway this should be OK on master now.

    While debugging, I found a slapos.recipe.cmmi bug: __buildout_installed__ is empty for shared parts.

    I'm not sure we're talking about the same thing, but if a part is installed in shared mode, the software which installed the part does not "own" the part and should not uninstall that part, because other softwares might also be using it. The only way to uninstall shared part is slapos node prune command that was added recently.

  • if a part is installed in shared mode, the software which installed the part does not "own" the part and should not uninstall that part

    Oops, yes, of course. The problem I encountered is that I manually deleted a shared part and expected it to be rebuilt. A missing __buildout_installed__ path is enough to trigger a reinstallation, but since __buildout_installed__ has to be empty, we'd need some other mechanism.

  • Oh, I've just remembered about zc.buildout.uninstall entry point. We could specify a function that does nothing if __buildout_installed__ points to a shared location.

  • The problem I encountered is that I manually deleted a shared part and expected it to be rebuilt.

    Maybe it works only if the shared part is directly referenced. If for example we have curl component which depends on zlib, and an erp5 software which uses curl:

    • if we delete curl and reinstall erp5 software, curl part should be reinstalled, because the directory is missing, slapos.recipe.cmmi's init should detect this
    • if we delete zlib and reinstall erp5 software, zlib part will not be reinstalled, because erp5 only depends on curl, and curl looks installed, the directory is present.

    This __buildout_installed__ trick could work, but not in all cases I think. If installing erp5 installed curl and zlib, they will be in erp5 software's .installed.cfg, but not in other softwares. Later if we delete zlib and reinstall another software using curl (for example cloudooo), then zlib would not be seen as missing. Or maybe we can find a way to write dependent parts in __buildout_installed__.

    I prefer if we can say that this use case is not supported, but maybe we could implement this use case with a script that would delete a shared part and all shared parts depending on it (or maybe slapos node prune could detect this and "repair" by deleting "orphean" parts).

  • Buildout does not work like this. It always establishes the full graph of dependencies, regardless parts are installed or not. In your example of ERP5 SR where zlib is deleted but not curl, both are uninstalled (the first one because of missing path, the second because of uninstalled dependency) and installed again if __buildout_installed__ was set.

    The only strange thing with shared parts is that the installation of curl would be a no-op, even though zlib was rebuilt. In theory, this should work because builds are supposed to be reproducible (well, modulo non-important things like profile-guided optimization).

    Edited by Julien Muchembled
  • Yes, you're right, if we make slapos.recipe.cmmi always return shared part location, not only on install but also when it detects the part was already installed (this was what I missed), it should be enough. About the zc.buildout.uninstall entry point, we are using it in slapos.recipe.build:gitclone (here) to prevent buildout from deleting a git repository with local changes.

Please register or sign in to reply
make-targets = ./b2 link=shared dll-path=${:location}/lib:${bzip2:location}/lib:${gcc:location}/lib:${gcc:location}/lib64:${zlib:location}/lib install make-options = $MAKEFLAGS link=shared dll-path=${:location}/lib:${bzip2:location}/lib:${gcc:location}/lib:${gcc:location}/lib64:${zlib:location}/lib
environment = environment =
PATH=${gcc:location}/bin:%(PATH)s PATH=${gcc:location}/bin:%(PATH)s
BZIP2_INCLUDE=${bzip2:location}/include BZIP2_INCLUDE=${bzip2:location}/include
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment