Tests do pass

https://nexedi.erp5.net/test_result_module/20180205-62704D70

(we can ignore git's hash in tests as I just rebased)

/reviewed-on nexedi/slapos!285

neotest: Actually add instance and teach it to run `neotest bench-local` under Nexedi testing infrastructure

Commit 31fe231f (neotest: Draft software-release to run NEO/go & friends
tests/benchmarks under webrunner) added "neotest" software release, but
there it added only software supplying - not instantiation - this way
allowing only manual play with neotest.

Recently @rafael approached me saying that one of my webrunner is red on
monitoring because "neotest" software inside fails to instantiate, and
suggested to create a simple instance with bin/runTestSuite inside, so
that this way neotest could be added to a testnode and run on regular
basis by our testing infrastructure.

So here it goes - simple instance with only bin/runTestSuite which tries
to adhere to testnode protocol.

Fixups discovered to be needed while running on testnodes are also included in this merge request.

/reviewed-on nexedi/slapos!282

lsblk is needed for neotest. Please see details in individual commit messages.

/cc @kazuhiko 
/reviewed-on nexedi/slapos!283

Similarly to coreutils we need recent util-linux because if e.g. a testnode has
old version of it its lsblk does not work to what neotest expects, e.g.:

	lsblk: unknown column: REV,SIZE

For example we need realpath and if base os provides only older coreutils there
is no such utility and neotest fails:

/srv/slapgrid/slappart8/srv/testnode/bct/soft/55158b6a5b027aa40954c3f974d5b3e9/go.work/src/lab.nexedi.com/kirr/neo/go/neo/t/neotest: line 589: realpath: command not found

Fix it by building & hooking into $PATH coreutils by ourselves.

neotest: Actually add instance and teach it to run `neotest bench-local` under Nexedi testing infrastructure

Commit 31fe231f (neotest: Draft software-release to run NEO/go & friends
tests/benchmarks under webrunner) added "neotest" software release, but
there it added only software supplying - not instantiation - this way
allowing only manual play with neotest.

Recently @rafael approached me saying that one of my webrunner is red on
monitoring because "neotest" software inside fails to instantiate, and
suggested to create a simple instance with bin/runTestSuite inside, so
that this way neotest could be added to a testnode and run on regular
basis by our testing infrastructure.

So here it goes - simple instance with only bin/runTestSuite which tries
to adhere to testnode protocol.

For the actual runTestSuite program I followed the same approach as for
bin/neotest itself - the main program is not processed via any templates
and is used as it was pristinely downloaded, and all buildout
integration is done via first sourcing $SR/neotest-env.sh .
So bin/runTestSuite inside instance follows this approach.

erp5.util is updated from 0.4.49 to latest 0.4.50 because there were
backward-incompatible changes in 0.4.50 release (2 @luke: which btw will
render neoppod and build-rina runTestSuite(s) broken when they try to
upgrade) and for me it would be a waste of time to later go through
updating for 0.4.49 -> 0.4.50 API change, so I used the latest release
right from start.

/cc @Tyagov
/see also kirr/neo@526491d3

lsblk is needed for neotest (see https://lab.nexedi.com/kirr/slapos/commit/7cc37e3c).
However lsblk is built only when the following dependecies are enabled:

	- libblkid
	- libmount
	- libsmartcols

	https://github.com/karelzak/util-linux/blob/v2.31-286-g8e782a839/configure.ac#L1483

we already have libblkid and libmount enabled, but libsmartcols was missing.

Enable it.

While debugging the change this print was useful to know whether the
patch was taken into effect or not, but now it seems to be too chatty in
practice. Let's disable it.

/reviewed-by TrustMe.

…l for activities is needed.

/reviewed-on nexedi/slapos!281

stack/monitor: Create srv/monitor/private/monitor-log directory. It is specified by slapos.monitor.monitor.Monitoring.

Please see description of the rationale in the added patch.

/co-developed-with @Tyagov

/reviewed-on nexedi/slapos!280

stack/monitor: Add a random delay of maximum 60 seconds to all monitor cron jobs to avoid high load at the first second of every minute.

Revert "stack/monitor: Add a random delay of maximum 60 seconds to all monitor cron jobs to avoid high load at the first second of every minute."

This reverts commit 677a80e4.

This reverts commit ee8e3fb2.

stack/monitor: Add a random delay of maximum 60 seconds to all monitor cron jobs to avoid high load at the first second of every minute.

It is useful to give a random delay on cron jobs.

For example one could want to build the software with race detector,
e.g.

	[gowork]
	...
	buildflags = -race

or if you want to build a plugin for fluent-bit

	[gowork]
	...
	install	   = your.package.import.path
	buildflags = -buildmode=c-shared

and then your package will be installed as shared library into

	${gowork:directory}/pkg/`go env GOOS`_`go env GOARCH`/<your.package.import.path>.a

One could pass that location explicitly to `fluentd-bit -e ...` or
symlink to it from some .so from global directory of fluent-bit plugins.

Maybe it is not very good to use `go install` for building plugins
(instead of `go build -o ...`) but we anyway need to have ability to
specify buildflags as a generic feature, hence this patch.

/reviewed-on nexedi/slapos!274

@alain.takoudjou , @rafael  this fixes broken webrunner tests.

Passing one: https://nexedi.erp5.net/test_result_module/20180126-5FF058E7/2

/reviewed-on nexedi/slapos!277

@Nicolas , @rafael , @klaus  please review.

/reviewed-on nexedi/slapos!276

@rafael , @Nicolas , @klaus  please review. It's followup of !272 

/reviewed-on nexedi/slapos!275

 zc.buildout version up to 2.5.2+slapos012
 slapos.recipe.template version up to 4.3
 slapos.core version up to 1.4.4

My mistake from 1b540151 (golang: Infrastructure to build Go workspaces / projects)

/reviewed-by TrustMe

If we don't specify MaxRequestWorkers, it seems like Apache 2.4
errors with:
AH03490: scoreboard is full, not at MaxRequestWorkers. Increase
ServerLimit.
Maybe because MaxRequestWorkers then defaults to a too high value
compared to ServerLimit and ThreadsPerChild.

/reviewed-on nexedi/slapos!263

This is useful to being able to peek for low level site result details.

getWorkingSet() is slow and it is not always needed. So, it is better not to call it in __init__.

Revert "slapos/recipe/librecipe/generic.py: GenericBaseRecipe's _ws is a lazy attribute now."

This reverts commit 281d3774.

getWorkingSet() is slow and it is not always needed. So, it is better not to call it in __init__.