Close the NBD socket when testing it so it won't mess up qemu.

/reviewed-on nexedi/slapos!516

- Add `buildout.hash.cfg` file so this way we can use the useful `update-hash` script
- Refactor a bit the `software.cfg` file sections as it was repeating itself

/reviewed-on nexedi/slapos!512

Extend the backend with Timeout configuration via headers and use it
to prove, that request taking more than try_duration is correctly served.

Also prove that try_duration and try_interval are correct passed to the slave
configuration.

try_duration and try_interval are Caddy proxy's switches which allow to deal
with non working backend (https://caddyserver.com/docs/proxy)

The non working backend is the one, to which connection is lost or was not
possible to make, without sending any data.

The default try_duration=5s and try_interval=250ms are chosen, so that in
normal network conditions (with all possible problems in the network, like
lost packets) the browser will have to wait up to 5 seconds to be informed
that backend is inaccessible or for the request to start being processed,
but only a bit more than 250ms if Caddy would have to reestablish connection
to faulty backend.

In order to check it out it is advisable to setup a system, with real backend,
like apache one, and configure iptables to randomly reject packets to it:

  iptables -A INPUT -m statistic --mode random -p tcp --dport <backend_port> \
  --probability 0.05 -j REJECT --reject-with tcp-reset

Using ab or any other tool will results with lot of 502 EOF in the Caddy error
log and also reported by ab. With this configuration there are no more
errors visible to the client, which come from the problems on the network
between Caddy and the backend.

Assert itself "check_output" of given command, so provide simple
subprocess_output with all possible information from the called command.

Since slapos.core 1.4.16+ slapos proxy is not able to consume or publish
non-strings, which makes tests not working.

Thanks to this 'None' is used in BuildoutSerializer

ZODB/go started to use it in tests:

	kirr/neo@532d014f

It started to be used in fs1/go.

https://github.com/kisielk/og-rek/compare/dd41cde712...8b25c4cefd

- support for persistent references,
- fixes for decoder and encoder,
- allow to specify protocol on encoding,
- support for protocols 3 and 4 including added support for Python bytes,
- package-level documentation.

update pep445 patch using https://raw.githubusercontent.com/vstinner/pytracemalloc/pytracemalloc-1.4/patches/2.7.15/pep445.patch

If qemu use physical disk device instead of created qemu image disk, we skip this promise.

/reviewed-on nexedi/slapos!507

While helping @romain to review my current work we tried to build helloworld and found that it is completely broken.

It was first minimally fixed to pin eggs and the like.

Then I've added Go1.11 to our stack, so that recent-enough compiler/stdlib could be used. A note goes here that currently our lab uses go1.9 which is outdated and not supported by upstream, and other bits are still using go1.10.3 while go1.10.7 was already released some time ago with bug and security fixes. I did not touched neither  go1.10 / nor go1.9, but imho it makes sense for infrastructure people to look into appropriate upgrades.

Finally component/helloweb/ is switched to use gowork infrastructure (see 1b540151 for gowork introduction). Today gowork is used to build eveything go-related, so building go bits manually via cmmi as example is  a bit misleading. Gowork is also required in case we'll need to use any third-party or our package a helloweb-go.

/cc @luke, @alain.takoudjou
/reviewed-by @jerome, @tomo
/reviewed-on nexedi/slapos!505

- it is gowork that is now used by Go-related bits in SlapOS, so using
  gowork is preferred as example. Besides we need gowork to be able to
  use any third-party Go package.

- other languages can use helloweb repository from under gowork/ tree as
  well.

The hash of helloweb.git is changed, becuase its layout had to be too
adjusted to match gowork mode:

helloweb@a072af78...8bfedac6

To pick up Python3 support.

helloweb@39fd89a3...a072af78

Don't drop support for Go1.9, as, even though Go1.9 is no longer
supported, software/gitlab depends on it.

Our Go1.10.X is also too outdated, but I'm not touching it here neither.

/cc @alain.takoudjou, @luke

	2019-01-21 17:56:18 slapos[13553] INFO While:
	2019-01-21 17:56:18 slapos[13553] INFO   Installing.
	2019-01-21 17:56:18 slapos[13553] INFO   Getting section instance-profile.
	2019-01-21 17:56:18 slapos[13553] INFO   Initializing section instance-profile.
	2019-01-21 17:56:18 slapos[13553] INFO   Installing recipe slapos.recipe.template.
	2019-01-21 17:56:18 slapos[13553] INFO   Getting distribution for 'slapos.recipe.template'.
	2019-01-21 17:56:18 slapos[13553] INFO Error: Picked: slapos.recipe.template = 4.3
	2019-01-21 17:56:19 slapos[13553] ERROR Failed to run buildout profile in directory '/srv/slapgrid/slappart7/srv/runner/software/19771f7b751ffc2a88162b15750c6069'

	2019-01-21 18:01:21 slapos[26771] INFO While:
	2019-01-21 18:01:21 slapos[26771] INFO   Installing.
	2019-01-21 18:01:21 slapos[26771] INFO   Getting section helloweb-ruby.
	2019-01-21 18:01:21 slapos[26771] INFO   Initializing section helloweb-ruby.
	2019-01-21 18:01:21 slapos[26771] INFO   Getting option helloweb-ruby:input.
	2019-01-21 18:01:21 slapos[26771] INFO   Getting section helloweb-ruby-bundle.
	2019-01-21 18:01:21 slapos[26771] INFO   Initializing section helloweb-ruby-bundle.
	2019-01-21 18:01:21 slapos[26771] INFO   Getting option helloweb-ruby-bundle:make-targets.
	2019-01-21 18:01:21 slapos[26771] INFO   Getting section bundler.
	2019-01-21 18:01:21 slapos[26771] INFO   Initializing section bundler.
	2019-01-21 18:01:21 slapos[26771] INFO   Installing recipe rubygemsrecipe.
	2019-01-21 18:01:21 slapos[26771] INFO   Getting distribution for 'rubygemsrecipe'.
	2019-01-21 18:01:21 slapos[26771] INFO Error: Picked: rubygemsrecipe = 0.2.2+slapos001

Ruby stuff was failing to download at all -> let's use the versions that
are the same as currently used in software/gitlab.

Based on patch by @romain.
Helped by @jerome.

openss1.1 caused problems with bundler:

Unable to require openssl, install OpenSSL and rebuild ruby (preferred)
or use non-HTTPS sources

/reviewed-on nexedi/slapos!504

They are dependencies of slapos.toolbox and we already agreed to put slapos.toolbox version in a shared file.

/cc @jerome @jm

/reviewed-on !502

One of solutions for random 502 errors from caddy is to fully disable
HTTP2 protocol ( https://github.com/mholt/caddy/issues/1080 )

We run Caddy with HTTP2 enabled by default, as we can enable/disable it per
each slave, but in some environments it might be just better to fully avoid
HTTP2 codepaths in Caddy.

/reviewed-on !495

The IP used by frontend can be different than the real endpoint, and unknown
for the frontend itself, so make it catch-all to allow access.

/reviewed-on nexedi/slapos!497

Because of misleading tests (Accept-Encoding with gzip was always set by
requests, fixed in "caddy-frontend/test: Workaround requests issue with
Accept-Encoding") the original commit "Fix/caddy frontend prefer gzip type
zope" did not really fixed the issue for type:zope backend.

requests set Accept-Encoding header, but in the testr environment we
want to have full control over its behaviour, thus not setting any header if
not really wanted.

As there is not known way to avoid setting the header (skip_accept_encoding is
internal to httplib) set dummy Accept-Encoding header, which is enough for our
environment.