hosts-dict values cannot be domain names.
memcached entries are implementation-independent (no mention of kumofs
anymore).
relational database information (which does not follow above memcached
rule, sadly, naming mariadb explicitly) is published in two lists.
Add minimal patterns to document what requester should expect.
Document deadlock-debugger-password".
Drop non-standard "optional" property from jupyter-url declaration.

slaprunner: fix resiliency test, add development-mode (bool) parameter for easy build webrunner in webrunner

This is backported to ZODB-3.10 https://github.com/zopefoundation/persistent/pull/44

The patch is needed for wendelin.core to being able to relase not needed
memory.

/reviewed-on nexedi/slapos!93

This reverts commit 605e564b.

Rationale: Stability matters:

Quoting 605e564b:
> Besides changing only recv window size at runtime breaks compatibility with
> openssh: if we only do `-W 1M` on server and try to upload data with openssh as
> client, dropbear complains
>
>     [3302] Apr 17 23:10:06 Exit (slapuser2): Bad packet size 32777
>
> and connection terminates. Thus RECV_MAX_PAYLOAD_LEN increase is also
> required, which cannot be done via option at runtime:
>
>     https://github.com/mkj/dropbear/blob/DROPBEAR_0.53.1/options.h#L268
>
>     ---- 8< ----
>     /* Maximum size of a received SSH data packet - this _MUST_ be >= 32768
>        in order to interoperate with other implementations */
>     #ifndef RECV_MAX_PAYLOAD_LEN
>     #define RECV_MAX_PAYLOAD_LEN 32768
>     #endif
>     ---- 8< ----
>
> So let's increase DEFAULT_RECV_WINDOW to 1M and RECV_MAX_PAYLOAD_LEN
> appropriately (experimentally found that at 512K the complain goes
> away).

It turned out that "Bad packet size" did not really went away. For example I've
recently hit the following:

    [14586] Aug 04 19:12:43 Pubkey auth succeeded for 'slapuser16' with key md5 b1:35:06:d3:a5:b1:0b:c6:7f:e6:59:31:ab:3a:e1:56 from 2001:67c:1254:c0::1:49886
    [14586] Aug 04 19:12:55 Exit (slapuser16): Integrity error (bad packet size 524500)

in .slappartX_runner_sshd.log of my upgraded webrunner with connection being broken.
( !68 (comment 17748) )

We could maybe try to play games with increasing RECV_MAX_PAYLOAD_LEN to
be more than DEFAULT_RECV_WINDOW but this already turned out to be error-prone.

Since when really needed we should be able to replace dropbear with openssh

    !68 (comment 7082)

which is both performant and good-compatible, to me the way is:

- make current dropbear run stable again,
- when we really need to sync large amounts of data (and we should be
  needing to do soon or already) -> work on replacing dropbear with
  openssh.

- GitLab Software + patches ported to GitLab 8.7.X;
- Configs synced with upstream;
- No base software upgrades this time because it was all recently
  upgraded during a590b03e;

TODO: allow configuration of trusted proxies

/reviewed-by TrustMe

Like for 2a835e63

    $ git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --
            files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

I've manually reviewed

    git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --    \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

and modulo trusted proxies there are no interesting changes for us.

- config.ru template is gone - pristine gitlab-ce/config.ru can do the
  job because it obtains unicorn OOM killer setting via environment
  variables.

  https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/672

- we put TODO there for configuring trusted proxies (gitlab.yml & nginx)

- we restore our slaposified configuration from config.ru to unicorn.rb

This does almost(*) only pure merge. We will slaposify / adjust config
and corresponding md5sum in the following patches.

(*) smtp ssl option is only added as comment.

Update GitLab software to

    - gitlab-ce 8.7.9 + NXD patches

    - gitlab-shell to 2.7.2 + 1 patch to remove unneeded hooks.old in *.git

    - gitlab-workhorse stays at 0.7.1 + NXD patches because gitlab-ce 8.7.x
      sticks to this version (i.e. no workhorse upgrade for gitlab 8.6 -> 8.7)

This only updates software and begins SR update to 8.7 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

Like f6f97d72 - pristine copy from omnibus-gitlab 8.7.9+ce.1-0-gf589ad7

Changes are:

    - database.yml.erb

      * db_sslca option to specify CA for cases when DB is accessed via
        SSL (we do not need it as we access DB over unix:// only)

    - gitconfig.erb

      * turns gc.auto=0

        This is questionable to me. What they needed is to adjust
        warning reporting in git, not completely disable gc.auto and control it
        with their hands from rails.

        context: https://gitlab.com/gitlab-org/gitlab-ce/issues/14357

    - gitlab-rails-config.ru.erb removed

      with unicorn OOM killer settings moved to unicorn.rb. See:

      https://gitlab.com/gitlab-org/omnibus-gitlab/commit/cfbe6c55
      https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/672

    - gitlab.yml.erb

      * +geo_bulk_notify_worker (EE only, we do not use gitlab geo)
      * +repository_archive_cache_worker.cron   (gitlab-ce defaults to "0 * * * *")
        https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3663
      * +update_all_remote_mirrors_worker.cron  (EE only ?)
      * +omniauth.external_providers (we do not use omniauth)
      * +trusted_proxies

        this adds ability to let gitlab know trusted proxies addresses
        from which it can get and trust things like X-Forwarded-For and the
        like.

    - nginx-gitlab-http.conf.erb

      * add support for using nginx's realip module
        (http://nginx.org/en/docs/http/ngx_http_realip_module.html) for
        configuring trusted proxies and letting requests from them to
        pass through nginx with e.g. X-Forwarded-For header.

    - smtp_settings.rb.erb

      * +ssl option

        https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/730

    - unicorn.rb: see above about "gitlab-rails-config.ru.erb removed"

The following files stay the same:

    - gitlab-shell-config.yml.erb
    - nginx.conf.erb
    - rack_attack.rb.erb
    - resque.yml.erb

  CFFI is added on this directory, and it is required to bootstrap slapos.toolbox.

- GitLab Software + patches ported to GitLab 8.6.X;
- Configs synced with upstream;
- Base software upgraded where appropriate;
- misc adjustments.

Demo instance: https://softinst64196.host.vifib.net/

@jerome @kazuhiko @iv Please have a look. I've verified it works but there is always a chance one can miss some detail. If all ok I'd like to deploy this tomorrow (3 Aug) evening to lab.nexedi.com

Thanks beforehand for feedback,  
Kirill


/reviewed-on nexedi/slapos!92

Allows easier parameter input.

Starting from GitLab 8.6 pg_trgm extension becomes hard dependency of
gitlab.

    https://gitlab.com/gitlab-org/gitlab-ce/commit/d24ee2a2

The extension can be activated only by db superuser, so gitlab db
migration scripts does not activate it - it has to be done by DB
administrator or is handled by integrating code in omnibus case.

As we already handle DB setup and migrations in unicorn startup script,
as pre-action there, let's activate pg_trgm.

We'll need to invoke psql connected to gitlab db in another place, so
before doing it let's factor out the code to call psql as connected to a
separate function.

Like for 0a72505e

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --
        files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

I manually reviewed

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --  \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

in omnibus-gitlab, and module proxy_cache and http2 changes, which we
already handled in 2 previous patches, there is nothing more interesting
for us.

Almost no changes this time: we only comment-out Nginx cache. See
details for why we do not need it in comments and in f6f97d72.

This does almost(*) only pure merge. We will slaposify / adjust config
and corresponding md5sum in the following patches.

(*) option to enable/disable HTTP/2 was in the same line as other nginx
    already jinja2'ified listen options.

    As already noted in f6f97d72 we are going to always support HTTP/2,
    that's why we do not merge-in upstream change only to through it away in
    the following patch.

Update GitLab software to

    - gitlab-ce 8.6.9 + NXD patches

      nexedi/gitlab-ce!1

    - gitlab-shell to 2.6.12 + 1 patch to remove unneeded hooks.old in *.git

      nexedi/gitlab-shell!1

    - gitlab-workhorse 0.7.1 + NXD patches.

      nexedi/gitlab-workhorse!1

      ( download speedup patches were reworked because of upstream changes.
	Please see details in the above MR and in fixup commits )

This only updates software and begins SR update to 8.6 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.