- GitLab Software + patches ported to GitLab 8.7.X;
- Configs synced with upstream;
- No base software upgrades this time because it was all recently
  upgraded during a590b03e;

TODO: allow configuration of trusted proxies

/reviewed-by TrustMe

Like for 2a835e63

    $ git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --
            files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

I've manually reviewed

    git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --    \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

and modulo trusted proxies there are no interesting changes for us.

- config.ru template is gone - pristine gitlab-ce/config.ru can do the
  job because it obtains unicorn OOM killer setting via environment
  variables.

  https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/672

- we put TODO there for configuring trusted proxies (gitlab.yml & nginx)

- we restore our slaposified configuration from config.ru to unicorn.rb

This does almost(*) only pure merge. We will slaposify / adjust config
and corresponding md5sum in the following patches.

(*) smtp ssl option is only added as comment.

Update GitLab software to

    - gitlab-ce 8.7.9 + NXD patches

    - gitlab-shell to 2.7.2 + 1 patch to remove unneeded hooks.old in *.git

    - gitlab-workhorse stays at 0.7.1 + NXD patches because gitlab-ce 8.7.x
      sticks to this version (i.e. no workhorse upgrade for gitlab 8.6 -> 8.7)

This only updates software and begins SR update to 8.7 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

Like f6f97d72 - pristine copy from omnibus-gitlab 8.7.9+ce.1-0-gf589ad7

Changes are:

    - database.yml.erb

      * db_sslca option to specify CA for cases when DB is accessed via
        SSL (we do not need it as we access DB over unix:// only)

    - gitconfig.erb

      * turns gc.auto=0

        This is questionable to me. What they needed is to adjust
        warning reporting in git, not completely disable gc.auto and control it
        with their hands from rails.

        context: https://gitlab.com/gitlab-org/gitlab-ce/issues/14357

    - gitlab-rails-config.ru.erb removed

      with unicorn OOM killer settings moved to unicorn.rb. See:

      https://gitlab.com/gitlab-org/omnibus-gitlab/commit/cfbe6c55
      https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/672

    - gitlab.yml.erb

      * +geo_bulk_notify_worker (EE only, we do not use gitlab geo)
      * +repository_archive_cache_worker.cron   (gitlab-ce defaults to "0 * * * *")
        https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3663
      * +update_all_remote_mirrors_worker.cron  (EE only ?)
      * +omniauth.external_providers (we do not use omniauth)
      * +trusted_proxies

        this adds ability to let gitlab know trusted proxies addresses
        from which it can get and trust things like X-Forwarded-For and the
        like.

    - nginx-gitlab-http.conf.erb

      * add support for using nginx's realip module
        (http://nginx.org/en/docs/http/ngx_http_realip_module.html) for
        configuring trusted proxies and letting requests from them to
        pass through nginx with e.g. X-Forwarded-For header.

    - smtp_settings.rb.erb

      * +ssl option

        https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/730

    - unicorn.rb: see above about "gitlab-rails-config.ru.erb removed"

The following files stay the same:

    - gitlab-shell-config.yml.erb
    - nginx.conf.erb
    - rack_attack.rb.erb
    - resque.yml.erb

  CFFI is added on this directory, and it is required to bootstrap slapos.toolbox.

- GitLab Software + patches ported to GitLab 8.6.X;
- Configs synced with upstream;
- Base software upgraded where appropriate;
- misc adjustments.

Demo instance: https://softinst64196.host.vifib.net/

@jerome @kazuhiko @iv Please have a look. I've verified it works but there is always a chance one can miss some detail. If all ok I'd like to deploy this tomorrow (3 Aug) evening to lab.nexedi.com

Thanks beforehand for feedback,  
Kirill


/reviewed-on nexedi/slapos!92

Allows easier parameter input.

Starting from GitLab 8.6 pg_trgm extension becomes hard dependency of
gitlab.

    https://gitlab.com/gitlab-org/gitlab-ce/commit/d24ee2a2

The extension can be activated only by db superuser, so gitlab db
migration scripts does not activate it - it has to be done by DB
administrator or is handled by integrating code in omnibus case.

As we already handle DB setup and migrations in unicorn startup script,
as pre-action there, let's activate pg_trgm.

We'll need to invoke psql connected to gitlab db in another place, so
before doing it let's factor out the code to call psql as connected to a
separate function.

Like for 0a72505e

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --
        files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

I manually reviewed

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --  \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

in omnibus-gitlab, and module proxy_cache and http2 changes, which we
already handled in 2 previous patches, there is nothing more interesting
for us.

Almost no changes this time: we only comment-out Nginx cache. See
details for why we do not need it in comments and in f6f97d72.

This does almost(*) only pure merge. We will slaposify / adjust config
and corresponding md5sum in the following patches.

(*) option to enable/disable HTTP/2 was in the same line as other nginx
    already jinja2'ified listen options.

    As already noted in f6f97d72 we are going to always support HTTP/2,
    that's why we do not merge-in upstream change only to through it away in
    the following patch.

Update GitLab software to

    - gitlab-ce 8.6.9 + NXD patches

      nexedi/gitlab-ce!1

    - gitlab-shell to 2.6.12 + 1 patch to remove unneeded hooks.old in *.git

      nexedi/gitlab-shell!1

    - gitlab-workhorse 0.7.1 + NXD patches.

      nexedi/gitlab-workhorse!1

      ( download speedup patches were reworked because of upstream changes.
	Please see details in the above MR and in fixup commits )

This only updates software and begins SR update to 8.6 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

The reason is: starting from GitLab 8.6 this extension becomes hard
dependency of GitLab.

References:

    https://about.gitlab.com/2016/03/22/gitlab-8-6-released/
    -> "Changes for Source installations with PostgreSQL"

    http://www.postgresql.org/docs/current/static/pgtrgm.html

NOTE

There is no way to activate only some extension building at configure
time - it is "all" or "all with all extensions" in postgresql speak (=
"world" make target).

PostgreSQL INSTALL explicitly suggests for selected-extensions install
to jump to appropriate dirs and do `make install` from there.

    http://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=contrib/README;h=5eaeb2451f29877e986f4683c57dd70edde942d5;hb=HEAD#l15

that's why we abuse slapos.recipe.cmmi a bit and do a double

    make install && make -C contrib/pg_trgm/ install

Compared to 9.2.16 postgresql 9.2.17 is a bugfix release:

    https://www.postgresql.org/docs/9.2/static/release-9-2-17.html

gitlab-workhorse works perfectly fine with it, so switch to current stable golang.

2.9.0 -> 2.9.2 is a bugfix release with several fixes:

    https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/2.9.1.txt
    https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/2.9.2.txt

/reviewed-by TrustMe

From upcoming https://golang.org/doc/devel/release.html#go1.6.minor

    go1.6.3 (released 2016/07/17) includes security fixes to the net/http/cgi
    package and net/http package when used in a CGI environment. This release also
    adds support for macOS Sierra.  See the Go 1.6.3 milestone[1] on our issue
    tracker for details.

    [1] https://github.com/golang/go/issues?q=milestone%3AGo1.6.3

/reviewed-by TrustMe    (tested with helloworld)

To pick up output \n and language/runtime version in output.

nexedi/helloweb@0487fa7b...39fd89a3

/reviewed-by TrustMe

@jerome says at 5f5d5102 (comment 17119):

before f4e51f77, we had:
`~/srv/runner/instance/slappart0/bin/gitlab-rake` containing:
    ```python
    ...
    if __name__ == '__main__':
        sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
    ```

after, `~/srv/runner/instance/slappart0/bin/gitlab-rake` contains:
    ```shell
    #!/bin/bash
    COMMAND=/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake.py

    # If the wrapped command uses a shebang, execute the referenced
    # executable passing the script path as first argument.
    # This is to workaround the limitation of 127 characters in #!
    if [[ -f $COMMAND && x$(head -c2 "$COMMAND") = x"#!" ]]; then
      SHEBANG=$(head -1 "$COMMAND")
      INTERPRETER=( ${SHEBANG#\#!} )
      COMMAND="${INTERPRETER[@]} $COMMAND"
    fi

    exec $COMMAND
    ```

which is a wrapper around `gitlab-rake.py` containing:
    ```python
    ...
    if __name__ == '__main__':
        sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
    ```

`gitlab-rake.py` after is same as `gitlab-rake` before.

This [slapos.cookbook:wrapper](https://lab.nexedi.com/nexedi/slapos/blob/cd9faac0/slapos/recipe/wrapper.py#L39) has an argument *parameters-extra* which if set to true, propagate command line arguments to the wrapped script. The default value for this parameter is false.

Before f4e51f77, the generated wrapper was also propagating arguments even when *parameters-extra* was not set, but since this commit, this *parameters-extra* option is now handled as expected.

This is the reason for this regression. In our case, when we see `/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake assets:clean`, it just calls `rake` without arguments.

So a simple patch that fix the problem would be jerome/slapos@d3d05f02 . This way, the generated wrapper becomes:

```shell
...
exec $COMMAND $@
```

and arguments are correctly propagated.

Feel free to cherry-pick that patch for now, but it may be nice to rethink this *parameters-extra* option, after this debugging session, I believe it should be true by default.

/cc @seb for introducing the parameter in 80bb4305 and @vpelletier for touching this code in e7083872

/reviewed-by @kirr

/reviewed-by: TrustMe