- GitLab Software + patches ported to GitLab 8.6.X;
- Configs synced with upstream;
- Base software upgraded where appropriate;
- misc adjustments.

Demo instance: https://softinst64196.host.vifib.net/

@jerome @kazuhiko @iv Please have a look. I've verified it works but there is always a chance one can miss some detail. If all ok I'd like to deploy this tomorrow (3 Aug) evening to lab.nexedi.com

Thanks beforehand for feedback,  
Kirill


/reviewed-on nexedi/slapos!92

Allows easier parameter input.

Starting from GitLab 8.6 pg_trgm extension becomes hard dependency of
gitlab.

    https://gitlab.com/gitlab-org/gitlab-ce/commit/d24ee2a2

The extension can be activated only by db superuser, so gitlab db
migration scripts does not activate it - it has to be done by DB
administrator or is handled by integrating code in omnibus case.

As we already handle DB setup and migrations in unicorn startup script,
as pre-action there, let's activate pg_trgm.

We'll need to invoke psql connected to gitlab db in another place, so
before doing it let's factor out the code to call psql as connected to a
separate function.

Like for 0a72505e

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --
        files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

I manually reviewed

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --  \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

in omnibus-gitlab, and module proxy_cache and http2 changes, which we
already handled in 2 previous patches, there is nothing more interesting
for us.

Almost no changes this time: we only comment-out Nginx cache. See
details for why we do not need it in comments and in f6f97d72.

This does almost(*) only pure merge. We will slaposify / adjust config
and corresponding md5sum in the following patches.

(*) option to enable/disable HTTP/2 was in the same line as other nginx
    already jinja2'ified listen options.

    As already noted in f6f97d72 we are going to always support HTTP/2,
    that's why we do not merge-in upstream change only to through it away in
    the following patch.

Update GitLab software to

    - gitlab-ce 8.6.9 + NXD patches

      nexedi/gitlab-ce!1

    - gitlab-shell to 2.6.12 + 1 patch to remove unneeded hooks.old in *.git

      nexedi/gitlab-shell!1

    - gitlab-workhorse 0.7.1 + NXD patches.

      nexedi/gitlab-workhorse!1

      ( download speedup patches were reworked because of upstream changes.
	Please see details in the above MR and in fixup commits )

This only updates software and begins SR update to 8.6 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

The reason is: starting from GitLab 8.6 this extension becomes hard
dependency of GitLab.

References:

    https://about.gitlab.com/2016/03/22/gitlab-8-6-released/
    -> "Changes for Source installations with PostgreSQL"

    http://www.postgresql.org/docs/current/static/pgtrgm.html

NOTE

There is no way to activate only some extension building at configure
time - it is "all" or "all with all extensions" in postgresql speak (=
"world" make target).

PostgreSQL INSTALL explicitly suggests for selected-extensions install
to jump to appropriate dirs and do `make install` from there.

    http://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=contrib/README;h=5eaeb2451f29877e986f4683c57dd70edde942d5;hb=HEAD#l15

that's why we abuse slapos.recipe.cmmi a bit and do a double

    make install && make -C contrib/pg_trgm/ install

Compared to 9.2.16 postgresql 9.2.17 is a bugfix release:

    https://www.postgresql.org/docs/9.2/static/release-9-2-17.html

gitlab-workhorse works perfectly fine with it, so switch to current stable golang.

2.9.0 -> 2.9.2 is a bugfix release with several fixes:

    https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/2.9.1.txt
    https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/2.9.2.txt

/reviewed-by TrustMe

From upcoming https://golang.org/doc/devel/release.html#go1.6.minor

    go1.6.3 (released 2016/07/17) includes security fixes to the net/http/cgi
    package and net/http package when used in a CGI environment. This release also
    adds support for macOS Sierra.  See the Go 1.6.3 milestone[1] on our issue
    tracker for details.

    [1] https://github.com/golang/go/issues?q=milestone%3AGo1.6.3

/reviewed-by TrustMe    (tested with helloworld)

To pick up output \n and language/runtime version in output.

helloweb@0487fa7b...39fd89a3

/reviewed-by TrustMe

@jerome says at nexedi/slapos@5f5d5102 (comment 17119):

before f4e51f77, we had:
`~/srv/runner/instance/slappart0/bin/gitlab-rake` containing:
    ```python
    ...
    if __name__ == '__main__':
        sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
    ```

after, `~/srv/runner/instance/slappart0/bin/gitlab-rake` contains:
    ```shell
    #!/bin/bash
    COMMAND=/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake.py

    # If the wrapped command uses a shebang, execute the referenced
    # executable passing the script path as first argument.
    # This is to workaround the limitation of 127 characters in #!
    if [[ -f $COMMAND && x$(head -c2 "$COMMAND") = x"#!" ]]; then
      SHEBANG=$(head -1 "$COMMAND")
      INTERPRETER=( ${SHEBANG#\#!} )
      COMMAND="${INTERPRETER[@]} $COMMAND"
    fi

    exec $COMMAND
    ```

which is a wrapper around `gitlab-rake.py` containing:
    ```python
    ...
    if __name__ == '__main__':
        sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
    ```

`gitlab-rake.py` after is same as `gitlab-rake` before.

This [slapos.cookbook:wrapper](https://lab.nexedi.com/nexedi/slapos/blob/cd9faac0/slapos/recipe/wrapper.py#L39) has an argument *parameters-extra* which if set to true, propagate command line arguments to the wrapped script. The default value for this parameter is false.

Before f4e51f77, the generated wrapper was also propagating arguments even when *parameters-extra* was not set, but since this commit, this *parameters-extra* option is now handled as expected.

This is the reason for this regression. In our case, when we see `/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake assets:clean`, it just calls `rake` without arguments.

So a simple patch that fix the problem would be jerome/slapos@d3d05f02 . This way, the generated wrapper becomes:

```shell
...
exec $COMMAND $@
```

and arguments are correctly propagated.

Feel free to cherry-pick that patch for now, but it may be nice to rethink this *parameters-extra* option, after this debugging session, I believe it should be true by default.

/cc @seb for introducing the parameter in 80bb4305 and @vpelletier for touching this code in e7083872

/reviewed-by @kirr

/reviewed-by: TrustMe

- logrotate: make stack with cfg.in containing sections for logrotating with dcron and gzip
- neoppod: use this logrotate stack

- apache: add generic configuration file for backend
- erp5: make balancer use generic apache backend configuration file

@Tyagov this is the slapos part of nexedi/erp5!131. It includes:

- Documentation on top on each Jupyter notebook about why and how we use the
environment object.

- astor egg, used to convert abstract syntax tree objects back to source
code.

/reviewed-on nexedi/slapos!76