There is no reason to git clone by default, as slapos.cookbook should be developed independently.

  Anyone an include develop, locally, to test on the edge changes, or include local modification
  when they use webrunner or a development server.

https://bitbucket.org/sirex/rubygemsrecipe/issues/9/gem-shebang-line-length-limitation

so that the client IP is no longer the frontend IP, so rack
attack won't be blacklisting fontend IPs

The watcher should also watch for signals like SIGTERM killing sidekiq, which
are trapped by sidekiq, with sidekiq exiting successfully (with exit code 0).

To achieve this we rework our watcher-sigkill to be a generic watcher -
that can be given a set of restart exit codes including signal names and
monitors whether child process terminated with matching for restart exit
code.

Example usage:

	watcher 0,SIGKILL prog ...

Based on patch by @iv.
Discussion: https://lab.nexedi.com/lab.nexedi.com/lab.nexedi.com/issues/25#note_22085

An issue on Gitlab SR when pushing on protected branches occurs with git 2.11.
Gitlab issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/25601

Fixes: https://lab.nexedi.com/lab.nexedi.com/lab.nexedi.com/issues/37

In prevision for gitlab-backup restore to be done in a script. In fact, it requires `sed` command to be available.
We have to add /bin to the PATH as `which sed` returns `/bin/sed`.

  - git-clone and build git-backup sources
  - add a cron job doing `gitlab-backup-pull` every 4 hours by default
  - add instance parameter to change backup frequency
  - add xnice for using less resources while backuping
  - add lock to avoid concurrent calls of backup script
  - add a specific software type `gitlab-export` doing all this instead of in
    `default`; this can be changed in Services > parameters (kirr's changes)

`coreutils`, `grep` and `tar` components are used by `gitlab-backup-pull`
script, as it is called from a cron and the PATH has to include these
binaries paths

This is needed until the patch regarding `pidfile` in wrapper.py
(571d6514) is added to released SlapOS cookbook egg.

Update GitLab software to

    - gitlab-ce 8.8.9 + NXD patches

    - gitlab-shell stays at 2.7.2 + 1 patch to remove unneeded hooks.old in *.git
      because gitlab-ce 8.7.x sticks to this version (i.e. no shell upgrade for
      gitlab 8.7 -> 8.8)

    - gitlab-workhorse stays at 0.7.1 + NXD patches because gitlab-ce 8.8.x
      sticks to this version (i.e. no workhorse upgrade for gitlab 8.7 -> 8.8)

This only updates software and begins SR update to 8.8 - for now gitlab
instance starts using gitlab-ce appropriate version.

it is no longer possible to use http:/ as curl now expect a hostname to be
given, see curl issue https://github.com/curl/curl/issues/936

Like for af0c5b11

    $ git diff 8-7-stable 8-8-stable --
            files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

Like for 61177775.

I've manually reviewed

    git diff 8-7-stable 8-8-stable -- files/gitlab-config-template/gitlab.rb.template \
                                      files/gitlab-cookbooks/gitlab/attributes/default.rb

on omnibus-gitlab repository.

There are only changes on postgresql and registry options, which are not relevant for us.

Do not use container_registry and registry_* options

Like f6f97d72 - pristine copy from omnibus-gitlab 8.8.7+ce.1-0-g5116476

Changes are:

    - gitlab.yml.erb
      Add gitlab_default_projects_features_container_registry variable to be used by docker containers

    - nginx.conf.erb
      Docker related password storage
      https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1218 (commit f74472d4)

    - rack_attack.rb.erb
      Disable Rack Attack throttling if admin disables it in config file

    - smtp_settings.rb.erb
      If authentication is not enabled for smtp, don't place it in the config.

The following files stay the same:

    - database.yml.erb
    - gitconfig.erb
    - gitlab-shell-config.yml.erb
    - nginx-gitlab-http.conf.erb
    - resque.yml.erb
    - unicorn.rb.erb

Like for 2a835e63

    $ git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --
            files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

I've manually reviewed

    git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --    \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

and modulo trusted proxies there are no interesting changes for us.

- config.ru template is gone - pristine gitlab-ce/config.ru can do the
  job because it obtains unicorn OOM killer setting via environment
  variables.

  https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/672

- we put TODO there for configuring trusted proxies (gitlab.yml & nginx)

- we restore our slaposified configuration from config.ru to unicorn.rb

Update GitLab software to

    - gitlab-ce 8.7.9 + NXD patches

    - gitlab-shell to 2.7.2 + 1 patch to remove unneeded hooks.old in *.git

    - gitlab-workhorse stays at 0.7.1 + NXD patches because gitlab-ce 8.7.x
      sticks to this version (i.e. no workhorse upgrade for gitlab 8.6 -> 8.7)

This only updates software and begins SR update to 8.7 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

Like f6f97d72 - pristine copy from omnibus-gitlab 8.7.9+ce.1-0-gf589ad7

Changes are:

    - database.yml.erb

      * db_sslca option to specify CA for cases when DB is accessed via
        SSL (we do not need it as we access DB over unix:// only)

    - gitconfig.erb

      * turns gc.auto=0

        This is questionable to me. What they needed is to adjust
        warning reporting in git, not completely disable gc.auto and control it
        with their hands from rails.

        context: https://gitlab.com/gitlab-org/gitlab-ce/issues/14357

    - gitlab-rails-config.ru.erb removed

      with unicorn OOM killer settings moved to unicorn.rb. See:

      https://gitlab.com/gitlab-org/omnibus-gitlab/commit/cfbe6c55
      https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/672

    - gitlab.yml.erb

      * +geo_bulk_notify_worker (EE only, we do not use gitlab geo)
      * +repository_archive_cache_worker.cron   (gitlab-ce defaults to "0 * * * *")
        https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3663
      * +update_all_remote_mirrors_worker.cron  (EE only ?)
      * +omniauth.external_providers (we do not use omniauth)
      * +trusted_proxies

        this adds ability to let gitlab know trusted proxies addresses
        from which it can get and trust things like X-Forwarded-For and the
        like.

    - nginx-gitlab-http.conf.erb

      * add support for using nginx's realip module
        (http://nginx.org/en/docs/http/ngx_http_realip_module.html) for
        configuring trusted proxies and letting requests from them to
        pass through nginx with e.g. X-Forwarded-For header.

    - smtp_settings.rb.erb

      * +ssl option

        https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/730

    - unicorn.rb: see above about "gitlab-rails-config.ru.erb removed"

The following files stay the same:

    - gitlab-shell-config.yml.erb
    - nginx.conf.erb
    - rack_attack.rb.erb
    - resque.yml.erb

Starting from GitLab 8.6 pg_trgm extension becomes hard dependency of
gitlab.

    https://gitlab.com/gitlab-org/gitlab-ce/commit/d24ee2a2

The extension can be activated only by db superuser, so gitlab db
migration scripts does not activate it - it has to be done by DB
administrator or is handled by integrating code in omnibus case.

As we already handle DB setup and migrations in unicorn startup script,
as pre-action there, let's activate pg_trgm.

We'll need to invoke psql connected to gitlab db in another place, so
before doing it let's factor out the code to call psql as connected to a
separate function.

Like for 0a72505e

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --
        files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

is empty.

I manually reviewed

    $ git diff 8.5.1+ce.0-1-ge732b39..8.6.5+ce.0-0-g342f8be --  \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

in omnibus-gitlab, and module proxy_cache and http2 changes, which we
already handled in 2 previous patches, there is nothing more interesting
for us.

Almost no changes this time: we only comment-out Nginx cache. See
details for why we do not need it in comments and in f6f97d72.

Update GitLab software to

    - gitlab-ce 8.6.9 + NXD patches

      gitlab-ce!1

    - gitlab-shell to 2.6.12 + 1 patch to remove unneeded hooks.old in *.git

      gitlab-shell!1

    - gitlab-workhorse 0.7.1 + NXD patches.

      gitlab-workhorse!1

      ( download speedup patches were reworked because of upstream changes.
	Please see details in the above MR and in fixup commits )

This only updates software and begins SR update to 8.6 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

gitlab-workhorse works perfectly fine with it, so switch to current stable golang.

@jerome says at 5f5d5102 (comment 17119):

before f4e51f77, we had:
`~/srv/runner/instance/slappart0/bin/gitlab-rake` containing:
    ```python
    ...
    if __name__ == '__main__':
        sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
    ```

after, `~/srv/runner/instance/slappart0/bin/gitlab-rake` contains:
    ```shell
    #!/bin/bash
    COMMAND=/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake.py

    # If the wrapped command uses a shebang, execute the referenced
    # executable passing the script path as first argument.
    # This is to workaround the limitation of 127 characters in #!
    if [[ -f $COMMAND && x$(head -c2 "$COMMAND") = x"#!" ]]; then
      SHEBANG=$(head -1 "$COMMAND")
      INTERPRETER=( ${SHEBANG#\#!} )
      COMMAND="${INTERPRETER[@]} $COMMAND"
    fi

    exec $COMMAND
    ```

which is a wrapper around `gitlab-rake.py` containing:
    ```python
    ...
    if __name__ == '__main__':
        sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
    ```

`gitlab-rake.py` after is same as `gitlab-rake` before.

This [slapos.cookbook:wrapper](https://lab.nexedi.com/nexedi/slapos/blob/cd9faac0/slapos/recipe/wrapper.py#L39) has an argument *parameters-extra* which if set to true, propagate command line arguments to the wrapped script. The default value for this parameter is false.

Before f4e51f77, the generated wrapper was also propagating arguments even when *parameters-extra* was not set, but since this commit, this *parameters-extra* option is now handled as expected.

This is the reason for this regression. In our case, when we see `/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake assets:clean`, it just calls `rake` without arguments.

So a simple patch that fix the problem would be jerome/slapos@d3d05f02 . This way, the generated wrapper becomes:

```shell
...
exec $COMMAND $@
```

and arguments are correctly propagated.

Feel free to cherry-pick that patch for now, but it may be nice to rethink this *parameters-extra* option, after this debugging session, I believe it should be true by default.

/cc @seb for introducing the parameter in 80bb4305 and @vpelletier for touching this code in e7083872

/reviewed-by @kirr

It was failing because gzip was not on PATH for gitlab-workhorse
service.

    2016/04/22 18:44:56 error: handleGetArchive: start [gzip -c -n]: exec: "gzip": executable file not found in $PATH

The same for bzip2. Fix it.

/reported-by @jm

it ends up being copied to the wrapper itself, like this:

{'PATH': '/srv/slapgrid/slappart6/srv/runner/software/f3701a18118c774c95b8ff0519e5de46/parts/git/bin:/srv/slapgrid/slappart6/srv/runner/software/f3701a18118c774c95b8ff0519e5de46/parts/ruby2.1/bin:%(PATH)s',

My mistake from 6ddc1c30 (gitlab: Setup gitlab-workhorse service)