* patch nltk egg so that the default data location is (buildout_directory)/parts/nltk-data without explicitly specifing by NLTK_DATA environment variable.
* download collection can be overridden like 'collection = aaa bbb ccc'.

/ref http://community.slapos.org/forum/NLTK-Buildout--Download-and-install-nltk_data

/cc @klaus, @Kreisel 

/reviewed-on !58

To pick up

    gitlab-workhorse@32ec77c4

/reviewed-by TrustMe  (I hope)

@rafael says:

    Use master instead.

    nexedi/slapos@000d0e5a (comment 4716)

So lamp-edge -> master

@rafael says:

    development stuff from marco, it was probably removed or not even
    included.

    In theory you can use master instead.

    nexedi/slapos@692729bc (comment 4713)

So lapp-resilient3 -> master

To pick up

    nexedi/gitlab-ce@827d3914

/reviewed-by TrustMe (I hope)

To pick up

    nexedi/gitlab-ce@1bfb586a

/reviewed-by @kazuhiko

kirr/ remains as my personal development place and nexedi/ ones become
official repositories.

Discussed with @kazuhiko
/reviewed-by TrustMe

version up: OpenSSL 1.0.2g, fixing CVE-2016-0800, CVE-2016-0705, CVE-2016-0798, CVE-2016-0797, CVE-2016-0799, and CVE-2016-0702.

To pick up

    kirr/gitlab-ce@8d922ec3
    kirr/gitlab-ce@5d6b7eba

See:

    kirr/gitlab-ce@5a6e6e55

for discussion.

/cc @kazuhiko, @jerome
/reviewed-by TrustMe

Shacache now uses CDN based URL for download.


  Update configuration to use CDN based domains for download date from CDN. 

See merge request !56

- GitLab Software + patches ported to latest stable GitLab 8.5.1
  (including fix for raw downloading to work in browser for private
repositories);
- Sync-with-upstream procedure streamlined (now only 1 branch for
  tracking upstream configuration files);
- Base software upgraded: Ruby, Redis, Nginx, Git;
- misc fixes.

/cc @jerome, @jm
/reviewed-by @kazuhiko
/reviewed-on nexedi/slapos!55

    $ git diff 8.4.4+ce.0-0-g1680742..8.5.1+ce.0-1-ge732b39 --  \
        files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

shows nothing.

I manually reviewed

    $ git diff 8.4.2+ce.0-3-g68d5ee8..8.5.1+ce.0-1-ge732b39 \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

in omnibus-gitlab, and module proxy_set_header change, which we already
addressed in previous patch in Nginx config, there are no more changes
for us.

    - relative URL support: comment out - we do not need it - gitlab is
      always located at /.

    - Nginx-http: restore our version for proxy_set_header - upstream
      turned to allowing users to configure this, see e.g.

        https://gitlab.com/gitlab-org/omnibus-gitlab/commit/e13d5e42
        https://gitlab.com/gitlab-org/omnibus-gitlab/commit/a450585e

      but doing this way creates more complexity for gitlab SR, so I've
      restored our version which essentially does the same as default in
      omnibus-gitlab, and if we'll need to tune it - we can do directly in
      Nginx config.

      In other words slapos version does not allow users to tune nginx
      headers as instance parameter.

This does only pure merge. We will slaposify / adjust config and
corresponding md5sum in the following patches.

/cc @kazuhiko, @jerome

As it is said in 97dcf455 (gitlab: Establish proper 1 branch for
tracking upstream configs) we are switching to a model where we track
upstream configureation files on only one branch.

This merge does not change files on master - because we already have all
current upstream changes in - just establish a proper structure for future
updates.

/cc @kazuhiko, @jerome

Update GitLab software to

    - gitlab-ce 8.5.1 + NXD patches

      https://lab.nexedi.com/kirr/gitlab-ce/commits/8-5-nxd

    - gitlab-shell to 2.6.10 + 1 patch to remove unneeded hooks.old in *.git

      https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/40

    - gitlab-workhorse 0.6.4 + NXD patches.

      https://lab.nexedi.com/kirr/gitlab-workhorse/commits/y/blobraw-4
      https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17

      ( download speedup patches got improved, and now also properly
        proxy _gitlab_session cookie to auth backend, so raw files for
        private repositories now open in browser ok )

This only updates software and begins SR update to 8.5 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

P.S. we also pin-up rubygems version, used to build gems, along the way.

GitLab uses git executable by full path as defined in gitlab.yml, but
not all places in code use it, e.g. here git is used just from $PATH

    https://gitlab.com/gitlab-org/gitlab_git/blob/2f0d3c1a/lib/gitlab_git/repository.rb#L259

So make sure to include our git into bundler-4gitlab PATH.

2.7.0 -> 2.7.2 is a bugfix release with several fixes:

    https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/2.7.1.txt
    https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/2.7.2.txt

1.9.4 -> 1.9.12 adds HTTP/2 support and removes SPDY support + other
bugfixes and improvements. We need HTTP/2 support for GitLab 8.5.

HTTP/2 details:

   http://hg.nginx.org/nginx/rev/257b51c37c5a

Full changelog:

---- 8< ---- http://nginx.org/en/CHANGES
Changes with nginx 1.9.12                                        24 Feb 2016

    *) Feature: Huffman encoding of response headers in HTTP/2.
       Thanks to Vlad Krasnov.

    *) Feature: the "worker_cpu_affinity" directive now supports more than
       64 CPUs.

    *) Bugfix: compatibility with 3rd party C++ modules; the bug had
       appeared in 1.9.11.
       Thanks to Piotr Sikora.

    *) Bugfix: nginx could not be built statically with OpenSSL on Linux;
       the bug had appeared in 1.9.11.

    *) Bugfix: the "add_header ... always" directive with an empty value did
       not delete "Last-Modified" and "ETag" header lines from error
       responses.

    *) Workaround: "called a function you should not call" and "shutdown
       while in init" messages might appear in logs when using OpenSSL
       1.0.2f.

    *) Bugfix: invalid headers might be logged incorrectly.

    *) Bugfix: socket leak when using HTTP/2.

    *) Bugfix: in the ngx_http_v2_module.

Changes with nginx 1.9.11                                        09 Feb 2016

    *) Feature: TCP support in resolver.

    *) Feature: dynamic modules.

    *) Bugfix: the $request_length variable did not include size of request
       headers when using HTTP/2.

    *) Bugfix: in the ngx_http_v2_module.

Changes with nginx 1.9.10                                        26 Jan 2016

    *) Security: invalid pointer dereference might occur during DNS server
       response processing if the "resolver" directive was used, allowing an
       attacker who is able to forge UDP packets from the DNS server to
       cause segmentation fault in a worker process (CVE-2016-0742).

    *) Security: use-after-free condition might occur during CNAME response
       processing if the "resolver" directive was used, allowing an attacker
       who is able to trigger name resolution to cause segmentation fault in
       a worker process, or might have potential other impact
       (CVE-2016-0746).

    *) Security: CNAME resolution was insufficiently limited if the
       "resolver" directive was used, allowing an attacker who is able to
       trigger arbitrary name resolution to cause excessive resource
       consumption in worker processes (CVE-2016-0747).

    *) Feature: the "auto" parameter of the "worker_cpu_affinity" directive.

    *) Bugfix: the "proxy_protocol" parameter of the "listen" directive did
       not work with IPv6 listen sockets.

    *) Bugfix: connections to upstream servers might be cached incorrectly
       when using the "keepalive" directive.

    *) Bugfix: proxying used the HTTP method of the original request after
       an "X-Accel-Redirect" redirection.

Changes with nginx 1.9.9                                         09 Dec 2015

    *) Bugfix: proxying to unix domain sockets did not work when using
       variables; the bug had appeared in 1.9.8.

Changes with nginx 1.9.8                                         08 Dec 2015

    *) Feature: pwritev() support.

    *) Feature: the "include" directive inside the "upstream" block.

    *) Feature: the ngx_http_slice_module.

    *) Bugfix: a segmentation fault might occur in a worker process when
       using LibreSSL; the bug had appeared in 1.9.6.

    *) Bugfix: nginx could not be built on OS X in some cases.

Changes with nginx 1.9.7                                         17 Nov 2015

    *) Feature: the "nohostname" parameter of logging to syslog.

    *) Feature: the "proxy_cache_convert_head" directive.

    *) Feature: the $realip_remote_addr variable in the
       ngx_http_realip_module.

    *) Bugfix: the "expires" directive might not work when using variables.

    *) Bugfix: a segmentation fault might occur in a worker process when
       using HTTP/2; the bug had appeared in 1.9.6.

    *) Bugfix: if nginx was built with the ngx_http_v2_module it was
       possible to use the HTTP/2 protocol even if the "http2" parameter of
       the "listen" directive was not specified.

    *) Bugfix: in the ngx_http_v2_module.

Changes with nginx 1.9.6                                         27 Oct 2015

    *) Bugfix: a segmentation fault might occur in a worker process when
       using HTTP/2.
       Thanks to Piotr Sikora and Denis Andzakovic.

    *) Bugfix: the $server_protocol variable was empty when using HTTP/2.

    *) Bugfix: backend SSL connections in the stream module might be timed
       out unexpectedly.

    *) Bugfix: a segmentation fault might occur in a worker process if
       different ssl_session_cache settings were used in different virtual
       servers.

    *) Bugfix: nginx/Windows could not be built with MinGW gcc; the bug had
       appeared in 1.9.4.
       Thanks to Kouhei Sutou.

    *) Bugfix: time was not updated when the timer_resolution directive was
       used on Windows.

    *) Miscellaneous minor fixes and improvements.
       Thanks to Markus Linnala, Kurtis Nusbaum and Piotr Sikora.

Changes with nginx 1.9.5                                         22 Sep 2015

    *) Feature: the ngx_http_v2_module (replaces ngx_http_spdy_module).
       Thanks to Dropbox and Automattic for sponsoring this work.

    *) Change: now the "output_buffers" directive uses two buffers by
       default.

    *) Change: now nginx limits subrequests recursion, not simultaneous
       subrequests.

    *) Change: now nginx checks the whole cache key when returning a
       response from cache.
       Thanks to Gena Makhomed and Sergey Brester.

    *) Bugfix: "header already sent" alerts might appear in logs when using
       cache; the bug had appeared in 1.7.5.

    *) Bugfix: "writev() failed (4: Interrupted system call)" errors might
       appear in logs when using CephFS and the "timer_resolution" directive
       on Linux.

    *) Bugfix: in invalid configurations handling.
       Thanks to Markus Linnala.

    *) Bugfix: a segmentation fault occurred in a worker process if the
       "sub_filter" directive was used at http level; the bug had appeared
       in 1.9.4.
---- 8< ----

Redis 2.8.23 -> 2.8.24 is a small bugfix release:

    --[ Redis 2.8.24 ] Release date: 18 Dec 2015

    Upgrade urgency: MODERATE. We fixed a crash that happens very rarely, so
                     updating does not hurt, but most users are unlikely to
                     experience this condition because it requires some odd
                     timing.

    * [FIX] lua_struct.c/getnum security issue fixed. (Luca Bruno discovered it,
            patched by Sun He and Chris Lamb)
    * [FIX] Fix a race condition in processCommand() because of interactions
            with freeMemoryIfNeeded(). Details in issue #2948 and especially
            in the commit message d999f5a. (Race found analytically by
            Oran Agra, patch by Salvatore Sanfilippo)

    * [NEW] Log offending memory access address on SIGSEGV/SIGBUS (Salvatore
            Sanfilippo)

https://raw.githubusercontent.com/antirez/redis/2.8/00-RELEASENOTES

No config changes.

Ruby 2.1.8 contains security and other bugfixes

    https://www.ruby-lang.org/en/news/2015/12/16/ruby-2-1-8-released/

Like 8c62b063, d17f1f5f and e8461571 - pristine copy from omnibus-gitlab
8.5.1+ce.0-1-ge732b39 .

Changes are in

    - gitlab.yml.erb, unicorn.rb.erb

      * Something related to relative URL root (we do not use)
      * Something related to SAML (we do not use)
      * Misc

    - nginx-gitlab-http.conf.erb

      * SPDY -> HTTP/2
      * Relative URL root
      * Configurable proxy_set_header passing

The following files stay the same:

    - database.yml.erb
    - gitconfig.erb
    - gitlab-rails-config.ru.erb
    - gitlab-shell-config.yml.erb
    - nginx.conf.erb
    - rack_attack.rb.erb
    - resque.yml.erb
    - smtp_settings.rb.erb

It was my mistake to establish several tracking lines for tracking
upstream changes - e.g. in

    61544d87    (gitlab: Import nginx http configuration from omnibus-gitlab)

we started not from

    6fd7b987    (gitlab: Import gitlab-ce & gitlab-shell configs from omnibus-gitlab)

-- the first upstream tracking commit on its own branch -- but from

    4c127fdd    (gitlab: Setup sidekiq service)

i.e. from after some changes which already tweaked upstream
configuration files.

This makes updating gitlab more work than necessary: instead of
switching to upstream branch only once, importing all files, and
then switching back to master and merging upstream changes only once, we
currently have to do that operation 3 times:

    - for main gitlab settings,
    - for nginx settings, and
    - for gitconfig settings

which is not convenient and wastes our time.

So establish a proper 1 branch for tracking upstream configs:

Here we cherry-pick the following commits

    61544d87    (gitlab: Import nginx http configuration from omnibus-gitlab)
    d17f1f5f    (gitlab: Sync nginx http configuration from omnibus gitlab)

    8f945bd2    (gitlab: Import gitconfig from omnibus-gitlab)
    e8461571    (gitlab: Sync gitconfig settings from omnibus-gitlab)

and later we'll be updating upstream files on a branch starting from
this commit and containing upstream changes only.

/cc @kazuhiko, @jerome