Shacache now uses CDN based URL for download.


  Update configuration to use CDN based domains for download date from CDN. 

See merge request !56

- GitLab Software + patches ported to latest stable GitLab 8.5.1
  (including fix for raw downloading to work in browser for private
repositories);
- Sync-with-upstream procedure streamlined (now only 1 branch for
  tracking upstream configuration files);
- Base software upgraded: Ruby, Redis, Nginx, Git;
- misc fixes.

/cc @jerome, @jm
/reviewed-by @kazuhiko
/reviewed-on nexedi/slapos!55

    $ git diff 8.4.4+ce.0-0-g1680742..8.5.1+ce.0-1-ge732b39 --  \
        files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

shows nothing.

I manually reviewed

    $ git diff 8.4.2+ce.0-3-g68d5ee8..8.5.1+ce.0-1-ge732b39 \
        files/gitlab-config-template/gitlab.rb.template \
        files/gitlab-cookbooks/gitlab/attributes/default.rb

in omnibus-gitlab, and module proxy_set_header change, which we already
addressed in previous patch in Nginx config, there are no more changes
for us.

    - relative URL support: comment out - we do not need it - gitlab is
      always located at /.

    - Nginx-http: restore our version for proxy_set_header - upstream
      turned to allowing users to configure this, see e.g.

        https://gitlab.com/gitlab-org/omnibus-gitlab/commit/e13d5e42
        https://gitlab.com/gitlab-org/omnibus-gitlab/commit/a450585e

      but doing this way creates more complexity for gitlab SR, so I've
      restored our version which essentially does the same as default in
      omnibus-gitlab, and if we'll need to tune it - we can do directly in
      Nginx config.

      In other words slapos version does not allow users to tune nginx
      headers as instance parameter.

This does only pure merge. We will slaposify / adjust config and
corresponding md5sum in the following patches.

/cc @kazuhiko, @jerome

As it is said in 97dcf455 (gitlab: Establish proper 1 branch for
tracking upstream configs) we are switching to a model where we track
upstream configureation files on only one branch.

This merge does not change files on master - because we already have all
current upstream changes in - just establish a proper structure for future
updates.

/cc @kazuhiko, @jerome

Update GitLab software to

    - gitlab-ce 8.5.1 + NXD patches

      https://lab.nexedi.com/kirr/gitlab-ce/commits/8-5-nxd

    - gitlab-shell to 2.6.10 + 1 patch to remove unneeded hooks.old in *.git

      https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/40

    - gitlab-workhorse 0.6.4 + NXD patches.

      https://lab.nexedi.com/kirr/gitlab-workhorse/commits/y/blobraw-4
      https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17

      ( download speedup patches got improved, and now also properly
        proxy _gitlab_session cookie to auth backend, so raw files for
        private repositories now open in browser ok )

This only updates software and begins SR update to 8.5 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

P.S. we also pin-up rubygems version, used to build gems, along the way.

GitLab uses git executable by full path as defined in gitlab.yml, but
not all places in code use it, e.g. here git is used just from $PATH

    https://gitlab.com/gitlab-org/gitlab_git/blob/2f0d3c1a/lib/gitlab_git/repository.rb#L259

So make sure to include our git into bundler-4gitlab PATH.

2.7.0 -> 2.7.2 is a bugfix release with several fixes:

    https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/2.7.1.txt
    https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/2.7.2.txt

1.9.4 -> 1.9.12 adds HTTP/2 support and removes SPDY support + other
bugfixes and improvements. We need HTTP/2 support for GitLab 8.5.

HTTP/2 details:

   http://hg.nginx.org/nginx/rev/257b51c37c5a

Full changelog:

---- 8< ---- http://nginx.org/en/CHANGES
Changes with nginx 1.9.12                                        24 Feb 2016

    *) Feature: Huffman encoding of response headers in HTTP/2.
       Thanks to Vlad Krasnov.

    *) Feature: the "worker_cpu_affinity" directive now supports more than
       64 CPUs.

    *) Bugfix: compatibility with 3rd party C++ modules; the bug had
       appeared in 1.9.11.
       Thanks to Piotr Sikora.

    *) Bugfix: nginx could not be built statically with OpenSSL on Linux;
       the bug had appeared in 1.9.11.

    *) Bugfix: the "add_header ... always" directive with an empty value did
       not delete "Last-Modified" and "ETag" header lines from error
       responses.

    *) Workaround: "called a function you should not call" and "shutdown
       while in init" messages might appear in logs when using OpenSSL
       1.0.2f.

    *) Bugfix: invalid headers might be logged incorrectly.

    *) Bugfix: socket leak when using HTTP/2.

    *) Bugfix: in the ngx_http_v2_module.

Changes with nginx 1.9.11                                        09 Feb 2016

    *) Feature: TCP support in resolver.

    *) Feature: dynamic modules.

    *) Bugfix: the $request_length variable did not include size of request
       headers when using HTTP/2.

    *) Bugfix: in the ngx_http_v2_module.

Changes with nginx 1.9.10                                        26 Jan 2016

    *) Security: invalid pointer dereference might occur during DNS server
       response processing if the "resolver" directive was used, allowing an
       attacker who is able to forge UDP packets from the DNS server to
       cause segmentation fault in a worker process (CVE-2016-0742).

    *) Security: use-after-free condition might occur during CNAME response
       processing if the "resolver" directive was used, allowing an attacker
       who is able to trigger name resolution to cause segmentation fault in
       a worker process, or might have potential other impact
       (CVE-2016-0746).

    *) Security: CNAME resolution was insufficiently limited if the
       "resolver" directive was used, allowing an attacker who is able to
       trigger arbitrary name resolution to cause excessive resource
       consumption in worker processes (CVE-2016-0747).

    *) Feature: the "auto" parameter of the "worker_cpu_affinity" directive.

    *) Bugfix: the "proxy_protocol" parameter of the "listen" directive did
       not work with IPv6 listen sockets.

    *) Bugfix: connections to upstream servers might be cached incorrectly
       when using the "keepalive" directive.

    *) Bugfix: proxying used the HTTP method of the original request after
       an "X-Accel-Redirect" redirection.

Changes with nginx 1.9.9                                         09 Dec 2015

    *) Bugfix: proxying to unix domain sockets did not work when using
       variables; the bug had appeared in 1.9.8.

Changes with nginx 1.9.8                                         08 Dec 2015

    *) Feature: pwritev() support.

    *) Feature: the "include" directive inside the "upstream" block.

    *) Feature: the ngx_http_slice_module.

    *) Bugfix: a segmentation fault might occur in a worker process when
       using LibreSSL; the bug had appeared in 1.9.6.

    *) Bugfix: nginx could not be built on OS X in some cases.

Changes with nginx 1.9.7                                         17 Nov 2015

    *) Feature: the "nohostname" parameter of logging to syslog.

    *) Feature: the "proxy_cache_convert_head" directive.

    *) Feature: the $realip_remote_addr variable in the
       ngx_http_realip_module.

    *) Bugfix: the "expires" directive might not work when using variables.

    *) Bugfix: a segmentation fault might occur in a worker process when
       using HTTP/2; the bug had appeared in 1.9.6.

    *) Bugfix: if nginx was built with the ngx_http_v2_module it was
       possible to use the HTTP/2 protocol even if the "http2" parameter of
       the "listen" directive was not specified.

    *) Bugfix: in the ngx_http_v2_module.

Changes with nginx 1.9.6                                         27 Oct 2015

    *) Bugfix: a segmentation fault might occur in a worker process when
       using HTTP/2.
       Thanks to Piotr Sikora and Denis Andzakovic.

    *) Bugfix: the $server_protocol variable was empty when using HTTP/2.

    *) Bugfix: backend SSL connections in the stream module might be timed
       out unexpectedly.

    *) Bugfix: a segmentation fault might occur in a worker process if
       different ssl_session_cache settings were used in different virtual
       servers.

    *) Bugfix: nginx/Windows could not be built with MinGW gcc; the bug had
       appeared in 1.9.4.
       Thanks to Kouhei Sutou.

    *) Bugfix: time was not updated when the timer_resolution directive was
       used on Windows.

    *) Miscellaneous minor fixes and improvements.
       Thanks to Markus Linnala, Kurtis Nusbaum and Piotr Sikora.

Changes with nginx 1.9.5                                         22 Sep 2015

    *) Feature: the ngx_http_v2_module (replaces ngx_http_spdy_module).
       Thanks to Dropbox and Automattic for sponsoring this work.

    *) Change: now the "output_buffers" directive uses two buffers by
       default.

    *) Change: now nginx limits subrequests recursion, not simultaneous
       subrequests.

    *) Change: now nginx checks the whole cache key when returning a
       response from cache.
       Thanks to Gena Makhomed and Sergey Brester.

    *) Bugfix: "header already sent" alerts might appear in logs when using
       cache; the bug had appeared in 1.7.5.

    *) Bugfix: "writev() failed (4: Interrupted system call)" errors might
       appear in logs when using CephFS and the "timer_resolution" directive
       on Linux.

    *) Bugfix: in invalid configurations handling.
       Thanks to Markus Linnala.

    *) Bugfix: a segmentation fault occurred in a worker process if the
       "sub_filter" directive was used at http level; the bug had appeared
       in 1.9.4.
---- 8< ----

Redis 2.8.23 -> 2.8.24 is a small bugfix release:

    --[ Redis 2.8.24 ] Release date: 18 Dec 2015

    Upgrade urgency: MODERATE. We fixed a crash that happens very rarely, so
                     updating does not hurt, but most users are unlikely to
                     experience this condition because it requires some odd
                     timing.

    * [FIX] lua_struct.c/getnum security issue fixed. (Luca Bruno discovered it,
            patched by Sun He and Chris Lamb)
    * [FIX] Fix a race condition in processCommand() because of interactions
            with freeMemoryIfNeeded(). Details in issue #2948 and especially
            in the commit message d999f5a. (Race found analytically by
            Oran Agra, patch by Salvatore Sanfilippo)

    * [NEW] Log offending memory access address on SIGSEGV/SIGBUS (Salvatore
            Sanfilippo)

https://raw.githubusercontent.com/antirez/redis/2.8/00-RELEASENOTES

No config changes.

Ruby 2.1.8 contains security and other bugfixes

    https://www.ruby-lang.org/en/news/2015/12/16/ruby-2-1-8-released/

Like 8c62b063, d17f1f5f and e8461571 - pristine copy from omnibus-gitlab
8.5.1+ce.0-1-ge732b39 .

Changes are in

    - gitlab.yml.erb, unicorn.rb.erb

      * Something related to relative URL root (we do not use)
      * Something related to SAML (we do not use)
      * Misc

    - nginx-gitlab-http.conf.erb

      * SPDY -> HTTP/2
      * Relative URL root
      * Configurable proxy_set_header passing

The following files stay the same:

    - database.yml.erb
    - gitconfig.erb
    - gitlab-rails-config.ru.erb
    - gitlab-shell-config.yml.erb
    - nginx.conf.erb
    - rack_attack.rb.erb
    - resque.yml.erb
    - smtp_settings.rb.erb

It was my mistake to establish several tracking lines for tracking
upstream changes - e.g. in

    61544d87    (gitlab: Import nginx http configuration from omnibus-gitlab)

we started not from

    6fd7b987    (gitlab: Import gitlab-ce & gitlab-shell configs from omnibus-gitlab)

-- the first upstream tracking commit on its own branch -- but from

    4c127fdd    (gitlab: Setup sidekiq service)

i.e. from after some changes which already tweaked upstream
configuration files.

This makes updating gitlab more work than necessary: instead of
switching to upstream branch only once, importing all files, and
then switching back to master and merging upstream changes only once, we
currently have to do that operation 3 times:

    - for main gitlab settings,
    - for nginx settings, and
    - for gitconfig settings

which is not convenient and wastes our time.

So establish a proper 1 branch for tracking upstream configs:

Here we cherry-pick the following commits

    61544d87    (gitlab: Import nginx http configuration from omnibus-gitlab)
    d17f1f5f    (gitlab: Sync nginx http configuration from omnibus gitlab)

    8f945bd2    (gitlab: Import gitconfig from omnibus-gitlab)
    e8461571    (gitlab: Sync gitconfig settings from omnibus-gitlab)

and later we'll be updating upstream files on a branch starting from
this commit and containing upstream changes only.

/cc @kazuhiko, @jerome

/reviewed-by @vpelletier

component/alsa: Do not disable pcm, firefox needs it otherwise it crashes when playing sounds

In a selenium suite for a project, we have some gadgets that play some sounds using html5 audio element.
We discovered that the firefox coming with SlapOS used in test nodes crashes as soon as it plays a sound, with this error:
```
parts/firefox/firefox: relocation error:
/srv/slapgrid/slappart11/srv/runner/software/88ef27d24c6e0a792f72f568146a3838/parts/firefox/libxul.so:
symbol snd_pcm_open, version ALSA_0.9 not defined in file libasound.so.2 with link time reference
```
indeed, libasound.so does not contain this symbol if built with --disable-pcm

Building libasound without this switch fixed this problem and did not seem to cause any trouble, but I have no idea why pcm was disabled in the first place. It has been like this since initial commit of this component profile at 82900a33


It's not urgent, but we would like to have this in testnode branch at some point. Shall I make another merge request for this commit in testnode branch or just merge master in testnode branch ?

See merge request !35

Conflicts:
	software/erp5testnode/software.cfg

Since this parameter is a json encoded string, request parameter must be []

@rafael says frontend infrastructure is not yet ready to handle slave
requests for frontends which come with lab.nexedi.com frontend SR URLs.

So revert apache-frontend-URL related part of 86deb9c6 (*: slapos.git
moved to https://lab.nexedi.com/nexedi/slapos.git) back for now.

/cc @jp
/reviewed-by @rafael
/reviewed-on nexedi/slapos!52

86deb9c6 (*: slapos.git moved to https://lab.nexedi.com/nexedi/slapos.git)
went over whole tree and updated slapos.git URL but when changing
non-top-software.cfg-like files forgot to update their md5sum in
software releases.

Fixes started to appear - e.g. 8b1496c3 (kvm: fix md5sum, thanks
@alain.takoudjou), but generally the whole tree remains more-or-less
broken.

Fix it. (not tested - manually checked files and updated md5sums)

/reported-by @rafael

This updates links to slapos.git in the tree to point to new location.

We do so whole-tree except one place in stack/monitor/ :

    [download-monitor-static]
    recipe = hexagonit.recipe.download
    url = http://git.erp5.org/gitweb/slapos.git/snapshot/930be99041ea26b7b1186830e5eb56ef0acc1bdf.tar.gz
    ...

(see d8800c0b "monitor: Download statics files from snapshot")

The reason we do not update that link yet, is that 930be99041ea26b7b1186830e5eb56ef0acc1bdf
is a tree object and gitlab does not allow to dowload tree object as archives (yet ?)

So for now that link stays unconverted - and we'll think how to do with it.

Fix image display on jupyter

This fix is for bug 
https://nexedi.erp5.net/bug_module/20160212-76EAAC/view?ignore_layout:int=1
where the image base64 string was only showing. The error in this case was that the `mime_type` of content code_result was not being updated by the kernel before display.

Reported by : @Tyagov 
Please review: @Tyagov @kirr 

For testing, you can use this jupyter instance with updated kernel running : 
https://[2001:67c:1254:e:8b::335d]:8888/tree

There are also some example notebooks in the aforementioned jupyter.

See merge request !47

This fix is for bug https://nexedi.erp5.net/bug_module/20160212-76EAAC/view?ignore_layout:int=1
where the image base64 string was only shoeing. The error in this case was that the mime_type
of content code_result was not being updated by the kernel before display.

Reported by : @ivan

To pick up kirr/gitlab-ce@32fa1180

/cc @kazuhiko, @jerome
/reviewed-by TrustMe

6a4e45c5 (gitlab: Sync sidekiq service launch with upstream) claimed to
sync sidekiq startup with upstream, but missed one upstream change in
between 8.2 and 8.4:

    https://gitlab.com/gitlab-org/omnibus-gitlab/commit/2eb7634f

My bad - I should have reviewed git log for

    https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

and I did not noticed that change because I did not and just compared
queues list visually.

Pick it up.

/cc @kazuhiko, @jerome
/reviewed-by TrustMe (I hope)

This brings slapos gitlab port up to date with current stable gitlab
version 8.4, and teaches Unicorn startup script to wait a bit for
PostgreSQL to start and be ready, before unicorn startup proceeds.

/cc @kazuhiko
/somewhat-reviewed-by @jerome
/reviewed-on nexedi/slapos!44