/reviewed-by @vpelletier

Conflicts:
	software/erp5testnode/software.cfg

@rafael says frontend infrastructure is not yet ready to handle slave
requests for frontends which come with lab.nexedi.com frontend SR URLs.

So revert apache-frontend-URL related part of 86deb9c6 (*: slapos.git
moved to https://lab.nexedi.com/nexedi/slapos.git) back for now.

/cc @jp
/reviewed-by @rafael
/reviewed-on !52

86deb9c6 (*: slapos.git moved to https://lab.nexedi.com/nexedi/slapos.git)
went over whole tree and updated slapos.git URL but when changing
non-top-software.cfg-like files forgot to update their md5sum in
software releases.

Fixes started to appear - e.g. 8b1496c3 (kvm: fix md5sum, thanks
@alain.takoudjou), but generally the whole tree remains more-or-less
broken.

Fix it. (not tested - manually checked files and updated md5sums)

/reported-by @rafael

This updates links to slapos.git in the tree to point to new location.

We do so whole-tree except one place in stack/monitor/ :

    [download-monitor-static]
    recipe = hexagonit.recipe.download
    url = http://git.erp5.org/gitweb/slapos.git/snapshot/930be99041ea26b7b1186830e5eb56ef0acc1bdf.tar.gz
    ...

(see d8800c0b "monitor: Download statics files from snapshot")

The reason we do not update that link yet, is that 930be99041ea26b7b1186830e5eb56ef0acc1bdf
is a tree object and gitlab does not allow to dowload tree object as archives (yet ?)

So for now that link stays unconverted - and we'll think how to do with it.

This fix is for bug https://nexedi.erp5.net/bug_module/20160212-76EAAC/view?ignore_layout:int=1
where the image base64 string was only shoeing. The error in this case was that the mime_type
of content code_result was not being updated by the kernel before display.

Reported by : @ivan

To pick up kirr/gitlab-ce@32fa1180

/cc @kazuhiko, @jerome
/reviewed-by TrustMe

6a4e45c5 (gitlab: Sync sidekiq service launch with upstream) claimed to
sync sidekiq startup with upstream, but missed one upstream change in
between 8.2 and 8.4:

    https://gitlab.com/gitlab-org/omnibus-gitlab/commit/2eb7634f

My bad - I should have reviewed git log for

    https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb

and I did not noticed that change because I did not and just compared
queues list visually.

Pick it up.

/cc @kazuhiko, @jerome
/reviewed-by TrustMe (I hope)

  This changes were squashed into a single commit and it includes:

    - install slapos_configurator by default
    - Allow define ssl authentication for a certain zope family
    - Allow define custom zope path per zope family

  Those changes are originally written by Alain Takoudjou and rewrited by Rafael Monnerat

To pick up kirr/gitlab-ce@a5965b54.

  The local changes are not yet generic enough to be shared among stack, so
  Introduce local changes here better

  There is no strong reason to use git.erp5.org to test code anymore

  slapos.package is not an egg anymore, it's implementation was replaced by
    the usage of ansible with playbooks.

As it was outlined in 5a744de7 (gitlab: Compile assets on instantiation
and make sure DB is properly setup/migrated before unicorn runs) we are
performing DB initialization and migration in pre-action as part of
unicorn startup script. But that currently has one drawback:

    if all services start at the same time - e.g. both PostgreSQL and
    Unicorn - and that is a common scenario when SR is compiled and
    instantiated / started, PostgreSQL is usually not yet ready to
    process queries from Unicorn startup script, and first-time Unicorn
    startup fails.

Until now this problem was workarounded by manually starting unicorn
second time - after some time postgresql is started and ready. But why
do it manually, if we can do the same logic automatically. So fix it:

    in Unicorn startup script wait a bit (up to 5 seconds) for
    PostgreSQL to become ready.

/cc @kazuhiko, @jerome

Even though we do not use LFS objects, builds and build artifacts,
gitlab backups corresponding directories and recreates them upon
restore.

Current state is: the directories are living under Rails root - that is
in our gitlab-work/ inside instance, and since gitlab-work is recreated
almost on every re-instantiation if we'll have some content in that
directories - it will be lost.

Don't allow even a chance for that to happen and move what is designed
to hold data to under var/ .

NOTE we create shared/lfs-objects/ and shared/artifacts/ because if we
    do not have this dirs, gitlab backup fails. Upstream code just
    assumes the directories are there.

/cc @kazuhiko, @jerome

Upstream added `pages` and `elasticsearch` queues to sidekiq

    https://gitlab.com/gitlab-org/omnibus-gitlab/commit/be6844ff
    https://gitlab.com/gitlab-org/omnibus-gitlab/commit/801d6ed7

but since those queues are used for EE-only services we do not enable
them.

/cc @kazuhiko, @jerome

As gitlab-workhorse is now serving everything, it proxies unknown-to-it
requests to unicorn, and that returns redirect to /users/sign_in for
/non-existent, not 403.

Adjust the promise and just verify whether we can get /static.css

NOTE nginx promise now effectively depends on gitlab-workhorse being up
    and running - because nginx is now configured to be only an SSL
    terminator and all requests processing is done by gitlab-workhorse and
    services below it.

    If in the future we'll need a pure nginx-is-up promise, we can
    introduce something to nginx config and verify that particular http
    endpoint.

/cc @kazuhiko, @jerome

add `-documentRoot ...` argument to gitlab-workhorse start as upstream
started to do:

    https://gitlab.com/gitlab-org/omnibus-gitlab/commit/d769f751

( gitlab-workhorse now needs to know where document root is as it now
  serves all content - e.g. static files too

  this option was introduced here:

    https://gitlab.com/gitlab-org/gitlab-workhorse/commit/1a7009e4  )

/cc @kazuhiko, @jerome

I manually reviewed

    files/gitlab-config-template/gitlab.rb.template
    files/gitlab-cookbooks/gitlab/attributes/default.rb

from omnibus-gitlab 8.4.2+ce.0-3-g68d5ee8 and picked up changes:

- unicorn OOM killer memory limits were raised up (because gitlab is now
  more memory hungry according to talks on gitlab.com)

  https://gitlab.com/gitlab-org/omnibus-gitlab/commit/f8ee4a19

- there is no default limit for nginx client_max_body_size - it just
  goes to gitlab-workhorse as is at any size

  https://gitlab.com/gitlab-org/omnibus-gitlab/commit/78862837

/cc @kazuhiko, @jerome

Direct merge for gitconfig.erb from omnibus-gitlab did not brought up
updates.

pack.threads=1 and receive.fsckObjects=true picked up from
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb
in 264d785a stay the same.

/cc @kazuhiko, @jerome

- there is a section for gitlab pages, which we stub-out;

- there is no longer a need to add /raw/... handling to nginx - as
  now nginx is just an SSL terminator for gitlab-workhorse, all URL
  handling is done inside gitlab-workhorse and is dealt with ok by our
  patches.

- as now nginx does not directly connect to unicorn, there is no need to
  pass unicorn section to nginx's template.

/cc @kazuhiko, @jerome

Slapos'ify config updates brought by sync with omnibus-gitlab (see merge
in the previous patch). Changes:

- default visibility levels gone away (see merge commit and
  https://gitlab.com/gitlab-org/omnibus-gitlab/commit/b5ebbab3 )

- there are new settings for GitLab Pages and Elasticsearch, which are EE
  only and thus we do not support them.

- there are new settings for auxiliary cron-like jobs, which we do not
  support for now, since they are used either for CI (not supported by
  us) or EE features.

Configuration files that were synced, but did not changed are also
marked as updated, so it is easier to track their changes to upstream in
the future.

/cc @kazuhiko, @jerome

Like 61544d87 - pristine copy from omnibus 8.4.4+ce.0-0-g1680742.

Most of http handling is moving to gitlab-workhorse. Nginx remains only
as ssl terminator.

/cc @kazuhiko, @jerome

Like 6fd7b987 - pristine copy from omnibus 8.4.4+ce.0-0-g1680742.

The only change is in

    gitlab.yml

The following files stay the same:

    database.yml.erb
    gitlab-rails-config.ru.erb
    gitlab-shell-config.yml.erb
    rack_attack.rb.erb
    resque.yml.erb
    smtp_settings.rb.erb
    unicorn.rb.erb

/cc @kazuhiko, @jerome

Update GitLab software to

    - gitlab-ce 8.4 + NXD patches

      https://lab.nexedi.com/kirr/gitlab-ce/commits/8-4-nxd

    - gitlab-shell to 2.6.10 which now is pure upstream, as all NXD
      patches were merged.

    - gitlab-workhorse 0.6.X + NXD patches.

      https://lab.nexedi.com/kirr/gitlab-workhorse/commits/y/blobraw-3
      https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17

This only updates software and begins SR update to 8.4 - for now gitlab
instance becomes non-working -- we'll pull in configuration files
updates and fixups in the following patches.

/cc @kazuhiko, @jerome

Query strings used on GET requests have size limitations
on servers and this causes big code cells to not be executed
at all, returning only an Internal Server Error with no
further explanation.

/reviewed-by @kirr, @Tyagov (on !43)

Pandas and scikit-learn are handy to have for data processing which we
are going to do more and more in ERP5 context. Matplotlib is very handy
to have when one works with Jupyter, but also can be used by just
backend code to generate graphs.

From this point of view it makes sense to have this eggs always
available out of the box. To do so move them from Wendelin to ERP5.

/suggested-by @tiwariayush
/reviewed-by @Tyagov (on !43)

This patch teaches ERP5 software release to automatically instantiate Jupyter
notebook web UI and tune it to connect to ERP5 by default. When Jupyter is
enabled, it also installs on-server erp5_data_notebook bt5 (erp5!29)
which handles code execution requested for Jupyter.

For ERP5 - for security and backward compatibility reasons - Jupyter
instantiation and erp5_data_notebook bt5 install happen only if jupyter is
explicitly enabled in instance parameters. The default is not to have Jupyter
out of the box.

On the other hand for Wendelin SR, which inherits from ERP5 SR, the
default is to have Jupyter out of the box, because Wendelin SR is fresh
enough without lots of backward compatibility needs, and Jupyter is
usually very handy for people who use Wendelin.

~~~~

For integration, we reuse already established in ERP5 infrastructure, to
request various slave instances, and request Jupyter in a way so it
automatically tunes and connects to balancer of one of Zope family.

Jupyter code itself is compiled by reusing
software/ipython_notebook/software.cfg, and Jupyter instance code is
reused by hooking software/ipython_notebook/instance.cfg.in into ERP5 SR
properly (the idea to override instance-jupyter not to render into
default template.cfg is taken from previous work by @tiwariayush).

~~~~

I tested this patch inside webrunner with create-erp5-site software type and
various configurations (whether to have or not have jupyter, to which zope
family to connect it, etc).

I have not tested frontend instantiation fully - because tests were done only
in webrunner, but I've tried to make sure generated buildout code is valid for
cases with frontend.

NOTE the code in this patch depends erp5_data_notebook bt5 (erp5!29) which just got merged to erp5.git recently (see erp5@f662b5a2)

NOTE even when erp5_data_notebook bt5 is installed, on a freshly installed ERP5, it
is required to "check site consistency" first, so that initial bt5(s) are
actually installed and erp5 is ready to function.

/cc @vpelletier, @Tyagov, @klaus, @Camata, @tiwariayush, @Kreisel, @jerome, @nexedi
/proposed-for-review-on !43

Gitlab uses github-markup to render various text-based markups
(markdown, rst, ...) into html. For rst github-markup wants to run
python and have docutils egg available:

    https://github.com/github/markup/blob/5393ae93/lib/github/markups.rb#L36

as we were not having docutils installed and path to proper python
interpreter setup, rst documents were not automatically rendered and
were show just as plain text.

We do a lot of documents in rst - that case is important for us.

So fix it by providing gitlab with properly setup python interpreter
with all needed eggs installed.

/cc @kazuhiko, @jerome
/proposed-for-review-on !39

The reason is: we are going to integrate Jupyter into ERP5 SR, and in
ERP5 SR code, there is already established infrastructure, which
requests slave instances. That infrastructure passes parameters in
serialized (= json) form, that's why we need to switch.

On the other hand, slapos.cookbook:slapconfiguration.serialised handles
non-json instance parameters as well: when there parameters format is
not

    {'_' -> <json>}

it decodes them as slapos.cookbook:slapconfiguration would do:

    https://lab.nexedi.com/nexedi/slapos/blob/69229988/slapos/recipe/slapconfiguration.py#L232
    https://lab.nexedi.com/nexedi/slapos/blob/69229988/slapos/recipe/librecipe/__init__.py#L51

So from this point of view, though we don't really need it here (Jupyter
SR is relatively new), we are staying backward compatible.

/cc @tiwariayush

Both numpy and ipython are included in ERP5 SR which wendelin inherits
from, and are pinned there. Here are e.g. latest pin-ups for numpy and
ipython in erp5:

    e3144a8a    (version up eggs.)
    135570c9    (version up eggs.)

Furthermore: this is not only a cleanup. As e3144a8a shows current
version of numpy in erp5 is 1.10.4 and in wendelin we still have 1.9.2
which is unintentional downgrade compared to erp5.

Don't do that.

/cc @kazuhiko
/reviewed-by @Tyagov  (on !41)