It turned out we cannot currently use slapos.cookbook in software part
of SR - the reason is slapos.cookbook egg depends on lxml egg, which in
turn needs libxml to be also installed via slapos.

For this reason stack/slapos.cfg has

    [slapos-cookbook]   # NOTE _not_ slapos.cookbook
    recipe = zc.recipe.egg
    eggs =
      ${lxml-python:egg}
    ...

and lxml-python is lxml building recipe to build it together with
libxml:

    [lxml-python]
    recipe = zc.recipe.egg:custom
    egg = lxml

    rpath =
      ${libxml2:location}/lib/
      ${libxslt:location}/lib/
      ${zlib:location}/lib/
    environment = lxml-python-env
    ...

So underlying idea, as I understand it, is: every SR contains
slapos-cookbook in parts and this way lxml-python gets build. Then when
there is slapos.cookbook egg usage, it is already correctly built.

BUT

This works when such slapos.cookbook egg usage happens _only_ in instance
part of an SR: otherwise, if buildout sees slapos.cookbook egg usage in
some recipe, e.g. like it currently is in helloweb-ruby:

    [helloweb-ruby]
    recipe  = slapos.cookbook:wrapper
    ...

it _first_ tries to install slapos.cookbook egg directly - as needed for
recipes eggs are installed as a first step, _before_ further buildout
processing. What happens then is that slapos.cookbook (note not "-") egg
sources and dependencies are downloaded from pypi, including lxml egg,
all are tried to build, and in lxml egg it fails this way:

    ...
    Processing lxml-3.5.0
    Writing /tmp/tmpLGK4xWbuild/lxml-3.5.0/setup.cfg
    Running setup.py -q bdist_egg --dist-dir /tmp/tmpLGK4xWbuild/lxml-3.5.0/egg-dist-tmp-DJvofa
    Building lxml version 3.5.0.
    Building without Cython.
    ERROR: /bin/sh: 1: xslt-config: not found

    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt
    In file included from src/lxml/lxml.etree.c:323:0:
    src/lxml/includes/etree_defs.h:14:31: fatal error: libxml/xmlversion.h: No such file or directory
     #include "libxml/xmlversion.h"
                                   ^
    compilation terminated.
    Compile failed: command 'gcc' failed with exit status 1
    /tmp/tmpLGK4xWbuild/lxml-3.5.0/temp/xmlXPathInitGlEAOF.c:1:26: fatal error: libxml/xpath.h: No such file or directory
     #include "libxml/xpath.h"
                              ^
    compilation terminated.
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: Setup script exited with error: command 'gcc' failed with exit status 1
    An error occurred when trying to install lxml 3.5.0. Look above this message for any errors that were output by easy_install.
    Could't load zc.buildout entry point wrapper
    from slapos.cookbook:
    Couldn't install: lxml 3.5.0.
    While:
      Installing.
      Getting section helloweb-ruby.
      Initializing section helloweb-ruby.
      Installing recipe slapos.cookbook.
    Error: Couldn't install: lxml 3.5.0

Previously it probably used to work because we had system libxml
installed, and this way lxml compilation succeeded (but was incorrect
from slapos point of view).

( The problem turned out to be already known somehow - see e.g. c7d00913
  "Initial neoppod commit." and look for "Note on LXML/END LXML" there )

Solution could be: either fix slapos.cookbook installation via e.g.
teaching buildout to take into account pre-dependencies for eggs (for
lxml) or just to avoid using slapos.cookbook:wrapper for executable
generation.

While @kazuhiko is working on the first more-generic solution, here goes
a simpler one to just make helloweb component alive again: like it is
done in a lot of places (e.g. in software/kvm/) let's use
collective.recipe.template to generate a short shell script.

NOTE

previously the command line was

    ${bundler:bundle} exec sh -c 'helloweb.rb "$@"' ${:_buildout_section_name_}

but now it is the same with "$@" appended:

    exec ${bundler:bundle} exec sh -c 'helloweb.rb "$@"' ${:_buildout_section_name_} "$@"

The reason is slapos.cookbook:wrapper uses slapos.recipe.librecipe.execute.generic_exec()
in generated script, which appends sys.argv[1:] to the command-line implicitly:

    def generic_exec(args):
      ...
      os.execve(exec_list[0], exec_list + sys.argv[1:], exec_env)
                                        ^^^^^^^^^^^^^^

    https://lab.nexedi.com/nexedi/slapos/blob/54bbe0a9/slapos/recipe/librecipe/execute.py#L84

that's why last "$@" was not present in original version.

P.S.

Otherwise currently slapos.cookbook is used only in instance parts of
recipes in whole slapos.git

/reviewed-by TrustMe
/debugged-with @kazuhiko
/cc @vpelletier

ctypes lib does not read the rpath of the current binary to look after the
desired library, so although it exits the import failed. Pass-by the first
find to make the import work

Initially, the condition checking rdiff backup status was right
after calling it. But then a condition about CORRUPTED_ARGS has
been inserted, making the result of $? different from initially expected.

Due to this, the cleanup of old versions was never done, making
the backup becoming too fat very quickly.

Thus clearly define variable for rdiff backup status to have expected
condition.

Some dists like SLE_12 don't seem to have it.

../../stack/slapos.cfg is removed from component/*/buildout.cfg
because we normally don't specify it in component/
The OBS package will need to extend it.

Pin versions required for ipython==4.0.0 with ipykernel separated
from ipython eggs.
The split was in accordance to : https://blog.jupyter.org/2015/04/15/the-big-split/

/reviewed-by @kirr  (on nexedi/slapos!33)

ipython_notebook SR hooked with ERP5 kernel.
This kernel helps in interaction between erp5 and Jupyter frontend.
The patches have been cleaned up

Features:

- All the code execution is being done at erp5 side, Jupyter just acts as dumb client.
- Receives result as string and its mime_type and thanks to kernel, displays it accordingly.
- Interactions b/w erp5 and Jupyter frontend are based on HTTP requests.

Major changes:

- Addition of erp5 kernel
- Improvement in code according to guidelines(name, section name)
- Use jinja template as instance file and make it more dynamic
- Debugging added for ipython_notebook service.

Note: The certificate authentication changed has been reverted to the previous
one(done by creating wrapper around openssl command) for now.

/cc @Tyagov
/reviewed-by @kirr, @jerome  (on nexedi/slapos!33)

@jerome added --matplotlib=inline in 48eefab5 (ipython notebook) but it is
really neither needed:

   @jerome
   I remember adding this --matplotlib=inline line, but I am not sure it was
   ever needed. Using magic %matplotlib in notebook should be enough.

   @tiwariayush
   Yeah, for inline matplotlib in default python kernel, magics do there
   work(therefore neither pylab nor matplotlib alias are needed while starting the
   server), so I'd say leave this commit as it is and regarding version updation:
   a new patch making change wherever required.

nor supported:

   $ cat .slappart0_ipython_notebook.log
   [W 15:51:35.454 NotebookApp] Unrecognized alias: '--matplotlib=inline', it will probably have no effect.

Remove it.

P.S.

'--logfile' isn't available for ipython version 3.2.0 but we are not removing
it since we are planning to upgrade IPython to versions 4.x where it is supported.

Based on patch by @tiwariayush  (see nexedi/slapos!33)

Jupyter: Change section name to instance-jupyter so as not to raise conflict in case of multiple extends

/reviewed-by @kirr  (on !33)

Maintain consistency with the slapOS SR format.
This SR can be hooked with other SR(ex:wendelin) and its better
to follow one way of publishing result parameters

[ kirr: This essentially changes publication format to JSON:

    $ xslapos proxy show --params

    # before
    slappart0: ipython_notebook (type default)
        url = https://[2001:67c:1254:e:49::952d]:8888
        monitor_url = https://[2001:67c:1254:e:49::952d]:9685

    # after
    slappart0: ipython_notebook (type default)
        _ = {"url": "https://[2001:67c:1254:e:49::952d]:8888", "monitor_url": "https://[2001:67c:1254:e:49::952d]:9685"}

  I'm not convinced we really need this, nor that the .serialized version is
  the most oftenly used one:

    slapos$ git grep 'slapos.cookbook:publish$' |wc -l
    59
    slapos$ git grep 'slapos.cookbook:publish.serialised$' |wc -l
    13

  but we can have it and see how it goes, reverting if needed ]

/cc @jerome
/proposed-for-review-on nexedi/slapos!33

This helps in logging up the requests made by ipython_notebook service

[ kirr: To be clear - until log-level is set to DEBUG, IPython notebook does
  not log HTTP requests, and since logging of HTTP requests is considered normal
  for most of our services (Zope, Apache, etc), it makes sense to enable such
  functionality for notebook too.

  There is not much additional noise produced by --log-level=DEBUG - in
  practice ipython only prints what config files it uses on startup, so this
  should be ok to go. ]

/reviewed-by @kirr, @jerome  (on !33)

ERP5 kernel basic info/workflow:

1. User enters code on notebook cell and executes
2. Code is sent to kernel via websockets
3. Kernel sends request to ERP5
4. Code is executed by ERP5 and the result is returned back via request.
5. Result is received and rendered on the notebook frontend.
6. Other message formats such as error and status are also conveyed by the Kernel.

[ kirr: in IPython notebook speak kernel is something that allows IPython
  notebook server side to talk to execution backend. ERP5 kernel is a thing that
  allows ipython notbook to talk to ERP5 (with help on-ERP5-server special bt5
  installed which accepts and executes commands).

  The bt5 to handle notebook calls on ERP5 side - erp5-data-notebook - is
  proposed to be merged into erp5.git on nexedi/erp5!29 ]

/initially-reviewed-by @kirr, @Tyagov  (in a lot of places, last time on nexedi/slapos!33)

IPython Notebook: Explicitly add environment variable around wrapper and use ipython directory inside instance in env

[ kirr: By default IPython keeps configuration and other files location in
  ~/.ipython . What this patch does is organize explicit directory in instance
  tree to keep such files  ]

/reviewed-by @kirr  (on nexedi/slapos!33)

IPython Notebook: Add dynamic-template-base section for common jinja related file section and extend them with this section

[ kirr: to-Jinja2 conversion is required because jinja is more suitable to
  describing instances compared to buildout, because jinja2 has e.g. control
  structures ]

/reviewed-by @kirr  (on nexedi/slapos!33)

Commit cee110b2 (IPython Notebook: Fixing coding crimes for section
names) changed IPython notebook section name to use '-' as word
delimiter but forgot to update users, and this way wendelin build
started to fail:

    INFO While:
    INFO   Installing.
    INFO   Getting section ipython_notebook.
    INFO Error: The referenced section, 'ipython_notebook', was not defined.

Fix it. (And I've made sure with whole-tree git grep that there is no
more ipython notebook users except wendelin in whole slapos.git so far)

/reported-by @Tyagov
/cc @tiwariayush
/reviewed-by TrustMe

This is required so that buildout does not fallback to installing
non-dev egg if version of wendelin.core from -dev differs from what has
been pinned.

For example the following

    [buildout]
    extends =
      https://lab.node.vifib.com/nexedi/slapos/raw/1.0.12/software/wendelin/software-dev.cfg

    # pin wendelin.core-dev to latest assumed-good revision with ZBlk1 support
    [wendelin.core-repository]
    revision = c507d9009f59fec2041bac9c31c5b08a48d3897d

will install wendelin.core-0.4.egg from pypi instead of installing
c507d9009f59fec2041bac9c31c5b08a48d3897d from repository, because that
latter revision says it is already version 0.5 and 1.0.12 wendelin SR
pins wendelin.core to 0.4 .

So unpin wendelin.core from versions and let software-dev.cfg work
always.

/cc @klaus
/reviewed-by @Tyagov  (on nexedi/slapos!36)

IPython Notebook: Add download-file-base section which would be extended by sections requiring common usage

/reviewed-by @kirr  (on nexedi/slapos!33)

/reviewed-by @kirr  (on nexedi/slapos!33)