slapos:13169cabd4cf602022be5a8d6447ce11528cc6c1 commitshttps://lab.nexedi.com/nexedi/slapos/-/commits/13169cabd4cf602022be5a8d6447ce11528cc6c12016-01-18T00:39:27+03:00https://lab.nexedi.com/nexedi/slapos/-/commit/13169cabd4cf602022be5a8d6447ce11528cc6c1gitlab: Hook gitlab- and gitlab-shell- configuration files into the system2016-01-18T00:39:27+03:00Kirill Smelkovkirr@nexedi.com
- Download them on SR build and pass info to instance
- Instance prepares to process them as jinja2 templates
- Instance hooks the files into configuration location as appropriate
Every file so far is renamed *.erb -> *.in and a header added showing
that this file is autogenerated with links about what was the base
gitlab and/or omnibus version and omnibus reference revision this
template was last updated for.
So far all result configuration files are invalid - because ERB syntax
is there. We will convert the configuration files to proper jinja2
syntax and to using slapos parameters incrementally in the upcoming
patches.
NOTE (again): md5 sums are not yet fixed - we will fix them in the end
of gitlab patches series after applying all tweaking changes.
/cc <a href="/kazuhiko" data-user="5" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kazuhiko Shiozaki">@kazuhiko</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>https://lab.nexedi.com/nexedi/slapos/-/commit/6fd7b9870aeeb9ad8ed7e2f0a1e6db8d6ea80746gitlab: Import gitlab-ce & gitlab-shell configs from omnibus-gitlab2016-01-18T00:39:27+03:00Kirill Smelkovkirr@nexedi.com
Pristine import of template configuration files from omnibus GitLab
package. All files were imported as-is in their ERB form and filenames
from omnibus-gitlab 8.2.3+ce.0-0-g8eda093 from here:
<a href="https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default" rel="nofollow noreferrer noopener" target="_blank">https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default</a>
We will convert the templates to jinja2 and adjust them to slapos
version in the following patches.
Scheme for synchronizing with future upstream changes is envisioned as this:
- checkout latest commit which updated pristine erb files
- copy updated files from omnibus-gitlab, and commit the updates
- checkout slapos master
- merge commit that updated erb
That should reasonably work with not too-many conflicts and even those
should be not hard to resolve (with `git mergetool` e.g. in kdiff3)
/cc <a href="/kazuhiko" data-user="5" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kazuhiko Shiozaki">@kazuhiko</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>https://lab.nexedi.com/nexedi/slapos/-/commit/2ddc5b0ca03cdb4030f0645d53a2f857e74f7beegitlab: Organize per-instance gitlab work tree2016-01-18T00:39:27+03:00Kirill Smelkovkirr@nexedi.com
Organize per-instance place for gitlab configuration and work directory.
Unfortunately as GitLab is Ruby-on-Rails application, it is not possible
to keep its code in one place and have multiple separate configuration
sets in different places and start that code for a configuration set -
GitLab and Rails insist to get configuration from relative to source
code tree.
GitLab omnibus "solves" this by having only one configuration set and
having symlinks from code to that only configiration set. In slapos we
can potentially have several instances for one software and thus we
cannot do that.
With such limitations a proper solution would be to bind-mount software
code into instance filesystem namespace close to configuration - that
way the code will be only one and will find proper per-instance config.
Currently we do not have namespaces available on slapos unfortunately,
thus something else is needed.
The workaround I decided to do is this: to clone cloned gitlab
repository from software/ space to instance/ space and adjust it in
instance space. This has the following drawbacks:
- code is duplicated
- code becomes read-write, instead of being read-only
but imho it is the most practical thing to do. Another solution could be
to patch GitLab / Rails to remove "config lives in code" assumption, but
the number of places where this needs to be done is really many.
NOTE gems which gitlab uses and which were installed during software
compilation are not duplicated - they are reused via bundler - via
pointing BUNDLE_GEMFILE to original location in software.
NOTE2 For instance tasks and also for maintanace convenience we establish
<instance>/bin/gitlab-* programs, e.g. gitlab-rake, which e.g. for
gitlab-rake will run rake with correctly loaded gitlab environment -
like in gitlab-omnibus.
/cc <a href="/kazuhiko" data-user="5" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kazuhiko Shiozaki">@kazuhiko</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>, <a href="/jp" data-user="2" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jean-Paul Smets">@jp</a>https://lab.nexedi.com/nexedi/slapos/-/commit/0d286c5da56b364c185c0a13b58010c2fa444d15gitlab: Redis service2016-01-18T00:39:27+03:00Kirill Smelkovkirr@nexedi.com
Organize internal Redis service, like with PostgreSQL in the previous
patch, with the help of slapos.cookbook:redis.server recipe.
Like with postgresql, and as we planned, redis listens only on
internal-to-partition unix socket.
The recipe establishes both service and promise to check it is alive;
we only need to setup log rotation manually.
/cc <a href="/kazuhiko" data-user="5" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kazuhiko Shiozaki">@kazuhiko</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>https://lab.nexedi.com/nexedi/slapos/-/commit/470719fe487871fd281a47db722e784b41175071gitlab: PostgreSQL service2016-01-18T00:39:20+03:00Kirill Smelkovkirr@nexedi.com
Organize internal PostgreSQL database which will be used as DB for
Roby-on-Rails GitLab and listens only on unix socket (for security and
performance reasons - see earlier intro patch).
To do it we use slapos.cookbook:postgres recipe, with disabling
"listen-to-network" via passing empty sets to ipv4 and ipv6 recipe
arguments.
The promise to check whether DB is alive is just `psql -c '\q'` which
will error if failing to connect to DB, but exit silently if connected ok.
Explicit log rotation is not needed - as postgresql logs to
stdout/stderr - not to a file - logs are handled by slapos - put into
.slappartX_postgresql.log and automatically rotated there.
XXX omnibus-gitlab tunes postgresql with shared_buffers and other
parameters, most likely for performance reasons - see e.g.
<a href="https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-2-stable/files/gitlab-cookbooks/gitlab/templates/default/postgresql.conf.erb#L113" rel="nofollow noreferrer noopener" target="_blank">https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-2-stable/files/gitlab-cookbooks/gitlab/templates/default/postgresql.conf.erb#L113</a>
I decided not to fine-tune postgresql for now, and get on-field feedback
first, and then, if needed, we can tune.
/cc <a href="/kazuhiko" data-user="5" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kazuhiko Shiozaki">@kazuhiko</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>https://lab.nexedi.com/nexedi/slapos/-/commit/5e4a181aba40280290490accf65213b49111a2e5gitlab: Add helper for setting up promises2016-01-07T16:44:32+03:00Kirill Smelkovkirr@nexedi.com
A recipe could do
[promise-<service>]
<= promise-wrapper
command-line = ...
and the wrapper will be put automatiaclly into etc/promise/<service>.
( for this to happen !py! magic is used again, like we did for logrotate
and cron entries before )
/cc <a href="/kazuhiko" data-user="5" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kazuhiko Shiozaki">@kazuhiko</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>https://lab.nexedi.com/nexedi/slapos/-/commit/e7c5c05aa34daa0730a5267038156feb81ec0f88gitlab: Make a plan to base instance layout on gitlab-omnibus and to...2016-01-07T16:44:14+03:00Kirill Smelkovkirr@nexedi.comgitlab: Make a plan to base instance layout on gitlab-omnibus and to interconnect all internal services via unix sockets
Upcoming changes will follow two points:
- we try to base our gitlab setup on how it is done in
gitlab-omnibus[1] with the idea to ease tracking upstream changes to
instance setup.
- we will interconnect all internal services via unix sockets only.
The reason to do it is twofold:
1. easier security: currently files on different slapos partitions
are isolated from each other, but there is no "in-between-partitions"
networking isolation - thus (potentially evil) programs can
access internal services on other slapos partition.
permissions to access unix sockets, on the other hand, are
managed by filesystem-level permissions, and thus unix sockets in
one partition will be, by default, isolated from programs on
another partitions.
2. It is well known that UNIX sockets are faster than TCP over
loopback. For example for our std shuttles they have 2 times lower
latency and ~ 2-3 times more throughput compared to TCP over loopback
More details on 1 & 2 can be found e.g. here:
<a href="https://lab.nexedi.com/nexedi/slapos/merge_requests/27" data-original="https://lab.nexedi.com/nexedi/slapos/merge_requests/27" data-link="false" data-link-reference="true" data-project="15" data-merge-request="178" data-project-path="nexedi/slapos" data-iid="27" data-mr-title="slapos/recipe/redis: Add support for UNIX sockets" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!27</a>
<a href="https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/30" rel="nofollow noreferrer noopener" target="_blank">https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/30</a>
/cc <a href="/kazuhiko" data-user="5" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kazuhiko Shiozaki">@kazuhiko</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>
[1] <a href="https://gitlab.com/gitlab-org/omnibus-gitlab" rel="nofollow noreferrer noopener" target="_blank">https://gitlab.com/gitlab-org/omnibus-gitlab</a>
https://lab.nexedi.com/nexedi/slapos/-/commit/ab6d2f28ee56d4f84f5c9757db80e18622b737fagitlab: Add empty instance2016-01-06T22:17:28+03:00Kirill Smelkovkirr@nexedi.com
Add stub instance configuration which just establishes a way to have
several software types(*), pass all needed info from software to
instance, organizes base directory and establishes log rotation base for
upcoming services.
Log rotation is done with the help of cron periodicallly calling
logrotate. The rotation is done in "copytruncate" mode - i.e. log file
is not moved away and signal sent for service to reopen it, but instead
log content is just copied to outside and there is no need for a service
to reopen it's log file.
The reason it is done this way, is that there is a chance of not
handling such "reopen-log-file" callbacks correctly on a service side,
and so the net is full of crashing reports, e.g. like this:
<a href="http://serverfault.com/questions/627521/why-is-logrotate-causing-apache-to-seg-fault-each-time" rel="nofollow noreferrer noopener" target="_blank">http://serverfault.com/questions/627521/why-is-logrotate-causing-apache-to-seg-fault-each-time</a>
That's why we take a safer approach instead, even if "copytruncate" mode
is risking to loose several log entries(**) on rotation.
NOTE services will organize log rotation with just
[logrotate-entry-<service>]
<= logrotate-entry
log = path/to/log/files/*.log
For this to work some "!py!" magic (our way to serialize object into
executable python and process it in buildout recipes) is used to process
section names.
The approach trick is also used for cron, e.g. logrotate registers to
cron this way:
[cron-entry-logrotate]
<= cron-entry
time = daily
command = ${logrotate:wrapper}
NOTE2 instance md5 are not fixed yet - we'll fix them after applying all
patches in gitlab series.
(*) for now there is only 1 - "gitlab", but we'll need to have "-export"
and "-import" for resiliency in the future.
(**) ideally such things should be done with logfs - a filesystem
specializeing in logging - for client services it will look like as
they just continue to write to log file, and on log service side, the
rotation can happen, all transparent to client service.
/cc <a href="/kazuhiko" data-user="5" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kazuhiko Shiozaki">@kazuhiko</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>https://lab.nexedi.com/nexedi/slapos/-/commit/5e971c58a6f555222ab1a435db4350fe54194ed7Start of GitLab Software Release2016-01-06T22:15:56+03:00Kirill Smelkovkirr@nexedi.com
First step - build all needed software. We build:
- Git
- PostgreSQL 9.2
- Redis 2.8
- Nginx
- gitlab-shell
- gitlab-workhorse
- gitlab-ce 8.2 itself
and everything which is needed to build the above programs.
Git is needed because GitLab is a git-hosting service and uses git
underneath. PostgreSQL is used as DB by gitlab and Redis as a cache.
GitLab-shell is a small project to manage ssh access to the service
(we'll disable ssh though) and to perform all "change a repository"
operations.
GitLab-workhorse is a service which offloads long-running or slow
request from main GitLab service.
GitLab-ce is the main Ruby-on-Rails-based web application.
Ruby- and Go- based programs are built in a way similar to:
- <a href="/iv/slapos/-/commit/31a45a94c6fe19416eadb464e488678ee79d5a53" data-original="31a45a94" data-link="false" data-link-reference="false" data-project="220" data-commit="31a45a94c6fe19416eadb464e488678ee79d5a53" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="helloworld & helloweb: Ruby version" class="gfm gfm-commit has-tooltip">31a45a94</a> (helloworld & helloweb: Ruby version), and
- <a href="/iv/slapos/-/commit/24e824149065f3c8bd82e626cc91c2ed0832c4ea" data-original="24e82414" data-link="false" data-link-reference="false" data-project="220" data-commit="24e824149065f3c8bd82e626cc91c2ed0832c4ea" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="helloworld & helloweb: Go version" class="gfm gfm-commit has-tooltip">24e82414</a> (helloworld & helloweb: Go version)
Version of all components, except Git, were picked the same, as used by
gitlab omnibus v8.2 .
/cc <a href="/kazuhiko" data-user="5" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kazuhiko Shiozaki">@kazuhiko</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>https://lab.nexedi.com/nexedi/slapos/-/commit/c222a5d851dda1ff0c56f081d61309133219604dversion up: ca-certificates 201512142016-01-04T18:54:56+01:00Julien Muchembledjm@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/21d8b0f6c76bedeb7751998347e940a836237601version up : MariaDB 10.1.10.2015-12-28T11:00:14+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/d5df9d8d9c63a0903bce06ba9a84dc931835aca7mariadb: enable innodb lzma compression and disable some options explicitly.2015-12-28T11:00:14+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/758fbc2ea21bb0b85eafe5946b07101d0d393924harfbuzz: build without ICU.2015-12-28T11:00:14+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/94ebe8a894afbfbc2c7497d33459ea2b176f1cdepython2.7: disable building some modules explicitly.2015-12-28T11:00:14+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/bf06cb766de9f32be5babfa1f5a6bffd9474681fNew component for OBS builds: chrpath2015-12-26T23:09:10+01:00Julien Muchembledjm@nexedi.com
Some dists like SLE_12 don't seem to have it.https://lab.nexedi.com/nexedi/slapos/-/commit/284612accf5cab16dfceb7c2f9798197cd2ac430re6stnet: update and cleanup2015-12-23T11:13:48+01:00Julien Muchembledjm@nexedi.com
../../stack/slapos.cfg is removed from component/*/buildout.cfg
because we normally don't specify it in component/
The OBS package will need to extend it.https://lab.nexedi.com/nexedi/slapos/-/commit/d88878e6738f78d806957684e6705f43d532abacJupyter: Version up eggs and update erp5_kernel accordingly2015-12-21T23:57:19+03:00Ayush Tiwariayush.tiwari@nexedi.com
Pin versions required for ipython==4.0.0 with ipykernel separated
from ipython eggs.
The split was in accordance to : <a href="https://blog.jupyter.org/2015/04/15/the-big-split/" rel="nofollow noreferrer noopener" target="_blank">https://blog.jupyter.org/2015/04/15/the-big-split/</a>
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a> (on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)https://lab.nexedi.com/nexedi/slapos/-/commit/1d3ef32b4f383541ea6ac9c173059ce932b69b76version up : Python 2.7.11.2015-12-21T11:44:02+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/84363d5c48ae4cddb70ad45893bbf478c8b74274version up eggs.2015-12-21T11:44:01+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/147f23005ad877816b43d30dfd1be971a5afa95dhaproxy: explicitly enable USE_DL, that is required at least in Ubuntu 15.10.2015-12-21T11:44:01+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/8206dcddad2ab34a5a5a0d95414b6140da06915aerp5 kernel jupyter2015-12-19T00:21:40+03:00Ayush Tiwariayush.tiwari@nexedi.com
ipython_notebook SR hooked with ERP5 kernel.
This kernel helps in interaction between erp5 and Jupyter frontend.
The patches have been cleaned up
Features:
- All the code execution is being done at erp5 side, Jupyter just acts as dumb client.
- Receives result as string and its mime_type and thanks to kernel, displays it accordingly.
- Interactions b/w erp5 and Jupyter frontend are based on HTTP requests.
Major changes:
- Addition of erp5 kernel
- Improvement in code according to guidelines(name, section name)
- Use jinja template as instance file and make it more dynamic
- Debugging added for ipython_notebook service.
Note: The certificate authentication changed has been reverted to the previous
one(done by creating wrapper around openssl command) for now.
/cc <a href="/Tyagov" data-user="15" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Ivan Tyagov">@Tyagov</a>
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a> (on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)https://lab.nexedi.com/nexedi/slapos/-/commit/837c05c64e59066970683ce95cf5862822708090Jupyter: --matplotlib=inline is nether supported nor needed2015-12-19T00:17:57+03:00Kirill Smelkovkirr@nexedi.com
<a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a> added --matplotlib=inline in <a href="/zf.huang/slapos/-/commit/48eefab5b360ae6be0e708496736773bc7548f5c" data-original="48eefab5" data-link="false" data-link-reference="false" data-project="1457" data-commit="48eefab5b360ae6be0e708496736773bc7548f5c" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="ipython notebook" class="gfm gfm-commit has-tooltip">48eefab5</a> (ipython notebook) but it is
really neither needed:
<a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>
I remember adding this --matplotlib=inline line, but I am not sure it was
ever needed. Using magic %matplotlib in notebook should be enough.
<a href="/tiwariayush" data-user="28" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Ayush Tiwari">@tiwariayush</a>
Yeah, for inline matplotlib in default python kernel, magics do there
work(therefore neither pylab nor matplotlib alias are needed while starting the
server), so I'd say leave this commit as it is and regarding version updation:
a new patch making change wherever required.
nor supported:
$ cat .slappart0_ipython_notebook.log
[W 15:51:35.454 NotebookApp] Unrecognized alias: '--matplotlib=inline', it will probably have no effect.
Remove it.
P.S.
'--logfile' isn't available for ipython version 3.2.0 but we are not removing
it since we are planning to upgrade IPython to versions 4.x where it is supported.
Based on patch by <a href="/tiwariayush" data-user="28" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Ayush Tiwari">@tiwariayush</a> (see <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)https://lab.nexedi.com/nexedi/slapos/-/commit/bd3d8e4882bc62ef59e1c540a733a49391289330Jupyter: Change section name to instance-jupyter so as not to raise conflict...2015-12-19T00:17:57+03:00Ayush Tiwariayush.tiwari@nexedi.comJupyter: Change section name to instance-jupyter so as not to raise conflict in case of multiple extends
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a> (on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">!33</a>)
https://lab.nexedi.com/nexedi/slapos/-/commit/0fb4f687e0a2f156051c97f747e47369b326be4eJupyter: Publish the serialized result2015-12-19T00:17:57+03:00Ayush Tiwariayush.tiwari@nexedi.com
Maintain consistency with the slapOS SR format.
This SR can be hooked with other SR(ex:wendelin) and its better
to follow one way of publishing result parameters
[ kirr: This essentially changes publication format to JSON:
$ xslapos proxy show --params
# before
slappart0: ipython_notebook (type default)
url = <a href="https://%5B2001:67c:1254:e:49::952d%5D:8888" rel="nofollow noreferrer noopener" target="_blank">https://[2001:67c:1254:e:49::952d]:8888</a>
monitor_url = <a href="https://%5B2001:67c:1254:e:49::952d%5D:9685" rel="nofollow noreferrer noopener" target="_blank">https://[2001:67c:1254:e:49::952d]:9685</a>
# after
slappart0: ipython_notebook (type default)
_ = {"url": "https://[2001:67c:1254:e:49::952d]:8888", "monitor_url": "https://[2001:67c:1254:e:49::952d]:9685"}
I'm not convinced we really need this, nor that the .serialized version is
the most oftenly used one:
slapos$ git grep 'slapos.cookbook:publish$' |wc -l
59
slapos$ git grep 'slapos.cookbook:publish.serialised$' |wc -l
13
but we can have it and see how it goes, reverting if needed ]
/cc <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a>
/proposed-for-review-on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>https://lab.nexedi.com/nexedi/slapos/-/commit/3bf1cce94f40a7a7b7cbd5e092608fbccec2ce4fJupyter: Set log-level alias to 'DEBUG' for ipython notebook service2015-12-19T00:17:57+03:00Ayush Tiwariayush.tiwari@nexedi.com
This helps in logging up the requests made by ipython_notebook service
[ kirr: To be clear - until log-level is set to DEBUG, IPython notebook does
not log HTTP requests, and since logging of HTTP requests is considered normal
for most of our services (Zope, Apache, etc), it makes sense to enable such
functionality for notebook too.
There is not much additional noise produced by --log-level=DEBUG - in
practice ipython only prints what config files it uses on startup, so this
should be ok to go. ]
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a>, <a href="/jerome" data-user="9" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Jérome Perrin">@jerome</a> (on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)https://lab.nexedi.com/nexedi/slapos/-/commit/33954c6616bddf9652f8ba70b31b2363f03e20f0Jupyter: Hook ERP5 Kernel2015-12-19T00:17:57+03:00Ayush Tiwariayush.tiwari@nexedi.com
ERP5 kernel basic info/workflow:
1. User enters code on notebook cell and executes
2. Code is sent to kernel via websockets
3. Kernel sends request to ERP5
4. Code is executed by ERP5 and the result is returned back via request.
5. Result is received and rendered on the notebook frontend.
6. Other message formats such as error and status are also conveyed by the Kernel.
[ kirr: in IPython notebook speak kernel is something that allows IPython
notebook server side to talk to execution backend. ERP5 kernel is a thing that
allows ipython notbook to talk to ERP5 (with help on-ERP5-server special bt5
installed which accepts and executes commands).
The bt5 to handle notebook calls on ERP5 side - erp5-data-notebook - is
proposed to be merged into erp5.git on <a href="/nexedi/erp5/-/merge_requests/29" data-original="nexedi/erp5!29" data-link="false" data-link-reference="false" data-project="1" data-merge-request="204" data-project-path="nexedi/erp5" data-iid="29" data-mr-title="Business Template : erp5_data_notebook (To execute remote python code from inside of erp5)" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/erp5!29</a> ]
/initially-reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a>, <a href="/Tyagov" data-user="15" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Ivan Tyagov">@Tyagov</a> (in a lot of places, last time on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)https://lab.nexedi.com/nexedi/slapos/-/commit/77ffa61eeb839fe88e3b32e88095d4c243a9b28eIPython Notebook: Explicitly add environment variable around wrapper and use...2015-12-19T00:17:57+03:00Ayush Tiwariayush.tiwari@nexedi.comIPython Notebook: Explicitly add environment variable around wrapper and use ipython directory inside instance in env
[ kirr: By default IPython keeps configuration and other files location in
~/.ipython . What this patch does is organize explicit directory in instance
tree to keep such files ]
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a> (on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)
https://lab.nexedi.com/nexedi/slapos/-/commit/2ff9b78410be8092725aeb6a34f4517fba5117e6IPython Notebook: Add dynamic-template-base section for common jinja related...2015-12-19T00:17:57+03:00Ayush Tiwariayush.tiwari@nexedi.comIPython Notebook: Add dynamic-template-base section for common jinja related file section and extend them with this section
https://lab.nexedi.com/nexedi/slapos/-/commit/fbfaaef84a9743fd79de2a8518b670a13023203fIPython Notebook: Convert instance to jinja2 template2015-12-19T00:17:57+03:00Ayush Tiwariayush.tiwari@nexedi.com
[ kirr: to-Jinja2 conversion is required because jinja is more suitable to
describing instances compared to buildout, because jinja2 has e.g. control
structures ]
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a> (on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)https://lab.nexedi.com/nexedi/slapos/-/commit/ae1aa162504d9277348c5e4f198d97cb2a780d70wendelin: Adjust to recent ipython-notebook section renaming2015-12-18T10:54:43+03:00Kirill Smelkovkirr@nexedi.com
Commit <a href="/iv/slapos/-/commit/cee110b20c6f324c065bd1b6895781afc8a7e81e" data-original="cee110b2" data-link="false" data-link-reference="false" data-project="220" data-commit="cee110b20c6f324c065bd1b6895781afc8a7e81e" data-reference-type="commit" data-container="body" data-placement="top" data-html="true" title="IPython Notebook: Fixing coding crimes for section names" class="gfm gfm-commit has-tooltip">cee110b2</a> (IPython Notebook: Fixing coding crimes for section
names) changed IPython notebook section name to use '-' as word
delimiter but forgot to update users, and this way wendelin build
started to fail:
INFO While:
INFO Installing.
INFO Getting section ipython_notebook.
INFO Error: The referenced section, 'ipython_notebook', was not defined.
Fix it. (And I've made sure with whole-tree git grep that there is no
more ipython notebook users except wendelin in whole slapos.git so far)
/reported-by <a href="/Tyagov" data-user="15" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Ivan Tyagov">@Tyagov</a>
/cc <a href="/tiwariayush" data-user="28" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Ayush Tiwari">@tiwariayush</a>
/reviewed-by TrustMehttps://lab.nexedi.com/nexedi/slapos/-/commit/02ae06e992a5a4537ffef9986b409c37fdaac493ncurse: add a patch for gcc 5.2015-12-17T10:09:48+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/202a8bec7a56f3682aa3462e3866bdbdf98326b3version up : apache httpd 2.4.18.2015-12-17T09:49:52+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/85d92907656f923bd5cc966c808d1d180071c954version up eggs.2015-12-17T09:41:37+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/abfd20e78c04c1460b1fcf69608ed5a5a9e74f88mariadb: stop disabling derived_merge globally.2015-12-17T09:41:37+01:00Kazuhiko Shiozakikazuhiko@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/3d72656bc43ab0fad0e9f968ff55646d8ce5765bMerge branch 'kvm-cluster'2015-12-16T16:08:52+01:00Alain Takoudjoualain.takoudjou@nexedi.comhttps://lab.nexedi.com/nexedi/slapos/-/commit/69f30c1da5b01801bb64323c7a4506f435c3dd97wendelin: Unpin wendelin.core from versions in -dev variant2015-12-16T11:49:25+03:00Kirill Smelkovkirr@nexedi.com
This is required so that buildout does not fallback to installing
non-dev egg if version of wendelin.core from -dev differs from what has
been pinned.
For example the following
[buildout]
extends =
<a href="https://lab.node.vifib.com/nexedi/slapos/raw/1.0.12/software/wendelin/software-dev.cfg" rel="nofollow noreferrer noopener" target="_blank">https://lab.node.vifib.com/nexedi/slapos/raw/1.0.12/software/wendelin/software-dev.cfg</a>
# pin wendelin.core-dev to latest assumed-good revision with ZBlk1 support
[wendelin.core-repository]
revision = c507d9009f59fec2041bac9c31c5b08a48d3897d
will install wendelin.core-0.4.egg from pypi instead of installing
c507d9009f59fec2041bac9c31c5b08a48d3897d from repository, because that
latter revision says it is already version 0.5 and 1.0.12 wendelin SR
pins wendelin.core to 0.4 .
So unpin wendelin.core from versions and let software-dev.cfg work
always.
/cc <a href="/klaus" data-user="7" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Klaus Wölfel">@klaus</a>
/reviewed-by <a href="/Tyagov" data-user="15" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Ivan Tyagov">@Tyagov</a> (on <a href="/nexedi/slapos/-/merge_requests/36" data-original="nexedi/slapos!36" data-link="false" data-link-reference="false" data-project="15" data-merge-request="226" data-project-path="nexedi/slapos" data-iid="36" data-mr-title="wendelin: Unpin wendelin.core from versions in -dev variant" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">!36</a>)https://lab.nexedi.com/nexedi/slapos/-/commit/c886abe6bab2c2db198c3a7dddc9989ca172d3f9IPython Notebook: Add download-file-base section which would be extended by...2015-12-15T20:42:36+03:00Ayush Tiwariayush.tiwari@nexedi.comIPython Notebook: Add download-file-base section which would be extended by sections requiring common usage
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a> (on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)
https://lab.nexedi.com/nexedi/slapos/-/commit/ff3d16381c8971cc090f2eea5b00383d9e7f4046IPython Notebook: Fix naming of files according to extensions, follow guidelines2015-12-15T20:41:43+03:00Ayush Tiwariayush.tiwari@nexedi.com
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a> (on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)https://lab.nexedi.com/nexedi/slapos/-/commit/cee110b20c6f324c065bd1b6895781afc8a7e81eIPython Notebook: Fixing coding crimes for section names2015-12-15T20:39:44+03:00Ayush Tiwariayush.tiwari@nexedi.com
Use spinal-case('-' separated) instead of snake_case('_' separated) in section names.
/reviewed-by <a href="/kirr" data-user="14" data-reference-type="user" data-container="body" data-placement="top" data-html="true" class="gfm gfm-project_member" title="Kirill Smelkov">@kirr</a> (on <a href="/nexedi/slapos/-/merge_requests/33" data-original="nexedi/slapos!33" data-link="false" data-link-reference="false" data-project="15" data-merge-request="206" data-project-path="nexedi/slapos" data-iid="33" data-mr-title="erp5 kernel jupyter" data-reference-type="merge_request" data-container="body" data-placement="top" data-html="true" title="" class="gfm gfm-merge_request">nexedi/slapos!33</a>)https://lab.nexedi.com/nexedi/slapos/-/commit/b1dcbf774baadd95023247c2baea3c55ec0fa065kvm: Add restrict mode for nat interface2015-12-14T17:00:48+01:00Alain Takoudjoualain.takoudjou@nexedi.com
If restrict mode is set to true, nat interface (eth0) will be isolated, no network access, only host and guest forward rules will work throught that interface.
This option is true by default for kvm-cluster, and false for single and resilient kvm.