- 17 Jan, 2016 21 commits
-
-
Kirill Smelkov authored
Like with [promise-wrapper] a recipe could do [promise-<service>] <= promise-byurl url = ... and a script to check such ur will be generated and automatically put into etc/promise/<service>. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Automatically configure unicorn to spawn as much worker processes as there are CPUs on the system by default. GitLab omnibus pre-hardcodes this value default to 2 (which we copied) and then also tweaks it this way in active code https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/attributes/default.rb#L230 which we also do here. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Explicitly point gitlab-shell to location where we keep secrets. We already pointeg gitlab to that place and now we do that for gitlab-shell so those 2 peieces can connect to each other ok. Regarding the setting itself - there is no such block in omnibus-gitlab, but it is present in gitlab-shell configuration example: https://gitlab.com/gitlab-org/gitlab-shell/blob/82b3a4e8/config.yml.example#L35 /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Convert gitlab-shell configuration file to slapos: - convert to jinja2, - connect gitlab-shell to unicorn & redis unix sockets NOTE - http_settings are left to be default (empty) ones - as that works ok. - `auth_file` is still configured to point to wont-be-used sshkeys file, as without it gitlab-shell check will fail. - support for audit_usernames and git_annex is disabled and remains not configurable. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Unicorn is a forking server with the idea that master process preloads heavy Ruby-on-Rails application, and then to handle new request a worker process is forked with application already loaded in its memory (and modification being tracked by OS via copy-on-write). From this point of view the only reasonable value for preload_app is always "true" and omnibus-gitlab does this: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/definitions/unicorn_service.rb#L65 Then unicorn documentation shows what code has to be there in pre-/post- forking event: http://bogomips.org/unicorn.git/tree/examples/unicorn.conf.rb?id=3312aca8#n57 GitLab uses only part of it that "allows a new master process to incrementally phase out the old master process with SIGTTOU to avoid a thundering herd": https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/definitions/unicorn_service.rb#L69 http://bogomips.org/unicorn.git/tree/examples/unicorn.conf.rb?id=3312aca8#n75 but strangely does not use code parts that are "highly recommended" or "require" for "Rails + "preload_app true"" case. For the reference I've added such codes, but kept them being commented out. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Convert unicorn parameters to slapos and configure it to listen on unix socket only. ( Omnibus configures unicorn to listen on unix socket and loopback TCP, mainly because gitlab-shell could not connect to unicorn via unix socket until recently: https://gitlab.com/gitlab-org/gitlab-shell/commit/184385ac But as it can now, there is no point to keep on TCP port open ) To be able to do such configuration we add stub to unicorn service section (to create needed directories where to keep the socket). There will be follow-up patch which configures unicorn pre/post-forking actions, which is not trivial and thus better be done on its own. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Convert the rest of this configuration file to slapos. It is straightforward conversion of parameters except: - access-via-ssh is disabled (gitlab slapos version does not support ssh access and supports HTTP(S) only by design on purpose) - we do not support restricting possible projects visibility via instance parameter (very low chance this will be needed in practice) - default issue-closing pattern is just ok for now and not configurable - support for builds, build artifacts & CI is disabled (we do not support CI (yet ?)) - some internal defaults are just ok (e.g. where to organize directory for keeping repositories archives for downloads) - reply-by-email is not supported (yet ?) - we do not support LFS (yet ?) - just plain git hosting is ok for now. - Gravatar defaults are ok for now and not configurable. - Support for LDAP is disabled - Support for Kerberos is disabled - Support for OmniAuth is disabled - Satellites path is just /dev/null as we start from version where satellites are already non-existent. - Uploading backups to somewhere via GitLab's builtin mechanism is not supported - we'll use SlapOS native backup and resiliency for this. - Support for Google analytics is disabled. - Support for Piwik is disabled. - we are ok (for now) with default rack-attack git settings /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
This user will need to be specified several times in configuration files, as by default gitlab uses 'git' user and does "sudo" to it if it is not current. We will use {{ backend_info.user }} in the upcoming patches. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
GitLab has a notion of "external URL" - the canonical "frontend" URL the server is reachable through: this URL is used as prefix to show e.g. git-clone URL for repositories, etc, even if a server can be reachable via several frontends. Add external_url handling to slapos instance. NOTE whether to use https or not is also defined by external_url, in particular by external_url scheme. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Convert to slapos SMTP settings for gitlab: - convert to jinja2 - remove support for gitlab CI (we do not support it (yet ?)) - add handling of `smtp_enable` parameter directly to that file ( omnibus handles this parameter externally and just removes smtp_settings.rb if it is true ) NOTE smtp_settings.rb contains SMTP password, so it is mode is set to 0600. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Just another 2 simple parameters (attack detection tunables) conversion to jinja2/slapos. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Just convert 2 parameters used in that file to jinja syntax and add those parameters (unicorn OOM killer tunables) to gitlab-parameters.cfg /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
A simple change just to point resque to redis unix socket. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
We tweak database.yml to point to our postgresql unix socket; set adapter to hardcoded postgresql, encoding to unicode and omit collation (which according to omnibus-gitlab is used for mysql only). The only instance parameter imported from omnibus is `db_pool` - how many connection to a DB to keep open in a RoR thread/process. XXX we use db's superuser as a user to connect. Is it ok to do even if the whole DB is used only for gitlab? (I think it is ok for the first iteration, but we'll probably need to refine this later) /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Introduce a library of Jinja2 macros that will be handy to use in templates. For now we add only 2 macros: cfg(name) - to get instance configuration parameter `name`, and cfg_bool - to get truth value of ----//---- The reason we introduce cfg() is that we will need to use a lot of parameters in many places and it is much more handy to write, e.g. cfg('email_enabled') compared to instance_parameter:configuration.email_enabled /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
We will be using a several dozens of parameters to control gitlab instance. It makes sense not to deviate in such parameters namings and defaults from omnibus version. Thus for such parameters - for clarity - we organize a separate file where we will be keeping them - gitlab-parameters.cfg. In this patch series all used parameters will be "imported" from omnibus-gitlab 8.2.3+ce.0-0-g8eda093. NOTE it is maybe better to try to autogenerate that file from upstream omnibus parameters definitions. If time will tell it becomes hard to maintain our copy - we'll consider going that way. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
- Download them on SR build and pass info to instance - Instance prepares to process them as jinja2 templates - Instance hooks the files into configuration location as appropriate Every file so far is renamed *.erb -> *.in and a header added showing that this file is autogenerated with links about what was the base gitlab and/or omnibus version and omnibus reference revision this template was last updated for. So far all result configuration files are invalid - because ERB syntax is there. We will convert the configuration files to proper jinja2 syntax and to using slapos parameters incrementally in the upcoming patches. NOTE (again): md5 sums are not yet fixed - we will fix them in the end of gitlab patches series after applying all tweaking changes. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Pristine import of template configuration files from omnibus GitLab package. All files were imported as-is in their ERB form and filenames from omnibus-gitlab 8.2.3+ce.0-0-g8eda093 from here: https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default We will convert the templates to jinja2 and adjust them to slapos version in the following patches. Scheme for synchronizing with future upstream changes is envisioned as this: - checkout latest commit which updated pristine erb files - copy updated files from omnibus-gitlab, and commit the updates - checkout slapos master - merge commit that updated erb That should reasonably work with not too-many conflicts and even those should be not hard to resolve (with `git mergetool` e.g. in kdiff3) /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Organize per-instance place for gitlab configuration and work directory. Unfortunately as GitLab is Ruby-on-Rails application, it is not possible to keep its code in one place and have multiple separate configuration sets in different places and start that code for a configuration set - GitLab and Rails insist to get configuration from relative to source code tree. GitLab omnibus "solves" this by having only one configuration set and having symlinks from code to that only configiration set. In slapos we can potentially have several instances for one software and thus we cannot do that. With such limitations a proper solution would be to bind-mount software code into instance filesystem namespace close to configuration - that way the code will be only one and will find proper per-instance config. Currently we do not have namespaces available on slapos unfortunately, thus something else is needed. The workaround I decided to do is this: to clone cloned gitlab repository from software/ space to instance/ space and adjust it in instance space. This has the following drawbacks: - code is duplicated - code becomes read-write, instead of being read-only but imho it is the most practical thing to do. Another solution could be to patch GitLab / Rails to remove "config lives in code" assumption, but the number of places where this needs to be done is really many. NOTE gems which gitlab uses and which were installed during software compilation are not duplicated - they are reused via bundler - via pointing BUNDLE_GEMFILE to original location in software. NOTE2 For instance tasks and also for maintanace convenience we establish <instance>/bin/gitlab-* programs, e.g. gitlab-rake, which e.g. for gitlab-rake will run rake with correctly loaded gitlab environment - like in gitlab-omnibus. /cc @kazuhiko, @jerome, @jp
-
Kirill Smelkov authored
Organize internal Redis service, like with PostgreSQL in the previous patch, with the help of slapos.cookbook:redis.server recipe. Like with postgresql, and as we planned, redis listens only on internal-to-partition unix socket. The recipe establishes both service and promise to check it is alive; we only need to setup log rotation manually. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Organize internal PostgreSQL database which will be used as DB for Roby-on-Rails GitLab and listens only on unix socket (for security and performance reasons - see earlier intro patch). To do it we use slapos.cookbook:postgres recipe, with disabling "listen-to-network" via passing empty sets to ipv4 and ipv6 recipe arguments. The promise to check whether DB is alive is just `psql -c '\q'` which will error if failing to connect to DB, but exit silently if connected ok. Explicit log rotation is not needed - as postgresql logs to stdout/stderr - not to a file - logs are handled by slapos - put into .slappartX_postgresql.log and automatically rotated there. XXX omnibus-gitlab tunes postgresql with shared_buffers and other parameters, most likely for performance reasons - see e.g. https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-2-stable/files/gitlab-cookbooks/gitlab/templates/default/postgresql.conf.erb#L113 I decided not to fine-tune postgresql for now, and get on-field feedback first, and then, if needed, we can tune. /cc @kazuhiko, @jerome
-
- 07 Jan, 2016 2 commits
-
-
Kirill Smelkov authored
A recipe could do [promise-<service>] <= promise-wrapper command-line = ... and the wrapper will be put automatiaclly into etc/promise/<service>. ( for this to happen !py! magic is used again, like we did for logrotate and cron entries before ) /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
gitlab: Make a plan to base instance layout on gitlab-omnibus and to interconnect all internal services via unix sockets Upcoming changes will follow two points: - we try to base our gitlab setup on how it is done in gitlab-omnibus[1] with the idea to ease tracking upstream changes to instance setup. - we will interconnect all internal services via unix sockets only. The reason to do it is twofold: 1. easier security: currently files on different slapos partitions are isolated from each other, but there is no "in-between-partitions" networking isolation - thus (potentially evil) programs can access internal services on other slapos partition. permissions to access unix sockets, on the other hand, are managed by filesystem-level permissions, and thus unix sockets in one partition will be, by default, isolated from programs on another partitions. 2. It is well known that UNIX sockets are faster than TCP over loopback. For example for our std shuttles they have 2 times lower latency and ~ 2-3 times more throughput compared to TCP over loopback More details on 1 & 2 can be found e.g. here: !27 https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/30 /cc @kazuhiko, @jerome [1] https://gitlab.com/gitlab-org/omnibus-gitlab
-
- 06 Jan, 2016 2 commits
-
-
Kirill Smelkov authored
Add stub instance configuration which just establishes a way to have several software types(*), pass all needed info from software to instance, organizes base directory and establishes log rotation base for upcoming services. Log rotation is done with the help of cron periodicallly calling logrotate. The rotation is done in "copytruncate" mode - i.e. log file is not moved away and signal sent for service to reopen it, but instead log content is just copied to outside and there is no need for a service to reopen it's log file. The reason it is done this way, is that there is a chance of not handling such "reopen-log-file" callbacks correctly on a service side, and so the net is full of crashing reports, e.g. like this: http://serverfault.com/questions/627521/why-is-logrotate-causing-apache-to-seg-fault-each-time That's why we take a safer approach instead, even if "copytruncate" mode is risking to loose several log entries(**) on rotation. NOTE services will organize log rotation with just [logrotate-entry-<service>] <= logrotate-entry log = path/to/log/files/*.log For this to work some "!py!" magic (our way to serialize object into executable python and process it in buildout recipes) is used to process section names. The approach trick is also used for cron, e.g. logrotate registers to cron this way: [cron-entry-logrotate] <= cron-entry time = daily command = ${logrotate:wrapper} NOTE2 instance md5 are not fixed yet - we'll fix them after applying all patches in gitlab series. (*) for now there is only 1 - "gitlab", but we'll need to have "-export" and "-import" for resiliency in the future. (**) ideally such things should be done with logfs - a filesystem specializeing in logging - for client services it will look like as they just continue to write to log file, and on log service side, the rotation can happen, all transparent to client service. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
First step - build all needed software. We build: - Git - PostgreSQL 9.2 - Redis 2.8 - Nginx - gitlab-shell - gitlab-workhorse - gitlab-ce 8.2 itself and everything which is needed to build the above programs. Git is needed because GitLab is a git-hosting service and uses git underneath. PostgreSQL is used as DB by gitlab and Redis as a cache. GitLab-shell is a small project to manage ssh access to the service (we'll disable ssh though) and to perform all "change a repository" operations. GitLab-workhorse is a service which offloads long-running or slow request from main GitLab service. GitLab-ce is the main Ruby-on-Rails-based web application. Ruby- and Go- based programs are built in a way similar to: - 31a45a94 (helloworld & helloweb: Ruby version), and - 24e82414 (helloworld & helloweb: Go version) Version of all components, except Git, were picked the same, as used by gitlab omnibus v8.2 . /cc @kazuhiko, @jerome
-
- 04 Jan, 2016 1 commit
-
-
Julien Muchembled authored
-
- 28 Dec, 2015 4 commits
-
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
- 26 Dec, 2015 1 commit
-
-
Julien Muchembled authored
Some dists like SLE_12 don't seem to have it.
-
- 23 Dec, 2015 1 commit
-
-
Julien Muchembled authored
../../stack/slapos.cfg is removed from component/*/buildout.cfg because we normally don't specify it in component/ The OBS package will need to extend it.
-
- 21 Dec, 2015 4 commits
-
-
Ayush Tiwari authored
Pin versions required for ipython==4.0.0 with ipykernel separated from ipython eggs. The split was in accordance to : https://blog.jupyter.org/2015/04/15/the-big-split/ /reviewed-by @kirr (on nexedi/slapos!33)
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
- 18 Dec, 2015 4 commits
-
-
Ayush Tiwari authored
ipython_notebook SR hooked with ERP5 kernel. This kernel helps in interaction between erp5 and Jupyter frontend. The patches have been cleaned up Features: - All the code execution is being done at erp5 side, Jupyter just acts as dumb client. - Receives result as string and its mime_type and thanks to kernel, displays it accordingly. - Interactions b/w erp5 and Jupyter frontend are based on HTTP requests. Major changes: - Addition of erp5 kernel - Improvement in code according to guidelines(name, section name) - Use jinja template as instance file and make it more dynamic - Debugging added for ipython_notebook service. Note: The certificate authentication changed has been reverted to the previous one(done by creating wrapper around openssl command) for now. /cc @Tyagov /reviewed-by @kirr, @jerome (on !33)
-
Kirill Smelkov authored
@jerome added --matplotlib=inline in 48eefab5 (ipython notebook) but it is really neither needed: @jerome I remember adding this --matplotlib=inline line, but I am not sure it was ever needed. Using magic %matplotlib in notebook should be enough. @tiwariayush Yeah, for inline matplotlib in default python kernel, magics do there work(therefore neither pylab nor matplotlib alias are needed while starting the server), so I'd say leave this commit as it is and regarding version updation: a new patch making change wherever required. nor supported: $ cat .slappart0_ipython_notebook.log [W 15:51:35.454 NotebookApp] Unrecognized alias: '--matplotlib=inline', it will probably have no effect. Remove it. P.S. '--logfile' isn't available for ipython version 3.2.0 but we are not removing it since we are planning to upgrade IPython to versions 4.x where it is supported. Based on patch by @tiwariayush (see nexedi/slapos!33)
-
Ayush Tiwari authored
Jupyter: Change section name to instance-jupyter so as not to raise conflict in case of multiple extends /reviewed-by @kirr (on nexedi/slapos!33)
-
Ayush Tiwari authored
Maintain consistency with the slapOS SR format. This SR can be hooked with other SR(ex:wendelin) and its better to follow one way of publishing result parameters [ kirr: This essentially changes publication format to JSON: $ xslapos proxy show --params # before slappart0: ipython_notebook (type default) url = https://[2001:67c:1254:e:49::952d]:8888 monitor_url = https://[2001:67c:1254:e:49::952d]:9685 # after slappart0: ipython_notebook (type default) _ = {"url": "https://[2001:67c:1254:e:49::952d]:8888", "monitor_url": "https://[2001:67c:1254:e:49::952d]:9685"} I'm not convinced we really need this, nor that the .serialized version is the most oftenly used one: slapos$ git grep 'slapos.cookbook:publish$' |wc -l 59 slapos$ git grep 'slapos.cookbook:publish.serialised$' |wc -l 13 but we can have it and see how it goes, reverting if needed ] /cc @jerome /proposed-for-review-on nexedi/slapos!33
-