- 01 Feb, 2016 3 commits
-
-
Douglas authored
Query strings used on GET requests have size limitations on servers and this causes big code cells to not be executed at all, returning only an Internal Server Error with no further explanation. /reviewed-by @kirr, @Tyagov (on nexedi/slapos!43)
-
Kirill Smelkov authored
Pandas and scikit-learn are handy to have for data processing which we are going to do more and more in ERP5 context. Matplotlib is very handy to have when one works with Jupyter, but also can be used by just backend code to generate graphs. From this point of view it makes sense to have this eggs always available out of the box. To do so move them from Wendelin to ERP5. /suggested-by @tiwariayush /reviewed-by @Tyagov (on !43)
-
Kirill Smelkov authored
This patch teaches ERP5 software release to automatically instantiate Jupyter notebook web UI and tune it to connect to ERP5 by default. When Jupyter is enabled, it also installs on-server erp5_data_notebook bt5 (nexedi/erp5!29) which handles code execution requested for Jupyter. For ERP5 - for security and backward compatibility reasons - Jupyter instantiation and erp5_data_notebook bt5 install happen only if jupyter is explicitly enabled in instance parameters. The default is not to have Jupyter out of the box. On the other hand for Wendelin SR, which inherits from ERP5 SR, the default is to have Jupyter out of the box, because Wendelin SR is fresh enough without lots of backward compatibility needs, and Jupyter is usually very handy for people who use Wendelin. ~~~~ For integration, we reuse already established in ERP5 infrastructure, to request various slave instances, and request Jupyter in a way so it automatically tunes and connects to balancer of one of Zope family. Jupyter code itself is compiled by reusing software/ipython_notebook/software.cfg, and Jupyter instance code is reused by hooking software/ipython_notebook/instance.cfg.in into ERP5 SR properly (the idea to override instance-jupyter not to render into default template.cfg is taken from previous work by @tiwariayush). ~~~~ I tested this patch inside webrunner with create-erp5-site software type and various configurations (whether to have or not have jupyter, to which zope family to connect it, etc). I have not tested frontend instantiation fully - because tests were done only in webrunner, but I've tried to make sure generated buildout code is valid for cases with frontend. NOTE the code in this patch depends erp5_data_notebook bt5 (nexedi/erp5!29) which just got merged to erp5.git recently (see nexedi/erp5@f662b5a2) NOTE even when erp5_data_notebook bt5 is installed, on a freshly installed ERP5, it is required to "check site consistency" first, so that initial bt5(s) are actually installed and erp5 is ready to function. /cc @vpelletier, @Tyagov, @klaus, @Camata, @tiwariayush, @Kreisel, @jerome, @nexedi /proposed-for-review-on nexedi/slapos!43
-
- 27 Jan, 2016 1 commit
-
-
Kirill Smelkov authored
The reason is: we are going to integrate Jupyter into ERP5 SR, and in ERP5 SR code, there is already established infrastructure, which requests slave instances. That infrastructure passes parameters in serialized (= json) form, that's why we need to switch. On the other hand, slapos.cookbook:slapconfiguration.serialised handles non-json instance parameters as well: when there parameters format is not {'_' -> <json>} it decodes them as slapos.cookbook:slapconfiguration would do: https://lab.nexedi.com/nexedi/slapos/blob/69229988/slapos/recipe/slapconfiguration.py#L232 https://lab.nexedi.com/nexedi/slapos/blob/69229988/slapos/recipe/librecipe/__init__.py#L51 So from this point of view, though we don't really need it here (Jupyter SR is relatively new), we are staying backward compatible. /cc @tiwariayush
-
- 26 Jan, 2016 1 commit
-
-
Kirill Smelkov authored
Both numpy and ipython are included in ERP5 SR which wendelin inherits from, and are pinned there. Here are e.g. latest pin-ups for numpy and ipython in erp5: e3144a8a (version up eggs.) 135570c9 (version up eggs.) Furthermore: this is not only a cleanup. As e3144a8a shows current version of numpy in erp5 is 1.10.4 and in wendelin we still have 1.9.2 which is unintentional downgrade compared to erp5. Don't do that. /cc @kazuhiko /reviewed-by @Tyagov (on nexedi/slapos!41)
-
- 25 Jan, 2016 2 commits
-
-
Julien Muchembled authored
-
-
- 24 Jan, 2016 1 commit
-
-
Kirill Smelkov authored
@rafael added this in 971d0bb7 (erp5: Make possible extent the list of initial business templates to install), but we dropped that change while merging erp5-cluster to master - see: 6bbb61a8 "Merge branch 'master' into erp5-cluster", and e84d5e83 "Merge branch 'erp5-cluster'" 6bbb61a8 claimed that it Dropped commit 971d0bb7 ("erp5: Make possible extent the list of initial business templates to install"). but it actually dropped changes only under stack/erp5/ , not software/wendelin/ Fix it. /cc @rafael, @jm, @Tyagov /reviewed-by TrustMe
-
- 22 Jan, 2016 1 commit
-
-
Julien Muchembled authored
-
- 21 Jan, 2016 1 commit
-
-
Jérome Perrin authored
-
- 20 Jan, 2016 3 commits
-
-
Julien Muchembled authored
In slapos.package.git/obs, we need to build binaries for specific paths, without changing where buildout actually install them.
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
- 17 Jan, 2016 27 commits
-
-
Kirill Smelkov authored
Hello up there, Here comes SlapOS port of GitLab. We start from GitLab 8.2.X as that is what we currently run on KVM on lab.nexedi.com, so that our data can be straightforwardly migrated. The SR compiles all needed software and organized all (sub-)services in one partition and interconnects them with unix sockets for security and speed reasons (see patch "gitlab: Make a plan to base instance layout on gitlab-omnibus and to interconnect all internal services"). Services configuration files are originally taken from omnibus-gitlab "distribution" and incrementally ported to slapos variant. This way we establish a (imho) good path on how to track upstream changes and minimize our delta & effort supporting it. GitLab itself is patched (above patches that were already applied by upstream): - to support HTTP(S) only - to show site's ICP number - to speedup raw blob downloading ~ 17x times ( see patch "gitlab: Optimize raw blob downloading" for details and https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17 ) Overall it should work and we should finally be able to migrate slapos.git (because of raw blob downloading is not slow now) to GitLab and all other Nexedi git repositories. Thanks, Kirill P.S. Somewhat outdated, but this picture on GitLab architecture might help to understand how parts are glued together: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/architecture.md P.P.S. Native resiliency is not implemented yet, but we should be able to use gitlab inside resilient webrunner already. /proposed-for-review-on nexedi/slapos!39 /partly-reviewed-by @kazuhiko, @jerome, @Yanni, @jp /cc @rafael, @jm
-
Kirill Smelkov authored
We've reached a state where first gitlab SR version should work. So as promised let's freeze the md5 checksums. All later patches should update corresponding md5 info when they change a file. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
In slapos we do a lot of automated software rebuild constantly, and thus there is constant flow of requests to get raw blobs from git service, e.g. like this https://lab.nexedi.com/nexedi/slapos/raw/master/software/wendelin/software.cfg A lot of requests comes to slapos.git repository and currently gitlab, out of the box, cannot keep up with that load. I've prepared patches to offload raw blobs download requests handling from unicorn (ruby) to gitlab-workhorse (go), and that resulted in ~ 17x speedup - e.g. previously our std shuttle can handle ~ 70 raw-blob requests/s and with my changes it is now ~ 1200 requests/s. The patches were sent upstream https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17 and we discussed with GitLab people and made a plan how to proceed incrementally. It will probably take some time for gitlab team to fully accept the approach though. For now we can use our gitlab-workhorse fork. The patches itself are: kirr/gitlab-workhorse@1b274d0d kirr/gitlab-workhorse@2beb8c95 /cc @kazuhiko, @jerome, @jm
-
Kirill Smelkov authored
GitLab Nexedi Edition is currently upstream 8.2.X + the following patches: - HTTP(S) is made to be default clone protocol kirr/gitlab-ce@5c1f2fb3 and SSH info is completely removed from UI kirr/gitlab-ce@dfe9fb16 kirr/gitlab-ce@f3f84743 so essentially the only way to access a repository is via HTTP(S). - Rake check tasks are adjusted to exit with non-zero code if there is a failure kirr/gitlab-ce@a93ae418 We need this for promises to work correctly with failures being detected, not silently skipped. The patch was sent upstream: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1885 - GitLab supports setting up site's ICP License in gitlab.yml and shows it in appropriate places together with info about GitLab itself: kirr/gitlab-ce@e7e0fd88 kirr/gitlab-ce@79c127e6 + other cosmetic/minor changes. More patches will probably come (e.g. apply a single patch from a merge-request with `git am` without creating merge commit for just 1 patch, etc) but for now that's all. NOTE ICP is non-ascii text with hieroglyphs. slapos.core was taught to be able to pass parameters with non-ascii values to instance: slapos.core@347d33d6 That patch is included in slapos.core 1.3.15, but as we currently have a lot of older slapos.core deployed (e.g. 1.3.5 on my development webrunner) a workaround is (hopefully temporarily) used to pass non-ascii values as URL-encoded strings. /cc @kazuhiko, @jerome, @rafael
-
Kirill Smelkov authored
In the previous patch we setup nginx service which listens to the world and as such gitlab service becomes to be ready to used - so publish backend URL. NOTE we'll need to optimise and tweak gitlab a bit further in upcoming patches, so it can be really used under load and with our use-cases, but even now it listens to http ok and generally works. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Go through nginx configuration templates and convert them to jinja2 with slapos parameters (reminder: names and default values are imported from omnibus-gitlab 8.2.3+ce.0-0-g8eda093), except commenting out features we do not want to support (yet ?). As nginx is a reverse-proxy, i.e. it integrates all internal services and works as frontend to them, our gitlab service is now ready to listen and talk to the world over (standard to slapos services backend) IPv6. Nginx also acts as SSL termination point - for it to work by default we setup self-signed certificate for the backend, which can be manually changed to proper certificate if needed. Backend certificate is used if gitlab is configured to work in HTTPS mode (and frontend certificate is another story). NOTE ssl certificate is generated with just `openssl req ...` - yes, there is slapos.cookbook:certificate_authority.request but it requires to start whole service and has up to 60 seconds latency to generate certificate. And we only need to run 1 command to do that... The features disabled are: - http -> https redirection not needed for us at nginx level - the frontend can do the redirection and also gitlab speaks HSTS on https port so when we access https port via http protocol, it gets redirected to https. - kerberos - ssl_dhparam - providing custom nginx configuration via instance parameter /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Like with Rails configuration files, hook nginx configuration files into SR / instance build process; rename *.erb -> *.in and add our header. The templates are still not valid - a lot of erb code is left there - we'll slapos'ify it incrementally in the following patches. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Like with Rails configuration this first step is pristine import of nginx configuration files from omnibus-gitlab. All files were imported as-is in their ERB form and filenames from omnibus-gitlab 8.2.3+ce.0-0-g8eda093 from here: https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default We import only nginx main http configuration - nginx's CI and Mattermost configurations are not imported, as we do not support CI and Mattermost (yet ?). As with Rails configuration files, we will convert the templates to jinja2 and adjust them to slapos version in the following patches. We will also use the same (commit from last-erb-mod commit + merge) approach to track upstream changes. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Sidekiq[1] is used in GitLab as background jobs manager - i.e. if a request handler needs to spawn some non-light job - it adds it to sidekiq queue (in Redis) and relies on sidekiq service to later pick this job up and execute it. The service is setup with just to run bin/gitlab-sidekiq with appropriate queues (extracted from omnibus-gitlab) and appropriate settings to controlling GitLab's sidekiq Out-Of-Memory killer[2]. NOTE Unlike unicorn OOM killer, Sidekiq memory killer just makes sidekiq processes to be SIGKILL terminated and relies on managing service to restart it. In slapos we don't have mechanism to set autorestart=true, nor bang/watchdog currently work with slapproxy, so we setup to do such monitoring ourselves manually with here-introduced watcher-sigkill program. NOTE2 sidekiq promise, because it is rake/gitlab based, is slow to load/run and thus is put into etc/promise.slow/ [1] http://sidekiq.org/ [2] https://gitlab.com/gitlab-org/gitlab-ce/blob/1322bd78/doc/operations/sidekiq_memory_killer.md /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
As was described in the previous patch, we need $HOME to be propagated by this programs so that git can find partition's .gitconfig. Specifically we need the following patches to be present in our build: https://gitlab.com/gitlab-org/gitlab-shell/commit/9e087f64 https://gitlab.com/gitlab-org/gitlab-workhorse/commit/b5f1b803 They both have been applied upstream very close to revisions we previously had in software.cfg, so we only need to update the revisions to get them. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Convert gitconfig template to jinja2 (reusing already-there `email_display_name` and `email_from` parameters for commits generated by gitlab). System-level git config from gitlab-omnibus is also imported to this file (on slapos we cannot tweak system-level git config - software/.../parts/git/... is read-only for programs in instance partitions - so we move all gitlab's system-wide git settings to this "user-level" gitconfig. System gitconfig in omnibus is defined here: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda093/files/gitlab-cookbooks/gitlab/attributes/default.rb#L23 so it is pack.threads = 1 and receive.fsckObjects = true which makes sense to not waste a lot of memory when packing and not to allow corrupt objects to enter to system by evil users intentionally. To make the file foundable by git - we put it into partition root directory and set $HOME to point to partition root when running appropriate programs / services. NOTE we'll need to upgrade gitlab-shell and gitlab-workhorse to propagate $HOME for this setting to actually have effect. See the next patch. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Like with Rails configuration files, this is pristine import of template gitconfig from omnibus GitLab from https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default This is only a "user" part of git configuration. System-wide configuration is generated dynamically: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda093/files/gitlab-cookbooks/gitlab/attributes/default.rb#L23 and we'll import it by hand in the follow-up patches. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Gitlab-workhorse[1] is a service which offloads Ruby-on-Rails based GitLab from long-running and slow requests. It is written in Go. Now as we have unicorn service set up, we can setup gitlab-workhorse service (which uses unicorn as authentication backend). Gitlab-workhorse setup is easy - it is just one program and several command line options to point to unicorn socket and to configure on which unix socket gitlab-workhorse will listen itself. NOTE we have to care that git and ruby to be on PATH when running gitlab-workhorse - because on e.g. git push'ing workhorse will run `git receive-pack` and a hook will be called which calls gitlab-shell, which is written in ruby. NOTE2 promise to check whether gitlab-workhorse is alive is to ping it via URL to non-existent endpoint and check for proper 403 HTTP code returned. [1] https://gitlab.com/gitlab-org/gitlab-workhorse /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
gitlab: Compile assets on instantiation and make sure DB is properly setup/migrated before unicorn runs There are several actions that needs to be done on gitlab instance upgrade: - we have to (re-)compile assets - we have to migrate DB and also before the first run - we have to initialize DB We can compile assets as part of instantiation process, but regarding DB migration / setup - it is not currently possible to do that as part of instantiation - for that operations we need PG & Redis to be already running, but the first time slapos instantiates an SR it first prepares all services, and only after instantiation is done, starts them all. There is currently no way to hook into starting process, and run some scripts after one service is started but before another service startup... So the solution is: to perform such actions in delayed mode as part of application - unicorn service - startup: it makes sure PG is running and initializes it and does other actions which needs to be done to migrate the DB. Only if/after they succeed the main application is started. NOTE the comment about unicorn/gitlab startup slowness from the previous patch still holds true - so in order to get "all ok" after instantiation, it is required to perform the instantiation several times, because unicorn promise initially fails. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Now that all gitlab Rails application configuration files are ready, we can setup unicorn service to start it. NOTE there is a promise to check unicorn by url which works, but there are also rake tasks to check gitlab itself, e.g. like gitlab:app:check gitlab:gitlab_shell:check gitlab:repo:check Unfortunately this tasks are slow to run (and gitlab:repo:check is very slow to run). That's why we do not put them into etc/promise/ - if we do - slapos reports promises time outs. What we do is we put them into etc/promise.slow/ so we have those scripts ready, but currently no one automatically checks them. Again, the promise to check unicorn just by accessing it by URL is there and is checked automatically out of the box. NOTE2 GitLab is very slow to load. That's why it can take some time after unicorn starts that it's promise start to report ok. This can show itself as temporary instantiation errors which say promise such and such failed. NOTE3 Unicorn start, but so far we did not cared to setup GitLab DB schema on instantiation. That's why unicorn remains not very usable and a lot of requests fail. We'll teach instance to setup DB and perform all other needed settings in the next patch. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Like with [promise-wrapper] a recipe could do [promise-<service>] <= promise-byurl url = ... and a script to check such ur will be generated and automatically put into etc/promise/<service>. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Automatically configure unicorn to spawn as much worker processes as there are CPUs on the system by default. GitLab omnibus pre-hardcodes this value default to 2 (which we copied) and then also tweaks it this way in active code https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/attributes/default.rb#L230 which we also do here. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Explicitly point gitlab-shell to location where we keep secrets. We already pointeg gitlab to that place and now we do that for gitlab-shell so those 2 peieces can connect to each other ok. Regarding the setting itself - there is no such block in omnibus-gitlab, but it is present in gitlab-shell configuration example: https://gitlab.com/gitlab-org/gitlab-shell/blob/82b3a4e8/config.yml.example#L35 /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Convert gitlab-shell configuration file to slapos: - convert to jinja2, - connect gitlab-shell to unicorn & redis unix sockets NOTE - http_settings are left to be default (empty) ones - as that works ok. - `auth_file` is still configured to point to wont-be-used sshkeys file, as without it gitlab-shell check will fail. - support for audit_usernames and git_annex is disabled and remains not configurable. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Unicorn is a forking server with the idea that master process preloads heavy Ruby-on-Rails application, and then to handle new request a worker process is forked with application already loaded in its memory (and modification being tracked by OS via copy-on-write). From this point of view the only reasonable value for preload_app is always "true" and omnibus-gitlab does this: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/definitions/unicorn_service.rb#L65 Then unicorn documentation shows what code has to be there in pre-/post- forking event: http://bogomips.org/unicorn.git/tree/examples/unicorn.conf.rb?id=3312aca8#n57 GitLab uses only part of it that "allows a new master process to incrementally phase out the old master process with SIGTTOU to avoid a thundering herd": https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/definitions/unicorn_service.rb#L69 http://bogomips.org/unicorn.git/tree/examples/unicorn.conf.rb?id=3312aca8#n75 but strangely does not use code parts that are "highly recommended" or "require" for "Rails + "preload_app true"" case. For the reference I've added such codes, but kept them being commented out. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Convert unicorn parameters to slapos and configure it to listen on unix socket only. ( Omnibus configures unicorn to listen on unix socket and loopback TCP, mainly because gitlab-shell could not connect to unicorn via unix socket until recently: https://gitlab.com/gitlab-org/gitlab-shell/commit/184385ac But as it can now, there is no point to keep on TCP port open ) To be able to do such configuration we add stub to unicorn service section (to create needed directories where to keep the socket). There will be follow-up patch which configures unicorn pre/post-forking actions, which is not trivial and thus better be done on its own. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Convert the rest of this configuration file to slapos. It is straightforward conversion of parameters except: - access-via-ssh is disabled (gitlab slapos version does not support ssh access and supports HTTP(S) only by design on purpose) - we do not support restricting possible projects visibility via instance parameter (very low chance this will be needed in practice) - default issue-closing pattern is just ok for now and not configurable - support for builds, build artifacts & CI is disabled (we do not support CI (yet ?)) - some internal defaults are just ok (e.g. where to organize directory for keeping repositories archives for downloads) - reply-by-email is not supported (yet ?) - we do not support LFS (yet ?) - just plain git hosting is ok for now. - Gravatar defaults are ok for now and not configurable. - Support for LDAP is disabled - Support for Kerberos is disabled - Support for OmniAuth is disabled - Satellites path is just /dev/null as we start from version where satellites are already non-existent. - Uploading backups to somewhere via GitLab's builtin mechanism is not supported - we'll use SlapOS native backup and resiliency for this. - Support for Google analytics is disabled. - Support for Piwik is disabled. - we are ok (for now) with default rack-attack git settings /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
This user will need to be specified several times in configuration files, as by default gitlab uses 'git' user and does "sudo" to it if it is not current. We will use {{ backend_info.user }} in the upcoming patches. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
GitLab has a notion of "external URL" - the canonical "frontend" URL the server is reachable through: this URL is used as prefix to show e.g. git-clone URL for repositories, etc, even if a server can be reachable via several frontends. Add external_url handling to slapos instance. NOTE whether to use https or not is also defined by external_url, in particular by external_url scheme. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Convert to slapos SMTP settings for gitlab: - convert to jinja2 - remove support for gitlab CI (we do not support it (yet ?)) - add handling of `smtp_enable` parameter directly to that file ( omnibus handles this parameter externally and just removes smtp_settings.rb if it is true ) NOTE smtp_settings.rb contains SMTP password, so it is mode is set to 0600. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Just another 2 simple parameters (attack detection tunables) conversion to jinja2/slapos. /cc @kazuhiko, @jerome
-
Kirill Smelkov authored
Just convert 2 parameters used in that file to jinja syntax and add those parameters (unicorn OOM killer tunables) to gitlab-parameters.cfg /cc @kazuhiko, @jerome
-