1. 24 Feb, 2017 2 commits
  2. 26 Jan, 2017 1 commit
  3. 20 Oct, 2016 1 commit
  4. 11 Aug, 2016 1 commit
    • iv's avatar
      gitlab: Sync upstream configs from omnibus-gitlab 8.8.7+ce.1-0-g5116476 · 5e227fdb
      iv authored
      Like f6f97d72 - pristine copy from omnibus-gitlab 8.8.7+ce.1-0-g5116476
      
      Changes are:
      
          - gitlab.yml.erb
            Add gitlab_default_projects_features_container_registry variable to be used by docker containers
      
          - nginx.conf.erb
            Docker related password storage
            https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1218 (commit f74472d4)
      
          - rack_attack.rb.erb
            Disable Rack Attack throttling if admin disables it in config file
      
          - smtp_settings.rb.erb
            If authentication is not enabled for smtp, don't place it in the config.
      
      The following files stay the same:
      
          - database.yml.erb
          - gitconfig.erb
          - gitlab-shell-config.yml.erb
          - nginx-gitlab-http.conf.erb
          - resque.yml.erb
          - unicorn.rb.erb
      5e227fdb
  5. 07 Aug, 2016 1 commit
  6. 05 Aug, 2016 1 commit
  7. 02 Aug, 2016 1 commit
  8. 11 Apr, 2016 1 commit
  9. 28 Feb, 2016 3 commits
    • Kirill Smelkov's avatar
      gitlab: Slapos'ify gitlab config updates · b19d2942
      Kirill Smelkov authored
          - relative URL support: comment out - we do not need it - gitlab is
            always located at /.
      
          - Nginx-http: restore our version for proxy_set_header - upstream
            turned to allowing users to configure this, see e.g.
      
              https://gitlab.com/gitlab-org/omnibus-gitlab/commit/e13d5e42
              https://gitlab.com/gitlab-org/omnibus-gitlab/commit/a450585e
      
            but doing this way creates more complexity for gitlab SR, so I've
            restored our version which essentially does the same as default in
            omnibus-gitlab, and if we'll need to tune it - we can do directly in
            Nginx config.
      
            In other words slapos version does not allow users to tune nginx
            headers as instance parameter.
      b19d2942
    • Kirill Smelkov's avatar
      gitlab: Sync upstream configs from omnibus-gitlab · 02d0063b
      Kirill Smelkov authored
      Like 8c62b063, d17f1f5f and e8461571 - pristine copy from omnibus-gitlab
      8.5.1+ce.0-1-ge732b39 .
      
      Changes are in
      
          - gitlab.yml.erb, unicorn.rb.erb
      
            * Something related to relative URL root (we do not use)
            * Something related to SAML (we do not use)
            * Misc
      
          - nginx-gitlab-http.conf.erb
      
            * SPDY -> HTTP/2
            * Relative URL root
            * Configurable proxy_set_header passing
      
      The following files stay the same:
      
          - database.yml.erb
          - gitconfig.erb
          - gitlab-rails-config.ru.erb
          - gitlab-shell-config.yml.erb
          - nginx.conf.erb
          - rack_attack.rb.erb
          - resque.yml.erb
          - smtp_settings.rb.erb
      02d0063b
    • Kirill Smelkov's avatar
      gitlab: Establish proper 1 branch for tracking upstream configs · 97dcf455
      Kirill Smelkov authored
      It was my mistake to establish several tracking lines for tracking
      upstream changes - e.g. in
      
          61544d87    (gitlab: Import nginx http configuration from omnibus-gitlab)
      
      we started not from
      
          6fd7b987    (gitlab: Import gitlab-ce & gitlab-shell configs from omnibus-gitlab)
      
      -- the first upstream tracking commit on its own branch -- but from
      
          4c127fdd    (gitlab: Setup sidekiq service)
      
      i.e. from after some changes which already tweaked upstream
      configuration files.
      
      This makes updating gitlab more work than necessary: instead of
      switching to upstream branch only once, importing all files, and
      then switching back to master and merging upstream changes only once, we
      currently have to do that operation 3 times:
      
          - for main gitlab settings,
          - for nginx settings, and
          - for gitconfig settings
      
      which is not convenient and wastes our time.
      
      So establish a proper 1 branch for tracking upstream configs:
      
      Here we cherry-pick the following commits
      
          61544d87    (gitlab: Import nginx http configuration from omnibus-gitlab)
          d17f1f5f    (gitlab: Sync nginx http configuration from omnibus gitlab)
      
          8f945bd2    (gitlab: Import gitconfig from omnibus-gitlab)
          e8461571    (gitlab: Sync gitconfig settings from omnibus-gitlab)
      
      and later we'll be updating upstream files on a branch starting from
      this commit and containing upstream changes only.
      
      /cc @kazuhiko, @jerome
      97dcf455
  10. 11 Feb, 2016 6 commits
  11. 17 Jan, 2016 20 commits
    • Kirill Smelkov's avatar
      gitlab: Optimize raw blob downloading · a913c2e4
      Kirill Smelkov authored
      In slapos we do a lot of automated software rebuild constantly, and thus
      there is constant flow of requests to get raw blobs from git service,
      e.g. like this
      
          https://lab.nexedi.com/nexedi/slapos/raw/master/software/wendelin/software.cfg
      
      A lot of requests comes to slapos.git repository and currently gitlab,
      out of the box, cannot keep up with that load.
      
      I've prepared patches to offload raw blobs download requests handling
      from unicorn (ruby) to gitlab-workhorse (go), and that resulted in ~ 17x
      speedup - e.g. previously our std shuttle can handle ~ 70 raw-blob
      requests/s and with my changes it is now ~ 1200 requests/s.
      
      The patches were sent upstream
      
          https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17
      
      and we discussed with GitLab people and made a plan how to proceed
      incrementally. It will probably take some time for gitlab team to fully
      accept the approach though.
      
      For now we can use our gitlab-workhorse fork. The patches itself are:
      
          kirr/gitlab-workhorse@1b274d0d
          kirr/gitlab-workhorse@2beb8c95
      
      /cc @kazuhiko, @jerome, @jm
      a913c2e4
    • Kirill Smelkov's avatar
      gitlab: Switch to "GitLab Nexedi Edition" · 74d4ea62
      Kirill Smelkov authored
      GitLab Nexedi Edition is currently upstream 8.2.X + the following
      patches:
      
          - HTTP(S) is made to be default clone protocol
      
              kirr/gitlab-ce@5c1f2fb3
      
            and SSH info is completely removed from UI
      
              kirr/gitlab-ce@dfe9fb16
              kirr/gitlab-ce@f3f84743
      
            so essentially the only way to access a repository is via HTTP(S).
      
          - Rake check tasks are adjusted to exit with non-zero code if there
            is a failure
      
              kirr/gitlab-ce@a93ae418
      
            We need this for promises to work correctly with failures being
            detected, not silently skipped. The patch was sent upstream:
      
              https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1885
      
          - GitLab supports setting up site's ICP License in gitlab.yml and
            shows it in appropriate places together with info about GitLab
            itself:
      
              kirr/gitlab-ce@e7e0fd88
              kirr/gitlab-ce@79c127e6
      
          + other cosmetic/minor changes.
      
      More patches will probably come (e.g. apply a single patch from a
      merge-request with `git am` without creating merge commit for just 1
      patch, etc) but for now that's all.
      
      NOTE ICP is non-ascii text with hieroglyphs. slapos.core was taught to
          be able to pass parameters with non-ascii values to instance:
      
              slapos.core@347d33d6
      
          That patch is included in slapos.core 1.3.15, but as we currently
          have a lot of older slapos.core deployed (e.g. 1.3.5 on my
          development webrunner) a workaround is (hopefully temporarily) used
          to pass non-ascii values as URL-encoded strings.
      
      /cc @kazuhiko, @jerome, @rafael
      74d4ea62
    • Kirill Smelkov's avatar
      gitlab: Import nginx http configuration from omnibus-gitlab · 61544d87
      Kirill Smelkov authored
      Like with Rails configuration this first step is pristine import of
      nginx configuration files from omnibus-gitlab. All files were imported
      as-is in their ERB form and filenames from omnibus-gitlab
      8.2.3+ce.0-0-g8eda093 from here:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default
      
      We import only nginx main http configuration - nginx's CI and Mattermost
      configurations are not imported, as we do not support CI and Mattermost (yet ?).
      
      As with Rails configuration files, we will convert the templates to
      jinja2 and adjust them to slapos version in the following patches.  We
      will also use the same (commit from last-erb-mod commit + merge)
      approach to track upstream changes.
      
      /cc @kazuhiko, @jerome
      61544d87
    • Kirill Smelkov's avatar
      gitlab/nginx: Slapos'ify config and turn nginx into a service · 85f7d7e3
      Kirill Smelkov authored
      Go through nginx configuration templates and convert them to jinja2 with
      slapos parameters (reminder: names and default values are imported from
      omnibus-gitlab 8.2.3+ce.0-0-g8eda093), except commenting out features we
      do not want to support (yet ?).
      
      As nginx is a reverse-proxy, i.e. it integrates all internal services
      and works as frontend to them, our gitlab service is now ready to listen
      and talk to the world over (standard to slapos services backend) IPv6.
      
      Nginx also acts as SSL termination point - for it to work by default we
      setup self-signed certificate for the backend, which can be manually
      changed to proper certificate if needed. Backend certificate is used
      if gitlab is configured to work in HTTPS mode (and frontend certificate
      is another story).
      
      NOTE ssl certificate is generated with just `openssl req ...` - yes, there
          is slapos.cookbook:certificate_authority.request but it requires
          to start whole service and has up to 60 seconds latency to generate
          certificate. And we only need to run 1 command to do that...
      
      The features disabled are:
      
          - http -> https redirection
      
            not needed for us at nginx level - the frontend can do the
            redirection and also gitlab speaks HSTS on https port so when we access
            https port via http protocol, it gets redirected to https.
      
          - kerberos
          - ssl_dhparam
          - providing custom nginx configuration via instance parameter
      
      /cc @kazuhiko, @jerome
      85f7d7e3
    • Kirill Smelkov's avatar
      gitlab: Hook nginx configuration files into SR system · 45127f6d
      Kirill Smelkov authored
      Like with Rails configuration files, hook nginx configuration files into
      SR / instance build process; rename *.erb -> *.in and add our header.
      
      The templates are still not valid - a lot of erb code is left there -
      we'll slapos'ify it incrementally in the following patches.
      
      /cc @kazuhiko, @jerome
      45127f6d
    • Kirill Smelkov's avatar
      gitlab: Import gitconfig from omnibus-gitlab · 8f945bd2
      Kirill Smelkov authored
      Like with Rails configuration files, this is pristine import of template
      gitconfig from omnibus GitLab from
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default
      
      This is only a "user" part of git configuration. System-wide
      configuration is generated dynamically:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda093/files/gitlab-cookbooks/gitlab/attributes/default.rb#L23
      
      and we'll import it by hand in the follow-up patches.
      
      /cc @kazuhiko, @jerome
      8f945bd2
    • Kirill Smelkov's avatar
      gitlab: Slapos'ify / tweak gitconfig and hook it into the system · 264d785a
      Kirill Smelkov authored
      Convert gitconfig template to jinja2 (reusing already-there
      `email_display_name` and `email_from` parameters for commits generated by
      gitlab).
      
      System-level git config from gitlab-omnibus is also imported to this
      file (on slapos we cannot tweak system-level git config -
      software/.../parts/git/... is read-only for programs in instance
      partitions - so we move all gitlab's system-wide git settings to this
      "user-level" gitconfig.
      
      System gitconfig in omnibus is defined here:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda093/files/gitlab-cookbooks/gitlab/attributes/default.rb#L23
      
      so it is
      
          pack.threads = 1    and
          receive.fsckObjects = true
      
      which makes sense to not waste a lot of memory when packing and not to
      allow corrupt objects to enter to system by evil users intentionally.
      
      To make the file foundable by git - we put it into partition root
      directory and set $HOME to point to partition root when running
      appropriate programs / services.
      
      NOTE we'll need to upgrade gitlab-shell and gitlab-workhorse to
          propagate $HOME for this setting to actually have effect.
          See the next patch.
      
      /cc @kazuhiko, @jerome
      264d785a
    • Kirill Smelkov's avatar
      gitlab/gitlab-shell-config.yml: Explicitly point it to secret file · b55d823d
      Kirill Smelkov authored
      Explicitly point gitlab-shell to location where we keep secrets.
      
      We already pointeg gitlab to that place and now we do that for
      gitlab-shell so those 2 peieces can connect to each other ok.
      
      Regarding the setting itself - there is no such block in omnibus-gitlab,
      but it is present in gitlab-shell configuration example:
      
          https://gitlab.com/gitlab-org/gitlab-shell/blob/82b3a4e8/config.yml.example#L35
      
      /cc @kazuhiko, @jerome
      b55d823d
    • Kirill Smelkov's avatar
      gitlab/gitlab-shell-config.yml: Slapos'ify it · 0cd14ef6
      Kirill Smelkov authored
      Convert gitlab-shell configuration file to slapos:
      
          - convert to jinja2,
          - connect gitlab-shell to unicorn & redis unix sockets
      
      NOTE
      
          - http_settings are left to be default (empty) ones - as that works ok.
          - `auth_file` is still configured to point to wont-be-used sshkeys
            file, as without it gitlab-shell check will fail.
          - support for audit_usernames and git_annex is disabled and
            remains not configurable.
      
      /cc @kazuhiko, @jerome
      0cd14ef6
    • Kirill Smelkov's avatar
      gitlab/unicorn.rb: Configure preload_app and pre-/post- forking actions · d599096a
      Kirill Smelkov authored
      Unicorn is a forking server with the idea that master process preloads
      heavy Ruby-on-Rails application, and then to handle new request a worker
      process is forked with application already loaded in its memory (and
      modification being tracked by OS via copy-on-write).
      
      From this point of view the only reasonable value for preload_app is
      always "true" and omnibus-gitlab does this:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/definitions/unicorn_service.rb#L65
      
      Then unicorn documentation shows what code has to be there in pre-/post-
      forking event:
      
          http://bogomips.org/unicorn.git/tree/examples/unicorn.conf.rb?id=3312aca8#n57
      
      GitLab uses only part of it that "allows a new master process to
      incrementally phase out the old master process with SIGTTOU to avoid a
      thundering herd":
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/definitions/unicorn_service.rb#L69
          http://bogomips.org/unicorn.git/tree/examples/unicorn.conf.rb?id=3312aca8#n75
      
      but strangely does not use code parts that are "highly recommended" or
      "require" for "Rails + "preload_app true"" case.
      
      For the reference I've added such codes, but kept them being commented
      out.
      
      /cc @kazuhiko, @jerome
      d599096a
    • Kirill Smelkov's avatar
      gitlab/unicorn.rb: First round of slaposification · 0aae33d9
      Kirill Smelkov authored
      Convert unicorn parameters to slapos and configure it to listen on unix
      socket only.
      
      ( Omnibus configures unicorn to listen on unix socket and
        loopback TCP, mainly because gitlab-shell could not connect to unicorn
        via unix socket until recently:
      
            https://gitlab.com/gitlab-org/gitlab-shell/commit/184385ac
      
        But as it can now, there is no point to keep on TCP port open )
      
      To be able to do such configuration we add stub to unicorn service
      section (to create needed directories where to keep the socket).
      
      There will be follow-up patch which configures unicorn pre/post-forking
      actions, which is not trivial and thus better be done on its own.
      
      /cc @kazuhiko, @jerome
      0aae33d9
    • Kirill Smelkov's avatar
      gitlab/gitlab.yml: Slapos'ify rest of it · c3f1f0a9
      Kirill Smelkov authored
      Convert the rest of this configuration file to slapos.
      
      It is straightforward conversion of parameters except:
      
          - access-via-ssh is disabled (gitlab slapos version does not support
            ssh access and supports HTTP(S) only by design on purpose)
      
          - we do not support restricting possible projects visibility via
            instance parameter (very low chance this will be needed in
            practice)
      
          - default issue-closing pattern is just ok for now and not
            configurable
      
          - support for builds, build artifacts & CI is disabled (we do not
            support CI (yet ?))
      
          - some internal defaults are just ok (e.g. where to organize
            directory for keeping repositories archives for downloads)
      
          - reply-by-email is not supported (yet ?)
      
          - we do not support LFS (yet ?) - just plain git hosting is ok for now.
      
          - Gravatar defaults are ok for now and not configurable.
      
          - Support for LDAP is disabled
      
          - Support for Kerberos is disabled
      
          - Support for OmniAuth is disabled
      
          - Satellites path is just /dev/null as we start from version where
            satellites are already non-existent.
      
          - Uploading backups to somewhere via GitLab's builtin mechanism is
            not supported - we'll use SlapOS native backup and resiliency for
            this.
      
          - Support for Google analytics is disabled.
      
          - Support for Piwik is disabled.
      
          - we are ok (for now) with default rack-attack git settings
      
      /cc @kazuhiko, @jerome
      c3f1f0a9
    • Kirill Smelkov's avatar
      gitlab/gitlab.yml: Handle "external URL" · 93362a08
      Kirill Smelkov authored
      GitLab has a notion of "external URL" - the canonical "frontend" URL the
      server is reachable through: this URL is used as prefix to show
      e.g. git-clone URL for repositories, etc, even if a server can be
      reachable via several frontends.
      
      Add external_url handling to slapos instance.
      
      NOTE whether to use https or not is also defined by external_url, in
      particular by external_url scheme.
      
      /cc @kazuhiko, @jerome
      93362a08
    • Kirill Smelkov's avatar
      gitlab/smtp_settings.rb: Convert/integrate to slapos · c64f7ece
      Kirill Smelkov authored
      Convert to slapos SMTP settings for gitlab:
      
          - convert to jinja2
          - remove support for gitlab CI (we do not support it (yet ?))
          - add handling of `smtp_enable` parameter directly to that file
            ( omnibus handles this parameter externally and just removes
              smtp_settings.rb if it is true )
      
      NOTE smtp_settings.rb contains SMTP password, so it is mode is set to 0600.
      
      /cc @kazuhiko, @jerome
      c64f7ece
    • Kirill Smelkov's avatar
      gitlab/rack_attack.rb: Convert/integrate to slapos · a44f5a43
      Kirill Smelkov authored
      Just another 2 simple parameters (attack detection tunables) conversion
      to jinja2/slapos.
      
      /cc @kazuhiko, @jerome
      a44f5a43
    • Kirill Smelkov's avatar
      gitlab/config.ru: Convert/integrate to slapos · 41b1edb5
      Kirill Smelkov authored
      Just convert 2 parameters used in that file to jinja syntax and add
      those parameters (unicorn OOM killer tunables) to gitlab-parameters.cfg
      
      /cc @kazuhiko, @jerome
      41b1edb5
    • Kirill Smelkov's avatar
      gitlab/resque.yml: Tweak to integrate gitlab with internal redis · b20c258b
      Kirill Smelkov authored
      A simple change just to point resque to redis unix socket.
      
      /cc @kazuhiko, @jerome
      b20c258b
    • Kirill Smelkov's avatar
      gitlab/database.yml: Tweak to integrate gitlab with internal postgresql · a73d20f4
      Kirill Smelkov authored
      We tweak database.yml to point to our postgresql unix socket; set
      adapter to hardcoded postgresql, encoding to unicode and omit collation
      (which according to omnibus-gitlab is used for mysql only).
      
      The only instance parameter imported from omnibus is `db_pool` - how
      many connection to a DB to keep open in a RoR thread/process.
      
      XXX we use db's superuser as a user to connect. Is it ok to do even if
          the whole DB is used only for gitlab? (I think it is ok for the
          first iteration, but we'll probably need to refine this later)
      
      /cc @kazuhiko, @jerome
      a73d20f4
    • Kirill Smelkov's avatar
      gitlab: Import gitlab-ce & gitlab-shell configs from omnibus-gitlab · 6fd7b987
      Kirill Smelkov authored
      Pristine import of template configuration files from omnibus GitLab
      package. All files were imported as-is in their ERB form and filenames
      from omnibus-gitlab 8.2.3+ce.0-0-g8eda093 from here:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default
      
      We will convert the templates to jinja2 and adjust them to slapos
      version in the following patches.
      
      Scheme for synchronizing with future upstream changes is envisioned as this:
      
          - checkout latest commit which updated pristine erb files
          - copy updated files from omnibus-gitlab, and commit the updates
          - checkout slapos master
          - merge commit that updated erb
      
      That should reasonably work with not too-many conflicts and even those
      should be not hard to resolve (with `git mergetool` e.g. in kdiff3)
      
      /cc @kazuhiko, @jerome
      6fd7b987
    • Kirill Smelkov's avatar
      gitlab: Hook gitlab- and gitlab-shell- configuration files into the system · 13169cab
      Kirill Smelkov authored
      - Download them on SR build and pass info to instance
      - Instance prepares to process them as jinja2 templates
      - Instance hooks the files into configuration location as appropriate
      
      Every file so far is renamed *.erb -> *.in and a header added showing
      that this file is autogenerated with links about what was the base
      gitlab and/or omnibus version and omnibus reference revision this
      template was last updated for.
      
      So far all result configuration files are invalid - because ERB syntax
      is there. We will convert the configuration files to proper jinja2
      syntax and to using slapos parameters incrementally in the upcoming
      patches.
      
      NOTE (again): md5 sums are not yet fixed - we will fix them in the end
          of gitlab patches series after applying all tweaking changes.
      
      /cc @kazuhiko, @jerome
      13169cab