1. 17 Jan, 2016 9 commits
    • Kirill Smelkov's avatar
      gitlab: Add helper to set up promise to check something via url · 2772191c
      Kirill Smelkov authored
      Like with [promise-wrapper] a recipe could do
      
          [promise-<service>]
          <= promise-byurl
          url     = ...
      
      and a script to check such ur will be generated and automatically put
      into etc/promise/<service>.
      
      /cc @kazuhiko, @jerome
      2772191c
    • Kirill Smelkov's avatar
      gitlab/unicorn: Automatically load all available CPUs by default · 5dc6321c
      Kirill Smelkov authored
      Automatically configure unicorn to spawn as much worker processes as
      there are CPUs on the system by default.
      
      GitLab omnibus pre-hardcodes this value default to 2 (which we copied)
      and then also tweaks it this way in active code
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/attributes/default.rb#L230
      
      which we also do here.
      
      /cc @kazuhiko, @jerome
      5dc6321c
    • Kirill Smelkov's avatar
      gitlab: Determine current slapuserX in instance · 34419064
      Kirill Smelkov authored
      This user will need to be specified several times in configuration
      files, as by default gitlab uses 'git' user and does "sudo" to it if it
      is not current.
      
      We will use {{ backend_info.user }} in the upcoming patches.
      
      /cc @kazuhiko, @jerome
      34419064
    • Kirill Smelkov's avatar
      gitlab: Introduce macro library · 2e81276b
      Kirill Smelkov authored
      Introduce a library of Jinja2 macros that will be handy to use in
      templates. For now we add only 2 macros:
      
          cfg(name)   - to get instance configuration parameter `name`,   and
          cfg_bool    - to get truth value of ----//----
      
      The reason we introduce cfg() is that we will need to use a lot of
      parameters in many places and it is much more handy to write, e.g.
      
          cfg('email_enabled')
      
      compared to
      
          instance_parameter:configuration.email_enabled
      
      /cc @kazuhiko, @jerome
      2e81276b
    • Kirill Smelkov's avatar
      gitlab: Organize place to keep parameters & their default imported from gitlab-omnibus · de860ba5
      Kirill Smelkov authored
      We will be using a several dozens of parameters to control gitlab
      instance. It makes sense not to deviate in such parameters namings and
      defaults from omnibus version.
      
      Thus for such parameters - for clarity - we organize a separate file
      where we will be keeping them - gitlab-parameters.cfg.
      
      In this patch series all used parameters will be "imported" from
      omnibus-gitlab 8.2.3+ce.0-0-g8eda093.
      
      NOTE it is maybe better to try to autogenerate that file from upstream
          omnibus parameters definitions. If time will tell it becomes hard to
          maintain our copy - we'll consider going that way.
      
      /cc @kazuhiko, @jerome
      de860ba5
    • Kirill Smelkov's avatar
      gitlab: Hook gitlab- and gitlab-shell- configuration files into the system · 13169cab
      Kirill Smelkov authored
      - Download them on SR build and pass info to instance
      - Instance prepares to process them as jinja2 templates
      - Instance hooks the files into configuration location as appropriate
      
      Every file so far is renamed *.erb -> *.in and a header added showing
      that this file is autogenerated with links about what was the base
      gitlab and/or omnibus version and omnibus reference revision this
      template was last updated for.
      
      So far all result configuration files are invalid - because ERB syntax
      is there. We will convert the configuration files to proper jinja2
      syntax and to using slapos parameters incrementally in the upcoming
      patches.
      
      NOTE (again): md5 sums are not yet fixed - we will fix them in the end
          of gitlab patches series after applying all tweaking changes.
      
      /cc @kazuhiko, @jerome
      13169cab
    • Kirill Smelkov's avatar
      gitlab: Organize per-instance gitlab work tree · 2ddc5b0c
      Kirill Smelkov authored
      Organize per-instance place for gitlab configuration and work directory.
      
      Unfortunately as GitLab is Ruby-on-Rails application, it is not possible
      to keep its code in one place and have multiple separate configuration
      sets in different places and start that code for a configuration set -
      GitLab and Rails insist to get configuration from relative to source
      code tree.
      
      GitLab omnibus "solves" this by having only one configuration set and
      having symlinks from code to that only configiration set. In slapos we
      can potentially have several instances for one software and thus we
      cannot do that.
      
      With such limitations a proper solution would be to bind-mount software
      code into instance filesystem namespace close to configuration - that
      way the code will be only one and will find proper per-instance config.
      Currently we do not have namespaces available on slapos unfortunately,
      thus something else is needed.
      
      The workaround I decided to do is this: to clone cloned gitlab
      repository from software/ space to instance/ space and adjust it in
      instance space. This has the following drawbacks:
      
          - code is duplicated
          - code becomes read-write, instead of being read-only
      
      but imho it is the most practical thing to do. Another solution could be
      to patch GitLab / Rails to remove "config lives in code" assumption, but
      the number of places where this needs to be done is really many.
      
      NOTE gems which gitlab uses and which were installed during software
          compilation are not duplicated - they are reused via bundler - via
          pointing BUNDLE_GEMFILE to original location in software.
      
      NOTE2 For instance tasks and also for maintanace convenience we establish
          <instance>/bin/gitlab-* programs, e.g. gitlab-rake, which e.g. for
          gitlab-rake will run rake with correctly loaded gitlab environment -
          like in gitlab-omnibus.
      
      /cc @kazuhiko, @jerome, @jp
      2ddc5b0c
    • Kirill Smelkov's avatar
      gitlab: Redis service · 0d286c5d
      Kirill Smelkov authored
      Organize internal Redis service, like with PostgreSQL in the previous
      patch, with the help of slapos.cookbook:redis.server recipe.
      
      Like with postgresql, and as we planned, redis listens only on
      internal-to-partition unix socket.
      
      The recipe establishes both service and promise to check it is alive;
      we only need to setup log rotation manually.
      
      /cc @kazuhiko, @jerome
      0d286c5d
    • Kirill Smelkov's avatar
      gitlab: PostgreSQL service · 470719fe
      Kirill Smelkov authored
      Organize internal PostgreSQL database which will be used as DB for
      Roby-on-Rails GitLab and listens only on unix socket (for security and
      performance reasons - see earlier intro patch).
      
      To do it we use slapos.cookbook:postgres recipe, with disabling
      "listen-to-network" via passing empty sets to ipv4 and ipv6 recipe
      arguments.
      
      The promise to check whether DB is alive is just `psql -c '\q'` which
      will error if failing to connect to DB, but exit silently if connected ok.
      
      Explicit log rotation is not needed - as postgresql logs to
      stdout/stderr - not to a file - logs are handled by slapos - put into
      .slappartX_postgresql.log and automatically rotated there.
      
      XXX omnibus-gitlab tunes postgresql with shared_buffers and other
      parameters, most likely for performance reasons - see e.g.
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-2-stable/files/gitlab-cookbooks/gitlab/templates/default/postgresql.conf.erb#L113
      
      I decided not to fine-tune postgresql for now, and get on-field feedback
      first, and then, if needed, we can tune.
      
      /cc @kazuhiko, @jerome
      470719fe
  2. 06 Jan, 2016 1 commit
    • Kirill Smelkov's avatar
      gitlab: Add empty instance · ab6d2f28
      Kirill Smelkov authored
      Add stub instance configuration which just establishes a way to have
      several software types(*), pass all needed info from software to
      instance, organizes base directory and establishes log rotation base for
      upcoming services.
      
      Log rotation is done with the help of cron periodicallly calling
      logrotate. The rotation is done in "copytruncate" mode - i.e. log file
      is not moved away and signal sent for service to reopen it, but instead
      log content is just copied to outside and there is no need for a service
      to reopen it's log file.
      
      The reason it is done this way, is that there is a chance of not
      handling such "reopen-log-file" callbacks correctly on a service side,
      and so the net is full of crashing reports, e.g. like this:
      
          http://serverfault.com/questions/627521/why-is-logrotate-causing-apache-to-seg-fault-each-time
      
      That's why we take a safer approach instead, even if "copytruncate" mode
      is risking to loose several log entries(**) on rotation.
      
      NOTE services will organize log rotation with just
      
          [logrotate-entry-<service>]
          <= logrotate-entry
          log     = path/to/log/files/*.log
      
      For this to work some "!py!" magic (our way to serialize object into
      executable python and process it in buildout recipes) is used to process
      section names.
      
      The approach trick is also used for cron, e.g. logrotate registers to
      cron this way:
      
          [cron-entry-logrotate]
          <= cron-entry
          time    = daily
          command = ${logrotate:wrapper}
      
      NOTE2 instance md5 are not fixed yet - we'll fix them after applying all
          patches in gitlab series.
      
      (*) for now there is only 1 - "gitlab", but we'll need to have "-export"
          and "-import" for resiliency in the future.
      
      (**) ideally such things should be done with logfs - a filesystem
          specializeing in logging - for client services it will look like as
          they just continue to write to log file, and on log service side, the
          rotation can happen, all transparent to client service.
      
      /cc @kazuhiko, @jerome
      ab6d2f28