1. 09 Aug, 2016 3 commits
    • Alain Takoudjou's avatar
    • Alain Takoudjou's avatar
      monitor: script moved to slapos.toolbox · 445a6b90
      Alain Takoudjou authored
      445a6b90
    • Kirill Smelkov's avatar
      Revert "dropbear: Don't waste transfer time in favour of small-memory machines defaults" · 55460a4a
      Kirill Smelkov authored
      This reverts commit 605e564b.
      
      Rationale: Stability matters:
      
      Quoting 605e564b:
      > Besides changing only recv window size at runtime breaks compatibility with
      > openssh: if we only do `-W 1M` on server and try to upload data with openssh as
      > client, dropbear complains
      >
      >     [3302] Apr 17 23:10:06 Exit (slapuser2): Bad packet size 32777
      >
      > and connection terminates. Thus RECV_MAX_PAYLOAD_LEN increase is also
      > required, which cannot be done via option at runtime:
      >
      >     https://github.com/mkj/dropbear/blob/DROPBEAR_0.53.1/options.h#L268
      >
      >     ---- 8< ----
      >     /* Maximum size of a received SSH data packet - this _MUST_ be >= 32768
      >        in order to interoperate with other implementations */
      >     #ifndef RECV_MAX_PAYLOAD_LEN
      >     #define RECV_MAX_PAYLOAD_LEN 32768
      >     #endif
      >     ---- 8< ----
      >
      > So let's increase DEFAULT_RECV_WINDOW to 1M and RECV_MAX_PAYLOAD_LEN
      > appropriately (experimentally found that at 512K the complain goes
      > away).
      
      It turned out that "Bad packet size" did not really went away. For example I've
      recently hit the following:
      
          [14586] Aug 04 19:12:43 Pubkey auth succeeded for 'slapuser16' with key md5 b1:35:06:d3:a5:b1:0b:c6:7f:e6:59:31:ab:3a:e1:56 from 2001:67c:1254:c0::1:49886
          [14586] Aug 04 19:12:55 Exit (slapuser16): Integrity error (bad packet size 524500)
      
      in .slappartX_runner_sshd.log of my upgraded webrunner with connection being broken.
      ( nexedi/slapos!68 (comment 17748) )
      
      We could maybe try to play games with increasing RECV_MAX_PAYLOAD_LEN to
      be more than DEFAULT_RECV_WINDOW but this already turned out to be error-prone.
      
      Since when really needed we should be able to replace dropbear with openssh
      
          nexedi/slapos!68 (comment 7082)
      
      which is both performant and good-compatible, to me the way is:
      
      - make current dropbear run stable again,
      - when we really need to sync large amounts of data (and we should be
        needing to do soon or already) -> work on replacing dropbear with
        openssh.
      55460a4a
  2. 07 Aug, 2016 6 commits
    • Kirill Smelkov's avatar
      gitlab: Upgrade to 8.7 · abc0873b
      Kirill Smelkov authored
      - GitLab Software + patches ported to GitLab 8.7.X;
      - Configs synced with upstream;
      - No base software upgrades this time because it was all recently
        upgraded during a590b03e;
      
      TODO: allow configuration of trusted proxies
      
      /reviewed-by TrustMe
      abc0873b
    • Kirill Smelkov's avatar
      gitlab: Sync sidekiq service to upstream · af0c5b11
      Kirill Smelkov authored
      Like for 2a835e63
      
          $ git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --
                  files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb
      
      is empty.
      af0c5b11
    • Kirill Smelkov's avatar
      gitlab: Sync gitlab-parameters.cfg to upstream · 61177775
      Kirill Smelkov authored
      I've manually reviewed
      
          git diff 8.6.5+ce.0-0-g342f8be..8.7.9+ce.1-0-gf589ad7 --    \
              files/gitlab-config-template/gitlab.rb.template \
              files/gitlab-cookbooks/gitlab/attributes/default.rb
      
      and modulo trusted proxies there are no interesting changes for us.
      61177775
    • Kirill Smelkov's avatar
      gitlab: Slapos'ify gitlab config updates · 35b142e0
      Kirill Smelkov authored
      - config.ru template is gone - pristine gitlab-ce/config.ru can do the
        job because it obtains unicorn OOM killer setting via environment
        variables.
      
        https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/672
      
      - we put TODO there for configuring trusted proxies (gitlab.yml & nginx)
      
      - we restore our slaposified configuration from config.ru to unicorn.rb
      35b142e0
    • Kirill Smelkov's avatar
      gitlab: Merge in upstream config updates · 0f85f570
      Kirill Smelkov authored
      This does almost(*) only pure merge. We will slaposify / adjust config
      and corresponding md5sum in the following patches.
      
      (*) smtp ssl option is only added as comment.
      0f85f570
    • Kirill Smelkov's avatar
      gitlab: Update software to gitlab 8.7 · bf688f44
      Kirill Smelkov authored
      Update GitLab software to
      
          - gitlab-ce 8.7.9 + NXD patches
      
          - gitlab-shell to 2.7.2 + 1 patch to remove unneeded hooks.old in *.git
      
          - gitlab-workhorse stays at 0.7.1 + NXD patches because gitlab-ce 8.7.x
            sticks to this version (i.e. no workhorse upgrade for gitlab 8.6 -> 8.7)
      
      This only updates software and begins SR update to 8.7 - for now gitlab
      instance becomes non-working -- we'll pull in configuration files
      updates and fixups in the following patches.
      bf688f44
  3. 05 Aug, 2016 8 commits
  4. 04 Aug, 2016 2 commits
  5. 03 Aug, 2016 2 commits
  6. 02 Aug, 2016 10 commits
  7. 01 Aug, 2016 1 commit
  8. 29 Jul, 2016 3 commits
  9. 25 Jul, 2016 2 commits
  10. 19 Jul, 2016 3 commits
    • Kirill Smelkov's avatar
      golang: v↑ (1.6.3) · e770e9f7
      Kirill Smelkov authored
      From upcoming https://golang.org/doc/devel/release.html#go1.6.minor
      
          go1.6.3 (released 2016/07/17) includes security fixes to the net/http/cgi
          package and net/http package when used in a CGI environment. This release also
          adds support for macOS Sierra.  See the Go 1.6.3 milestone[1] on our issue
          tracker for details.
      
          [1] https://github.com/golang/go/issues?q=milestone%3AGo1.6.3
      
      /reviewed-by TrustMe    (tested with helloworld)
      e770e9f7
    • Kirill Smelkov's avatar
      helloweb: v↑ (39fd89a3) · ca093d0d
      Kirill Smelkov authored
      To pick up output \n and language/runtime version in output.
      
      nexedi/helloweb@0487fa7b...39fd89a3
      
      /reviewed-by TrustMe
      ca093d0d
    • Jérome Perrin's avatar
      gitlab: enable parameters-extra options when creating wrapper · 6082d6e9
      Jérome Perrin authored
      @jerome says at 5f5d5102 (comment 17119):
      
      before f4e51f77, we had:
      `~/srv/runner/instance/slappart0/bin/gitlab-rake` containing:
          ```python
          ...
          if __name__ == '__main__':
              sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
          ```
      
      after, `~/srv/runner/instance/slappart0/bin/gitlab-rake` contains:
          ```shell
          #!/bin/bash
          COMMAND=/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake.py
      
          # If the wrapped command uses a shebang, execute the referenced
          # executable passing the script path as first argument.
          # This is to workaround the limitation of 127 characters in #!
          if [[ -f $COMMAND && x$(head -c2 "$COMMAND") = x"#!" ]]; then
            SHEBANG=$(head -1 "$COMMAND")
            INTERPRETER=( ${SHEBANG#\#!} )
            COMMAND="${INTERPRETER[@]} $COMMAND"
          fi
      
          exec $COMMAND
          ```
      
      which is a wrapper around `gitlab-rake.py` containing:
          ```python
          ...
          if __name__ == '__main__':
              sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
          ```
      
      `gitlab-rake.py` after is same as `gitlab-rake` before.
      
      This [slapos.cookbook:wrapper](https://lab.nexedi.com/nexedi/slapos/blob/cd9faac0/slapos/recipe/wrapper.py#L39) has an argument *parameters-extra* which if set to true, propagate command line arguments to the wrapped script. The default value for this parameter is false.
      
      Before f4e51f77, the generated wrapper was also propagating arguments even when *parameters-extra* was not set, but since this commit, this *parameters-extra* option is now handled as expected.
      
      This is the reason for this regression. In our case, when we see `/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake assets:clean`, it just calls `rake` without arguments.
      
      So a simple patch that fix the problem would be jerome/slapos@d3d05f02 . This way, the generated wrapper becomes:
      
      ```shell
      ...
      exec $COMMAND $@
      ```
      
      and arguments are correctly propagated.
      
      Feel free to cherry-pick that patch for now, but it may be nice to rethink this *parameters-extra* option, after this debugging session, I believe it should be true by default.
      
      /cc @seb for introducing the parameter in 80bb4305 and @vpelletier for touching this code in e7083872
      
      /reviewed-by @kirr
      6082d6e9