Increase HTTP timeout and log request durations

On some GitLab deployments internal API calls regularly take more than
60 seconds (the default HTTP read timeout of Ruby's Net::HTTP). Until
we understand the cause of this slowness, by raising the client
timeout in gitlab-shell we can at least spare end users having to
retry their `git pull` or `git push`.

See merge request !37

Add relative URL info

Relevant MR https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2770

See merge request !36

[ci skip]

Added git gc for housekeeping

This merge request will add the gc functionality used by the housekeeping function in the project settings page.

see gitlab-org/gitlab-ce#3041 and https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1658

See merge request !23

Conflicts:
	lib/gitlab_projects.rb

Remove trailing slashes from gitlab_url

They do not play nice with gitlab-workhorse (or rather Golang net/http
DefaultServemux).

See merge request !35

They do not play nice with gitlab-workhorse (or rather Golang net/http
DefaultServemux).

Stricter exec cmd

In response to the gitlab-shell 2.6.6-2.6.7 remote code execution
vulnerability.

See merge request !33

Pass $HOME to git as well

[ this patch has the same rationale and reasoning as
  https://gitlab.com/gitlab-org/gitlab-workhorse/commit/0d0bd209

  details follow ]

Git has 3 places for configs:

    - system
    - global (per user), and
    - local  (per repository)

System config location is hardcoded at git compile time (to usually
$prefix/etc/gitconfig). Local configuration is usually picked because we
pass full repo path to subcommand. But global configuration is currently not
picked at all, because HOME env variable is not passed to git.

Pass $HOME through and let git see it's "global" config.

Currently GitLab omnibus stores gitlab user name/email  + "autocrlf =
true" in global config, so missing it should not be a blocker for
receive/send-pack operations. But having it is more correct and can be
handy in the future if/when more git operations are done from-under
gitlab-shell.

Having $HOME properly set is also needed when one cannot change system
git config and have to put site-wide configuration into global git
config under $HOME.

That was the case I've hit and the reason for this patch.

/cc @dzaporozhets, @jacobvosmaer

See merge request !32

Hoping this makes it more obvious when code touches the very
unsafe contents of this variable.

Passing strings to Kernel::exec leads to remote code execution.

Revert "Run git-lfs-authenticate script with original command line arguments"

This reverts commit 8449979f.
Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Related to: #35, gitlab/gitlabhq#2635

See merge request !68

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

This reverts commit 8449979f.
Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

[ this patch has the same rationale and reasoning as
  https://gitlab.com/gitlab-org/gitlab-workhorse/commit/0d0bd209

  details follow ]

Git has 3 places for configs:

    - system
    - global (per user), and
    - local  (per repository)

System config location is hardcoded at git compile time (to usually
$prefix/etc/gitconfig). Local configuration is usually picked because we
pass full repo path to subcommand. But global configuration is currently not
picked at all, because HOME env variable is not passed to git.

Pass $HOME through and let git see it's "global" config.

Currently GitLab omnibus stores gitlab user name/email  + "autocrlf =
true" in global config, so missing it should not be a blocker for
receive/send-pack operations. But having it is more correct and can be
handy in the future if/when more git operations are done from-under
gitlab-shell.

Having $HOME properly set is also needed when one cannot change system
git config and have to put site-wide configuration into global git
config under $HOME.

That was the case I've hit and the reason for this patch.

Add fetch-remote command for repo mirroring

Also exits `import-repository` with non-zero status when import fails.

See merge request !29

This reverts commit ae498b6c, reversing
changes made to 79fdf65c.

Add support to connect gitlab-shell to Unicorn via UNIX socket  (v2)

Hello up there.

I'm doing SlapOS port of GitLab, and that means several different services could be running on the same machine, including several GitLabs.

So far all internal GitLab subservices could be glued together via UNIX sockets except gitlab-shell -> Unicorn link, which, when done via local TCP, requires firewall/network namespaces to protect services on one machine from each other.

On the other hand access to UNIX domain sockets is managed via regular UNIX permissions on filesystem, and thus is easier to manage. Besides UNIX domain sockets are well known to be faster compared to TCP over loopback - in particular to have ~ 2 times less latency and ~ 2 times more throughput.

From this point of view it makes sense to teach gitlab-shell to talk to Unicorn via UNIX socket and switch to that mode by default eventually.

I've just made a patch for this. Please apply.

Thanks beforehand,  
Kirill

/cc @dzaporozhets, @jacobvosmaer, @rspeicher 

See merge request !30

Use load path, set ROOT_PATH via ENV

Uses load path instead of require_relative, adds an environment variable to set gitlab-shell ROOT_PATH variable (Fixes #30)

See merge request !28

It is well known that UNIX sockets are faster than TCP over loopback.

E.g. on my machine according to lmbench[1] they have ~ 2 times
lower latency and ~ 2-3 times more throughput compared to TCP over
loopback:

    *Local* Communication latencies in microseconds - smaller is better
    ---------------------------------------------------------------------
    Host                 OS 2p/0K  Pipe AF     UDP  RPC/   TCP  RPC/ TCP
                            ctxsw       UNIX         UDP         TCP conn
    --------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
    teco      Linux 4.2.0-1  13.8  29.2 26.8  45.0  47.9  48.5  55.5  45.

    *Local* Communication bandwidths in MB/s - bigger is better
    -----------------------------------------------------------------------------
    Host                OS  Pipe AF    TCP  File   Mmap  Bcopy  Bcopy  Mem   Mem
                                 UNIX      reread reread (libc) (hand) read write
    --------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
    teco      Linux 4.2.0-1 1084 4353 1493 2329.1 3720.7 1613.8 1109.2 3402 1404.

The same ratio usually holds for servers.

Also UNIX sockets, since they reside on filesystem, besides being faster with
less latency, have one another nice property: access permissions to them are
managed the same way access to files is.

Because of lower latencies and higher throughput - for performance reasons, and
for easier security, it makes sense to interconnect services on one machine via
UNIX sockets and talk via TCP only to outside world.

All internal services inside GitLab can talk to each other via UNIX socket
already and only gitlab-shell was missing support to talk to Unicorn via UNIX
socket.

Let's teach gitlab-shell to talk via UNIX sockets.

[1] http://www.bitmover.com/lmbench/

~~~~

In this patch we

- add URI::HTTPUNIX to handle http+unix:// URI scheme
- add Net::HTTPUNIX to handle "connect via unix socket and then talk http"
- adjust GitlabNet#http_client_for() accordingly
- adjust documentation in config.yml.example

The http+unix:// scheme is not reinvented anew: the idea about its structure is
quite logical an was already established at least in requests-unixsocket python
package:

    http://fixall.online/theres-no-need-to-reinvent-the-wheelhttpsgithubcommsabramorequests-unixsocketurl/241810/
    https://github.com/msabramo/requests-unixsocket