Add support to connect gitlab-shell to Unicorn via UNIX socket

It is well known that UNIX sockets are faster than TCP over loopback.

E.g. on my machine according to lmbench[1] they have ~ 2 times
lower latency and ~ 2-3 times more throughput compared to TCP over
loopback:

    *Local* Communication latencies in microseconds - smaller is better
    ---------------------------------------------------------------------
    Host                 OS 2p/0K  Pipe AF     UDP  RPC/   TCP  RPC/ TCP
                            ctxsw       UNIX         UDP         TCP conn
    --------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
    teco      Linux 4.2.0-1  13.8  29.2 26.8  45.0  47.9  48.5  55.5  45.

    *Local* Communication bandwidths in MB/s - bigger is better
    -----------------------------------------------------------------------------
    Host                OS  Pipe AF    TCP  File   Mmap  Bcopy  Bcopy  Mem   Mem
                                 UNIX      reread reread (libc) (hand) read write
    --------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
    teco      Linux 4.2.0-1 1084 4353 1493 2329.1 3720.7 1613.8 1109.2 3402 1404.

The same ratio usually holds for servers.

Also UNIX sockets, since they reside on filesystem, besides being faster with
less latency, have one another nice property: access permissions to them are
managed the same way access to files is.

Because of lower latencies and higher throughput - for performance reasons, and
for easier security, it makes sense to interconnect services on one machine via
UNIX sockets and talk via TCP only to outside world.

All internal services inside GitLab can talk to each other via UNIX socket
already and only gitlab-shell was missing support to talk to Unicorn via UNIX
socket.

Let's teach gitlab-shell to talk via UNIX sockets.

[1] http://www.bitmover.com/lmbench/

~~~~

In this patch we

- add URI::HTTPUNIX to handle http+unix:// URI scheme
- add Net::HTTPUNIX to handle "connect via unix socket and then talk http"
- adjust GitlabNet#http_client_for() accordingly
- adjust documentation in config.yml.example

The http+unix:// scheme is not reinvented anew: the idea about its structure is
quite logical an was already established at least in requests-unixsocket python
package:

    http://fixall.online/theres-no-need-to-reinvent-the-wheelhttpsgithubcommsabramorequests-unixsocketurl/241810/
    https://github.com/msabramo/requests-unixsocket

config.yml.example

@@ -10,7 +10,9 @@ user: git
 # Default: http://localhost:8080/
 # You only have to change the default if you have configured Unicorn
 # to listen on a custom port, or if you have configured Unicorn to
-# only listen on a Unix domain socket.
+# only listen on a Unix domain socket. For Unix domain sockets use
+# "http+unix://<urlquoted-path-to-socket>/", e.g.
+# "http+unix://%2Fpath%2Fto%2Fsocket/"
 gitlab_url: "http://localhost:8080/"
 
 # See installation.md#using-https for additional HTTPS configuration details.

lib/gitlab_net.rb

@@ -5,6 +5,7 @@ require 'json'
 require_relative 'gitlab_config'
 require_relative 'gitlab_logger'
 require_relative 'gitlab_access'
+require_relative 'httpunix'
 
 class GitlabNet
   class ApiUnreachableError < StandardError; end
@@ -63,7 +64,11 @@ class GitlabNet
   end
 
   def http_client_for(uri)
-    http = Net::HTTP.new(uri.host, uri.port)
+    if uri.is_a?(URI::HTTPUNIX)
+      http = Net::HTTPUNIX.new(uri.hostname)
+    else
+      http = Net::HTTP.new(uri.host, uri.port)
+    end
 
     if uri.is_a?(URI::HTTPS)
       http.use_ssl = true

lib/httpunix.rb

+# support for http+unix://... connection scheme
+#
+# The URI scheme has the same structure as the similar one for python requests. See:
+#   http://fixall.online/theres-no-need-to-reinvent-the-wheelhttpsgithubcommsabramorequests-unixsocketurl/241810/
+#   https://github.com/msabramo/requests-unixsocket
+
+require 'uri'
+require 'net/http'
+
+module URI
+  class HTTPUNIX < HTTP
+    def hostname
+      # decode %XX from path to file
+      v = self.host
+      URI.decode(v)
+    end
+
+    # port is not allowed in URI
+    DEFAULT_PORT = nil
+    def set_port(v)
+      return v unless v
+      raise InvalidURIError, "http+unix:// cannot contain port"
+    end
+  end
+  @@schemes['HTTP+UNIX'] = HTTPUNIX
+end
+
+# Based on:
+# - http://stackoverflow.com/questions/15637226/ruby-1-9-3-simple-get-request-to-unicorn-through-socket
+# - Net::HTTP::connect
+module Net
+  class HTTPUNIX < HTTP
+    def initialize(socketpath, port=nil)
+      super(socketpath, port)
+      @port    = nil    # HTTP will set it to default - override back -> set DEFAULT_PORT
+    end
+
+    # override to prevent ":<port>" being appended to HTTP_HOST
+    def addr_port
+      address
+    end
+
+    def connect
+      D "opening connection to #{address} ..."
+      s = UNIXSocket.new(address)
+      D "opened"
+      @socket = BufferedIO.new(s)
+      @socket.read_timeout = @read_timeout
+      @socket.continue_timeout = @continue_timeout
+      @socket.debug_output = @debug_output
+      on_connect
+    end
+  end
+end

spec/httpunix_spec.rb

+require_relative 'spec_helper'
+require_relative '../lib/httpunix'
+require 'webrick'
+
+describe URI::HTTPUNIX do
+  describe :parse do
+    uri = URI::parse('http+unix://%2Fpath%2Fto%2Fsocket/img.jpg')
+    subject { uri }
+
+    it { should be_an_instance_of(URI::HTTPUNIX) }
+    its(:scheme) { should eq('http+unix') }
+    its(:hostname) { should eq('/path/to/socket') }
+    its(:path) { should eq('/img.jpg') }
+  end
+end
+
+
+# like WEBrick::HTTPServer, but listens on UNIX socket
+class HTTPUNIXServer < WEBrick::HTTPServer
+  def listen(address, port)
+    socket = Socket.unix_server_socket(address)
+    socket.autoclose = false
+    server = UNIXServer.for_fd(socket.fileno)
+    socket.close
+    @listeners << server
+  end
+end
+
+def tmp_socket_path
+  File.join(ROOT_PATH, 'tmp', 'socket')
+end
+
+describe Net::HTTPUNIX do
+  # "hello world" over unix socket server in background thread
+  FileUtils.mkdir_p(File.dirname(tmp_socket_path))
+  server = HTTPUNIXServer.new(:BindAddress => tmp_socket_path)
+  server.mount_proc '/' do |req, resp|
+    resp.body = "Hello World  (at #{req.path})"
+  end
+  Thread.start { server.start }
+
+  it "talks via HTTP ok" do
+    VCR.turned_off do
+      begin
+        WebMock.allow_net_connect!
+        http = Net::HTTPUNIX.new(tmp_socket_path)
+        expect(http.get('/').body).to eq('Hello World  (at /)')
+        expect(http.get('/path').body).to eq('Hello World  (at /path)')
+
+      ensure
+        WebMock.disable_net_connect!
+      end
+    end
+  end
+end