Merge branch 'y/httpunix2' into 'master'

Add support to connect gitlab-shell to Unicorn via UNIX socket  (v2)

Hello up there.

I'm doing SlapOS port of GitLab, and that means several different services could be running on the same machine, including several GitLabs.

So far all internal GitLab subservices could be glued together via UNIX sockets except gitlab-shell -> Unicorn link, which, when done via local TCP, requires firewall/network namespaces to protect services on one machine from each other.

On the other hand access to UNIX domain sockets is managed via regular UNIX permissions on filesystem, and thus is easier to manage. Besides UNIX domain sockets are well known to be faster compared to TCP over loopback - in particular to have ~ 2 times less latency and ~ 2 times more throughput.

From this point of view it makes sense to teach gitlab-shell to talk to Unicorn via UNIX socket and switch to that mode by default eventually.

I've just made a patch for this. Please apply.

Thanks beforehand,  
Kirill

/cc @dzaporozhets, @jacobvosmaer, @rspeicher 

See merge request !30

config.yml.example

@@ -10,7 +10,9 @@ user: git
 # Default: http://localhost:8080/
 # You only have to change the default if you have configured Unicorn
 # to listen on a custom port, or if you have configured Unicorn to
-# only listen on a Unix domain socket.
+# only listen on a Unix domain socket. For Unix domain sockets use
+# "http+unix://<urlquoted-path-to-socket>/", e.g.
+# "http+unix://%2Fpath%2Fto%2Fsocket/"
 gitlab_url: "http://localhost:8080/"
 
 # See installation.md#using-https for additional HTTPS configuration details.

lib/gitlab_net.rb

@@ -5,6 +5,7 @@ require 'json'
 require_relative 'gitlab_config'
 require_relative 'gitlab_logger'
 require_relative 'gitlab_access'
+require_relative 'httpunix'
 
 class GitlabNet
   class ApiUnreachableError < StandardError; end
@@ -63,7 +64,11 @@ class GitlabNet
   end
 
   def http_client_for(uri)
-    http = Net::HTTP.new(uri.host, uri.port)
+    if uri.is_a?(URI::HTTPUNIX)
+      http = Net::HTTPUNIX.new(uri.hostname)
+    else
+      http = Net::HTTP.new(uri.host, uri.port)
+    end
 
     if uri.is_a?(URI::HTTPS)
       http.use_ssl = true

lib/httpunix.rb

+# support for http+unix://... connection scheme
+#
+# The URI scheme has the same structure as the similar one for python requests. See:
+#   http://fixall.online/theres-no-need-to-reinvent-the-wheelhttpsgithubcommsabramorequests-unixsocketurl/241810/
+#   https://github.com/msabramo/requests-unixsocket
+
+require 'uri'
+require 'net/http'
+
+module URI
+  class HTTPUNIX < HTTP
+    def hostname
+      # decode %XX from path to file
+      v = self.host
+      URI.decode(v)
+    end
+
+    # port is not allowed in URI
+    DEFAULT_PORT = nil
+    def set_port(v)
+      return v unless v
+      raise InvalidURIError, "http+unix:// cannot contain port"
+    end
+  end
+  @@schemes['HTTP+UNIX'] = HTTPUNIX
+end
+
+# Based on:
+# - http://stackoverflow.com/questions/15637226/ruby-1-9-3-simple-get-request-to-unicorn-through-socket
+# - Net::HTTP::connect
+module Net
+  class HTTPUNIX < HTTP
+    def initialize(socketpath, port=nil)
+      super(socketpath, port)
+      @port    = nil    # HTTP will set it to default - override back -> set DEFAULT_PORT
+    end
+
+    # override to prevent ":<port>" being appended to HTTP_HOST
+    def addr_port
+      address
+    end
+
+    def connect
+      D "opening connection to #{address} ..."
+      s = UNIXSocket.new(address)
+      D "opened"
+      @socket = BufferedIO.new(s)
+      @socket.read_timeout = @read_timeout
+      @socket.continue_timeout = @continue_timeout
+      @socket.debug_output = @debug_output
+      on_connect
+    end
+  end
+end

spec/httpunix_spec.rb

+require_relative 'spec_helper'
+require_relative '../lib/httpunix'
+require 'webrick'
+
+describe URI::HTTPUNIX do
+  describe :parse do
+    uri = URI::parse('http+unix://%2Fpath%2Fto%2Fsocket/img.jpg')
+    subject { uri }
+
+    it { should be_an_instance_of(URI::HTTPUNIX) }
+    its(:scheme) { should eq('http+unix') }
+    its(:hostname) { should eq('/path/to/socket') }
+    its(:path) { should eq('/img.jpg') }
+  end
+end
+
+
+# like WEBrick::HTTPServer, but listens on UNIX socket
+class HTTPUNIXServer < WEBrick::HTTPServer
+  def listen(address, port)
+    socket = Socket.unix_server_socket(address)
+    socket.autoclose = false
+    server = UNIXServer.for_fd(socket.fileno)
+    socket.close
+    @listeners << server
+  end
+end
+
+def tmp_socket_path
+  File.join(ROOT_PATH, 'tmp', 'socket')
+end
+
+describe Net::HTTPUNIX do
+  # "hello world" over unix socket server in background thread
+  FileUtils.mkdir_p(File.dirname(tmp_socket_path))
+  server = HTTPUNIXServer.new(:BindAddress => tmp_socket_path)
+  server.mount_proc '/' do |req, resp|
+    resp.body = "Hello World  (at #{req.path})"
+  end
+  Thread.start { server.start }
+
+  it "talks via HTTP ok" do
+    VCR.turned_off do
+      begin
+        WebMock.allow_net_connect!
+        http = Net::HTTPUNIX.new(tmp_socket_path)
+        expect(http.get('/').body).to eq('Hello World  (at /)')
+        expect(http.get('/path').body).to eq('Hello World  (at /path)')
+
+      ensure
+        WebMock.disable_net_connect!
+      end
+    end
+  end
+end