- 21 Jan, 2016 16 commits
-
-
Kirill Smelkov authored
In previous patch we added code to serve blob content via running `git cat-file ...` directly, but for every such request a request to slow RoR-based auth backend is made, which is bad for performance. Let's cache auth backend reply for small period of time, e.g. 30 seconds, which will change the situation dramatically: If we have a lot of requests to the same repository, we query auth backend only for every Nth request and with e.g. 100 raw blob request/s N=3000 which means that previous load to RoR code essentially goes away. On the other hand as we query auth backend only once in a while and refresh the cache, we will not miss potential changes in project settings. I mean potential e.g. 25 seconds delay for a project to become public, or vise versa to become private does no real harm. The cache is done with the idea to allow the read side codepath to execute in parallel and to be not blocked by eventual cache updates. Overall this improves performance a lot: (on a 8-CPU i7-3770S with 16GB of RAM, 2001:67c:1254:e:89::fa34 is on localhost) # request is handled by gitlab-workhorse, but without auth caching $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg Running 10s test @ http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg 1 threads and 40 connections Thread Stats Avg Stdev Max +/- Stdev Latency 622.99ms 200.08ms 1.40s 77.03% Req/Sec 62.65 22.37 120.00 55.00% Latency Distribution 50% 589.51ms 75% 726.88ms 90% 896.09ms 99% 1.18s 626 requests in 10.01s, 1.11MB read Requests/sec: 62.55 Transfer/sec: 113.73KB # request goes to gitlab-workhorse with auth caching (this patch) $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg Running 10s test @ http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg 1 threads and 40 connections Thread Stats Avg Stdev Max +/- Stdev Latency 36.62ms 25.39ms 351.14ms 72.02% Req/Sec 1.16k 93.73 1.36k 77.00% Latency Distribution 50% 36.30ms 75% 47.02ms 90% 66.36ms 99% 122.46ms 11580 requests in 10.01s, 20.56MB read Requests/sec: 1156.85 Transfer/sec: 2.05MB i.e. it is ~ 17x improvement.
-
Kirill Smelkov authored
Currently GitLab serves requests to get raw blobs via Ruby-on-Rails code and Unicorn. Because RoR/Unicorn is relatively heavyweight, in environment where there are a lot of simultaneous requests to get raw blobs, this works very slow and server is constantly overloaded. On the other hand, to get raw blob content, we do not need anything from RoR framework - we only need to have access to project git repository on filesystem, and knowing whether access for getting data from there should be granted or not. That means it is possible to handle '.../raw/....' request directly in more lightweight and performant gitlab-workhorse. As gitlab-workhorse is written in Go, and Go has good concurrency/parallelism support and is generally much faster than Ruby, moving raw blob serving task to it makes sense and should be a net win. In this patch: we add infrastructure to process GET request for '/raw/...': - extract project / ref and path from URL - query auth backend for whether download access should be granted or not - emit blob content via spawning external `git cat-file` I've tried to mimic the output to be as close as the one emitted by RoR code, with the idea that for users the change should be transparent. As in this patch we do auth backend query for every request to get a blob, RoR code is still loaded very much, so essentially there is no speedup yet: (on a 8-CPU i7-3770S with 16GB of RAM, 2001:67c:1254:e:89::fa34 is on localhost) # without patch: request eventually goes to unicorn (9 unicorn workers) $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg Running 10s test @ http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg 1 threads and 40 connections Thread Stats Avg Stdev Max +/- Stdev Latency 609.34ms 156.92ms 1.18s 79.60% Req/Sec 64.22 19.90 120.00 67.00% Latency Distribution 50% 596.50ms 75% 678.23ms 90% 805.72ms 99% 1.04s 642 requests in 10.01s, 1.24MB read Requests/sec: 64.16 Transfer/sec: 127.00KB # with this patch: request handled by gitlab-workhorse $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg Running 10s test @ http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg 1 threads and 40 connections Thread Stats Avg Stdev Max +/- Stdev Latency 622.99ms 200.08ms 1.40s 77.03% Req/Sec 62.65 22.37 120.00 55.00% Latency Distribution 50% 589.51ms 75% 726.88ms 90% 896.09ms 99% 1.18s 626 requests in 10.01s, 1.11MB read Requests/sec: 62.55 Transfer/sec: 113.73KB In the next patch we'll cache requests to auth backend and that will improve performance dramatically.
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
Use gitlab-zip-cat to send zip entries Fixes https://gitlab.com/gitlab-org/gitlab-workhorse/issues/17 See merge request !31
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
Configure URLPrefix before building routes Fixes #19 See merge request !32
-
Jacob Vosmaer authored
-
- 20 Jan, 2016 3 commits
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
Add support for new artifacts features It is proof of concept implementation that misses following: - [x] Saving metadata to file, and passing that file to rails, - [x] Making it look nice, - [x] Have `refactor-upstream` merged: https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/20 - [x] Add tests for metadata and single file downloads, - [ ] Spawning separate process to verify zip archive **ONHOLD**: https://gitlab.com/gitlab-org/gitlab-workhorse/issues/17, - [ ] Spawning separate process to download file from zip archive **ONHOLD**: https://gitlab.com/gitlab-org/gitlab-workhorse/issues/17 See merge request !28
-
- 19 Jan, 2016 4 commits
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
This test duplicates some of what is in internal/artifacts but @grzesiek and I felt it was important to have an integration test that also covers the route to this resource.
-
Jacob Vosmaer authored
-
Grzegorz Bizon authored
-
- 18 Jan, 2016 8 commits
-
-
Kamil Trzcinski authored
Uses escaper from mime/multipart/writer.go.
-
Kamil Trzcinski authored
-
Kamil Trzcinski authored
* commit '3bbb1be7': Version 0.6.0 Gotta commit those moved files Develomentmode ended up the wrong way around Move test helpers into seperate package Prevent package testing from going into the build
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
Develomentmode ended up the wrong way around See merge request !29
-
Grzegorz Bizon authored
-
Grzegorz Bizon authored
-
Grzegorz Bizon authored
-
- 15 Jan, 2016 6 commits
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
Refactor upstream Rationale: the code has become a tangled mess of global variables and types that hang together when they need not. For example: every HTTP handler uses a 'gitRequest'?? I want to clean this up and see if I can move some things into internal packages. Apart from using internal packages we now use http.Handler where we can, and fewer global variables. See merge request !20
-
- 14 Jan, 2016 3 commits
-
-
Kamil Trzcinski authored
-
Kamil Trzcinski authored
-
Grzegorz Bizon authored
-