• Kirill Smelkov's avatar
    NXD blob/auth: Cache auth backend reply for 30s · 19649275
    Kirill Smelkov authored
    [ Sent upstream: https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17
    
      This patch was sent upstream but was not accepted for "complexity"
      reason of auth cache, despite that provides more than an order of magnitude
      speedup. Just carry it with us as NXD ]
    
    In previous patch we added code to serve blob content via running `git cat-file
    ...` directly, but for every such request a request to slow RoR-based auth
    backend is made, which is bad for performance.
    
    Let's cache auth backend reply for small period of time, e.g. 30 seconds, which
    will change the situation dramatically:
    
    If we have a lot of requests to the same repository, we query auth backend only
    for every Nth request and with e.g. 100 raw blob request/s N=3000 which means
    that previous load to RoR code essentially goes away.
    
    On the other hand as we query auth backend only once in a while and refresh the
    cache, we will not miss potential changes in project settings. I mean potential
    e.g. 25 seconds delay for a project to become public, or vise versa to become
    private does no real harm.
    
    The cache is done with the idea to allow the read side codepath to execute in
    parallel and to be not blocked by eventual cache updates.
    
    Overall this improves performance a lot:
    
      (on a 8-CPU i7-3770S with 16GB of RAM, 2001:67c:1254:e:8b::c776 is on localhost)
    
      # request is handled by gitlab-workhorse, but without auth caching
      $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:8b::c776]:7777/nexedi/slapos/raw/master/software/wendelin/software.cfg
      Running 10s test @ http://[2001:67c:1254:e:8b::c776]:7777/nexedi/slapos/raw/master/software/wendelin/software.cfg
        1 threads and 40 connections
        Thread Stats   Avg      Stdev     Max   +/- Stdev
          Latency   458.42ms   66.26ms 766.12ms   84.76%
          Req/Sec    85.38     16.59   120.00     82.00%
        Latency Distribution
           50%  459.26ms
           75%  490.09ms
           90%  523.95ms
           99%  611.33ms
        853 requests in 10.01s, 1.51MB read
      Requests/sec:     85.18
      Transfer/sec:    154.90KB
    
      # request goes to gitlab-workhorse with auth caching (this patch)
      $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:8b::c776]:7777/nexedi/slapos/raw/master/software/wendelin/software.cfg
      Running 10s test @ http://[2001:67c:1254:e:8b::c776]:7777/nexedi/slapos/raw/master/software/wendelin/software.cfg
        1 threads and 40 connections
        Thread Stats   Avg      Stdev     Max   +/- Stdev
          Latency    34.52ms   19.28ms 288.63ms   74.74%
          Req/Sec     1.20k   127.21     1.39k    85.00%
        Latency Distribution
           50%   32.67ms
           75%   42.73ms
           90%   56.26ms
           99%   99.86ms
        11961 requests in 10.01s, 21.24MB read
      Requests/sec:   1194.51
      Transfer/sec:      2.12MB
    
    i.e. it is ~ 14x improvement.
    19649275
api.go 5.51 KB