EE: Improve performance of searching for and auto completing of users

See merge request gitlab-org/gitlab-ee!4575

CE upstream - 2018-02-23 12:10 UTC

See merge request gitlab-org/gitlab-ee!4692

# Conflicts:
#	app/controllers/groups/application_controller.rb
#	app/finders/issuable_finder.rb
#	app/policies/base_policy.rb
#	app/policies/merge_request_policy.rb
#	app/views/layouts/nav/sidebar/_group.html.haml
#	locale/gitlab.pot

[ci skip]

This optimises searching for users when using queries consisting out of
one or two characters such as "ab". We optimise such cases by searching
for `LOWER(name)` and `LOWER(username)` instead of using `ILIKE`. Using
`LOWER` produces a _much_ better performing query.

For example, when searching for all users matching the term "a" we'd
produce the following plan:

     Limit  (cost=637.69..637.74 rows=20 width=805) (actual time=41.983..41.995 rows=20 loops=1)
       Buffers: shared hit=8330
       ->  Sort  (cost=637.69..638.61 rows=368 width=805) (actual time=41.982..41.990 rows=20 loops=1)
             Sort Key: (CASE WHEN ((name)::text = 'a'::text) THEN 0 WHEN ((username)::text = 'a'::text) THEN 1 WHEN ((email)::text = 'a'::text) THEN 2 ELSE 3 END), name
             Sort Method: top-N heapsort  Memory: 35kB
             Buffers: shared hit=8330
             ->  Bitmap Heap Scan on users  (cost=75.47..627.89 rows=368 width=805) (actual time=9.452..41.305 rows=277 loops=1)
                   Recheck Cond: (((name)::text ~~* 'a'::text) OR ((username)::text ~~* 'a'::text) OR ((email)::text = 'a'::text))
                   Rows Removed by Index Recheck: 7601
                   Heap Blocks: exact=7636
                   Buffers: shared hit=8327
                   ->  BitmapOr  (cost=75.47..75.47 rows=368 width=0) (actual time=8.290..8.290 rows=0 loops=1)
                         Buffers: shared hit=691
                         ->  Bitmap Index Scan on index_users_on_name_trigram  (cost=0.00..38.85 rows=180 width=0) (actual time=4.369..4.369 rows=4071 loops=1)
                               Index Cond: ((name)::text ~~* 'a'::text)
                               Buffers: shared hit=360
                         ->  Bitmap Index Scan on index_users_on_username_trigram  (cost=0.00..34.41 rows=188 width=0) (actual time=3.896..3.896 rows=4140 loops=1)
                               Index Cond: ((username)::text ~~* 'a'::text)
                               Buffers: shared hit=328
                         ->  Bitmap Index Scan on users_email_key  (cost=0.00..1.94 rows=1 width=0) (actual time=0.022..0.022 rows=0 loops=1)
                               Index Cond: ((email)::text = 'a'::text)
                               Buffers: shared hit=3
     Planning time: 3.912 ms
     Execution time: 42.171 ms

With the changes in this commit we now produce the following plan
instead:

     Limit  (cost=13257.48..13257.53 rows=20 width=805) (actual time=1.567..1.579 rows=20 loops=1)
       Buffers: shared hit=287
       ->  Sort  (cost=13257.48..13280.93 rows=9379 width=805) (actual time=1.567..1.572 rows=20 loops=1)
             Sort Key: (CASE WHEN ((name)::text = 'a'::text) THEN 0 WHEN ((username)::text = 'a'::text) THEN 1 WHEN ((email)::text = 'a'::text) THEN 2 ELSE 3 END), name
             Sort Method: top-N heapsort  Memory: 35kB
             Buffers: shared hit=287
             ->  Bitmap Heap Scan on users  (cost=135.66..13007.91 rows=9379 width=805) (actual time=0.194..1.107 rows=277 loops=1)
                   Recheck Cond: ((lower((name)::text) = 'a'::text) OR (lower((username)::text) = 'a'::text) OR ((email)::text = 'a'::text))
                   Heap Blocks: exact=277
                   Buffers: shared hit=287
                   ->  BitmapOr  (cost=135.66..135.66 rows=9379 width=0) (actual time=0.152..0.152 rows=0 loops=1)
                         Buffers: shared hit=10
                         ->  Bitmap Index Scan on yorick_test_users  (cost=0.00..124.75 rows=9377 width=0) (actual time=0.101..0.101 rows=277 loops=1)
                               Index Cond: (lower((name)::text) = 'a'::text)
                               Buffers: shared hit=4
                         ->  Bitmap Index Scan on index_on_users_lower_username  (cost=0.00..1.94 rows=1 width=0) (actual time=0.035..0.035 rows=1 loops=1)
                               Index Cond: (lower((username)::text) = 'a'::text)
                               Buffers: shared hit=3
                         ->  Bitmap Index Scan on users_email_key  (cost=0.00..1.94 rows=1 width=0) (actual time=0.014..0.014 rows=0 loops=1)
                               Index Cond: ((email)::text = 'a'::text)
                               Buffers: shared hit=3
     Planning time: 0.303 ms
     Execution time: 1.687 ms

Here we can see the new query is 25 times faster compared to the old
query.

The frontend code doesn't use this so there's no practical point in
supporting this. We also hardcode the limit to 20 so users can no longer
request their own limit, which could overload the database (depending on
any upper bounds perhaps enforced by Kaminari).

We can instead just use a UNION. This removes the need for plucking
hundreds if not thousands of IDs into memory when a project has many
members.

CE upstream - 2018-02-23 00:25 UTC

See merge request gitlab-org/gitlab-ee!4685

Allow token authentication on go-get request

Closes #42817

See merge request gitlab-org/gitlab-ce!17148

Optimize number of queries for approvals

Closes #4903

See merge request gitlab-org/gitlab-ee!4492

Keep "Import project" tab/form active when validation fails trying to import  "Repo by URL"

Closes #43261

See merge request gitlab-org/gitlab-ce!17136

External authorization service

Closes #4216

See merge request gitlab-org/gitlab-ee!4675

Port `read_cross_project` ability from EE

See merge request gitlab-org/gitlab-ce!17208

Add Tip about Push to Create project on New Project page

Closes #42938

See merge request gitlab-org/gitlab-ce!17154

Add a button on the project page to set up a Kubernetes cluster and enable Auto DevOps -- EE merge edition

See merge request gitlab-org/gitlab-ee!4543

Enable "Legacy Authorization" by default when creating a Cluster on GCP

Closes #41619

See merge request gitlab-org/gitlab-ce!17302

Add a button on the project page to set up a Kubernetes cluster and enable  Auto DevOps

Closes #42431

See merge request gitlab-org/gitlab-ce!16900

CE upstream - 2018-02-22 18:24 UTC

Closes #4998

See merge request gitlab-org/gitlab-ee!4683

Remove bundle tag for ui help

Closes #41341

See merge request gitlab-org/gitlab-ce!17301

Add license ID number to usage ping

See merge request gitlab-org/gitlab-ee!4679

GKE 1.8 has RBAC on by default, since GKE has bumped the default version
to 1.8.7 RBAC is now on, and we don't support that out of the box.

This was also avoiding cluster applications (tiller, prometheus, etc), to
be installed.

Closes #41619

Remove ee dynamic imports

See merge request gitlab-org/gitlab-ee!4680

Resolve "cache key for CI stopped working"

Closes #43373

See merge request gitlab-org/gitlab-ce!17260

[ci skip]

[ci skip]

[ci skip]