Allow the Rails cookie to be used for API authentication

Makes the Rails cookie into a valid authentication token for the Grape
API, and uses it instead of token authentication in frontend code that
uses the API.

Rendering the private token into client-side javascript is a security
risk; it may be stolen through XSS or other attacks. In general,
re-using API code in the frontend is more desirable than implementing
endless actions that return JSON.

Closes #18302

See merge request !1995
Signed-off-by: Rémy Coutable <remy@rymai.me>

[ci skip]
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Exclude some pending or inactivated rows in Member scopes

An unapproved request or not-yet-accepted invite should not give access rights. Neither should a blocked user be considered a member of anything.

One visible outcome of this behaviour is that owners and masters of a group or project may be blocked, yet still receive notification emails for access requests.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/21650

See merge request !1994
Signed-off-by: Rémy Coutable <remy@rymai.me>

lib/backup: fix broken permissions when creating repo dir

## What does this MR do?

This MR fixes an issue where gitlab:backup:restore will not work because the repositories directory is set up with bad permissions. The bad permissions will prevent access to the repositories by the git user, causing all kinds of troubles (e.g. gitlab-shell won't install hooks).

## Are there points in the code the reviewer needs to double check?

## Why was this MR needed?

Rake task gitlab:backup:restore would complete successfully, but gitlab:check will report errors and gitlab cannot access repos.

## Screenshots (if relevant)

* repositories.old.1472478726 was created without the fix
* repositories was created with the fix

![gitlab-repo-permissions](/uploads/6cf2de57fe4f1be0bd5b5e1a13234889/gitlab-repo-permissions.png)

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- Tests
  - [ ] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?

See merge request !6098

Update doorkeeper to 4.2.0

Changelog: https://git.io/v6PnV

See merge request !5881
(cherry picked from commit c5aa31c8)

Update Hamlit to 2.6.1

Fixes gitlab-org/gitlab-ce#21025 and gitlab-org/gitlab-ce#21017

See merge request !5873
(cherry picked from commit e26ce27d)

Allow to sort by project size on admin projects page

Fixes #20317

See merge request !5797
(cherry picked from commit dd2088b0)

Set permissions to admin for importing a project via Import/Export

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/20802

In order to import a project, it is now required to be an admin

Moved from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5766

See merge request !1983
(cherry picked from commit 966b3038)

Fix attribute inclusion import/export config ignored in some cases

In the `import_export.yml` file we define the inclusion of some of the attributes. For some reason, this isn't working in certain cases - very unfortunate this includes `user`. This has been introduced in 8.10.3.

Related https://gitlab.com/gitlab-org/gitlab-ce/issues/20802

See merge request !1982

[ci skip]

Upgrade Rails to 4.2.7.1 for security fixes.

Upgrades Rails from 4.2.7 to 4.2.7.1 for security fixes.

For more information: http://weblog.rubyonrails.org/2016/8/11/Rails-5-0-0-1-4-2-7-2-and-3-2-22-3-have-been-released/

This should be backported to all currently-supported releases.

See merge request !5781

Used cached value of project count to reduce DB load

Following !5746, backport a change from https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/637/diffs#diff-2.

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5754
Signed-off-by: Rémy Coutable <remy@rymai.me>

Add a method in Project to return a cached value of total count of projects

This is in preparation to address the DB load caused by the counting
in gitlab-com/infrastructure#303.

See merge request !5746

Revert "Defend against 'Host' header injection"

We reverted this in omnibus but forgot to do it in the source NGINX template.

See merge request !5706
Signed-off-by: Rémy Coutable <remy@rymai.me>

Add a data migration to fix some missing timestamps in the members table (again)

Repeats an earlier migration to fix historic bad data in the members table (missing created_at and updated_at fields)

I'm expecting the WHERE clauses to be fast enough, and to return few enough rows, that the migration doesn't need to use batches, but I'm not too familiar with the size of these tables in the wild, so perhaps that's a poor assumption.

8.10 introduced a dependency on the  `members.created_at` field in the project and namespace member view. If bad data is present, viewing the list of members now results in an NoMethodError and a 500 response from GitLab. Although the previous migration should have fixed all bad rows, we have evidence that it didn't in at least one case, despite the migration claiming to have run in the past.

Closes #20568

See merge request !5670
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix Import/Export error checking versions

Fixes small bug preventing the correct error message about Import/Export version being displayed.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/20536

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5638
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix Import/Export not working in HA mode

Use a shared path instead of `Tempfile` default `/tmp` so the import file is accessible by any GitLab instance.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/20506

See merge request !5618
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Don’t close issues on original project

Closes #20527 -> https://gitlab.com/gitlab-org/gitlab-ce/issues/20527

See merge request !1981
Signed-off-by: Rémy Coutable <remy@rymai.me>

Dropdown enter key fix

## What does this MR do?

Fixes issue with dropdowns enter key not working correctly

See merge request !5544

Signed-off-by: Rémy Coutable <remy@rymai.me>

Trim extra displayed carriage returns in diffs and files with CRLFs

## What does this MR do?

This MR strips out extra carriage returns at the end of each tokenized string.

## Are there points in the code the reviewer needs to double check?

Does this remove more than we need?

## Why was this MR needed?

Windows users would see extra lines in their files. Newline characters are already added with every line in the HTML formatter.

## What are the relevant issue numbers?

#20440, gitlab-com/support-forum#892, #20468

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] Added for this feature/bug
  - [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5588
Signed-off-by: Rémy Coutable <remy@rymai.me>

Ignore invalid IPs in X-Forwarded-For when trusted proxies are configured.

## What does this MR do?

Catches IPAddr::InvalidAddressError exceptions in `trusted_proxy?` when a) a trusted proxy is set up in the gitlab config and b) an invalid IP address is passed to the method (e.g. one with a port attached). When caught, returns `false` from the method. Prevents a 500 error in this situation.

## What are the relevant issue numbers?

Closes gitlab-org/gitlab-ce#20466.

## Does this MR meet the acceptance criteria?

- [X] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [N/A] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [N/A] API support added
- Tests
  - [X] Added for this feature/bug
  - [X] All builds are passing
- [X] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [X] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [X] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5584
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix attr reader to force the intended values for source and target shas

## What does this MR do?

When importing a pull request from GitHub, the old and new branches may no longer actually exist by those names, but we need to recreate the merge
request diff with the right source and target shas.  We use these `target_branch_sha` and `source_branch_sha` attributes to force  these to the intended values. But the reader methods were always looking up to the target/source branch head instead of check if these values was  previously set, this MR applies a memoization pattern to both of them.

## Are there points in the code the reviewer needs to double check?

This [commit](https://gitlab.com/gitlab-org/gitlab-ce/commit/6ce25e7b4caa9e94de74378729178c7060d640b2) introduced this bug in the importer.

## What are the relevant issue numbers?

gitlab-org/gitlab-ce#20385

## Does this MR meet the acceptance criteria?

- [X] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [X] ~~[Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~
- [X] ~~API support added~~
- Tests
  - [X] Added for this feature/bug
  - [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

/cc @akitaonrails @DouweM

See merge request !5573
Signed-off-by: Rémy Coutable <remy@rymai.me>

Properly abort a merge when merge conflicts occur

If somehow a user attempted to accept a merge request that had conflicts (e.g. the "Accept Merge Request" button or the MR itself was not updated), `MergeService` did not properly detect that a conflict
occurred. It would assume that the MR went through without any issues and close the MR as though everything was fine. This could cause data loss if the source branch were removed.

Closes #20425

See merge request !5569
Signed-off-by: Rémy Coutable <remy@rymai.me>

fix repo hooks missing on import

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/19556

Repo hooks are missing from imported projects

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5549
Signed-off-by: Rémy Coutable <remy@rymai.me>

Add a log message when a project is scheduled for destruction for debugging

We have a lot of projects that are in `pending_delete` state. It's not clear whether they were ever scheduled for destruction, or whether Sidekiq just dropped the job due to `MemoryKiller` or some other reason.

Also this will provide a record of which user destroys a project.

#20365

See merge request !5540
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix timing problems running imports on production

Fixes https://gitlab.com/gitlab-com/infrastructure/issues/151

I've found out that in staging, the imported file is not copied fully by the time sidekiq runs the job, this should hopefully fix it. (Tested against staging).

See merge request !5523
Signed-off-by: Rémy Coutable <remy@rymai.me>