If Redis were not running or USE_DB were set to false, the
application settings retrieval would fail completely. This
change only attempts to use the cache if the system actually
wants to connect to the DB and rescues any failures in talking to
Redis.

Closes #17557

(cherry picked from commit c600cf83)

Teach GitLab not only to merge changes from a merge-request, but also to
apply patches posted to merge-request in a way like `git am` would do -
without merge commit and directly on top of current branch. Which way to
go is selected by user in web UI, and apply patches is the first option.

There are 3 cases:

- only 1 commit is present in MR -> the only available option is to
  apply that single commit as one patch without a merge

  ( There is no need for merge commit in this case at all: information
    about user who applied the patch goes to "Committer" field in resultant
    commit. Avoiding 1 merge per 1 patch results in cleaner history )

  It is also possible to review patch description directly in web UI,
  before doing the actual application, and correct / amend it as needed.

- several commits are present in MR:

  * it is possible to apply the patches directly on top of current
    branch. Again information about who applied what goes to "Committer"
    field.

  * it is possible to merge MR changes with making a merge commit.

    This variant is useful, when patches from a MR do several logical
    steps to reach one goal, and MR description contain cover letter for
    whole patch series.

    in this case original commits stay untouched and resulting merge
    will contain MR author as author, user who accepted MR as committer,
    and cover letter as merge commit message.

    NOTE we avoid useless "Merge branch X into Y" in merge message, and
        just put MR title into merge subject and MR description into merge
        description.

        This way it is more logical with more important information in
        merge subject and thus e.g. more handy to oversee what a merge brings,
        just by it subject, e.g. via looking at updates via

            gitk --first-parent ...

        or via web.

NOTE for pre-generated references to merge-request we now use full MR
    URL, instead of !<MR-n>. Full URLs work everywhere, not only on
    original site where MR was created, or even only in original repo
    and not its fork on the same site.

TODO detect whether request comes from China and only then show ICP (?).

We show in small font size the same info that is shown on sign_in page:

    "GitLab Nexedi Edition", "About GitLab" and "About Nexedi"

This is good to have and hereby-introduced about-footer area will be
also used in the next patch for ICP too.

XXX placement of .about-footer to be near bottom is done not very
correctly.

Like Omnibus, SlapOS version does not have init script - nothing to
check here.

This is handy for monitoring tools, which could e.g. periodically call check
tasks and instead of parsing output, rely on exit code.

The way we detect if something failed is via hooking into String#red, and if
anything was ever printed in red - that's an error.

The default was switched to HTTP in the previous patch, but let's completely
remove SSH option - we support only HTTP for git fetch/push.

Both fetch and push are possible over https, which is selected by http if
gitlab was configured to use https in external url.

This way to reduce security vectors and possible ways to interact with gitlab
we use https only without ssh at all.

= GitLab Community Edition + Nexedi patches

Update doorkeeper to 4.2.0

Changelog: https://git.io/v6PnV

See merge request !5881
(cherry picked from commit c5aa31c8)

[ci skip]

Upgrade Rails to 4.2.7.1 for security fixes.

Upgrades Rails from 4.2.7 to 4.2.7.1 for security fixes.

For more information: http://weblog.rubyonrails.org/2016/8/11/Rails-5-0-0-1-4-2-7-2-and-3-2-22-3-have-been-released/

This should be backported to all currently-supported releases.

See merge request !5781

Ensure logged-out users can't see private refs

https://gitlab.com/gitlab-org/gitlab-ce/issues/18033

I'm still not sure what to do about the CHANGELOG on security issues - should I add to a patch release? This issue was assigned to 8.10.

See merge request !1974
(cherry picked from commit 3a6ebb1f)

Fix privilege escalation issue with OAuth external users

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/19312

This MR fixes a privilege escalation issue, where manually set external users would be reverted back to internal users if they logged in via OAuth and that provider was not in the `external_providers` list.

/cc @douwe

See merge request !1975
(cherry picked from commit 5e6342b7)

Fix visibility of snippets when searching

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/18997

See merge request !1972
(cherry picked from commit 8a197c15)

Update omniauth-saml to 1.6.0 to address a security vulnerability in ruby-saml

Updates `omniauth-saml` to bring in the new `ruby-saml` dependency that addresses [CVE-2016-5697](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-5697)

Fixes #19206

See merge request !4951
(cherry picked from commit c3a8b252)

Only show notes through JSON on confidential issues that the user has access to

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/18535

See merge request !1970

Forbid scripting for wiki files

Wiki files (not pages - files in the repo) are just sent to the browser
with whatever content-type the mime_types gem assigns to them based on
their extension. As this is from the same domain as the GitLab
application, this is an XSS vulnerability.

Set a CSP forbidding all sources for scripting, CSS, XHR, etc. on these
files.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/17298.

See merge request !1969

Remove 'unscoped' from project builds selection

This is a fix for this security bug: https://gitlab.com/gitlab-org/gitlab-ce/issues/18188

/cc @kamil @grzegorz @stanhu

See merge request !1968

Fix UTF-8 handling in incremental trace update API

## What does this MR do?

This MR fixes invalid UTF-8 handling in incremental trace update API (used by GitLab Runner).

## Why was this MR needed?

Current version is using `.length` method to determine current trace size where Runner is using the trace size in bytes. Also this byte size is used in headers and file operations to agree the trace part to send. This is a problem when build trace contains any multi-byte UTF-8 characters. This MR is fixing this situation so all parts are using the same size in bytes.

### Runner -> API communication before fix:
```
Checking for builds... received                     runner=_token_
gitlab-ci-multi-runner 1.3.0~beta.26.gcfd63b9 (cfd63b9)  build=25 runner=_token_
Using Docker executor with image debian:jessie ...  build=25 runner=_token_
Pulling docker image debian:jessie ...              build=25 runner=_token_
25 Submitting build to coordinator... ok            runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-158 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=0-158 runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-491 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=158-505 runner=_token_
WARNING: 25 Appending trace to coordinator... range missmatch  RemoteRange=0-491 RemoteState= ResponseMessage=416 Requested Range Not Satisfiable ResponseStatusCode=416 SentRange=505-584 runner=_token_
WARNING: 25 Resending trace patch due to range missmatch  runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-556 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=491-584 runner=_token_
WARNING: 25 Appending trace to coordinator... range missmatch  RemoteRange=0-556 RemoteState= ResponseMessage=416 Requested Range Not Satisfiable ResponseStatusCode=416 SentRange=584-663 runner=_token_
WARNING: 25 Resending trace patch due to range missmatch  runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-621 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=556-663 runner=_token_
Build succeeded                                     build=25 runner=_token_
WARNING: 25 Appending trace to coordinator... range missmatch  RemoteRange=0-621 RemoteState= ResponseMessage=416 Requested Range Not Satisfiable ResponseStatusCode=416 SentRange=663-797 runner=_token_
WARNING: 25 Resending trace patch due to range missmatch  runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-741 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=621-797 runner=_token_
25 Submitting build to coordinator... ok            runner=_token_
```

### Runner -> API communication after fix:
```
Checking for builds... received                     runner=_token_
gitlab-ci-multi-runner 1.3.0~beta.26.gcfd63b9 (cfd63b9)  build=26 runner=_token_
Using Docker executor with image debian:jessie ...  build=26 runner=_token_
Pulling docker image debian:jessie ...              build=26 runner=_token_
26 Submitting build to coordinator... ok            runner=_token_
26 Appending trace to coordinator... ok             RemoteRange=0-158 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=0-158 runner=_token_
26 Appending trace to coordinator... ok             RemoteRange=0-505 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=158-505 runner=_token_
26 Appending trace to coordinator... ok             RemoteRange=0-584 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=505-584 runner=_token_
26 Appending trace to coordinator... ok             RemoteRange=0-663 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=584-663 runner=_token_
Build succeeded                                     build=26 runner=_token_
26 Submitting build to coordinator... ok            runner=_token_
```

See merge request !4541

Check if GitHub rate limite API was reached before update Webhooks

## What does this MR do?

Checks if the job needs to sleep, and wait for the rate limit to be reseted before update each Webhook.

## Are there points in the code the reviewer needs to double check?

No.

## Why was this MR needed?

The import process can fail if the API rate limit was reached during the import process.

## What are the relevant issue numbers?

https://gitlab.com/gitlab-org/gitlab-ce/issues/17498

## Screenshots (if relevant)

Not relevant.

See merge request !4509

Adjust the SAML control flow to allow LDAP identities to be added to an existing SAML user.

It correctly lets an existing SAML user to add their LDAP identity automatically at login.

A customer had issues with the `auto_link_ldap_user` feature. The flow was not working if there was an account with a SAML identity, but no LDAP identity. GitLab would pick up the correct LDAP person, but due to the order of the flow, that LDAP person was never associated with the user.

Fixes #17346

/cc @dblessing @balameb @stanhu

See merge request !4498

Ensure we don't show TODOS for projects pending delete

Joins the todos on the projects table in order to run the default scope. Also includes a where clause because the default scope is being removed soon.

An alternative approach, more like the Issues page, would be to filter down the list by passing user.authorized_projects into the where clause.

Or we could just be more defensive in the view when iterating.

Todos page throws 500 error for users with todos in a project pending deletion.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/17813

cc\ @stanhu

See merge request !4300

[ci skip]

Fix 2FA-based login for LDAP users

The OTP input form is shared by both LDAP and standard logins, but when
coming from an LDAP-based form, the form parameters aren't nested in a
Hash based on the `resource_name` value.

Now we check for a nested `remember_me` parameter and use that if it
exists, or fall back to the non-nested parameters if it doesn't.

Somewhat confusingly, the OTP input form _does_ nest parameters under
the `resource_name`, regardless of what type of login we're coming from,
so that allows everything else to work as normal.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/18185

See merge request !4493

Fixes missing number on generated ordered list

Closes #18102

See merge request !4437

Fix serious performance bug with rendering Markdown with InlineDiffFilter

Nokogiri's `node.replace` was being unnecessarily called for every text node in
the document due to a comparison bug. The code previously was comparing the
HTML representation of the full document against the text node, which would
always fail. Fix the comparison to just compare the modified text.

Closes #18011

See merge request !4392