Add latest changes from gitlab-org/gitlab@master

changelogs/unreleased/update-ado-image-to-0-10-0.yml

+---
+title: Update Auto DevOps deployment template's auto-deploy-image to v0.10.0 (updates the included glibc)
+merge_request: 25920
+author:
+type: other

doc/api/releases/index.md

@@ -9,7 +9,7 @@
 
 Paginated list of Releases, sorted by `released_at`.
 
-```
+```plaintext
 GET /projects/:id/releases
 ```
 
@@ -191,7 +191,7 @@ Example response:
 
 Get a Release for the given tag.
 
-```
+```plaintext
 GET /projects/:id/releases/:tag_name
 ```
 
@@ -303,7 +303,7 @@ Example response:
 
 Create a Release. You need push access to the repository to create a Release.
 
-```
+```plaintext
 POST /projects/:id/releases
 ```
 
@@ -430,7 +430,7 @@ Example response:
 
 Update a Release.
 
-```
+```plaintext
 PUT /projects/:id/releases/:tag_name
 ```
 
@@ -533,7 +533,7 @@ Example response:
 
 Delete a Release. Deleting a Release will not delete the associated tag.
 
-```
+```plaintext
 DELETE /projects/:id/releases/:tag_name
 ```
 

doc/api/repositories.md

@@ -7,7 +7,7 @@ be accessed without authentication if the repository is publicly accessible.
 
 This command provides essentially the same functionality as the `git ls-tree` command. For more information, see the section _Tree Objects_ in the [Git internals documentation](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects/#_tree_objects).
 
-```
+```plaintext
 GET /projects/:id/repository/tree
 ```
 
@@ -79,7 +79,7 @@ Allows you to receive information about blob in repository like size and
 content. Note that blob content is Base64 encoded. This endpoint can be accessed
 without authentication if the repository is publicly accessible.
 
-```
+```plaintext
 GET /projects/:id/repository/blobs/:sha
 ```
 
@@ -93,7 +93,7 @@ Parameters:
 Get the raw file contents for a blob by blob SHA. This endpoint can be accessed
 without authentication if the repository is publicly accessible.
 
-```
+```plaintext
 GET /projects/:id/repository/blobs/:sha/raw
 ```
 
@@ -107,7 +107,7 @@ Parameters:
 Get an archive of the repository. This endpoint can be accessed without
 authentication if the repository is publicly accessible.
 
-```
+```plaintext
 GET /projects/:id/repository/archive[.format]
 ```
 
@@ -130,7 +130,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.com/api/v4/pro
 This endpoint can be accessed without authentication if the repository is
 publicly accessible. Note that diffs could have an empty diff string if [diff limits](../development/diffs.md#diff-limits) are reached.
 
-```
+```plaintext
 GET /projects/:id/repository/compare
 ```
 
@@ -141,7 +141,7 @@ Parameters:
 - `to` (required) - the commit SHA or branch name
 - `straight` (optional) - comparison method, `true` for direct comparison between `from` and `to` (`from`..`to`), `false` to compare using merge base (`from`...`to`)'. Default is `false`.
 
-```
+```plaintext
 GET /projects/:id/repository/compare?from=master&to=feature
 ```
 
@@ -186,7 +186,7 @@ Response:
 Get repository contributors list. This endpoint can be accessed without
 authentication if the repository is publicly accessible.
 
-```
+```plaintext
 GET /projects/:id/repository/contributors
 ```
 
@@ -198,16 +198,16 @@ Parameters:
 
 Response:
 
-```
+```json
 [{
-  "name": "Dmitriy Zaporozhets",
-  "email": "dmitriy.zaporozhets@gmail.com",
+  "name": "Example User",
+  "email": "example@example.com",
   "commits": 117,
   "additions": 2097,
   "deletions": 517
 }, {
-  "name": "Jacob Vosmaer",
-  "email": "contact@jacobvosmaer.nl",
+  "name": "Sample User",
+  "email": "sample@example.com",
   "commits": 33,
   "additions": 338,
   "deletions": 244
@@ -218,7 +218,7 @@ Response:
 
 Get the common ancestor for 2 or more refs (commit SHAs, branch names or tags).
 
-```
+```plaintext
 GET /projects/:id/repository/merge_base
 ```
 

doc/api/repository_files.md

@@ -20,7 +20,7 @@ Allows you to receive information about file in repository like name, size,
 content. Note that file content is Base64 encoded. This endpoint can be accessed
 without authentication if the repository is publicly accessible.
 
-```
+```plaintext
 GET /projects/:id/repository/files/:file_path
 ```
 
@@ -55,7 +55,7 @@ NOTE: **Note:**
 
 In addition to the `GET` method, you can also use `HEAD` to get just file metadata.
 
-```
+```plaintext
 HEAD /projects/:id/repository/files/:file_path
 ```
 
@@ -84,7 +84,7 @@ X-Gitlab-Size: 1476
 
 Allows you to receive blame information. Each blame range contains lines and corresponding commit info.
 
-```
+```plaintext
 GET /projects/:id/repository/files/:file_path/blame
 ```
 
@@ -151,7 +151,7 @@ X-Gitlab-Size: 1476
 
 ## Get raw file from repository
 
-```
+```plaintext
 GET /projects/:id/repository/files/:file_path/raw
 ```
 
@@ -171,7 +171,7 @@ Like [Get file from repository](repository_files.md#get-file-from-repository) yo
 
 This allows you to create a single file. For creating multiple files with a single request see the [commits API](commits.html#create-a-commit-with-multiple-files-and-actions).
 
-```
+```plaintext
 POST /projects/:id/repository/files/:file_path
 ```
 
@@ -206,7 +206,7 @@ Parameters:
 
 This allows you to update a single file. For updating multiple files with a single request see the [commits API](commits.html#create-a-commit-with-multiple-files-and-actions).
 
-```
+```plaintext
 PUT /projects/:id/repository/files/:file_path
 ```
 
@@ -252,7 +252,7 @@ Currently GitLab Shell has a boolean return code, preventing GitLab from specify
 
 This allows you to delete a single file. For deleting multiple files with a single request, see the [commits API](commits.html#create-a-commit-with-multiple-files-and-actions).
 
-```
+```plaintext
 DELETE /projects/:id/repository/files/:file_path
 ```
 

doc/api/repository_submodules.md

@@ -9,7 +9,7 @@ submodule's reference to keep up to date other projects that use it.
 This endpoint allows you to update a [Git submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules) reference in a
 specific branch.
 
-```
+```plaintext
 PUT /projects/:id/repository/submodules/:submodule
 ```
 

doc/api/resource_label_events.md

@@ -8,7 +8,7 @@ Resource label events keep track about who, when, and which label was added to,
 
 Gets a list of all label events for a single issue.
 
-```
+```plaintext
 GET /projects/:id/issues/:issue_iid/resource_label_events
 ```
 
@@ -72,7 +72,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/ap
 
 Returns a single label event for a specific project issue
 
-```
+```plaintext
 GET /projects/:id/issues/:issue_iid/resource_label_events/:resource_label_event_id
 ```
 
@@ -94,7 +94,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/ap
 
 Gets a list of all label events for a single epic.
 
-```
+```plaintext
 GET /groups/:id/epics/:epic_id/resource_label_events
 ```
 
@@ -158,7 +158,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/ap
 
 Returns a single label event for a specific group epic
 
-```
+```plaintext
 GET /groups/:id/epics/:epic_id/resource_label_events/:resource_label_event_id
 ```
 
@@ -180,7 +180,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab
 
 Gets a list of all label events for a single merge request.
 
-```
+```plaintext
 GET /projects/:id/merge_requests/:merge_request_iid/resource_label_events
 ```
 
@@ -244,7 +244,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/ap
 
 Returns a single label event for a specific project merge request
 
-```
+```plaintext
 GET /projects/:id/merge_requests/:merge_request_iid/resource_label_events/:resource_label_event_id
 ```
 

doc/api/runners.md

@@ -31,7 +31,7 @@ GitLab and Runner are then connected.
 
 Get a list of specific runners available to the user.
 
-```
+```plaintext
 GET /runners
 GET /runners?scope=active
 GET /runners?type=project_type
@@ -46,7 +46,7 @@ GET /runners?tag_list=tag1,tag2
 | `status`    | string         | no       | The status of runners to show, one of: `active`, `paused`, `online`, `offline` |
 | `tag_list`  | string array   | no       | List of of the runner's tags |
 
-```
+```shell
 curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners"
 ```
 
@@ -82,7 +82,7 @@ Example response:
 Get a list of all runners in the GitLab instance (specific and shared). Access
 is restricted to users with `admin` privileges.
 
-```
+```plaintext
 GET /runners/all
 GET /runners/all?scope=online
 GET /runners/all?type=project_type
@@ -97,7 +97,7 @@ GET /runners/all?tag_list=tag1,tag2
 | `status`    | string         | no       | The status of runners to show, one of: `active`, `paused`, `online`, `offline` |
 | `tag_list`  | string array   | no       | List of of the runner's tags |
 
-```
+```shell
 curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/all"
 ```
 
@@ -152,7 +152,7 @@ Example response:
 
 Get details of a runner.
 
-```
+```plaintext
 GET /runners/:id
 ```
 
@@ -160,7 +160,7 @@ GET /runners/:id
 |-----------|---------|----------|---------------------|
 | `id`      | integer | yes      | The ID of a runner  |
 
-```
+```shell
 curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/6"
 ```
 
@@ -204,7 +204,7 @@ Example response:
 
 Update details of a runner.
 
-```
+```plaintext
 PUT /runners/:id
 ```
 
@@ -219,7 +219,7 @@ PUT /runners/:id
 | `access_level` | string | no       | The access_level of the runner; `not_protected` or `ref_protected` |
 | `maximum_timeout` | integer | no   | Maximum timeout set when this Runner will handle the job |
 
-```
+```shell
 curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/6" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2"
 ```
 
@@ -265,7 +265,7 @@ Example response:
 
 Remove a runner.
 
-```
+```plaintext
 DELETE /runners/:id
 ```
 
@@ -273,7 +273,7 @@ DELETE /runners/:id
 |-----------|---------|----------|---------------------|
 | `id`      | integer | yes      | The ID of a runner  |
 
-```
+```shell
 curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/6"
 ```
 
@@ -283,7 +283,7 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git
 
 List jobs that are being processed or were processed by specified Runner.
 
-```
+```plaintext
 GET /runners/:id/jobs
 ```
 
@@ -294,7 +294,7 @@ GET /runners/:id/jobs
 | `order_by`| string  | no       | Order jobs by `id`. |
 | `sort`    | string  | no       | Sort jobs in `asc` or `desc` order (default: `desc`) |
 
-```
+```shell
 curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/1/jobs?status=running"
 ```
 
@@ -373,7 +373,7 @@ Example response:
 List all runners (specific and shared) available in the project. Shared runners
 are listed if at least one shared runner is defined.
 
-```
+```plaintext
 GET /projects/:id/runners
 GET /projects/:id/runners?scope=active
 GET /projects/:id/runners?type=project_type
@@ -389,7 +389,7 @@ GET /projects/:id/runners?tag_list=tag1,tag2
 | `status`   | string         | no       | The status of runners to show, one of: `active`, `paused`, `online`, `offline` |
 | `tag_list` | string array   | no       | List of of the runner's tags |
 
-```
+```shell
 curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/9/runners"
 ```
 
@@ -424,7 +424,7 @@ Example response:
 
 Enable an available specific runner in the project.
 
-```
+```plaintext
 POST /projects/:id/runners
 ```
 
@@ -433,7 +433,7 @@ POST /projects/:id/runners
 | `id`        | integer/string | yes      | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user |
 | `runner_id` | integer | yes      | The ID of a runner  |
 
-```
+```shell
 curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/9/runners" --form "runner_id=9"
 ```
 
@@ -458,7 +458,7 @@ Disable a specific runner from the project. It works only if the project isn't
 the only project associated with the specified runner. If so, an error is
 returned. Use the [Remove a runner](#remove-a-runner) call instead.
 
-```
+```plaintext
 DELETE /projects/:id/runners/:runner_id
 ```
 
@@ -467,7 +467,7 @@ DELETE /projects/:id/runners/:runner_id
 | `id`        | integer/string | yes      | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user |
 | `runner_id` | integer | yes      | The ID of a runner  |
 
-```
+```shell
 curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/9/runners/9"
 ```
 
@@ -475,7 +475,7 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git
 
 Register a new Runner for the instance.
 
-```
+```plaintext
 POST /runners
 ```
 
@@ -491,7 +491,7 @@ POST /runners
 | `access_level`    | string | no   | The access_level of the runner; `not_protected` or `ref_protected` |
 | `maximum_timeout` | integer | no  | Maximum timeout set when this Runner will handle the job |
 
-```
+```shell
 curl --request POST "https://gitlab.example.com/api/v4/runners" --form "token=<registration_token>" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2"
 ```
 
@@ -514,7 +514,7 @@ Example response:
 
 Deletes a registered Runner.
 
-```
+```plaintext
 DELETE /runners
 ```
 
@@ -522,7 +522,7 @@ DELETE /runners
 |-------------|---------|----------|---------------------|
 | `token`     | string  | yes      | Runner's [authentication token](#registration-and-authentication-tokens).  |
 
-```
+```shell
 curl --request DELETE "https://gitlab.example.com/api/v4/runners" --form "token=<authentication_token>"
 ```
 
@@ -536,7 +536,7 @@ Response:
 
 Validates authentication credentials for a registered Runner.
 
-```
+```plaintext
 POST /runners/verify
 ```
 
@@ -544,7 +544,7 @@ POST /runners/verify
 |-------------|---------|----------|---------------------|
 | `token`     | string  | yes      | Runner's [authentication token](#registration-and-authentication-tokens).  |
 
-```
+```shell
 curl --request POST "https://gitlab.example.com/api/v4/runners/verify" --form "token=<authentication_token>"
 ```
 

doc/api/scim.md

@@ -219,7 +219,7 @@ They match an expression as specified in [the RFC7644 filtering section](https:/
 
 Example:
 
-```
+```plaintext
 id eq a-b-c-d
 ```
 

doc/api/search.md

@@ -8,7 +8,7 @@ Every API call to search must be authenticated.
 
 Search globally across the GitLab instance.
 
-```
+```plaintext
 GET /search
 ```
 
@@ -410,7 +410,7 @@ Search within the specified group.
 
 If a user is not a member of a group and the group is private, a `GET` request on that group will result to a `404` status code.
 
-```
+```plaintext
 GET /groups/:id/search
 ```
 

doc/ci/services/postgres.md

@@ -22,22 +22,23 @@ services:
 variables:
   POSTGRES_DB: nice_marmot
   POSTGRES_USER: runner
-  POSTGRES_PASSWORD: "runner-password"
+  POSTGRES_PASSWORD: ""
+  POSTGRES_HOST_AUTH_METHOD: trust
 ```
 
 NOTE: **Note:**
-The `POSTGRES_DB`, `POSTGRES_USER`, and `POSTGRES_PASSWORD` variables can't be set in
-the GitLab UI. To set them, assign them to a variable
+The `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` and `POSTGRES_HOST_AUTH_METHOD`
+variables can't be set in the GitLab UI. To set them, assign them to a variable
 [in the UI](../variables/README.md#via-the-ui), and then assign that
-variable to the `POSTGRES_DB`, `POSTGRES_USER`, and `POSTGRES_PASSWORD` variables in
-your `.gitlab-ci.yml`.
+variable to the `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` and `POSTGRES_HOST_AUTH_METHOD`
+variables in your `.gitlab-ci.yml`.
 
 And then configure your application to use the database, for example:
 
 ```yaml
 Host: postgres
 User: runner
-Password: runner-password
+Password: ''
 Database: nice_marmot
 ```
 

doc/development/database/add_foreign_key_to_existing_column.md

+# Adding foreign key constraint to an existing column
+
+Foreign keys help ensure consistency between related database tables. The current database review process **always** encourages you to add [foreign keys](../foreign_keys.md) when creating tables that reference records from other tables.
+
+Starting with Rails version 4, Rails includes migration helpers to add foreign key constraints to database tables. Before Rails 4, the only way for ensuring some level of consistency was the [`dependent`](https://guides.rubyonrails.org/association_basics.html#options-for-belongs-to-dependent) option within the association definition. Ensuring data consistency on the application level could fail in some unfortunate cases, so we might end up with inconsistent data in the table. This is mostly affecting older tables, where we simply didn't have the framework support to ensure consistency on the database level. These data inconsistencies can easily cause unexpected application behavior or bugs.
+
+Adding a foreign key to an existing database column requires database structure changes and potential data changes. In case the table is in use, we should always assume that there is inconsistent data.
+
+To add a foreign key constraint to an existing column:
+
+1. GitLab version `N.M`: Add a `NOT VALID` foreign key constraint to the column to ensure GitLab doesn't create inconsistent records.
+1. GitLab version `N.M`: Add a data migration, to fix or clean up existing records.
+1. GitLab version `N.M+1`: Validate the whole table by making the foreign key `VALID`.
+
+## Example
+
+Consider the following table structures:
+
+`users` table:
+
+- `id` (integer, primary key)
+- `name` (string)
+
+`emails` table:
+
+- `id` (integer, primary key)
+- `user_id` (integer)
+- `email` (string)
+
+Express the relationship in `ActiveRecord`:
+
+```ruby
+class User < ActiveRecord::Base
+  has_many :emails
+end
+
+class Email < ActiveRecord::Base
+  belongs_to :user
+end
+```
+
+Problem: when the user is removed, the email records related to the removed user will stay in the `emails` table:
+
+```ruby
+user = User.find(1)
+user.destroy
+
+emails = Email.where(user_id: 1) # returns emails for the deleted user
+```
+
+### Prevent invalid records
+
+Add a `NOT VALID` foreign key constraint to the table, which enforces consistency on the record changes.
+
+In the example above, you'd be still able to update records in the `emails` table. However, when you'd try to update the `user_id` with non-existent value, the constraint causes a database error.
+
+Migration file for adding `NOT VALID` foreign key:
+
+```ruby
+class AddNotValidForeignKeyToEmailsUser < ActiveRecord::Migration[5.2]
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+
+  def up
+    # safe to use: it requires short lock on the table since we don't validate the foreign key
+    add_foreign_key :emails, :users, on_delete: :cascade, validate: false # rubocop:disable Migration/AddConcurrentForeignKey
+  end
+
+  def down
+    remove_foreign_key_if_exists :emails, column: :user_id
+  end
+end
+```
+
+CAUTION: **Caution:**
+Avoid using the `add_foreign_key` constraint more than once per migration file, unless the source and target tables are identical.
+
+#### Data migration to fix existing records
+
+The approach here depends on the data volume and the cleanup strategy. If we can easily find "invalid" records by doing a simple database query and the record count is not that high, then the data migration can be executed within a Rails migration.
+
+In case the data volume is higher (>1000 records), it's better to create a background migration. If unsure, please contact the database team for advice.
+
+Example for cleaning up records in the `emails` table within a database migration:
+
+```ruby
+class RemoveRecordsWithoutUserFromEmailsTable < ActiveRecord::Migration[5.2]
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  class Email < ActiveRecord::Base
+    include EachBatch
+  end
+
+  def up
+    Email.where('user_id NOT IN (SELECT id FROM users)').each_batch do |relation|
+      relation.delete_all
+    end
+  end
+
+  def down
+    # Can be a no-op when data inconsistency is not affecting the pre and post deploymnet version of the application.
+    # In this case we might have records in the `emails` table where the associated record in the `users` table is not there anymore.
+  end
+end
+```
+
+### Validate the foreign key
+
+Validating the foreign key will scan the whole table and make sure that each relation is correct.
+
+NOTE: **Note:** When using [background migrations](../background_migrations.md), foreign key validation should happen in the next GitLab release.
+
+Migration file for validating the foreign key:
+
+```ruby
+# frozen_string_literal: true
+
+class ValidateForeignKeyOnEmailUsers < ActiveRecord::Migration[5.2]
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+
+  def up
+    validate_foreign_key :emails, :user_id
+  end
+
+  def down
+    # Can be safely a no-op if we don't roll back the inconsistent data.
+  end
+end
+```

doc/development/migration_style_guide.md

@@ -399,6 +399,8 @@ For an empty table (such as a fresh one), it is recommended to use
 `add_reference` in a single-transaction migration, combining it with other
 operations that don't require `disable_ddl_transaction!`.
 
+You can read more about adding [foreign key constraints to an existing column](database/add_foreign_key_to_existing_column.md).
+
 ## Adding Columns With Default Values
 
 When adding columns with default values to non-empty tables, you must use

doc/user/project/integrations/webhooks.md

@@ -1292,7 +1292,7 @@ Markdown features, like link labels.
 
 ## Testing webhooks
 
-You can trigger the webhook manually. Sample data from the project will be used. Sample data will take from the project.
+You can trigger the webhook manually. Sample data from the project will be used.
 > For example: for triggering `Push Events` your project should have at least one commit.
 
 ![Webhook testing](img/webhook_testing.png)

lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml

 .auto-deploy:
-  image: "registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v0.9.3"
+  image: "registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v0.10.0"
 
 review:
   extends: .auto-deploy