Merge branch 'docs-job-run-control' into 'master'

Create a new page about controlling CI/CD jobs

See merge request gitlab-org/gitlab!58880

doc/ci/environments/index.md

@@ -400,8 +400,8 @@ stop_review:
       when: manual
 ```
 
-Both jobs must have the same [`rules`](../yaml/README.md#onlyexcept-basic)
-or [`only/except`](../yaml/README.md#onlyexcept-basic) configuration. Otherwise,
+Both jobs must have the same [`rules`](../yaml/README.md#only--except)
+or [`only/except`](../yaml/README.md#only--except) configuration. Otherwise,
 the `stop_review` job might not be included in all pipelines that include the
 `deploy_review` job, and you cannot trigger `action: stop` to stop the environment automatically.
 

doc/ci/jobs/job_control.md

+---
+stage: Verify
+group: Continuous Integration
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Choose when to run jobs **(FREE)**
+
+When a new pipeline starts, GitLab checks the pipeline configuration to determine
+which jobs should run in that pipeline. You can configure jobs to run depending on
+the status of variables, the pipeline type, and so on.
+
+To configure a job to be included or excluded from certain pipelines, you can use:
+
+- [`rules`](../yaml/README.md#rules)
+- [`only`](../yaml/README.md#only--except)
+- [`except`](../yaml/README.md#only--except)
+
+Use [`needs`](../yaml/README.md#needs) to configure a job to run as soon as the
+earlier jobs it depends on finish running.
+
+## Specify when jobs run with `only` and `except`
+
+You can use [`only`](../yaml/README.md#only--except) and [`except`](../yaml/README.md#only--except)
+to control when to add jobs to pipelines.
+
+- Use `only` to define when a job runs.
+- Use `except` to define when a job **does not** run.
+
+### `only:refs` / `except:refs` examples
+
+`only` or `except` used without `refs` is the same as
+[`only:refs` / `except/refs`](../yaml/README.md#onlyrefs--exceptrefs)
+
+In the following example, `job` runs only for:
+
+- Git tags
+- [Triggers](../triggers/README.md#trigger-token)
+- [Scheduled pipelines](../pipelines/schedules.md)
+
+```yaml
+job:
+  # use special keywords
+  only:
+    - tags
+    - triggers
+    - schedules
+```
+
+To execute jobs only for the parent repository and not forks:
+
+```yaml
+job:
+  only:
+    - branches@gitlab-org/gitlab
+  except:
+    - main@gitlab-org/gitlab
+    - /^release/.*$/@gitlab-org/gitlab
+```
+
+This example runs `job` for all branches on `gitlab-org/gitlab`,
+except `main` and branches that start with `release/`.
+
+### `only: variables` / `except: variables` examples
+
+You can use `except:variables` to exclude jobs based on a commit message:
+
+```yaml
+end-to-end:
+  script: rake test:end-to-end
+  except:
+    variables:
+      - $CI_COMMIT_MESSAGE =~ /skip-end-to-end-tests/
+```
+
+You can use [parentheses](../variables/README.md#parentheses) with `&&` and `||`
+to build more complicated variable expressions:
+
+```yaml
+job1:
+  script:
+    - echo This rule uses parentheses.
+  only:
+    variables:
+      - ($CI_COMMIT_BRANCH == "master" || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE
+```
+
+### `only:changes` / `except:changes` examples
+
+You can skip a job if a change is detected in any file with a
+`.md` extension in the root directory of the repository:
+
+```yaml
+build:
+  script: npm run build
+  except:
+    changes:
+      - "*.md"
+```
+
+If you change multiple files, but only one file ends in `.md`,
+the `build` job is still skipped. The job does not run for any of the files.
+
+Read more about how to use `only:changes` and `except:changes`:
+
+- [New branches or tags *without* pipelines for merge requests](#use-onlychanges-without-pipelines-for-merge-requests).
+- [Scheduled pipelines](#use-onlychanges-with-scheduled-pipelines).
+
+#### Use `only:changes` with pipelines for merge requests
+
+With [pipelines for merge requests](../merge_request_pipelines/index.md),
+it's possible to define a job to be created based on files modified
+in a merge request.
+
+Use this keyword with `only: [merge_requests]` so GitLab can find the correct base
+SHA of the source branch. File differences are correctly calculated from any further
+commits, and all changes in the merge requests are properly tested in pipelines.
+
+For example:
+
+```yaml
+docker build service one:
+  script: docker build -t my-service-one-image:$CI_COMMIT_REF_SLUG .
+  only:
+    refs:
+      - merge_requests
+    changes:
+      - Dockerfile
+      - service-one/**/*
+```
+
+In this scenario, if a merge request changes
+files in the `service-one` directory or the `Dockerfile`, GitLab creates
+the `docker build service one` job.
+
+For example:
+
+```yaml
+docker build service one:
+  script: docker build -t my-service-one-image:$CI_COMMIT_REF_SLUG .
+  only:
+    changes:
+      - Dockerfile
+      - service-one/**/*
+```
+
+In this example, the pipeline might fail because of changes to a file in `service-one/**/*`.
+
+A later commit that doesn't have changes in `service-one/**/*`
+but does have changes to the `Dockerfile` can pass. The job
+only tests the changes to the `Dockerfile`.
+
+GitLab checks the **most recent pipeline** that **passed**. If the merge request is mergeable,
+it doesn't matter that an earlier pipeline failed because of a change that has not been corrected.
+
+When you use this configuration, ensure that the most recent pipeline
+properly corrects any failures from previous pipelines.
+
+#### Use `only:changes` without pipelines for merge requests
+
+Without [pipelines for merge requests](../merge_request_pipelines/index.md), pipelines
+run on branches or tags that don't have an explicit association with a merge request.
+In this case, a previous SHA is used to calculate the diff, which is equivalent to `git diff HEAD~`.
+This can result in some unexpected behavior, including:
+
+- When pushing a new branch or a new tag to GitLab, the policy always evaluates to true.
+- When pushing a new commit, the changed files are calculated by using the previous commit
+  as the base SHA.
+
+#### Use `only:changes` with scheduled pipelines
+
+`only:changes` always evaluates as true in [Scheduled pipelines](../pipelines/schedules.md).
+All files are considered to have changed when a scheduled pipeline runs.
+
+### Combine multiple keywords with `only` or `except`
+
+If you use multiple keywords with `only` or `except`, the keywords are evaluated
+as a single conjoined expression. That is:
+
+- `only:` includes the job if **all** of the keys have at least one condition that matches.
+- `except:` excludes the job if **any** of the keys have at least one condition that matches.
+
+With `only`, individual keys are logically joined by an `AND`. A job is added to
+the pipeline if the following is true:
+
+- `(any listed refs are true) AND (any listed variables are true) AND (any listed changes are true) AND (any chosen Kubernetes status matches)`
+
+In the following example, the `test` job is only created when **all** of the following are true:
+
+- The pipeline is [scheduled](../pipelines/schedules.md) **or** runs for `main`.
+- The `variables` keyword matches.
+- The `kubernetes` service is active on the project.
+
+```yaml
+test:
+  script: npm run test
+  only:
+    refs:
+      - main
+      - schedules
+    variables:
+      - $CI_COMMIT_MESSAGE =~ /run-end-to-end-tests/
+    kubernetes: active
+```
+
+With `except`, individual keys are logically joined by an `OR`. A job is **not**
+added if the following is true:
+
+- `(any listed refs are true) OR (any listed variables are true) OR (any listed changes are true) OR (a chosen Kubernetes status matches)`
+
+In the following example, the `test` job is **not** created when **any** of the following are true:
+
+- The pipeline runs for the `main` branch.
+- There are changes to the `README.md` file in the root directory of the repository.
+
+```yaml
+test:
+  script: npm run test
+  except:
+    refs:
+      - main
+    changes:
+      - "README.md"
+```
+
+## Regular expressions
+
+The `@` symbol denotes the beginning of a ref's repository path.
+To match a ref name that contains the `@` character in a regular expression,
+you must use the hex character code match `\x40`.
+
+Only the tag or branch name can be matched by a regular expression.
+The repository path, if given, is always matched literally.
+
+To match the tag or branch name,
+the entire ref name part of the pattern must be a regular expression surrounded by `/`.
+For example, you can't use `issue-/.*/` to match all tag names or branch names
+that begin with `issue-`, but you can use `/issue-.*/`.
+
+Regular expression flags must be appended after the closing `/`. Pattern matching
+is case-sensitive by default. Use the `i` flag modifier, like `/pattern/i`, to make
+a pattern case-insensitive:
+
+```yaml
+job:
+  # use regexp
+  only:
+    - /^issue-.*$/i
+  # use special keyword
+  except:
+    - branches
+```
+
+Use anchors `^` and `$` to avoid the regular expression
+matching only a substring of the tag name or branch name.
+For example, `/^issue-.*$/` is equivalent to `/^issue-/`,
+while just `/issue/` would also match a branch called `severe-issues`.
+
+### `only` / `except` regex syntax
+
+In GitLab 11.9.4, GitLab began internally converting the regexp used
+in `only` and `except` keywords to [RE2](https://github.com/google/re2/wiki/Syntax).
+
+[RE2](https://github.com/google/re2/wiki/Syntax) limits the set of available features
+due to computational complexity, and some features, like negative lookaheads, became unavailable.
+Only a subset of features provided by [Ruby Regexp](https://ruby-doc.org/core/Regexp.html)
+are now supported.
+
+From GitLab 11.9.7 to GitLab 12.0, GitLab provided a feature flag to
+let you use unsafe regexp syntax. After migrating to safe syntax, you should disable
+this feature flag again:
+
+```ruby
+Feature.enable(:allow_unsafe_ruby_regexp)
+```

doc/ci/merge_request_pipelines/index.md

@@ -137,7 +137,7 @@ save resources.
 #### Excluding certain branches
 
 Pipelines for merge requests require special treatment when
-using [`only`/`except`](../yaml/README.md#onlyexcept-basic). Unlike ordinary
+using [`only`/`except`](../yaml/README.md#only--except). Unlike ordinary
 branch refs (for example `refs/heads/my-feature-branch`), merge request refs
 use a special Git reference that looks like `refs/merge-requests/:iid/head`. Because
 of this, the following configuration will **not** work as expected:
@@ -153,7 +153,7 @@ Instead, you can use the
 [`$CI_COMMIT_REF_NAME` predefined environment
 variable](../variables/predefined_variables.md) in
 combination with
-[`only:variables`](../yaml/README.md#onlyvariablesexceptvariables) to
+[`only:variables`](../yaml/README.md#onlyvariables--exceptvariables) to
 accomplish this behavior:
 
 ```yaml

doc/ci/migration/jenkins.md

@@ -349,8 +349,8 @@ variable entry.
 
 GitLab does support a [`when` keyword](../yaml/README.md#when) which is used to indicate when a job should be
 run in case of (or despite) failure, but most of the logic for controlling pipelines can be found in
-our very powerful [`only/except` rules system](../yaml/README.md#onlyexcept-basic)
-(see also our [advanced syntax](../yaml/README.md#onlyexcept-basic)):
+our very powerful [`only/except` rules system](../yaml/README.md#only--except)
+(see also our [advanced syntax](../yaml/README.md#only--except)):
 
 ```yaml
 my_job:

doc/ci/multi_project_pipelines.md

@@ -67,7 +67,7 @@ When using:
 - CI/CD Variables or [`rules`](yaml/README.md#rulesif) to control job behavior, the value of
   the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md) is
   `pipeline` for multi-project pipeline triggered through the API with `CI_JOB_TOKEN`.
-- [`only/except`](yaml/README.md#onlyexcept-basic) to control job behavior, use the
+- [`only/except`](yaml/README.md#only--except) to control job behavior, use the
   `pipelines` keyword.
 
 ## Creating multi-project pipelines from `.gitlab-ci.yml`
@@ -113,7 +113,7 @@ When using:
   the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md) is
   `pipeline` for multi-project pipelines triggered with a bridge job (using the
   [`trigger:`](yaml/README.md#trigger) keyword).
-- [`only/except`](yaml/README.md#onlyexcept-basic) to control job behavior, use the
+- [`only/except`](yaml/README.md#only--except) to control job behavior, use the
   `pipelines` keyword.
 
 In the example, `staging` is marked as successful as soon as a downstream pipeline

doc/ci/parent_child_pipelines.md

@@ -38,7 +38,7 @@ set of concurrently running child pipelines, but within the same project:
 
 Child pipelines work well with other GitLab CI/CD features:
 
-- Use [`only: changes`](yaml/README.md#onlychangesexceptchanges) to trigger pipelines only when
+- Use [`only: changes`](yaml/README.md#onlychanges--exceptchanges) to trigger pipelines only when
   certain files change. This is useful for monorepos, for example.
 - Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal
   pipelines, they can have their own behaviors and sequencing in relation to triggers.

doc/ci/pipelines/schedules.md

@@ -65,7 +65,7 @@ GitLab CI/CD so that they can be used in your [`.gitlab-ci.yml` file](../../ci/y
 
 To configure a job to be executed only when the pipeline has been
 scheduled (or the opposite), use
-[only and except](../yaml/README.md#onlyexcept-basic) configuration keywords.
+[only and except](../yaml/README.md#only--except) configuration keywords.
 
 In the example below `make world` runs in scheduled pipelines, and `make build` runs in pipelines that are not scheduled:
 

doc/ci/triggers/README.md

@@ -26,7 +26,7 @@ depending on which trigger method is used.
 | `pipeline`                  | Using the `trigger:` keyword in the CI/CD configuration file, or using the trigger API with `$CI_JOB_TOKEN`. |
 | `trigger`                   | Using the trigger API using a generated trigger token |
 
-This also applies when using the `pipelines` or `triggers` keywords with the legacy [`only/except` basic syntax](../yaml/README.md#onlyexcept-basic).
+This also applies when using the `pipelines` or `triggers` keywords with the legacy [`only/except` basic syntax](../yaml/README.md#only--except).
 
 ### Trigger token
 

doc/ci/troubleshooting.md

@@ -123,7 +123,7 @@ This is usually caused by the `rules` configuration, and there are several ways
 
 #### A job is not in the pipeline
 
-GitLab determines if a job is added to a pipeline based on the [`only/except`](yaml/README.md#onlyexcept-basic)
+GitLab determines if a job is added to a pipeline based on the [`only/except`](yaml/README.md#only--except)
 or [`rules`](yaml/README.md#rules) defined for the job. If it didn't run, it's probably
 not evaluating as you expect.
 
@@ -150,7 +150,7 @@ A common reason a job is added to a pipeline unexpectedly is because the `change
 keyword always evaluates to true in certain cases. For example, `changes` is always
 true in certain pipeline types, including scheduled pipelines and pipelines for tags.
 
-The `changes` keyword is used in combination with [`only/except`](yaml/README.md#onlychangesexceptchanges)
+The `changes` keyword is used in combination with [`only/except`](yaml/README.md#onlychanges--exceptchanges)
 or [`rules`](yaml/README.md#ruleschanges)). It's recommended to use `changes` with
 `rules` or `only/except` configuration that ensures the job is only added to branch
 pipelines or merge request pipelines.

doc/ci/variables/README.md

@@ -642,7 +642,7 @@ CI/CD variables with multi-line values are not supported.
 
 ## CI/CD variable expressions
 
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyexcept-advanced)
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyvariables--exceptvariables)
 > - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules)
 
 Use variable expressions to limit which jobs are created
@@ -651,7 +651,7 @@ in a pipeline after changes are pushed to GitLab.
 In `.gitlab-ci.yml`, variable expressions work with both:
 
 - [`rules`](../yaml/README.md#rules), which is the recommended approach, and
-- [`only` and `except`](../yaml/README.md#onlyexcept-basic), which are candidates for deprecation.
+- [`only` and `except`](../yaml/README.md#only--except), which are candidates for deprecation.
 
 This is particularly useful in combination with variables and triggered
 pipeline variables.
@@ -672,7 +672,7 @@ If any of the conditions in `variables` evaluates to true when using `only`,
 a new job is created. If any of the expressions evaluates to true
 when `except` is being used, a job is not created.
 
-This follows the usual rules for [`only` / `except` policies](../yaml/README.md#onlyexcept-advanced).
+This follows the usual rules for [`only` / `except` policies](../yaml/README.md#onlyvariables--exceptvariables).
 
 ### Syntax of CI/CD variable expressions
 

doc/development/cicd/templates.md

@@ -127,7 +127,7 @@ job2:
 
 #### Use `rules` instead of `only` or `except`
 
-Avoid using [`only` or `except`](../../ci/yaml/README.md#onlyexcept-basic) if possible.
+Avoid using [`only` or `except`](../../ci/yaml/README.md#only--except) if possible.
 Only and except is not being developed any more, and [`rules`](../../ci/yaml/README.md#rules)
 is now the preferred syntax:
 

doc/topics/autodevops/customize.md

@@ -244,8 +244,8 @@ include:
 See the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) for information on available jobs.
 
 WARNING:
-Auto DevOps templates using the [`only`](../../ci/yaml/README.md#onlyexcept-basic) or
-[`except`](../../ci/yaml/README.md#onlyexcept-basic) syntax have switched
+Auto DevOps templates using the [`only`](../../ci/yaml/README.md#only--except) or
+[`except`](../../ci/yaml/README.md#only--except) syntax have switched
 to the [`rules`](../../ci/yaml/README.md#rules) syntax, starting in
 [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213336).
 If your `.gitlab-ci.yml` extends these Auto DevOps templates and override the `only` or

doc/user/application_security/container_scanning/index.md

@@ -270,7 +270,7 @@ container_scanning_new:
 ```
 
 WARNING:
-GitLab 13.0 and later doesn't support [`only` and `except`](../../../ci/yaml/README.md#onlyexcept-basic).
+GitLab 13.0 and later doesn't support [`only` and `except`](../../../ci/yaml/README.md#only--except).
 When overriding the template, you must use [`rules`](../../../ci/yaml/README.md#rules)
 instead.
 

doc/user/application_security/dast/index.md

@@ -656,7 +656,7 @@ To view details of vulnerabilities detected by DAST:
 ### Customizing the DAST settings
 
 WARNING:
-Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#onlyexcept-basic)
+Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#only--except)
 is no longer supported. When overriding the template, you must use [`rules`](../../../ci/yaml/README.md#rules) instead.
 
 The DAST settings can be changed through CI/CD variables by using the

doc/user/application_security/dependency_scanning/index.md

@@ -130,7 +130,7 @@ configuration, the last mention of the variable takes precedence.
 ### Overriding dependency scanning jobs
 
 WARNING:
-Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#onlyexcept-basic)
+Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#only--except)
 is no longer supported. When overriding the template, you must use [`rules`](../../../ci/yaml/README.md#rules) instead.
 
 To override a job definition (for example, to change properties like `variables` or `dependencies`),

doc/user/application_security/index.md

@@ -451,7 +451,7 @@ Found errors in your .gitlab-ci.yml:
 ```
 
 This error appears when the included job's `rules` configuration has been [overridden](sast/index.md#overriding-sast-jobs)
-with [the deprecated `only` or `except` syntax.](../../ci/yaml/README.md#onlyexcept-basic)
+with [the deprecated `only` or `except` syntax.](../../ci/yaml/README.md#only--except)
 To fix this issue, you must either:
 
 - [Transition your `only/except` syntax to `rules`](#transitioning-your-onlyexcept-syntax-to-rules).
@@ -462,7 +462,7 @@ To fix this issue, you must either:
 #### Transitioning your `only/except` syntax to `rules`
 
 When overriding the template to control job execution, previous instances of
-[`only` or `except`](../../ci/yaml/README.md#onlyexcept-basic) are no longer compatible
+[`only` or `except`](../../ci/yaml/README.md#only--except) are no longer compatible
 and must be transitioned to [the `rules` syntax](../../ci/yaml/README.md#rules).
 
 If your override is aimed at limiting jobs to only run on `master`, the previous syntax

doc/user/application_security/sast/index.md

@@ -222,7 +222,7 @@ the pipeline configuration, the last mention of the variable takes precedence.
 ### Overriding SAST jobs
 
 WARNING:
-Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#onlyexcept-basic)
+Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#only--except)
 is no longer supported. When overriding the template, you must use [`rules`](../../../ci/yaml/README.md#rules) instead.
 
 To override a job definition, (for example, change properties like `variables` or `dependencies`),

doc/user/application_security/secret_detection/index.md

@@ -169,7 +169,7 @@ declare a job with the same name as the SAST job to override. Place this new job
 inclusion and specify any additional keys under it.
 
 WARNING:
-Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#onlyexcept-basic)
+Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#only--except)
 is no longer supported. When overriding the template, you must use [`rules`](../../../ci/yaml/README.md#rules) instead.
 
 #### GIT_DEPTH

doc/user/compliance/license_compliance/index.md

@@ -180,7 +180,7 @@ directory of your project.
 ### Overriding the template
 
 WARNING:
-Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#onlyexcept-basic)
+Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/README.md#only--except)
 is no longer supported. When overriding the template, you must use [`rules`](../../../ci/yaml/README.md#rules) instead.
 
 If you want to override the job definition (for example, change properties like

doc/user/project/merge_requests/merge_when_pipeline_succeeds.md

@@ -67,7 +67,7 @@ You should be careful to configure CI/CD so that pipelines run for every merge r
 ### Limitations
 
 When this setting is enabled, a merge request is prevented from being merged if there
-is no pipeline. This may conflict with some use cases where [`only/except`](../../../ci/yaml/README.md#onlyexcept-advanced)
+is no pipeline. This may conflict with some use cases where [`only/except`](../../../ci/yaml/README.md#only--except)
 or [`rules`](../../../ci/yaml/README.md#rules) are used and they don't generate any pipelines.
 
 You should ensure that [there is always a pipeline](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/54226)

doc/user/project/pages/introduction.md

@@ -129,7 +129,7 @@ See this document for a [step-by-step guide](getting_started/pages_from_scratch.
 
 Remember that GitLab Pages are by default branch/tag agnostic and their
 deployment relies solely on what you specify in `.gitlab-ci.yml`. You can limit
-the `pages` job with the [`only` parameter](../../../ci/yaml/README.md#onlyexcept-basic),
+the `pages` job with the [`only` parameter](../../../ci/yaml/README.md#only--except),
 whenever a new commit is pushed to a branch used specifically for your
 pages.