Merge branch '38325-variables-usage-in-gitlab-ci-yml-docs' into 'master'

Add documentation about variables usage in GitLab CI

Closes #38325

See merge request gitlab-org/gitlab-ce!19075

doc/ci/README.md

@@ -19,7 +19,7 @@ Here's some info we've gathered to get you started.
 The first steps towards your GitLab CI/CD journey.
 
 - [Getting started with GitLab CI/CD](quick_start/README.md): understand how GitLab CI/CD works.
-- GitLab CI/CD configuration file: [`.gitlab-ci.yml`](yaml/README.md) - Learn all about the ins and outs of `.gitlab-ci.yml`.
+- [GitLab CI/CD configuration file: `.gitlab-ci.yml`](yaml/README.md) - Learn all about the ins and outs of `.gitlab-ci.yml`.
 - [Pipelines and jobs](pipelines.md): configure your GitLab CI/CD pipelines to build, test, and deploy your application.
 - Runners: The [GitLab Runner](https://docs.gitlab.com/runner/) is responsible by running the jobs in your CI/CD pipeline. On GitLab.com, Shared Runners are enabled by default, so
 you don't need to set up anything to start to use them with GitLab CI/CD.
@@ -46,7 +46,9 @@ you don't need to set up anything to start to use them with GitLab CI/CD.
 ## Exploring GitLab CI/CD
 
 - [CI/CD Variables](variables/README.md) - Learn how to use variables defined in
-  your `.gitlab-ci.yml` or secured ones defined in your project's settings
+  your `.gitlab-ci.yml` or the ones defined in your project's settings
+  - [Where variables can be used](variables/where_variables_can_be_used.md) - A
+    deeper look on where and how the CI/CD variables can be used
 - **The permissions model** - Learn about the access levels a user can have for
   performing certain CI actions
   - [User permissions](../user/permissions.md#gitlab-ci)

doc/ci/environments.md

@@ -246,23 +246,14 @@ As the name suggests, it is possible to create environments on the fly by just
 declaring their names dynamically in `.gitlab-ci.yml`. Dynamic environments is
 the basis of [Review apps](review_apps/index.md).
 
->**Note:**
-The `name` and `url` parameters can use most of the defined CI variables,
-including predefined, secure variables and `.gitlab-ci.yml`
-[`variables`](yaml/README.md#variables). You however cannot use variables
-defined under `script` or on the Runner's side. There are other variables that
-are unsupported in environment name context:
-- `CI_PIPELINE_ID`
-- `CI_JOB_ID`
-- `CI_JOB_TOKEN`
-- `CI_BUILD_ID`
-- `CI_BUILD_TOKEN`
-- `CI_REGISTRY_USER`
-- `CI_REGISTRY_PASSWORD`
-- `CI_REPOSITORY_URL`
-- `CI_ENVIRONMENT_URL`
-- `CI_DEPLOY_USER`
-- `CI_DEPLOY_PASSWORD`
+NOTE: **Note:**
+The `name` and `url` parameters can use most of the CI/CD variables,
+including [predefined](variables/README.md#predefined-variables-environment-variables),
+[secret](variables/README.md#secret-variables) and
+[`.gitlab-ci.yml` variables](yaml/README.md#variables). You however cannot use variables
+defined under `script` or on the Runner's side. There are also other variables that
+are unsupported in the context of `environment:name`. You can read more about
+[where variables can be used](variables/where_variables_can_be_used.md).
 
 GitLab Runner exposes various [environment variables][variables] when a job runs,
 and as such, you can use them as environment names. Let's add another job in

doc/ci/variables/README.md

@@ -22,6 +22,12 @@ For example, if you define `API_TOKEN=secure` as a secret variable and
 `API_TOKEN=yaml` in your `.gitlab-ci.yml`, the `API_TOKEN` will take the value
 `secure` as the secret variables are higher in the chain.
 
+## Unsupported variables
+
+There are cases where some variables cannot be used in the context of a
+`.gitlab-ci.yml` definition (for example under `script`). Read more
+about which variables are [not supported](where_variables_can_be_used.md).
+
 ## Predefined variables (Environment variables)
 
 Some of the predefined environment variables are available only if a minimum
@@ -36,6 +42,7 @@ future GitLab releases.**
 
 | Variable                        | GitLab | Runner | Description |
 |-------------------------------- |--------|--------|-------------|
+| **ARTIFACT_DOWNLOAD_ATTEMPTS**  | 8.15   | 1.9    | Number of attempts to download artifacts running a job |
 | **CI**                          | all    | 0.4    | Mark that job is executed in CI environment |
 | **CI_COMMIT_REF_NAME**          | 9.0    | all    | The branch or tag name for which project is built |
 | **CI_COMMIT_REF_SLUG**          | 9.0    | all    | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. |
@@ -46,6 +53,8 @@ future GitLab releases.**
 | **CI_COMMIT_DESCRIPTION**       | 10.8   | all    | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. |
 | **CI_CONFIG_PATH**              | 9.4    | 0.5    | The path to CI config file. Defaults to `.gitlab-ci.yml` |
 | **CI_DEBUG_TRACE**              | all    | 1.7    | Whether [debug tracing](#debug-tracing) is enabled |
+| **CI_DEPLOY_USER**              | 10.8   | all    | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.|
+| **CI_DEPLOY_PASSWORD**          | 10.8   | all    | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.|
 | **CI_DISPOSABLE_ENVIRONMENT**   | all    | 10.1   | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. |
 | **CI_ENVIRONMENT_NAME**         | 8.15   | all    | The name of the environment for this job |
 | **CI_ENVIRONMENT_SLUG**         | 8.15   | all    | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. |
@@ -82,16 +91,13 @@ future GitLab releases.**
 | **CI_SERVER_REVISION**          | all    | all    | GitLab revision that is used to schedule jobs |
 | **CI_SERVER_VERSION**           | all    | all    | GitLab version that is used to schedule jobs |
 | **CI_SHARED_ENVIRONMENT**       | all    | 10.1   | Marks that the job is executed in a shared environment (something that is persisted across CI invocations like `shell` or `ssh` executor). If the environment is shared, it is set to true, otherwise it is not defined at all. |
-| **ARTIFACT_DOWNLOAD_ATTEMPTS**  | 8.15   | 1.9    | Number of attempts to download artifacts running a job |
 | **GET_SOURCES_ATTEMPTS**        | 8.15   | 1.9    | Number of attempts to fetch sources running a job |
 | **GITLAB_CI**                   | all    | all    | Mark that job is executed in GitLab CI environment |
-| **GITLAB_USER_ID**              | 8.12   | all    | The id of the user who started the job |
 | **GITLAB_USER_EMAIL**           | 8.12   | all    | The email of the user who started the job |
+| **GITLAB_USER_ID**              | 8.12   | all    | The id of the user who started the job |
 | **GITLAB_USER_LOGIN**           | 10.0   | all    | The login username of the user who started the job |
 | **GITLAB_USER_NAME**            | 10.0   | all    | The real name of the user who started the job |
 | **RESTORE_CACHE_ATTEMPTS**      | 8.15   | 1.9    | Number of attempts to restore the cache running a job |
-| **CI_DEPLOY_USER**              | 10.8   | all    | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.|
-| **CI_DEPLOY_PASSWORD**          | 10.8   | all    | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.|
 
 ## 9.0 Renaming
 
@@ -540,34 +546,6 @@ Below you can find supported syntax reference:
     Pattern matching is case-sensitive by default. Use `i` flag modifier, like
     `/pattern/i` to make a pattern case-insensitive.
 
-### Unsupported predefined variables
-
-Because GitLab evaluates variables before creating jobs, we do not support a
-few variables that depend on persistence layer, like `$CI_JOB_ID`.
-
-Environments (like `production` or `staging`) are also being created based on
-what jobs pipeline consists of, thus some environment-specific variables are
-not supported as well.
-
-We do not support variables containing tokens because of security reasons.
-
-You can find a full list of unsupported variables below:
-
-- `CI_PIPELINE_ID`
-- `CI_JOB_ID`
-- `CI_JOB_TOKEN`
-- `CI_BUILD_ID`
-- `CI_BUILD_TOKEN`
-- `CI_REGISTRY_USER`
-- `CI_REGISTRY_PASSWORD`
-- `CI_REPOSITORY_URL`
-- `CI_ENVIRONMENT_URL`
-- `CI_DEPLOY_USER`
-- `CI_DEPLOY_PASSWORD`
-
-These variables are also not supported in a context of a
-[dynamic environment name][dynamic-environments].
-
 [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI secret variables"
 [eep]: https://about.gitlab.com/products/ "Available only in GitLab Premium"
 [envs]: ../environments.md
@@ -579,5 +557,4 @@ These variables are also not supported in a context of a
 [triggers]: ../triggers/README.md#pass-job-variables-to-a-trigger
 [subgroups]: ../../user/group/subgroups/index.md
 [builds-policies]: ../yaml/README.md#only-and-except-complex
-[dynamic-environments]: ../environments.md#dynamic-environments
 [gitlab-deploy-token]: ../../user/project/deploy_tokens/index.md#gitlab-deploy-token

doc/ci/variables/where_variables_can_be_used.md

+# Where variables can be used
+
+As it's described in the [CI/CD variables](README.md) docs, you can
+define many different variables. Some of them can be used for all GitLab CI/CD
+features, but some of them are more or less limited.
+
+This document describes where and how the different types of variables can be used.
+
+## Variables usage
+
+There are basically two places where you can use any defined variables:
+
+1. On GitLab's side there's `.gitlab-ci.yml`
+1. On the Runner's side there's `config.toml`
+
+### `.gitlab-ci.yml` file
+
+| Definition                           | Can be expanded?  | Expansion place | Description  |
+|--------------------------------------|-------------------|-----------------|--------------|
+| `environment:url`                    | yes               | GitLab           | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<ul><li>**Supported:** all variables defined for a job (secret variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules)</li><li>**Not suported:** variables defined in Runner's `config.toml` and variables created in job's `script`</li></ul> |
+| `environment:name`                   | yes               | GitLab           | Similar to `environment:url`, but the variables expansion **doesn't support**: <ul><li>variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`)</li><li>[persisted variables](#persisted-variables)</li></ul> |
+| `variables` | yes               | Runner          | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) |
+| `image`          | yes               | Runner          | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) |
+| `services:[]` | yes           | Runner          | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) |
+| `services:[]:name` | yes  | Runner          | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) |
+| `cache:key` | yes               | Runner          | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) |
+| `artifacts:name`                     | yes               | Runner          | The variable expansion is made by GitLab Runner's shell environment  |
+| `script`, `before_script`, `after_script` | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) |
+| `only:variables:[]`, `except:variables:[]`    | no                | n/a             | The variable must be in the form of `$variable`.<br/>**Not supported:**<ul><li>variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`)</li><li>[persisted variables](#persisted-variables)</li></ul> |
+
+### `config.toml` file
+
+NOTE: **Note:**
+You can read more about `config.toml` in the [Runner's docs](https://docs.gitlab.com/runner/configuration/advanced-configuration.html).
+
+| Definition                           | Can be expanded? | Description |
+|--------------------------------------|------------------|-------------|
+| `runners.environment`                | yes              | The variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) |
+| `runners.kubernetes.pod_labels`      | yes              | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) |
+| `runners.kubernetes.pod_annotations` | yes              | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) |
+
+## Expansion mechanisms
+
+There are three expansion mechanisms:
+
+- GitLab
+- GitLab Runner
+- Execution shell environment
+
+### GitLab internal variable expansion mechanism
+
+The expanded part needs to be in a form of `$variable`, or `${variable}` or `%variable%`.
+Each form is handled in the same way, no matter which OS/shell will finally handle the job,
+since the expansion is done in GitLab before any Runner will get the job.
+
+### GitLab Runner internal variable expansion mechanism
+
+- **Supported:** secret variables, `.gitlab-ci.yml` variables, `config.toml` variables, and
+  variables from triggers and pipeline schedules
+- **Not supported:** variables defined inside of scripts (e.g., `export MY_VARIABLE="test"`)
+
+The Runner uses Go's `os.Expand()` method for variable expansion. It means that it will handle
+only variables defined as `$variable` and `${variable}`. What's also important, is that
+the expansion is done only once, so nested variables may or may not work, depending on the
+ordering of variables definitions.
+
+### Execution shell environment
+
+This is an expansion that takes place during the `script` execution.
+How it works depends on the used shell (bash/sh/cmd/PowerShell). For example, if the job's
+`script` contains a line `echo $MY_VARIABLE-${MY_VARIABLE_2}`, it should be properly handled
+by bash/sh (leaving empty strings or some values depending whether the variables were
+defined or not), but will not work with Windows' cmd/PowerShell, since these shells
+are using a different variables syntax.
+
+**Supported:**
+
+- The `script` may use all available variables that are default for the shell (e.g., `$PATH` which
+  should be present in all bash/sh shells) and all variables defined by GitLab CI/CD (secret variables,
+  `.gitlab-ci.yml` variables, `config.toml` variables, and variables from triggers and pipeline schedules).
+- The `script` may also use all variables defined in the lines before. So, for example, if you define
+  a variable `export MY_VARIABLE="test"`:
+
+  - in `before_script`, it will work in the following lines of `before_script` and
+    all lines of the related `script`
+  - in `script`, it will work in the following lines of `script`
+  - in `after_script`, it will work in following lines of `after_script`
+
+## Persisted variables
+
+NOTE: **Note:**
+Some of the persisted variables contain tokens and cannot be used by some definitions
+due to security reasons.
+
+The following variables are known as "persisted":
+
+- `CI_PIPELINE_ID`
+- `CI_JOB_ID`
+- `CI_JOB_TOKEN`
+- `CI_BUILD_ID`
+- `CI_BUILD_TOKEN`
+- `CI_REGISTRY_USER`
+- `CI_REGISTRY_PASSWORD`
+- `CI_REPOSITORY_URL`
+- `CI_DEPLOY_USER`
+- `CI_DEPLOY_PASSWORD`
+
+They are:
+
+- **supported** for all definitions as [described in the table](#gitlab-ci-yml-file) where the "Expansion place" is "Runner"
+- **not supported:**
+  - by the definitions [described in the table](#gitlab-ci-yml-file) where the "Expansion place" is "GitLab"
+  - in the `only` and `except` [variables expressions](README.md#variables-expressions)