Skip to content

G
gitlab-ce
Commit fb31b832 authored by Marcel Amirault's avatar Marcel Amirault
Browse files
Options

Merge branch 'selhorn-multi-project-pipeline-edit' into 'master' 

Edited and reordered topics in Multi-project pipelines

See merge request gitlab-org/gitlab!64454
parents b1b4055c a62d48c3
Show whitespace changes
Inline Side-by-side
Showing with 107 additions and 134 deletions
doc/ci/jobs/job_control.md
View file @ fb31b832
......@@ -222,7 +222,7 @@ check the value of the `$CI_PIPELINE_SOURCE` variable:
| `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). |
| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). |
| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../parent_child_pipelines.md) with `rules`. Use this pipeline source in the child pipeline configuration so that it can be triggered by the parent pipeline. |
| `pipeline` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](../yaml/README.md#trigger) keyword. |
| `pipeline` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](../yaml/README.md#trigger) keyword. |
| `push` | For pipelines triggered by a `git push` event, including for branches and tags. |
| `schedule` | For [scheduled pipelines](../pipelines/schedules.md). |
| `trigger` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). |
......
doc/ci/multi_project_pipelines.md
View file @ fb31b832
......@@ -10,79 +10,39 @@ type: reference
> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8.
You can set up [GitLab CI/CD](README.md) across multiple projects, so that a pipeline
in one project can trigger a pipeline in another project.
in one project can trigger a pipeline in another project. You can visualize the entire pipeline
in one place, including all cross-project interdependencies.
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
For an overview see the [Multi-project pipelines demo](https://www.youtube.com/watch?v=g_PIwBM1J84).
GitLab CI/CD is a powerful continuous integration tool that works not only per project,
but also across projects with multi-project pipelines.
Multi-project pipelines are useful for larger products that require cross-project inter-dependencies, such as those
adopting a [microservices architecture](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/).
Cross-functional development teams can use cross-pipeline
triggering to trigger multiple pipelines for different microservices projects. Learn more
in the [Cross-project Pipeline Triggering and Visualization demo](https://about.gitlab.com/learn/)
at GitLab@learn, in the Continuous Integration (CI) section.
Additionally, it's possible to visualize the entire pipeline, including all cross-project
inter-dependencies. **(PREMIUM)**
## Use cases
Let's assume you deploy your web app from different projects in GitLab:
- One for the free version, which has its own pipeline that builds and tests your app
- One for the paid version add-ons, which also pass through builds and tests
- One for the documentation, which also builds, tests, and deploys with an SSG
With Multi-Project Pipelines you can visualize the entire pipeline, including all build and test stages for the three projects.
## Multi-project pipeline visualization **(PREMIUM)**
When you configure GitLab CI/CD for your project, you can visualize the stages of your
[jobs](pipelines/index.md#configure-a-pipeline) on a [pipeline graph](pipelines/index.md#visualize-pipelines).
![Multi-project pipeline graph](img/multi_project_pipeline_graph.png)
In the Merge Request Widget, multi-project pipeline mini-graphs are displayed,
and when hovering or tapping (on touchscreen devices) they expand and are shown adjacent to each other.
For example, you might deploy your web application from three different projects in GitLab.
Each project has its own build, test, and deploy process. With multi-project pipelines you can
visualize the entire pipeline, including all build and test stages for all three projects.
![Multi-project mini graph](img/multi_pipeline_mini_graph.gif)
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
For an overview, see the [Multi-project pipelines demo](https://www.youtube.com/watch?v=g_PIwBM1J84).
## Triggering multi-project pipelines through API
Multi-project pipelines are also useful for larger products that require cross-project interdependencies, like those
with a [microservices architecture](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/).
Learn more in the [Cross-project Pipeline Triggering and Visualization demo](https://about.gitlab.com/learn/)
at GitLab@learn, in the Continuous Integration section.
> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) to GitLab Free in 12.4.
If you trigger a pipeline in a downstream private project, the name of the project
and the status of the pipeline is visible in the upstream project's pipelines page.
Make sure that there are no confidentiality problems if you have a public project
that can trigger downstream pipelines in a private project.
When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab
recognizes the source of the job token, and thus internally ties these pipelines
together, allowing you to visualize their relationships on pipeline graphs.
## Create multi-project pipelines
These relationships are displayed in the pipeline graph by showing inbound and
outbound connections for upstream and downstream pipeline dependencies.
To create multi-project pipelines, you can:
When using:
- [Use the CI/CD configuration syntax](#create-multi-project-pipelines-in-your-gitlab-ciyml-file).
- [Use the API](#create-multi-project-pipelines-by-using-the-api).
- CI/CD Variables or [`rules`](yaml/README.md#rulesif) to control job behavior, the value of
the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md) is
`pipeline` for multi-project pipeline triggered through the API with `CI_JOB_TOKEN`.
- [`only/except`](yaml/README.md#only--except) to control job behavior, use the
`pipelines` keyword.
## Creating multi-project pipelines from `.gitlab-ci.yml`
### Create multi-project pipelines in your `.gitlab-ci.yml` file
> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8.
### Triggering a downstream pipeline using a bridge job
Before GitLab 11.8, it was necessary to implement a pipeline job that was
responsible for making the API request [to trigger a pipeline](#triggering-multi-project-pipelines-through-api)
in a different project.
In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this
task easier, and avoid needing GitLab Runner for triggering cross-project
pipelines. The following illustrates configuring a bridge job:
When you create a multi-project pipeline in your `.gitlab-ci.yml` file,
you create what is called a "trigger job". For example:
```yaml
rspec:
......@@ -96,38 +56,39 @@ staging:
trigger: my/deployment
```
In the example above, as soon as the `rspec` job succeeds in the `test` stage,
the `staging` bridge job is started. The initial status of this
job is `pending`. GitLab then creates a downstream pipeline in the
`my/deployment` project and, as soon as the pipeline is created, the
`staging` job succeeds. `my/deployment` is a full path to that project.
In this example, after the `rspec` job succeeds in the `test` stage,
the `staging` trigger job starts. The initial status of this
job is `pending`.
The user that created the upstream pipeline needs to have access rights to the
downstream project (`my/deployment` in this case). If a downstream project is
not found, or a user does not have access rights to create a pipeline there,
the `staging` job is marked as _failed_.
GitLab then creates a downstream pipeline in the
`my/deployment` project and, as soon as the pipeline is created, the
`staging` job succeeds. The full path to the project is `my/deployment`.
When using:
You can view the status for the pipeline, or you can display
[the downstream pipeline's status instead](#mirror-status-of-a-triggered-pipeline-in-the-trigger-job).
- CI/CD variables or [`rules`](yaml/README.md#rulesif) to control job behavior, the value of
the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md) is
`pipeline` for multi-project pipelines triggered with a bridge job (using the
[`trigger:`](yaml/README.md#trigger) keyword).
- [`only/except`](yaml/README.md#only--except) to control job behavior, use the
`pipelines` keyword.
The user that creates the upstream pipeline must be able to create pipelines in the
downstream project (`my/deployment`) too. If the downstream project is not found,
or the user does not have [permission](../user/permissions.md) to create a pipeline there,
the `staging` job is marked as _failed_.
In the example, `staging` is marked as successful as soon as a downstream pipeline
gets created. If you want to display the downstream pipeline's status instead, see
[Mirroring status from triggered pipeline](#mirroring-status-from-triggered-pipeline).
Trigger jobs can use only a limited set of the GitLab CI/CD [configuration keywords](yaml/README.md).
The keywords available for use in trigger jobs are:
NOTE:
Bridge jobs [do not support every configuration keyword](#limitations) that can be used
with other jobs. If a user tries to use unsupported configuration keywords, YAML
validation fails on pipeline creation.
- [`trigger`](yaml/README.md#trigger)
- [`stage`](yaml/README.md#stage)
- [`allow_failure`](yaml/README.md#allow_failure)
- [`rules`](yaml/README.md#rules)
- [`only` and `except`](yaml/README.md#only--except)
- [`when`](yaml/README.md#when) (only with a value of `on_success`, `on_failure`, or `always`)
- [`extends`](yaml/README.md#extends)
- [`needs`](yaml/README.md#needs)
### Specifying a downstream pipeline branch
#### Specify a downstream pipeline branch
It is possible to specify a branch name that a downstream pipeline uses:
You can specify a branch name for the downstream pipeline to use.
GitLab uses the commit on the head of the branch to
create the downstream pipeline.
```yaml
rspec:
......@@ -145,25 +106,18 @@ Use:
- The `project` keyword to specify the full path to a downstream project.
- The `branch` keyword to specify the name of a branch in the project specified by `project`.
[From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/10126), variable expansion is
[In GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/10126) and later, variable expansion is
supported.
GitLab uses a commit that is on the head of the branch when
creating a downstream pipeline.
NOTE:
Pipelines triggered on a protected branch in a downstream project use the [permissions](../user/permissions.md)
Pipelines triggered on a protected branch in a downstream project use the [role](../user/permissions.md)
of the user that ran the trigger job in the upstream project. If the user does not
have permission to run CI/CD pipelines against the protected branch, the pipeline fails. See
[pipeline security for protected branches](pipelines/index.md#pipeline-security-on-protected-branches).
### Passing CI/CD variables to a downstream pipeline
#### With the `variables` keyword
#### Pass CI/CD variables to a downstream pipeline by using the `variables` keyword
Sometimes you might want to pass CI/CD variables to a downstream pipeline.
You can do that using the `variables` keyword, just like you would when
defining a regular job.
You can do that by using the `variables` keyword, just like you would for any other job.
```yaml
rspec:
......@@ -206,14 +160,14 @@ downstream-job:
```
In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the
upstream pipeline is passed to the `downstream-job` job, and is available
within the context of all downstream builds.
upstream pipeline is passed to the `downstream-job` job. It is available
in the context of all downstream builds.
Upstream pipelines take precedence over downstream ones. If there are two
variables with the same name defined in both upstream and downstream projects,
the ones defined in the upstream project take precedence.
#### With variable inheritance
#### Pass CI/CD variables to a downstream pipeline by using variable inheritance
You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](variables/README.md#pass-an-environment-variable-to-another-job) and [cross project artifact downloads](yaml/README.md#cross-project-artifact-downloads-with-needs).
......@@ -253,13 +207,24 @@ test:
artifacts: true
```
### Mirroring status from triggered pipeline
#### Use `rules` or `only`/`except` with multi-project pipelines
You can use CI/CD variables or the [`rules`](yaml/README.md#rulesif) keyword to
[control job behavior](jobs/job_control.md) for multi-project pipelines. When a
downstream pipeline is triggered with the [`trigger`](yaml/README.md#trigger) keyword,
the value of the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md)
is `pipeline` for all its jobs.
If you use [`only/except`](yaml/README.md#only--except) to control job behavior, use the
[`pipelines`](yaml/README.md#onlyrefs--exceptrefs) keyword.
#### Mirror status of a triggered pipeline in the trigger job
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11238) in GitLab Premium 12.3.
> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8.
You can mirror the pipeline status from the triggered pipeline to the source
bridge job by using `strategy: depend`. For example:
trigger job by using `strategy: depend`. For example:
```yaml
trigger_job:
......@@ -268,7 +233,7 @@ trigger_job:
strategy: depend
```
### Mirroring status from upstream pipeline
#### Mirror status from upstream pipeline
You can mirror the pipeline status from an upstream pipeline to a bridge job by
using the `needs:pipeline` keyword. The latest pipeline status from the default branch is
......@@ -283,24 +248,24 @@ upstream_bridge:
pipeline: other/project
```
### Limitations
### Create multi-project pipelines by using the API
Bridge jobs are a little different from regular jobs. It is not
possible to use exactly the same configuration syntax as when defining regular jobs
that are picked by a runner.
> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) to GitLab Free in 12.4.
Some features are not implemented yet. For example, support for environments.
When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token),
GitLab recognizes the source of the job token. The pipelines become related,
so you can visualize their relationships on pipeline graphs.
[Configuration keywords](yaml/README.md) available for bridge jobs are:
These relationships are displayed in the pipeline graph by showing inbound and
outbound connections for upstream and downstream pipeline dependencies.
- `trigger` (to define a downstream pipeline trigger)
- `stage`
- `allow_failure`
- [`rules`](yaml/README.md#rules)
- `only` and `except`
- `when` (only with `on_success`, `on_failure`, and `always` values)
- `extends`
- `needs`
When using:
- CI/CD Variables or [`rules`](yaml/README.md#rulesif) to control job behavior, the value of
the [`$CI_PIPELINE_SOURCE` predefined variable](variables/predefined_variables.md) is
`pipeline` for multi-project pipeline triggered through the API with `CI_JOB_TOKEN`.
- [`only/except`](yaml/README.md#only--except) to control job behavior, use the
`pipelines` keyword.
## Trigger a pipeline when an upstream project is rebuilt **(PREMIUM)**
......@@ -320,13 +285,17 @@ number of upstream pipeline subscriptions is 2 by default, for both the upstream
downstream projects. This [application limit](../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) can be changed on self-managed instances by a GitLab administrator.
The upstream project needs to be [public](../public_access/public_access.md)
and the user must have [developer permissions](../user/permissions.md#project-members-permissions)
for the upstream project.
and the user must have the [developer role](../user/permissions.md#project-members-permissions)
in the upstream project.
## Multi-project pipeline visualization **(PREMIUM)**
## Downstream private projects confidentiality concern
When you configure GitLab CI/CD for your project, you can visualize the stages of your
[jobs](pipelines/index.md#configure-a-pipeline) on a [pipeline graph](pipelines/index.md#visualize-pipelines).
If you trigger a pipeline in a downstream private project, the name of the project
and the status of the pipeline is visible in the upstream project's pipelines page.
![Multi-project pipeline graph](img/multi_project_pipeline_graph.png)
If you have a public project that can trigger downstream pipelines in a private
project, make sure to check that there are no confidentiality problems.
In the Merge Request Widget, multi-project pipeline mini-graphs are displayed.
They expand and are shown adjacent to each other when hovering (or tapping on touchscreen devices).
![Multi-project mini graph](img/multi_pipeline_mini_graph.gif)
doc/ci/parent_child_pipelines.md
View file @ fb31b832
......@@ -85,7 +85,7 @@ microservice_a:
The maximum number of entries that are accepted for `trigger:include:` is three.
Similar to [multi-project pipelines](multi_project_pipelines.md#mirroring-status-from-triggered-pipeline),
Similar to [multi-project pipelines](multi_project_pipelines.md#mirror-status-of-a-triggered-pipeline-in-the-trigger-job),
we can set the parent pipeline to depend on the status of the child pipeline upon completion:
```yaml
......@@ -185,5 +185,8 @@ For an overview, see [Nested Dynamic Pipelines](https://youtu.be/C5j3ju9je2M).
## Pass CI/CD variables to a child pipeline
You can [pass CI/CD variables to a downstream pipeline](multi_project_pipelines.md#passing-cicd-variables-to-a-downstream-pipeline)
the same way as for multi-project pipelines.
You can pass CI/CD variables to a downstream pipeline using the same methods as
multi-project pipelines:
- [By using the `variable` keyword](multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword).
- [By using variable inheritance](multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-variable-inheritance).
doc/ci/variables/README.md
View file @ fb31b832
......@@ -581,7 +581,8 @@ You can override the value of a variable when you:
1. Run a job manually in the UI.
1. Use [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd).
1. Trigger a pipeline by using [the API](../triggers/README.md#making-use-of-trigger-variables).
1. Pass variables to a [downstream pipeline](../multi_project_pipelines.md#passing-cicd-variables-to-a-downstream-pipeline).
1. Pass variables to a downstream pipeline [by using the `variable` keyword](../multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword)
or [by using variable inheritance](../multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-variable-inheritance).
The pipeline variables declared in these events take [priority over other variables](#cicd-variable-precedence).
......
doc/ci/yaml/README.md
View file @ fb31b832
......@@ -1336,7 +1336,7 @@ pipeline based on branch names or pipeline types.
| `external` | When you use CI services other than GitLab. |
| `external_pull_requests` | When an external pull request on GitHub is created or updated (See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests)). |
| `merge_requests` | For pipelines created when a merge request is created or updated. Enables [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). |
| `pipelines` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](#trigger) keyword. |
| `pipelines` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#create-multi-project-pipelines-by-using-the-api), or the [`trigger`](#trigger) keyword. |
| `pushes` | For pipelines triggered by a `git push` event, including for branches and tags. |
| `schedules` | For [scheduled pipelines](../pipelines/schedules.md). |
| `tags` | When the Git reference for a pipeline is a tag. |
......@@ -3561,13 +3561,13 @@ deploystacks: [vultr, data]
Use `trigger` to define a downstream pipeline trigger. When GitLab starts a `trigger` job,
a downstream pipeline is created.
Jobs with `trigger` can only use a [limited set of keywords](../multi_project_pipelines.md#limitations).
Jobs with `trigger` can only use a [limited set of keywords](../multi_project_pipelines.md#create-multi-project-pipelines-in-your-gitlab-ciyml-file).
For example, you can't run commands with [`script`](#script), [`before_script`](#before_script),
or [`after_script`](#after_script).
You can use this keyword to create two different types of downstream pipelines:
- [Multi-project pipelines](../multi_project_pipelines.md#creating-multi-project-pipelines-from-gitlab-ciyml)
- [Multi-project pipelines](../multi_project_pipelines.md#create-multi-project-pipelines-in-your-gitlab-ciyml-file)
- [Child pipelines](../parent_child_pipelines.md)
[In GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/) and later, you can
......@@ -3642,7 +3642,7 @@ trigger_job:
include: path/to/child-pipeline.yml
```
Similar to [multi-project pipelines](../multi_project_pipelines.md#mirroring-status-from-triggered-pipeline),
Similar to [multi-project pipelines](../multi_project_pipelines.md#mirror-status-of-a-triggered-pipeline-in-the-trigger-job),
it's possible to mirror the status from a triggered pipeline:
```yaml
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7