Skip to content

G
gitlab-ce
Commit 478b4354 authored by Rémy Coutable's avatar Rémy Coutable
Browse files
Options

Simplify and clarify further pipeline type graphs and explanation 

Signed-off-by: default avatarRémy Coutable <remy@rymai.me>
parent 0d9577be
Hide whitespace changes
Inline Side-by-side
Showing with 57 additions and 161 deletions
doc/development/pipelines.md
View file @ 478b4354
...@@ -266,227 +266,123 @@ We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administratio ...@@ -266,227 +266,123 @@ We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administratio
## Pipelines types for merge requests ## Pipelines types for merge requests
In general, pipelines for an MR fall into one or more of the following types, In general, pipelines for an MR fall into one of the following types (from shorter to longer), depending on the changes made in the MR:
depending on the changes made in the MR:
- [Documentation only MR pipeline](#documentation-only-mr-pipeline): This is typically created for an MR that only changes documentation. - [Documentation pipeline](#documentation-pipeline): For MRs that touch documentation.
- [Code-only MR pipeline](#code-only-mr-pipeline): This is typically created for an MR that only changes code, either backend or frontend. - [Backend pipeline](#backend-pipeline): For MRs that touch backend code.
- [Frontend-only MR pipeline](#frontend-only-mr-pipeline): This is typically created for an MR that only changes frontend code. - [Frontend pipeline](#frontend-pipeline): For MRs that touch frontend code.
- [QA-only MR pipeline](#qa-only-mr-pipeline): This is typically created for an MR that only changes end to end tests related code. - [End-to-end pipeline](#end-to-end-pipeline): For MRs that touch code in the `qa/` folder.
A "pipeline type" is an abstract term that mostly describes the "critical path" (i.e. the chain of jobs for which the sum
of individual duration equals the pipeline's duration).
We use these "pipeline types" in [metrics dashboards](https://app.periscopedata.com/app/gitlab/858266/GitLab-Pipeline-Durations)
in order to detect what types and jobs need to be optimized first.
An MR that touches multiple areas would be associated with the longest type applicable. For instance, an MR that touches backend
and frontend would fall into the "Frontend" pipeline type since this type takes longer to finish than the "Backend" pipeline type.
We use the [`rules:`](../ci/yaml/index.md#rules) and [`needs:`](../ci/yaml/index.md#needs) keywords extensively We use the [`rules:`](../ci/yaml/index.md#rules) and [`needs:`](../ci/yaml/index.md#needs) keywords extensively
to determine the jobs that need to be run in a pipeline. Note that an MR that includes multiple types of changes would to determine the jobs that need to be run in a pipeline. Note that an MR that includes multiple types of changes would
have a pipelines that include jobs from multiple types (for example, a combination of docs-only and code-only pipelines). have a pipelines that include jobs from multiple types (for example, a combination of docs-only and code-only pipelines).
### Documentation only MR pipeline Following are graphs of the critical paths for each pipeline type. Jobs that aren't part of the critical path are ommitted.
### Documentation pipeline
[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/250546928): [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/432049110).
```mermaid ```mermaid
graph LR graph LR
classDef criticalPath fill:#f66; classDef criticalPath fill:#f66;
subgraph "Jobs starting immediately"; 1-3["docs-lint links (5 minutes)"];
1-1["danger-review (2.2 minutes)"]; class 1-3 criticalPath;
click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0"
1-2["docs-lint markdown (1.5 minutes)"];
click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=10224335&udv=0"
1-3["docs-lint links (5 minutes)"];
class 1-3 criticalPath;
click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0"
1-4["ui-docs-links lint (2.5 minutes)"];
click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=10823717&udv=1020379"
end
``` ```
### Code-only MR pipeline ### Backend pipeline
[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/136295694) [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/433316063).
```mermaid ```mermaid
graph RL; graph RL;
classDef criticalPath fill:#f66; classDef criticalPath fill:#f66;
subgraph "Jobs starting immediately"; 1-3["compile-test-assets (6 minutes)"];
1-2["build-qa-image"]; class 1-3 criticalPath;
click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0"
1-3["compile-test-assets (6 minutes)"]; 1-6["setup-test-env (4 minutes)"];
class 1-3 criticalPath; click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0"
click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" 1-14["retrieve-tests-metadata"];
1-6["setup-test-env (4 minutes)"]; click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0"
click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" 1-15["detect-tests"];
1-14["retrieve-tests-metadata"]; click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715"
click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0"
end
2_2-2["rspec-all frontend_fixture (7 minutes)"];
click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0"
subgraph "Needs `setup-test-env` & `compile-test-assets`";
2_2-2 --> 1-6 & 1-3;
end
2_4-1["package-and-qa (manual)"];
click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0"
subgraph "Needs `build-qa-image`";
2_4-1 --> 1-2;
end
2_5-1["rspec & db jobs (24 minutes)"]; 2_5-1["rspec & db jobs (24 minutes)"];
class 2_5-1 criticalPath; class 2_5-1 criticalPath;
click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations"
subgraph "Needs `compile-test-assets`, `setup-test-env`, & `retrieve-tests-metadata`"; 2_5-1 --> 1-3 & 1-6 & 1-14 & 1-15;
2_5-1 --> 1-3 & 1-6 & 1-14;
end
3_1-1["jest (15.3 minutes)"];
click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0"
subgraph "Needs `rspec-all frontend_fixture`";
3_1-1 --> 2_2-2;
end
3_2-1["rspec:coverage (5.35 minutes)"]; 3_2-1["rspec:coverage (5.35 minutes)"];
class 3_2-1 criticalPath; class 3_2-1 criticalPath;
click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0"
subgraph "Depends on `rspec` jobs"; 3_2-1 -.->|"(don't use needs<br/>because of limitations)"| 2_5-1;
3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1;
end
4_3-1["rspec:undercoverage (3.5 minutes)"]; 4_3-1["rspec:undercoverage (3.5 minutes)"];
class 4_3-1 criticalPath; class 4_3-1 criticalPath;
click 4_3-1 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=13446492&udv=1005715" click 4_3-1 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=13446492&udv=1005715"
subgraph "Needs `rspec:coverage`"; 4_3-1 --> 3_2-1;
4_3-1 --> 3_2-1;
end
4_1-1["coverage-frontend (2.15 minutes)"];
click 4_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910777&udv=0"
subgraph "Needs `jest`";
4_1-1 --> 3_1-1;
end
``` ```
### Frontend-only MR pipeline ### Frontend pipeline
[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/134661039): [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/431913287).
```mermaid ```mermaid
graph RL; graph RL;
classDef criticalPath fill:#f66; classDef criticalPath fill:#f66;
subgraph "Jobs starting immediately"; 1-2["build-qa-image (2 minutes)"];
1-2["build-qa-image (2 minutes)"]; click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0"
click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" 1-5["compile-production-assets (16 minutes)"];
1-3["compile-test-assets (6 minutes)"]; class 1-5 criticalPath;
click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0"
1-5["compile-production-assets (16 minutes)"];
class 1-5 criticalPath;
click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0"
1-6["setup-test-env (4 minutes)"];
click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0"
1-14["retrieve-tests-metadata"];
click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0"
end
2_2-2["rspec-all frontend_fixture (7 minutes)"];
click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0"
subgraph "Needs `setup-test-env` & `compile-test-assets`";
2_2-2 --> 1-6 & 1-3;
end
2_3-1["build-assets-image (1.3 minutes)"]; 2_3-1["build-assets-image (1.3 minutes)"];
class 2_3-1 criticalPath; class 2_3-1 criticalPath;
subgraph "Needs `compile-production-assets`"; 2_3-1 --> 1-5
2_3-1 --> 1-5
end
2_4-1["package-and-qa (manual)"];
click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0"
subgraph "Needs `build-qa-image` & `build-assets-image`";
2_4-1 --> 1-2 & 2_3-1;
end
2_5-1["rspec & db jobs (24 minutes)"];
click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations"
subgraph "Needs `compile-test-assets`, `setup-test-env, & `retrieve-tests-metadata`";
2_5-1 --> 1-3 & 1-6 & 1-14;
end
2_6-1["review-build-cng (32 minutes)"]; 2_6-1["start-review-app-pipeline (49 minutes)"];
class 2_6-1 criticalPath; class 2_6-1 criticalPath;
click 2_6-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914314&udv=0" click 2_6-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations"
subgraph "Needs `build-assets-image`"; 2_6-1 --> 2_3-1 & 1-2;
2_6-1 --> 2_3-1;
end
3_1-1["jest (15.3 minutes)"];
click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0"
subgraph "Needs `rspec-all frontend_fixture`";
3_1-1 --> 2_2-2;
end
3_2-1["rspec:coverage (5.35 minutes)"];
click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0"
subgraph "Depends on `rspec` jobs";
3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1;
end
4_3-1["rspec:undercoverage (3.5 minutes)"];
click 4_3-1 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=13446492&udv=1005715"
subgraph "Needs `rspec:coverage`";
4_3-1 --> 3_2-1;
end
4_1-1["coverage-frontend (2.15 minutes)"];
click 4_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910777&udv=0"
subgraph "Needs `jest`";
4_1-1 --> 3_1-1;
end
3_3-1["review-deploy (9 minutes)"];
class 3_3-1 criticalPath;
click 3_3-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6721130&udv=0"
subgraph "Played by `review-build-cng`";
3_3-1 --> 2_6-1;
end
4_2-1["review-qa-smoke (8.3 minutes)"];
class 4_2-1 criticalPath;
click 4_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6729805&udv=0"
4_2-2["review-performance (2.5 minutes)"];
click 4_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356817&udv=0"
subgraph "Played by `review-deploy`";
4_2-1 & 4_2-2 --> 3_3-1;
end
``` ```
### QA-only MR pipeline ### End-to-end pipeline
[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/134645109): [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/431918463).
```mermaid ```mermaid
graph RL; graph RL;
classDef criticalPath fill:#f66; classDef criticalPath fill:#f66;
subgraph "Jobs starting immediately"; 1-2["build-qa-image (2 minutes)"];
1-2["build-qa-image (2 minutes)"]; click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0"
click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" 1-5["compile-production-assets (16 minutes)"];
1-5["compile-production-assets (16 minutes)"]; class 1-5 criticalPath;
class 1-5 criticalPath; click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0"
click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" 1-15["detect-tests"];
end click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715"
2_3-1["build-assets-image (1.3 minutes)"]; 2_3-1["build-assets-image (1.3 minutes)"];
class 2_3-1 criticalPath; class 2_3-1 criticalPath;
subgraph "Needs `compile-production-assets`"; 2_3-1 --> 1-5
2_3-1 --> 1-5
end
2_4-1["package-and-qa (102 minutes)"]; 2_4-1["package-and-qa (102 minutes)"];
class 2_4-1 criticalPath; class 2_4-1 criticalPath;
click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0"
subgraph "Needs `build-qa-image` & `build-assets-image`"; 2_4-1 --> 1-2 & 2_3-1 & 1-15;
2_4-1 --> 1-2 & 2_3-1;
end
``` ```
## CI configuration internals ## CI configuration internals
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7