Commit 8918cc05 authored by Marcel Amirault's avatar Marcel Amirault

Merge branch 'selhorn-parallel-update' into 'master'

Updating parallel keyword to match standards

See merge request gitlab-org/gitlab!73791
parents 7ada6239 04afd852
......@@ -640,6 +640,106 @@ This job can no longer be scheduled to run automatically. You can, however, exec
To start a delayed job immediately, select **Play** (**{play}**).
Soon GitLab Runner starts the job.
## Parallelize large jobs
To split a large job into multiple smaller jobs that run in parallel, use the
[`parallel`](../yaml/index.md#parallel) keyword in your `.gitlab-ci.yml` file.
Different languages and test suites have different methods to enable parallelization.
For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters)
and RSpec to run Ruby tests in parallel:
```ruby
# Gemfile
source 'https://rubygems.org'
gem 'rspec'
gem 'semaphore_test_boosters'
```
```yaml
test:
parallel: 3
script:
- bundle
- bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL
```
You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec
job split into three separate jobs.
WARNING:
Test Boosters reports usage statistics to the author.
### Run a one-dimensional matrix of parallel jobs
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5.
You can create a one-dimensional matrix of parallel jobs:
```yaml
deploystacks:
stage: deploy
script:
- bin/deploy
parallel:
matrix:
- PROVIDER: [aws, ovh, gcp, vultr]
```
You can also [create a multi-dimensional matrix](../yaml/index.md#parallelmatrix).
### Run a matrix of parallel trigger jobs
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10.
You can run a [trigger](../yaml/index.md#trigger) job multiple times in parallel in a single pipeline,
but with different variable values for each instance of the job.
```yaml
deploystacks:
stage: deploy
trigger:
include: path/to/child-pipeline.yml
parallel:
matrix:
- PROVIDER: aws
STACK: [monitoring, app1]
- PROVIDER: ovh
STACK: [monitoring, backup]
- PROVIDER: [gcp, vultr]
STACK: [data]
```
This example generates 6 parallel `deploystacks` trigger jobs, each with different values
for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables.
```plaintext
deploystacks: [aws, monitoring]
deploystacks: [aws, app1]
deploystacks: [ovh, monitoring]
deploystacks: [ovh, backup]
deploystacks: [gcp, data]
deploystacks: [vultr, data]
```
In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/239737), you can
use the variables defined in `parallel: matrix` with the [`tags`](../yaml/index.md#tags) keyword for
dynamic runner selection.
```yaml
deploystacks:
stage: deploy
parallel:
matrix:
- PROVIDER: aws
STACK: [monitoring, app1]
- PROVIDER: gcp
STACK: [data]
tags:
- ${PROVIDER}-${STACK}
```
## Use predefined CI/CD variables to run jobs only in specific pipeline types
You can use [predefined CI/CD variables](../variables/predefined_variables.md) to choose
......
......@@ -3679,11 +3679,17 @@ test:
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5.
Use `parallel` to configure how many instances of a job to run in parallel.
The value can be from 2 to 50.
Use `parallel` to run a job multiple times in parallel in a single pipeline.
The `parallel` keyword creates N instances of the same job that run in parallel.
They are named sequentially from `job_name 1/N` to `job_name N/N`:
Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently.
Parallel jobs are named sequentially from `job_name 1/N` to `job_name N/N`.
**Keyword type**: Job keyword. You can use it only as part of a job.
**Possible inputs**: A numeric value from `2` to `50`.
**Example of `parallel`**:
```yaml
test:
......@@ -3691,47 +3697,32 @@ test:
parallel: 5
```
Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL`
[predefined CI/CD variable](../variables/index.md#predefined-cicd-variables) set.
This example creates 5 jobs that run in parallel, named `test 1/5` to `test 5/5`.
Different languages and test suites have different methods to enable parallelization.
For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters)
and RSpec to run Ruby tests in parallel:
```ruby
# Gemfile
source 'https://rubygems.org'
gem 'rspec'
gem 'semaphore_test_boosters'
```
**Additional details**:
```yaml
test:
parallel: 3
script:
- bundle
- bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL
```
- Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL`
[predefined CI/CD variable](../variables/index.md#predefined-cicd-variables) set.
WARNING:
Test Boosters reports usage statistics to the author.
**Related topics**:
You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec
job split into three separate jobs.
- [Parallelize large jobs](../jobs/job_control.md#parallelize-large-jobs).
#### Parallel `matrix` jobs
#### `parallel:matrix`
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3.
> - The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452).
Use `matrix:` to run a job multiple times in parallel in a single pipeline,
Use `parallel:matrix` to run a job multiple times in parallel in a single pipeline,
but with different variable values for each instance of the job.
There can be from 2 to 50 jobs.
Jobs can only run in parallel if there are multiple runners, or a single runner is
configured to run multiple jobs concurrently.
Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently.
**Keyword type**: Job keyword. You can use it only as part of a job.
**Possible inputs**: A numeric value from `2` to `50`.
Every job gets the same `CI_NODE_TOTAL` [CI/CD variable](../variables/index.md#predefined-cicd-variables) value, and a unique `CI_NODE_INDEX` value.
**Example of `parallel:matrix`**:
```yaml
deploystacks:
......@@ -3751,7 +3742,7 @@ deploystacks:
STACK: [data, processing]
```
The following example generates 10 parallel `deploystacks` jobs, each with different values
The example generates 10 parallel `deploystacks` jobs, each with different values
for `PROVIDER` and `STACK`:
```plaintext
......@@ -3767,74 +3758,10 @@ deploystacks: [vultr, data]
deploystacks: [vultr, processing]
```
The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452).
##### One-dimensional `matrix` jobs
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5.
You can also have one-dimensional matrices with a single job:
```yaml
deploystacks:
stage: deploy
script:
- bin/deploy
parallel:
matrix:
- PROVIDER: [aws, ovh, gcp, vultr]
```
##### Parallel `matrix` trigger jobs
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10.
Use `matrix:` to run a [trigger](#trigger) job multiple times in parallel in a single pipeline,
but with different variable values for each instance of the job.
```yaml
deploystacks:
stage: deploy
trigger:
include: path/to/child-pipeline.yml
parallel:
matrix:
- PROVIDER: aws
STACK: [monitoring, app1]
- PROVIDER: ovh
STACK: [monitoring, backup]
- PROVIDER: [gcp, vultr]
STACK: [data]
```
This example generates 6 parallel `deploystacks` trigger jobs, each with different values
for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables.
```plaintext
deploystacks: [aws, monitoring]
deploystacks: [aws, app1]
deploystacks: [ovh, monitoring]
deploystacks: [ovh, backup]
deploystacks: [gcp, data]
deploystacks: [vultr, data]
```
In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/239737), you can
use the variables defined in `parallel: matrix` with the [`tags`](#tags) keyword for
dynamic runner selection.
**Related topics**:
```yaml
deploystacks:
stage: deploy
parallel:
matrix:
- PROVIDER: aws
STACK: [monitoring, app1]
- PROVIDER: gcp
STACK: [data]
tags:
- ${PROVIDER}-${STACK}
```
- [Run a one-dimensional matrix of parallel jobs](../jobs/job_control.md#run-a-one-dimensional-matrix-of-parallel-jobs).
- [Run a matrix of triggered parallel jobs](../jobs/job_control.md#run-a-matrix-of-parallel-trigger-jobs).
### `trigger`
......
......@@ -137,7 +137,7 @@ This is typically necessary, since gems or Ruby applications that we maintain ou
update these repositories for the GitLab Rails application to work with a new Ruby,
it is good practice to keep Ruby versions in lock-step across all our repositories. For minor and major
upgrades, add new CI/CD jobs to these repositories using the new Ruby.
A [build matrix definition](../ci/yaml/index.md#parallel-matrix-jobs) can do this efficiently.
A [build matrix definition](../ci/yaml/index.md#parallelmatrix) can do this efficiently.
#### Decide which repositories to update
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment