Commit 8918cc05 authored by Marcel Amirault's avatar Marcel Amirault

Merge branch 'selhorn-parallel-update' into 'master'

Updating parallel keyword to match standards

See merge request gitlab-org/gitlab!73791
parents 7ada6239 04afd852
...@@ -640,6 +640,106 @@ This job can no longer be scheduled to run automatically. You can, however, exec ...@@ -640,6 +640,106 @@ This job can no longer be scheduled to run automatically. You can, however, exec
To start a delayed job immediately, select **Play** (**{play}**). To start a delayed job immediately, select **Play** (**{play}**).
Soon GitLab Runner starts the job. Soon GitLab Runner starts the job.
## Parallelize large jobs
To split a large job into multiple smaller jobs that run in parallel, use the
[`parallel`](../yaml/index.md#parallel) keyword in your `.gitlab-ci.yml` file.
Different languages and test suites have different methods to enable parallelization.
For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters)
and RSpec to run Ruby tests in parallel:
```ruby
# Gemfile
source 'https://rubygems.org'
gem 'rspec'
gem 'semaphore_test_boosters'
```
```yaml
test:
parallel: 3
script:
- bundle
- bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL
```
You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec
job split into three separate jobs.
WARNING:
Test Boosters reports usage statistics to the author.
### Run a one-dimensional matrix of parallel jobs
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5.
You can create a one-dimensional matrix of parallel jobs:
```yaml
deploystacks:
stage: deploy
script:
- bin/deploy
parallel:
matrix:
- PROVIDER: [aws, ovh, gcp, vultr]
```
You can also [create a multi-dimensional matrix](../yaml/index.md#parallelmatrix).
### Run a matrix of parallel trigger jobs
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10.
You can run a [trigger](../yaml/index.md#trigger) job multiple times in parallel in a single pipeline,
but with different variable values for each instance of the job.
```yaml
deploystacks:
stage: deploy
trigger:
include: path/to/child-pipeline.yml
parallel:
matrix:
- PROVIDER: aws
STACK: [monitoring, app1]
- PROVIDER: ovh
STACK: [monitoring, backup]
- PROVIDER: [gcp, vultr]
STACK: [data]
```
This example generates 6 parallel `deploystacks` trigger jobs, each with different values
for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables.
```plaintext
deploystacks: [aws, monitoring]
deploystacks: [aws, app1]
deploystacks: [ovh, monitoring]
deploystacks: [ovh, backup]
deploystacks: [gcp, data]
deploystacks: [vultr, data]
```
In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/239737), you can
use the variables defined in `parallel: matrix` with the [`tags`](../yaml/index.md#tags) keyword for
dynamic runner selection.
```yaml
deploystacks:
stage: deploy
parallel:
matrix:
- PROVIDER: aws
STACK: [monitoring, app1]
- PROVIDER: gcp
STACK: [data]
tags:
- ${PROVIDER}-${STACK}
```
## Use predefined CI/CD variables to run jobs only in specific pipeline types ## Use predefined CI/CD variables to run jobs only in specific pipeline types
You can use [predefined CI/CD variables](../variables/predefined_variables.md) to choose You can use [predefined CI/CD variables](../variables/predefined_variables.md) to choose
......
...@@ -3679,11 +3679,17 @@ test: ...@@ -3679,11 +3679,17 @@ test:
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5.
Use `parallel` to configure how many instances of a job to run in parallel. Use `parallel` to run a job multiple times in parallel in a single pipeline.
The value can be from 2 to 50.
The `parallel` keyword creates N instances of the same job that run in parallel. Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently.
They are named sequentially from `job_name 1/N` to `job_name N/N`:
Parallel jobs are named sequentially from `job_name 1/N` to `job_name N/N`.
**Keyword type**: Job keyword. You can use it only as part of a job.
**Possible inputs**: A numeric value from `2` to `50`.
**Example of `parallel`**:
```yaml ```yaml
test: test:
...@@ -3691,47 +3697,32 @@ test: ...@@ -3691,47 +3697,32 @@ test:
parallel: 5 parallel: 5
``` ```
Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL` This example creates 5 jobs that run in parallel, named `test 1/5` to `test 5/5`.
[predefined CI/CD variable](../variables/index.md#predefined-cicd-variables) set.
Different languages and test suites have different methods to enable parallelization. **Additional details**:
For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters)
and RSpec to run Ruby tests in parallel:
```ruby
# Gemfile
source 'https://rubygems.org'
gem 'rspec'
gem 'semaphore_test_boosters'
```
```yaml - Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL`
test: [predefined CI/CD variable](../variables/index.md#predefined-cicd-variables) set.
parallel: 3
script:
- bundle
- bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL
```
WARNING: **Related topics**:
Test Boosters reports usage statistics to the author.
You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec - [Parallelize large jobs](../jobs/job_control.md#parallelize-large-jobs).
job split into three separate jobs.
#### Parallel `matrix` jobs #### `parallel:matrix`
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3.
> - The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452).
Use `matrix:` to run a job multiple times in parallel in a single pipeline, Use `parallel:matrix` to run a job multiple times in parallel in a single pipeline,
but with different variable values for each instance of the job. but with different variable values for each instance of the job.
There can be from 2 to 50 jobs.
Jobs can only run in parallel if there are multiple runners, or a single runner is Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently.
configured to run multiple jobs concurrently.
**Keyword type**: Job keyword. You can use it only as part of a job.
**Possible inputs**: A numeric value from `2` to `50`.
Every job gets the same `CI_NODE_TOTAL` [CI/CD variable](../variables/index.md#predefined-cicd-variables) value, and a unique `CI_NODE_INDEX` value. **Example of `parallel:matrix`**:
```yaml ```yaml
deploystacks: deploystacks:
...@@ -3751,7 +3742,7 @@ deploystacks: ...@@ -3751,7 +3742,7 @@ deploystacks:
STACK: [data, processing] STACK: [data, processing]
``` ```
The following example generates 10 parallel `deploystacks` jobs, each with different values The example generates 10 parallel `deploystacks` jobs, each with different values
for `PROVIDER` and `STACK`: for `PROVIDER` and `STACK`:
```plaintext ```plaintext
...@@ -3767,74 +3758,10 @@ deploystacks: [vultr, data] ...@@ -3767,74 +3758,10 @@ deploystacks: [vultr, data]
deploystacks: [vultr, processing] deploystacks: [vultr, processing]
``` ```
The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452). **Related topics**:
##### One-dimensional `matrix` jobs
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5.
You can also have one-dimensional matrices with a single job:
```yaml
deploystacks:
stage: deploy
script:
- bin/deploy
parallel:
matrix:
- PROVIDER: [aws, ovh, gcp, vultr]
```
##### Parallel `matrix` trigger jobs
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10.
Use `matrix:` to run a [trigger](#trigger) job multiple times in parallel in a single pipeline,
but with different variable values for each instance of the job.
```yaml
deploystacks:
stage: deploy
trigger:
include: path/to/child-pipeline.yml
parallel:
matrix:
- PROVIDER: aws
STACK: [monitoring, app1]
- PROVIDER: ovh
STACK: [monitoring, backup]
- PROVIDER: [gcp, vultr]
STACK: [data]
```
This example generates 6 parallel `deploystacks` trigger jobs, each with different values
for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables.
```plaintext
deploystacks: [aws, monitoring]
deploystacks: [aws, app1]
deploystacks: [ovh, monitoring]
deploystacks: [ovh, backup]
deploystacks: [gcp, data]
deploystacks: [vultr, data]
```
In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/239737), you can
use the variables defined in `parallel: matrix` with the [`tags`](#tags) keyword for
dynamic runner selection.
```yaml - [Run a one-dimensional matrix of parallel jobs](../jobs/job_control.md#run-a-one-dimensional-matrix-of-parallel-jobs).
deploystacks: - [Run a matrix of triggered parallel jobs](../jobs/job_control.md#run-a-matrix-of-parallel-trigger-jobs).
stage: deploy
parallel:
matrix:
- PROVIDER: aws
STACK: [monitoring, app1]
- PROVIDER: gcp
STACK: [data]
tags:
- ${PROVIDER}-${STACK}
```
### `trigger` ### `trigger`
......
...@@ -137,7 +137,7 @@ This is typically necessary, since gems or Ruby applications that we maintain ou ...@@ -137,7 +137,7 @@ This is typically necessary, since gems or Ruby applications that we maintain ou
update these repositories for the GitLab Rails application to work with a new Ruby, update these repositories for the GitLab Rails application to work with a new Ruby,
it is good practice to keep Ruby versions in lock-step across all our repositories. For minor and major it is good practice to keep Ruby versions in lock-step across all our repositories. For minor and major
upgrades, add new CI/CD jobs to these repositories using the new Ruby. upgrades, add new CI/CD jobs to these repositories using the new Ruby.
A [build matrix definition](../ci/yaml/index.md#parallel-matrix-jobs) can do this efficiently. A [build matrix definition](../ci/yaml/index.md#parallelmatrix) can do this efficiently.
#### Decide which repositories to update #### Decide which repositories to update
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment