@@ -118,14 +118,14 @@ This section describes the earlier configuration format.
...
@@ -118,14 +118,14 @@ This section describes the earlier configuration format.
For source installations the following settings are nested under `artifacts:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `artifacts_object_store_`.
For source installations the following settings are nested under `artifacts:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `artifacts_object_store_`.
| `remote_directory`| The bucket name where Artifacts will be stored| |
| `remote_directory` | | The bucket name where Artifacts are stored |
| `direct_upload`| Set to `true` to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` |
| `direct_upload` | `false` | Set to `true` to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. |
| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 | `true` |
| `background_upload` | `true` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 |
| `proxy_download`| Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
| `proxy_download` | `false` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. |
| `connection`| Various connection options described below | |
| `connection` | | Various connection options described below |
#### Connection settings
#### Connection settings
...
@@ -336,7 +336,7 @@ To migrate back to local storage:
...
@@ -336,7 +336,7 @@ To migrate back to local storage:
If [`artifacts:expire_in`](../ci/yaml/README.md#artifactsexpire_in) is used to set
If [`artifacts:expire_in`](../ci/yaml/README.md#artifactsexpire_in) is used to set
an expiry for the artifacts, they are marked for deletion right after that date passes.
an expiry for the artifacts, they are marked for deletion right after that date passes.
Otherwise, they will expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md).
Otherwise, they expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md).
Artifacts are cleaned up by the `expire_build_artifacts_worker` cron job which Sidekiq
Artifacts are cleaned up by the `expire_build_artifacts_worker` cron job which Sidekiq
runs every hour at 50 minutes (`50 * * * *`).
runs every hour at 50 minutes (`50 * * * *`).
...
@@ -367,7 +367,7 @@ steps below.
...
@@ -367,7 +367,7 @@ steps below.
1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
If the `expire` directive is not set explicitly in your pipeline, artifacts will expire per the
If the `expire` directive is not set explicitly in your pipeline, artifacts expire per the
default artifacts expiration setting, which you can find in the [CI/CD Admin settings](../user/admin_area/settings/continuous_integration.md).
default artifacts expiration setting, which you can find in the [CI/CD Admin settings](../user/admin_area/settings/continuous_integration.md).
## Validation for dependencies
## Validation for dependencies
...
@@ -444,7 +444,7 @@ reasons are:
...
@@ -444,7 +444,7 @@ reasons are:
- The number of jobs run, and hence artifacts generated, is higher than expected.
- The number of jobs run, and hence artifacts generated, is higher than expected.
- Job logs are larger than expected, and have accumulated over time.
- Job logs are larger than expected, and have accumulated over time.
In these and other cases, you'll need to identify the projects most responsible
In these and other cases, identify the projects most responsible
for disk space usage, figure out what types of artifacts are using the most
for disk space usage, figure out what types of artifacts are using the most
space, and in some cases, manually delete job artifacts to reclaim disk space.
space, and in some cases, manually delete job artifacts to reclaim disk space.
...
@@ -508,7 +508,7 @@ If you need to manually remove job artifacts associated with multiple jobs while
...
@@ -508,7 +508,7 @@ If you need to manually remove job artifacts associated with multiple jobs while
1. Delete job artifacts older than a specific date:
1. Delete job artifacts older than a specific date:
NOTE: **Note:**
NOTE: **Note:**
This step will also erase artifacts that users have chosen to
This step also erases artifacts that users have chosen to
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
---
...
@@ -17,7 +17,7 @@ GET /projects/:id/jobs/:job_id/artifacts
...
@@ -17,7 +17,7 @@ GET /projects/:id/jobs/:job_id/artifacts
| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. |
| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. |
| `job_id` | integer | yes | ID of a job. |
| `job_id` | integer | yes | ID of a job. |
| `job_token`**(PREMIUM)** | string | no | To be used with [triggers](../ci/triggers/README.md#when-a-pipeline-depends-on-the-artifacts-of-another-pipeline) for multi-project pipelines. It should be invoked only inside `.gitlab-ci.yml`. Its value is always `$CI_JOB_TOKEN`. |
| `job_token`**(PREMIUM)** | string | no | To be used with [triggers](../ci/triggers/README.md#when-a-pipeline-depends-on-the-artifacts-of-another-pipeline) for multi-project pipelines. It should be invoked only inside `.gitlab-ci.yml`. Its value is always `$CI_JOB_TOKEN`. |
...
@@ -32,7 +32,7 @@ To use this in a [`script` definition](../ci/yaml/README.md#script) inside
...
@@ -32,7 +32,7 @@ To use this in a [`script` definition](../ci/yaml/README.md#script) inside
`.gitlab-ci.yml`**(PREMIUM)**, you can use either:
`.gitlab-ci.yml`**(PREMIUM)**, you can use either:
- The `JOB-TOKEN` header with the GitLab-provided `CI_JOB_TOKEN` variable.
- The `JOB-TOKEN` header with the GitLab-provided `CI_JOB_TOKEN` variable.
For example, the following job will download the artifacts of the job with ID
For example, the following job downloads the artifacts of the job with ID
`42`. Note that the command is wrapped into single quotes since it contains a
`42`. Note that the command is wrapped into single quotes since it contains a
colon (`:`):
colon (`:`):
...
@@ -44,7 +44,7 @@ To use this in a [`script` definition](../ci/yaml/README.md#script) inside
...
@@ -44,7 +44,7 @@ To use this in a [`script` definition](../ci/yaml/README.md#script) inside
```
```
- Or the `job_token` attribute with the GitLab-provided `CI_JOB_TOKEN` variable.
- Or the `job_token` attribute with the GitLab-provided `CI_JOB_TOKEN` variable.
For example, the following job will download the artifacts of the job with ID `42`:
For example, the following job downloads the artifacts of the job with ID `42`:
```yaml
```yaml
artifact_download:
artifact_download:
...
@@ -72,7 +72,7 @@ defining the job's name instead of its ID.
...
@@ -72,7 +72,7 @@ defining the job's name instead of its ID.
NOTE: **Note:**
NOTE: **Note:**
If a pipeline is [parent of other child pipelines](../ci/parent_child_pipelines.md), artifacts
If a pipeline is [parent of other child pipelines](../ci/parent_child_pipelines.md), artifacts
are searched in hierarchical order from parent to child. For example, if both parent and
are searched in hierarchical order from parent to child. For example, if both parent and
child pipelines have a job with the same name, the artifact from the parent pipeline will be returned.
child pipelines have a job with the same name, the artifact from the parent pipeline is returned.
```plaintext
```plaintext
GET /projects/:id/jobs/artifacts/:ref_name/download?job=name
GET /projects/:id/jobs/artifacts/:ref_name/download?job=name
...
@@ -81,7 +81,7 @@ GET /projects/:id/jobs/artifacts/:ref_name/download?job=name
...
@@ -81,7 +81,7 @@ GET /projects/:id/jobs/artifacts/:ref_name/download?job=name
| `description`| string | no | The description of pipeline schedule |
| `description` | string | no | The description of the pipeline schedule. |
| `ref`| string | no | The branch/tag name will be triggered |
| `ref` | string | no | The branch or tag name that is triggered. |
| `cron`| string | no | The cron (e.g. `0 1 * * *`) ([Cron syntax](https://en.wikipedia.org/wiki/Cron)) |
| `cron` | string | no | The [cron](https://en.wikipedia.org/wiki/Cron) schedule, for example: `0 1 * * *`. |
| `cron_timezone`| string | no | The timezone supported by `ActiveSupport::TimeZone` (e.g. `Pacific Time (US & Canada)`) or `TZInfo::Timezone` (e.g. `America/Los_Angeles`) |
| `cron_timezone` | string | no | The timezone supported by `ActiveSupport::TimeZone` (for example `Pacific Time (US & Canada)`), or `TZInfo::Timezone` (for example `America/Los_Angeles`). |
| `active`| boolean | no | The activation of pipeline schedule. If false is set, the pipeline schedule will deactivated initially. |
| `active` | boolean | no | The activation of pipeline schedule. If false is set, the pipeline schedule is initially deactivated. |
```shell
```shell
curl --request PUT --header"PRIVATE-TOKEN: <your_access_token>"--formcron="0 2 * * *""https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13"
curl --request PUT --header"PRIVATE-TOKEN: <your_access_token>"--formcron="0 2 * * *""https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13"
GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [Unit test reports](unit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change.
GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [Unit test reports](unit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change.
You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log.
You can configure your job to use custom Metrics Reports, and GitLab displays a report on the merge request so that it's easier and faster to identify changes without having to check the entire log.
In the Merge Request Widget, multi-project pipeline mini-graphs are displayed,
In the Merge Request Widget, multi-project pipeline mini-graphs are displayed,
and when hovering or tapping (on touchscreen devices) they will expand and be shown adjacent to each other.
and when hovering or tapping (on touchscreen devices) they expand and are shown adjacent to each other.
...
@@ -97,16 +97,16 @@ staging:
...
@@ -97,16 +97,16 @@ staging:
trigger:my/deployment
trigger:my/deployment
```
```
In the example above, as soon as `rspec` job succeeds in the `test` stage,
In the example above, as soon as the `rspec` job succeeds in the `test` stage,
the `staging` bridge job is going to be started. The initial status of this
the `staging` bridge job is started. The initial status of this
job will be `pending`. GitLab will create a downstream pipeline in the
job is `pending`. GitLab then creates a downstream pipeline in the
`my/deployment` project and, as soon as the pipeline gets created, the
`my/deployment` project and, as soon as the pipeline is created, the
`staging` job will succeed. `my/deployment` is a full path to that project.
`staging` job succeeds. `my/deployment` is a full path to that project.
The user that created the upstream pipeline needs to have access rights to the
The user that created the upstream pipeline needs to have access rights to the
downstream project (`my/deployment` in this case). If a downstream project can
downstream project (`my/deployment` in this case). If a downstream project is
not be found, or a user does not have access rights to create pipeline there,
not found, or a user does not have access rights to create a pipeline there,
the `staging` job is going to be marked as _failed_.
the `staging` job is marked as _failed_.
When using:
When using:
...
@@ -117,21 +117,18 @@ When using:
...
@@ -117,21 +117,18 @@ When using:
-[`only/except`](yaml/README.md#onlyexcept-basic) to control job behavior, use the
-[`only/except`](yaml/README.md#onlyexcept-basic) to control job behavior, use the
`pipelines` keyword.
`pipelines` keyword.
CAUTION: **Caution:**
In the example, `staging` is marked as successful as soon as a downstream pipeline
In the example, `staging` will be marked as succeeded as soon as a downstream pipeline
gets created. If you want to display the downstream pipeline's status instead, see
gets created. If you want to display the downstream pipeline's status instead, see
[Mirroring status from triggered pipeline](#mirroring-status-from-triggered-pipeline).
[Mirroring status from triggered pipeline](#mirroring-status-from-triggered-pipeline).
NOTE: **Note:**
NOTE: **Note:**
Bridge jobs do not support every configuration entry that a user can use
Bridge jobs [do not support every configuration keyword](#limitations) that can be used
in the case of regular jobs. Bridge jobs will not be picked by a runner,
with other jobs. If a user tries to use unsupported configuration keywords, YAML
so there is no point in adding support for `script`, for example. If a user
validation fails on pipeline creation.
tries to use unsupported configuration syntax, YAML validation will fail upon
pipeline creation.
### Specifying a downstream pipeline branch
### Specifying a downstream pipeline branch
It is possible to specify a branch name that a downstream pipeline will use:
It is possible to specify a branch name that a downstream pipeline uses:
```yaml
```yaml
rspec:
rspec:
...
@@ -152,7 +149,7 @@ Use:
...
@@ -152,7 +149,7 @@ Use:
[From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/10126), variable expansion is
[From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/10126), variable expansion is
supported.
supported.
GitLab will use a commit that is currently on the HEAD of the branch when
GitLab uses a commit that is on the head of the branch when
creating a downstream pipeline.
creating a downstream pipeline.
NOTE: **Note:**
NOTE: **Note:**
...
@@ -181,10 +178,10 @@ staging:
...
@@ -181,10 +178,10 @@ staging:
trigger:my/deployment
trigger:my/deployment
```
```
The `ENVIRONMENT` variable will be passed to every job defined in a downstream
The `ENVIRONMENT` variable is passed to every job defined in a downstream
pipeline. It will be available as an environment variable when GitLab Runner picks a job.
pipeline. It is available as an environment variable when GitLab Runner picks a job.
In the following configuration, the `MY_VARIABLE` variable will be passed to the downstream pipeline
In the following configuration, the `MY_VARIABLE` variable is passed to the downstream pipeline
that is created when the `trigger-downstream` job is queued. This is because `trigger-downstream`
that is created when the `trigger-downstream` job is queued. This is because `trigger-downstream`
job inherits variables declared in global variables blocks, and then we pass these variables to a downstream pipeline.
job inherits variables declared in global variables blocks, and then we pass these variables to a downstream pipeline.
...
@@ -210,7 +207,7 @@ downstream-job:
...
@@ -210,7 +207,7 @@ downstream-job:
```
```
In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the
In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the
upstream pipeline will be passed to the `downstream-job` job, and will be available
upstream pipeline is passed to the `downstream-job` job, and is available
within the context of all downstream builds.
within the context of all downstream builds.
Upstream pipelines take precedence over downstream ones. If there are two
Upstream pipelines take precedence over downstream ones. If there are two
...
@@ -289,9 +286,9 @@ upstream_bridge:
...
@@ -289,9 +286,9 @@ upstream_bridge:
### Limitations
### Limitations
Because bridge jobs are a little different to regular jobs, it is not
Bridge jobs are a little different from regular jobs. It is not
possible to use exactly the same configuration syntax here, as one would
possible to use exactly the same configuration syntax as when defining regular jobs
normally do when defining a regular job that will be picked by a runner.
that are picked by a runner.
Some features are not implemented yet. For example, support for environments.
Some features are not implemented yet. For example, support for environments.
...
@@ -318,7 +315,7 @@ tag in a different project:
...
@@ -318,7 +315,7 @@ tag in a different project:
1. Click subscribe.
1. Click subscribe.
Any pipelines that complete successfully for new tags in the subscribed project
Any pipelines that complete successfully for new tags in the subscribed project
will now trigger a pipeline on the current project's default branch. The maximum
now trigger a pipeline on the current project's default branch. The maximum
number of upstream pipeline subscriptions is 2 by default, for both the upstream and
number of upstream pipeline subscriptions is 2 by default, for both the upstream and
downstream projects. This [application limit](../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) can be changed on self-managed instances by a GitLab administrator.
downstream projects. This [application limit](../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) can be changed on self-managed instances by a GitLab administrator.
@@ -32,8 +32,8 @@ On the left side we have the events that can trigger a pipeline based on various
...
@@ -32,8 +32,8 @@ On the left side we have the events that can trigger a pipeline based on various
- When GitHub integration is used with [external pull requests](../../ci/ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests).
- When GitHub integration is used with [external pull requests](../../ci/ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests).
- When an upstream pipeline contains a [bridge job](../../ci/yaml/README.md#trigger) which triggers a downstream pipeline.
- When an upstream pipeline contains a [bridge job](../../ci/yaml/README.md#trigger) which triggers a downstream pipeline.
Triggering any of these events will invoke the [`CreatePipelineService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/create_pipeline_service.rb)
Triggering any of these events invokes the [`CreatePipelineService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/create_pipeline_service.rb)
which takes as input event data and the user triggering it, then will attempt to create a pipeline.
which takes as input event data and the user triggering it, then attempts to create a pipeline.
The `CreatePipelineService` relies heavily on the [`YAML Processor`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/yaml_processor.rb)
The `CreatePipelineService` relies heavily on the [`YAML Processor`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/yaml_processor.rb)
component, which is responsible for taking in a YAML blob as input and returns the abstract data structure of a
component, which is responsible for taking in a YAML blob as input and returns the abstract data structure of a
...
@@ -65,20 +65,20 @@ the `Runner API Gateway`.
...
@@ -65,20 +65,20 @@ the `Runner API Gateway`.
We can register, delete, and verify runners, which also causes read/write queries to the database. After a runner is connected,
We can register, delete, and verify runners, which also causes read/write queries to the database. After a runner is connected,
it keeps asking for the next job to execute. This invokes the [`RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb)
it keeps asking for the next job to execute. This invokes the [`RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb)
which will pick the next job and assign it to the runner. At this point the job will transition to a
which picks the next job and assigns it to the runner. At this point the job transitions to a
`running` state, which again triggers `ProcessPipelineService` due to the status change.
`running` state, which again triggers `ProcessPipelineService` due to the status change.
For more details read [Job scheduling](#job-scheduling)).
For more details read [Job scheduling](#job-scheduling)).
While a job is being executed, the runner sends logs back to the server as well any possible artifacts
While a job is being executed, the runner sends logs back to the server as well any possible artifacts
that need to be stored. Also, a job may depend on artifacts from previous jobs in order to run. In this
that need to be stored. Also, a job may depend on artifacts from previous jobs in order to run. In this
case the runner will download them using a dedicated API endpoint.
case the runner downloads them using a dedicated API endpoint.
Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts
Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts
are reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request.
are reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request.
Job status transitions are not all automated. A user may run [manual jobs](../../ci/yaml/README.md#whenmanual), cancel a pipeline, retry
Job status transitions are not all automated. A user may run [manual jobs](../../ci/yaml/README.md#whenmanual), cancel a pipeline, retry
specific failed jobs or the entire pipeline. Anything that
specific failed jobs or the entire pipeline. Anything that
causes a job to change status will trigger`ProcessPipelineService`, as it's responsible for
causes a job to change status triggers`ProcessPipelineService`, as it's responsible for
tracking the status of the entire pipeline.
tracking the status of the entire pipeline.
A special type of job is the [bridge job](../../ci/yaml/README.md#trigger) which is executed server-side
A special type of job is the [bridge job](../../ci/yaml/README.md#trigger) which is executed server-side
...
@@ -90,7 +90,7 @@ from the `CreatePipelineService` every time a downstream pipeline is triggered.
...
@@ -90,7 +90,7 @@ from the `CreatePipelineService` every time a downstream pipeline is triggered.
When a Pipeline is created all its jobs are created at once for all stages, with an initial state of `created`. This makes it possible to visualize the full content of a pipeline.
When a Pipeline is created all its jobs are created at once for all stages, with an initial state of `created`. This makes it possible to visualize the full content of a pipeline.
A job with the `created` state won't be seen by the runner yet. To make it possible to assign a job to a runner, the job must transition first into the `pending` state, which can happen if:
A job with the `created` state isn't seen by the runner yet. To make it possible to assign a job to a runner, the job must transition first into the `pending` state, which can happen if:
1. The job is created in the very first stage of the pipeline.
1. The job is created in the very first stage of the pipeline.
1. The job required a manual start and it has been triggered.
1. The job required a manual start and it has been triggered.
...
@@ -135,7 +135,7 @@ There are 3 top level queries that this service uses to gather the majority of t
...
@@ -135,7 +135,7 @@ There are 3 top level queries that this service uses to gather the majority of t
This list of jobs is then filtered further by matching tags between job and runner tags.
This list of jobs is then filtered further by matching tags between job and runner tags.
NOTE: **Note:**
NOTE: **Note:**
If a job contains tags, the runner will not pick the job if it does not match **all** the tags.
If a job contains tags, the runner doesn't pick the job if it does not match **all** the tags.
The runner may have more tags than defined for the job, but not vice-versa.
The runner may have more tags than defined for the job, but not vice-versa.
Finally if the runner can only pick jobs that are tagged, all untagged jobs are filtered out.
Finally if the runner can only pick jobs that are tagged, all untagged jobs are filtered out.
