Commit e98110f3 authored by Marcel Amirault's avatar Marcel Amirault

Merge branch 'selhorn-cache-123' into 'master'

Edited cache topic for CTRT

See merge request gitlab-org/gitlab!62356
parents a23717e2 c9945923
...@@ -5,73 +5,35 @@ info: To determine the technical writer assigned to the Stage/Group associated w ...@@ -5,73 +5,35 @@ info: To determine the technical writer assigned to the Stage/Group associated w
type: index, concepts, howto type: index, concepts, howto
--- ---
# Cache dependencies in GitLab CI/CD # Caching in GitLab CI/CD
GitLab CI/CD provides a caching mechanism that can be used to save time A cache is one or more files that a job downloads and saves. Subsequent jobs that use
when your jobs are running. the same cache don't have to download the files again, so they execute more quickly.
Caching is about speeding the time a job is executed by reusing the same To learn how to define the cache in your `.gitlab-ci.yml` file,
content of a previous job. Use caching when you are see the [`cache` reference](../yaml/README.md#cache).
developing software that depends on other libraries which are fetched via the
internet during build time.
If caching is enabled, it's shared between pipelines and jobs at the project ## How cache is different from artifacts
level by default. Caches are not shared across projects.
Make sure you read the [`cache` reference](../yaml/README.md#cache) to learn Use cache for dependencies, like packages you download from the internet.
how it is defined in `.gitlab-ci.yml`. Cache is stored where GitLab Runner is installed and uploaded to S3 if
[distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching).
## Cache vs artifacts - You can define it per job by using the `cache:` keyword. Otherwise it is disabled.
- You can define it per job so that:
- Subsequent pipelines can use it.
- Subsequent jobs in the same pipeline can use it, if the dependencies are identical.
- You cannot share it between projects.
If you use cache and artifacts to store the same path in your jobs, the cache might Use artifacts to pass intermediate build results between stages.
be overwritten because caches are restored before artifacts. Artifacts are generated by a job, stored in GitLab, and can be downloaded.
Don't use caching for passing artifacts between stages, as it is designed to store
runtime dependencies needed to compile the project:
- `cache`: **For storing project dependencies**
Caches can increase the speed of a given job in subsequent pipelines. You can
store downloaded dependencies so that they don't have to be fetched from the
internet again. Dependencies include things like npm packages, Go vendor packages, and so on.
You can configure a cache to pass intermediate build results between stages,
but you should use artifacts instead.
- `artifacts`: **Use for stage results that are passed between stages.**
Artifacts are files that are generated by a job so they can be stored and uploaded. You can
fetch and use artifacts in jobs in later stages of the same pipeline. You can't
create an artifact in a job in one stage, and use this artifact in a different job in
the same stage. This data is not available in different pipelines, but can be downloaded
from the UI.
If you download modules while building your application, you can declare them as
artifacts and subsequent stage jobs can use them.
You can define an [expiry time](../yaml/README.md#artifactsexpire_in) so artifacts - You can define artifacts per job. Subsequent jobs in later stages of the same
are deleted after a defined time. Use [dependencies](../yaml/README.md#dependencies) pipeline can use them.
to control which jobs fetch the artifacts. - You can't use the artifacts in a different pipeline.
Artifacts can also be used to make files available for download after a pipeline Artifacts expire after 30 days unless you define an [expiration time](../yaml/README.md#artifactsexpire_in).
completes, like a build image. Use [dependencies](../yaml/README.md#dependencies) to control which jobs fetch the artifacts.
Caches:
- Are disabled if not defined globally or per job (using `cache:`).
- Are available for all jobs in your `.gitlab-ci.yml` if enabled globally.
- Can be used in subsequent pipelines by the same job in which the cache was created (if not defined globally).
- Are stored where GitLab Runner is installed **and** uploaded to S3 if [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching).
- If defined per job, are used:
- By the same job in a subsequent pipeline.
- By subsequent jobs in the same pipeline, if they have identical dependencies.
Artifacts:
- Are disabled if not defined per job (using `artifacts:`).
- Can only be enabled per job, not globally.
- Are created during a pipeline and can be used by subsequent jobs in the same pipeline.
- Are always uploaded to GitLab (known as coordinator).
- Can have an expiration value for controlling disk usage (30 days by default).
Both artifacts and caches define their paths relative to the project directory, and Both artifacts and caches define their paths relative to the project directory, and
can't link to files outside it. can't link to files outside it.
...@@ -83,8 +45,7 @@ use one or more of the following: ...@@ -83,8 +45,7 @@ use one or more of the following:
- [Tag your runners](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs - [Tag your runners](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs
that share their cache. that share their cache.
- [Use sticky runners](../runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) - [Use runners that are only available to a particular project](../runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects).
that are only available to a particular project.
- [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example, - [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example,
different caches on each branch). For that, you can take advantage of the different caches on each branch). For that, you can take advantage of the
[predefined CI/CD variables](../variables/README.md#predefined-cicd-variables). [predefined CI/CD variables](../variables/README.md#predefined-cicd-variables).
...@@ -102,7 +63,7 @@ For runners to work with caches efficiently, you must do one of the following: ...@@ -102,7 +63,7 @@ For runners to work with caches efficiently, you must do one of the following:
Read about the [availability of the cache](#availability-of-the-cache) Read about the [availability of the cache](#availability-of-the-cache)
to learn more about the internals and get a better idea how cache works. to learn more about the internals and get a better idea how cache works.
### Share caches across the same branch ### Share caches between jobs in the same branch
Define a cache with the `key: ${CI_COMMIT_REF_SLUG}` so that jobs of each Define a cache with the `key: ${CI_COMMIT_REF_SLUG}` so that jobs of each
branch always use the same cache: branch always use the same cache:
...@@ -130,7 +91,7 @@ cache: ...@@ -130,7 +91,7 @@ cache:
key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG"
``` ```
### Share caches across different branches ### Share caches across jobs in different branches
To share a cache across all branches and all jobs, use the same key for everything: To share a cache across all branches and all jobs, use the same key for everything:
...@@ -146,7 +107,7 @@ cache: ...@@ -146,7 +107,7 @@ cache:
key: ${CI_JOB_NAME} key: ${CI_JOB_NAME}
``` ```
### Disable cache on specific jobs ### Disable cache for specific jobs
If you have defined the cache globally, it means that each job uses the If you have defined the cache globally, it means that each job uses the
same definition. You can override this behavior per-job, and if you want to same definition. You can override this behavior per-job, and if you want to
...@@ -219,7 +180,7 @@ test_async: ...@@ -219,7 +180,7 @@ test_async:
- node ./specs/start.js ./specs/async.spec.js - node ./specs/start.js ./specs/async.spec.js
``` ```
### Caching PHP dependencies ### Cache PHP dependencies
Assuming your project is using [Composer](https://getcomposer.org/) to install Assuming your project is using [Composer](https://getcomposer.org/) to install
the PHP dependencies, the following example defines `cache` globally so that the PHP dependencies, the following example defines `cache` globally so that
...@@ -248,7 +209,7 @@ test: ...@@ -248,7 +209,7 @@ test:
- vendor/bin/phpunit --configuration phpunit.xml --coverage-text --colors=never - vendor/bin/phpunit --configuration phpunit.xml --coverage-text --colors=never
``` ```
### Caching Python dependencies ### Cache Python dependencies
Assuming your project is using [pip](https://pip.pypa.io/en/stable/) to install Assuming your project is using [pip](https://pip.pypa.io/en/stable/) to install
the Python dependencies, the following example defines `cache` globally so that the Python dependencies, the following example defines `cache` globally so that
...@@ -289,7 +250,7 @@ test: ...@@ -289,7 +250,7 @@ test:
- flake8 . - flake8 .
``` ```
### Caching Ruby dependencies ### Cache Ruby dependencies
Assuming your project is using [Bundler](https://bundler.io) to install the Assuming your project is using [Bundler](https://bundler.io) to install the
gem dependencies, the following example defines `cache` globally so that all gem dependencies, the following example defines `cache` globally so that all
...@@ -347,7 +308,7 @@ deploy_job: ...@@ -347,7 +308,7 @@ deploy_job:
- bundle exec deploy - bundle exec deploy
``` ```
### Caching Go dependencies ### Cache Go dependencies
Assuming your project is using [Go Modules](https://github.com/golang/go/wiki/Modules) to install Assuming your project is using [Go Modules](https://github.com/golang/go/wiki/Modules) to install
Go dependencies, the following example defines `cache` in a `go-cache` template, that Go dependencies, the following example defines `cache` in a `go-cache` template, that
...@@ -396,6 +357,9 @@ machine where the runner is installed and depends on the type of the executor. ...@@ -396,6 +357,9 @@ machine where the runner is installed and depends on the type of the executor.
| [Docker](https://docs.gitlab.com/runner/executors/docker.html) | Locally, stored under [Docker volumes](https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage): `/var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip`. | | [Docker](https://docs.gitlab.com/runner/executors/docker.html) | Locally, stored under [Docker volumes](https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage): `/var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip`. |
| [Docker machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale runners) | Behaves the same as the Docker executor. | | [Docker machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale runners) | Behaves the same as the Docker executor. |
If you use cache and artifacts to store the same path in your jobs, the cache might
be overwritten because caches are restored before artifacts.
### How archiving and extracting works ### How archiving and extracting works
This example has two jobs that belong to two consecutive stages: This example has two jobs that belong to two consecutive stages:
......
...@@ -226,7 +226,7 @@ deploy_prod: ...@@ -226,7 +226,7 @@ deploy_prod:
### Caching ### Caching
GitLab provides a caching mechanism to speed up build times for your jobs by reusing previously downloaded dependencies. It's important to know the different between [cache and artifacts](../caching/index.md#cache-vs-artifacts) to make the best use of these features. GitLab provides a caching mechanism to speed up build times for your jobs by reusing previously downloaded dependencies. It's important to know the different between [cache and artifacts](../caching/index.md#how-cache-is-different-from-artifacts) to make the best use of these features.
CircleCI example of a job using a cache: CircleCI example of a job using a cache:
......
...@@ -159,7 +159,7 @@ In some cases, such as building a Go project for it to act as a dependency of a ...@@ -159,7 +159,7 @@ In some cases, such as building a Go project for it to act as a dependency of a
CI run for another project, removing the `vendor/` directory means the code must CI run for another project, removing the `vendor/` directory means the code must
be downloaded repeatedly, which can lead to intermittent problems due to rate be downloaded repeatedly, which can lead to intermittent problems due to rate
limiting or network failures. In these circumstances, you should [cache the limiting or network failures. In these circumstances, you should [cache the
downloaded code between](../../ci/caching/index.md#caching-go-dependencies). downloaded code between](../../ci/caching/index.md#cache-go-dependencies).
There was a There was a
[bug on modules checksums](https://github.com/golang/go/issues/29278) in Go versions earlier than v1.11.4, so make [bug on modules checksums](https://github.com/golang/go/issues/29278) in Go versions earlier than v1.11.4, so make
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment