Skip to content

G
gitlab-ce
Commit ca3fc229 authored by Douwe Maan's avatar Douwe Maan
Browse files
Options

Merge branch 'gitlab-ci-yaml-updates' into 'master' 

New CI YAML features

This introduces a couple of small `.gitlab-ci.yml` features:

1. Documentation for: Allow to use YAML anchors when parsing the `.gitlab-ci.yml`: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2958
2. Ignore jobs that start with `.`
3. Allow to pass name of created artifacts archive in `.gitlab-ci.yml`
4. Allow to define on which builds the current one depends on

These are really small changes so it makes not sense to create a separate merge requests for them.

@axil Could you review the documentation part?

The implementation on GitLab Runner side: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/113.

Fixes: https://gitlab.com/gitlab-org/gitlab-ce/issues/13755 https://gitlab.com/gitlab-org/gitlab-ce/issues/14211 https://gitlab.com/gitlab-org/gitlab-ce/issues/3423

cc @grzesiek @axil @DouweM 


See merge request !3182
parents 29b186e4 7ebb9320
Hide whitespace changes
Inline Side-by-side
Showing with 400 additions and 13 deletions
CHANGELOG
View file @ ca3fc229
......@@ -18,10 +18,14 @@ v 8.6.0 (unreleased)
- Return empty array instead of 404 when commit has no statuses in commit status API
- Decrease the font size and the padding of the `.anchor` icons used in the README (Roberto Dip)
- Rewrite logo to simplify SVG code (Sean Lang)
- Allow to use YAML anchors when parsing the `.gitlab-ci.yml` (Pascal Bach)
- Ignore jobs that start with `.` (hidden jobs)
- Allow to pass name of created artifacts archive in `.gitlab-ci.yml`
- Refactor and greatly improve search performance
- Add support for cross-project label references
- Update documentation to reflect Guest role not being enforced on internal projects
- Allow search for logged out users
- Allow to define on which builds the current one depends on
- Fix bug where Bitbucket `closed` issues were imported as `opened` (Iuri de Silvio)
- Don't show Issues/MRs from archived projects in Groups view
- Increase the notes polling timeout over time (Roberto Dip)
......
doc/ci/yaml/README.md
View file @ ca3fc229
......@@ -116,7 +116,8 @@ Alias for [stages](#stages).
### variables
_**Note:** Introduced in GitLab Runner v0.5.0._
>**Note:**
Introduced in GitLab Runner v0.5.0.
GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build
environment. The variables are stored in the git repository and are meant to
......@@ -153,7 +154,8 @@ cache:
#### cache:key
_**Note:** Introduced in GitLab Runner v1.0.0._
>**Note:**
Introduced in GitLab Runner v1.0.0.
The `key` directive allows you to define the affinity of caching
between jobs, allowing to have a single cache for all jobs,
......@@ -234,13 +236,14 @@ job_name:
| Keyword | Required | Description |
|---------------|----------|-------------|
| script | yes | Defines a shell script which is executed by runner |
| stage | no (default: `test`) | Defines a build stage |
| stage | no | Defines a build stage (default: `test`) |
| type | no | Alias for `stage` |
| only | no | Defines a list of git refs for which build is created |
| except | no | Defines a list of git refs for which build is not created |
| tags | no | Defines a list of tags which are used to select runner |
| allow_failure | no | Allow build to fail. Failed build doesn't contribute to commit status |
| when | no | Define when to run build. Can be `on_success`, `on_failure` or `always` |
| dependencies | no | Define other builds that a build depends on so that you can pass artifacts between them|
| artifacts | no | Define list build artifacts |
| cache | no | Define list of files that should be cached between subsequent runs |
......@@ -393,15 +396,18 @@ The above script will:
### artifacts
_**Note:** Introduced in GitLab Runner v0.7.0 for non-Windows platforms._
_**Note:** Limited Windows support was added in GitLab Runner v.1.0.0.
Currently not all executors are supported._
_**Note:** Build artifacts are only collected for successful builds._
>**Notes:**
>
> - Introduced in GitLab Runner v0.7.0 for non-Windows platforms.
> - Limited Windows support was added in GitLab Runner v.1.0.0.
> - Currently not all executors are supported.
> - Build artifacts are only collected for successful builds.
`artifacts` is used to specify list of files and directories which should be
attached to build after success. Below are some examples.
attached to build after success. To pass artifacts between different builds,
see [dependencies](#dependencies).
Below are some examples.
Send all files in `binaries` and `.config`:
......@@ -453,9 +459,130 @@ release-job:
The artifacts will be sent to GitLab after a successful build and will
be available for download in the GitLab UI.
#### artifacts:name
>**Note:**
Introduced in GitLab 8.6 and GitLab Runner v1.1.0.
The `name` directive allows you to define the name of the created artifacts
archive. That way, you can have a unique name of every archive which could be
useful when you'd like to download the archive from GitLab. The `artifacts:name`
variable can make use of any of the [predefined variables](../variables/README.md).
---
**Example configurations**
To create an archive with a name of the current build:
```yaml
job:
artifacts:
name: "$CI_BUILD_NAME"
```
To create an archive with a name of the current branch or tag including only
the files that are untracked by Git:
```yaml
job:
artifacts:
name: "$CI_BUILD_REF_NAME"
untracked: true
```
To create an archive with a name of the current build and the current branch or
tag including only the files that are untracked by Git:
```yaml
job:
artifacts:
name: "${CI_BUILD_NAME}_${CI_BUILD_REF_NAME}"
untracked: true
```
To create an archive with a name of the current [stage](#stages) and branch name:
```yaml
job:
artifacts:
name: "${CI_BUILD_STAGE}_${CI_BUILD_REF_NAME}"
untracked: true
```
---
If you use **Windows Batch** to run your shell scripts you need to replace
`$` with `%`:
```yaml
job:
artifacts:
name: "%CI_BUILD_STAGE%_%CI_BUILD_REF_NAME%"
untracked: true
```
### dependencies
>**Note:**
Introduced in GitLab 8.6 and GitLab Runner v1.1.1.
This feature should be used in conjunction with [`artifacts`](#artifacts) and
allows you to define the artifacts to pass between different builds.
Note that `artifacts` from previous [stages](#stages) are passed by default.
To use this feature, define `dependencies` in context of the job and pass
a list of all previous builds from which the artifacts should be downloaded.
You can only define builds from stages that are executed before the current one.
An error will be shown if you define builds from the current stage or next ones.
---
In the following example, we define two jobs with artifacts, `build:osx` and
`build:linux`. When the `test:osx` is executed, the artifacts from `build:osx`
will be downloaded and extracted in the context of the build. The same happens
for `test:linux` and artifacts from `build:linux`.
The job `deploy` will download artifacts from all previous builds because of
the [stage](#stages) precedence:
```yaml
build:osx:
stage: build
script: make build:osx
artifacts:
paths:
- binaries/
build:linux:
stage: build
script: make build:linux
artifacts:
paths:
- binaries/
test:osx:
stage: test
script: make test:osx
dependencies:
- build:osx
test:linux:
stage: test
script: make test:linux
dependencies:
- build:linux
deploy:
stage: deploy
script: make deploy
```
### cache
_**Note:** Introduced in GitLab Runner v0.7.0._
>**Note:**
Introduced in GitLab Runner v0.7.0.
`cache` is used to specify list of files and directories which should be cached
between builds. Below are some examples:
......@@ -509,6 +636,155 @@ rspec:
The cache is provided on best effort basis, so don't expect that cache will be
always present. For implementation details please check GitLab Runner.
## Hidden jobs
>**Note:**
Introduced in GitLab 8.6 and GitLab Runner v1.1.1.
Jobs that start with a dot (`.`) will be not processed by GitLab CI. You can
use this feature to ignore jobs, or use the
[special YAML features](#special-yaml-features) and transform the hidden jobs
into templates.
In the following example, `.job_name` will be ignored:
```yaml
.job_name:
script:
- rake spec
```
## Special YAML features
It's possible to use special YAML features like anchors (`&`), aliases (`*`)
and map merging (`<<`), which will allow you to greatly reduce the complexity
of `.gitlab-ci.yml`.
Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/).
### Anchors
>**Note:**
Introduced in GitLab 8.6 and GitLab Runner v1.1.1.
YAML also has a handy feature called 'anchors', which let you easily duplicate
content across your document. Anchors can be used to duplicate/inherit
properties, and is a perfect example to be used with [hidden jobs](#hidden-jobs)
to provide templates for your jobs.
The following example uses anchors and map merging. It will create two jobs,
`test1` and `test2`, that will inherit the parameters of `.job_template`, each
having their own custom `script` defined:
```yaml
.job_template: &job_definition # Hidden job that defines an anchor named 'job_definition'
image: ruby:2.1
services:
- postgres
- redis
test1:
<<: *job_definition # Merge the contents of the 'job_definition' alias
script:
- test1 project
test2:
<<: *job_definition # Merge the contents of the 'job_definition' alias
script:
- test2 project
```
`&` sets up the name of the anchor (`job_definition`), `<<` means "merge the
given hash into the current one", and `*` includes the named anchor
(`job_definition` again). The expanded version looks like this:
```yaml
.job_template:
image: ruby:2.1
services:
- postgres
- redis
test1:
image: ruby:2.1
services:
- postgres
- redis
script:
- test1 project
test2:
image: ruby:2.1
services:
- postgres
- redis
script:
- test2 project
```
Let's see another one example. This time we will use anchors to define two sets
of services. This will create two jobs, `test:postgres` and `test:mysql`, that
will share the `script` directive defined in `.job_template`, and the `services`
directive defined in `.postgres_services` and `.mysql_services` respectively:
```yaml
.job_template: &job_definition
script:
- test project
.postgres_services:
services: &postgres_definition
- postgres
- ruby
.mysql_services:
services: &mysql_definition
- mysql
- ruby
test:postgres:
<< *job_definition
services: *postgres_definition
test:mysql:
<< *job_definition
services: *mysql_definition
```
The expanded version looks like this:
```yaml
.job_template:
script:
- test project
.postgres_services:
services:
- postgres
- ruby
.mysql_services:
services:
- mysql
- ruby
test:postgres:
script:
- test project
services:
- postgres
- ruby
test:mysql:
script:
- test project
services:
- mysql
- ruby
```
You can see that the hidden jobs are conveniently used as templates.
## Validate the .gitlab-ci.yml
Each instance of GitLab CI has an embedded debug tool called Lint.
......
lib/ci/gitlab_ci_yaml_processor.rb
View file @ ca3fc229
......@@ -5,7 +5,9 @@ module Ci
DEFAULT_STAGES = %w(build test deploy)
DEFAULT_STAGE = 'test'
ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables, :cache]
ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services, :allow_failure, :type, :stage, :when, :artifacts, :cache]
ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services,
:allow_failure, :type, :stage, :when, :artifacts, :cache,
:dependencies]
attr_reader :before_script, :image, :services, :variables, :path, :cache
......@@ -60,6 +62,7 @@ module Ci
@jobs = {}
@config.each do |key, job|
next if key.to_s.start_with?('.')
stage = job[:stage] || job[:type] || DEFAULT_STAGE
@jobs[key] = { stage: stage }.merge(job)
end
......@@ -81,6 +84,7 @@ module Ci
services: job[:services] || @services,
artifacts: job[:artifacts],
cache: job[:cache] || @cache,
dependencies: job[:dependencies],
}.compact
}
end
......@@ -143,6 +147,7 @@ module Ci
validate_job_stage!(name, job) if job[:stage]
validate_job_cache!(name, job) if job[:cache]
validate_job_artifacts!(name, job) if job[:artifacts]
validate_job_dependencies!(name, job) if job[:dependencies]
end
private
......@@ -216,6 +221,10 @@ module Ci
end
def validate_job_artifacts!(name, job)
if job[:artifacts][:name] && !validate_string(job[:artifacts][:name])
raise ValidationError, "#{name} job: artifacts:name parameter should be a string"
end
if job[:artifacts][:untracked] && !validate_boolean(job[:artifacts][:untracked])
raise ValidationError, "#{name} job: artifacts:untracked parameter should be an boolean"
end
......@@ -225,6 +234,22 @@ module Ci
end
end
def validate_job_dependencies!(name, job)
if !validate_array_of_strings(job[:dependencies])
raise ValidationError, "#{name} job: dependencies parameter should be an array of strings"
end
stage_index = stages.index(job[:stage])
job[:dependencies].each do |dependency|
raise ValidationError, "#{name} job: undefined dependency: #{dependency}" unless @jobs[dependency]
unless stages.index(@jobs[dependency][:stage]) < stage_index
raise ValidationError, "#{name} job: dependency #{dependency} is not defined in prior stages"
end
end
end
def validate_array_of_strings(values)
values.is_a?(Array) && values.all? { |value| validate_string(value) }
end
......
spec/lib/ci/gitlab_ci_yaml_processor_spec.rb
View file @ ca3fc229
......@@ -397,7 +397,7 @@ module Ci
services: ["mysql"],
before_script: ["pwd"],
rspec: {
artifacts: { paths: ["logs/", "binaries/"], untracked: true },
artifacts: { paths: ["logs/", "binaries/"], untracked: true, name: "custom_name" },
script: "rspec"
}
})
......@@ -417,6 +417,7 @@ module Ci
image: "ruby:2.1",
services: ["mysql"],
artifacts: {
name: "custom_name",
paths: ["logs/", "binaries/"],
untracked: true
}
......@@ -427,6 +428,73 @@ module Ci
end
end
describe "Dependencies" do
let(:config) do
{
build1: { stage: 'build', script: 'test' },
build2: { stage: 'build', script: 'test' },
test1: { stage: 'test', script: 'test', dependencies: dependencies },
test2: { stage: 'test', script: 'test' },
deploy: { stage: 'test', script: 'test' }
}
end
subject { GitlabCiYamlProcessor.new(YAML.dump(config)) }
context 'no dependencies' do
let(:dependencies) { }
it { expect { subject }.to_not raise_error }
end
context 'dependencies to builds' do
let(:dependencies) { [:build1, :build2] }
it { expect { subject }.to_not raise_error }
end
context 'undefined dependency' do
let(:dependencies) { [:undefined] }
it { expect { subject }.to raise_error(GitlabCiYamlProcessor::ValidationError, 'test1 job: undefined dependency: undefined') }
end
context 'dependencies to deploy' do
let(:dependencies) { [:deploy] }
it { expect { subject }.to raise_error(GitlabCiYamlProcessor::ValidationError, 'test1 job: dependency deploy is not defined in prior stages') }
end
end
describe "Hidden jobs" do
let(:config) do
YAML.dump({
'.hidden_job' => { script: 'test' },
'normal_job' => { script: 'test' }
})
end
let(:config_processor) { GitlabCiYamlProcessor.new(config) }
subject { config_processor.builds_for_stage_and_ref("test", "master") }
it "doesn't create jobs that starts with dot" do
expect(subject.size).to eq(1)
expect(subject.first).to eq({
except: nil,
stage: "test",
stage_idx: 1,
name: :normal_job,
only: nil,
commands: "\ntest",
tag_list: [],
options: {},
when: "on_success",
allow_failure: false
})
end
end
describe "YAML Alias/Anchor" do
it "is correctly supported for jobs" do
config = <<EOT
......@@ -629,6 +697,13 @@ EOT
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: when parameter should be on_success, on_failure or always")
end
it "returns errors if job artifacts:name is not an a string" do
config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", artifacts: { name: 1 } } })
expect do
GitlabCiYamlProcessor.new(config)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: artifacts:name parameter should be a string")
end
it "returns errors if job artifacts:untracked is not an array of strings" do
config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", artifacts: { untracked: "string" } } })
expect do
......@@ -684,6 +759,13 @@ EOT
GitlabCiYamlProcessor.new(config)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: cache:paths parameter should be an array of strings")
end
it "returns errors if job dependencies is not an array of strings" do
config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", dependencies: "string" } })
expect do
GitlabCiYamlProcessor.new(config)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: dependencies parameter should be an array of strings")
end
end
end
end
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7