Merge branch 'gitlab-ci-yaml-updates' into 'master'

New CI YAML features

This introduces a couple of small `.gitlab-ci.yml` features:

1. Documentation for: Allow to use YAML anchors when parsing the `.gitlab-ci.yml`: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2958
2. Ignore jobs that start with `.`
3. Allow to pass name of created artifacts archive in `.gitlab-ci.yml`
4. Allow to define on which builds the current one depends on

These are really small changes so it makes not sense to create a separate merge requests for them.

@axil Could you review the documentation part?

The implementation on GitLab Runner side: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/113.

Fixes: https://gitlab.com/gitlab-org/gitlab-ce/issues/13755 https://gitlab.com/gitlab-org/gitlab-ce/issues/14211 https://gitlab.com/gitlab-org/gitlab-ce/issues/3423

cc @grzesiek @axil @DouweM 


See merge request !3182

CHANGELOG

@@ -18,10 +18,14 @@ v 8.6.0 (unreleased)
   - Return empty array instead of 404 when commit has no statuses in commit status API
   - Decrease the font size and the padding of the `.anchor` icons used in the README (Roberto Dip)
   - Rewrite logo to simplify SVG code (Sean Lang)
+  - Allow to use YAML anchors when parsing the `.gitlab-ci.yml` (Pascal Bach)
+  - Ignore jobs that start with `.` (hidden jobs)
+  - Allow to pass name of created artifacts archive in `.gitlab-ci.yml`
   - Refactor and greatly improve search performance
   - Add support for cross-project label references
   - Update documentation to reflect Guest role not being enforced on internal projects
   - Allow search for logged out users
+  - Allow to define on which builds the current one depends on
   - Fix bug where Bitbucket `closed` issues were imported as `opened` (Iuri de Silvio)
   - Don't show Issues/MRs from archived projects in Groups view
   - Increase the notes polling timeout over time (Roberto Dip)

doc/ci/yaml/README.md

@@ -116,7 +116,8 @@ Alias for [stages](#stages).
 
 ### variables
 
-_**Note:** Introduced in GitLab Runner v0.5.0._
+>**Note:**
+Introduced in GitLab Runner v0.5.0.
 
 GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in build
 environment. The variables are stored in the git repository and are meant to
@@ -153,7 +154,8 @@ cache:
 
 #### cache:key
 
-_**Note:** Introduced in GitLab Runner v1.0.0._
+>**Note:**
+Introduced in GitLab Runner v1.0.0.
 
 The `key` directive allows you to define the affinity of caching
 between jobs, allowing to have a single cache for all jobs,
@@ -234,13 +236,14 @@ job_name:
 | Keyword       | Required | Description |
 |---------------|----------|-------------|
 | script        | yes | Defines a shell script which is executed by runner |
-| stage         | no (default: `test`) | Defines a build stage |
+| stage         | no | Defines a build stage (default: `test`) |
 | type          | no | Alias for `stage` |
 | only          | no | Defines a list of git refs for which build is created |
 | except        | no | Defines a list of git refs for which build is not created |
 | tags          | no | Defines a list of tags which are used to select runner |
 | allow_failure | no | Allow build to fail. Failed build doesn't contribute to commit status |
 | when          | no | Define when to run build. Can be `on_success`, `on_failure` or `always` |
+| dependencies  | no | Define other builds that a build depends on so that you can pass artifacts between them|
 | artifacts     | no | Define list build artifacts |
 | cache         | no | Define list of files that should be cached between subsequent runs |
 
@@ -393,15 +396,18 @@ The above script will:
 
 ### artifacts
 
-_**Note:** Introduced in GitLab Runner v0.7.0 for non-Windows platforms._
-
-_**Note:** Limited Windows support was added in GitLab Runner v.1.0.0. 
-Currently not all executors are supported._ 
-
-_**Note:** Build artifacts are only collected for successful builds._
+>**Notes:**
+>
+> - Introduced in GitLab Runner v0.7.0 for non-Windows platforms.
+> - Limited Windows support was added in GitLab Runner v.1.0.0.
+> - Currently not all executors are supported.
+> - Build artifacts are only collected for successful builds.
 
 `artifacts` is used to specify list of files and directories which should be
-attached to build after success. Below are some examples.
+attached to build after success. To pass artifacts between different builds,
+see [dependencies](#dependencies).
+
+Below are some examples.
 
 Send all files in `binaries` and `.config`:
 
@@ -453,9 +459,130 @@ release-job:
 The artifacts will be sent to GitLab after a successful build and will
 be available for download in the GitLab UI.
 
+#### artifacts:name
+
+>**Note:**
+Introduced in GitLab 8.6 and GitLab Runner v1.1.0.
+
+The `name` directive allows you to define the name of the created artifacts
+archive. That way, you can have a unique name of every archive which could be
+useful when you'd like to download the archive from GitLab. The `artifacts:name`
+variable can make use of any of the [predefined variables](../variables/README.md).
+
+---
+
+**Example configurations**
+
+To create an archive with a name of the current build:
+
+```yaml
+job:
+  artifacts:
+    name: "$CI_BUILD_NAME"
+```
+
+To create an archive with a name of the current branch or tag including only
+the files that are untracked by Git:
+
+```yaml
+job:
+   artifacts:
+     name: "$CI_BUILD_REF_NAME"
+     untracked: true
+```
+
+To create an archive with a name of the current build and the current branch or
+tag including only the files that are untracked by Git:
+
+```yaml
+job:
+  artifacts:
+    name: "${CI_BUILD_NAME}_${CI_BUILD_REF_NAME}"
+    untracked: true
+```
+
+To create an archive with a name of the current [stage](#stages) and branch name:
+
+```yaml
+job:
+  artifacts:
+    name: "${CI_BUILD_STAGE}_${CI_BUILD_REF_NAME}"
+    untracked: true
+```
+
+---
+
+If you use **Windows Batch** to run your shell scripts you need to replace
+`$` with `%`:
+
+```yaml
+job:
+  artifacts:
+    name: "%CI_BUILD_STAGE%_%CI_BUILD_REF_NAME%"
+    untracked: true
+```
+
+### dependencies
+
+>**Note:**
+Introduced in GitLab 8.6 and GitLab Runner v1.1.1.
+
+This feature should be used in conjunction with [`artifacts`](#artifacts) and
+allows you to define the artifacts to pass between different builds.
+
+Note that `artifacts` from previous [stages](#stages) are passed by default.
+
+To use this feature, define `dependencies` in context of the job and pass
+a list of all previous builds from which the artifacts should be downloaded.
+You can only define builds from stages that are executed before the current one.
+An error will be shown if you define builds from the current stage or next ones.
+
+---
+
+In the following example, we define two jobs with artifacts, `build:osx` and
+`build:linux`. When the `test:osx` is executed, the artifacts from `build:osx`
+will be downloaded and extracted in the context of the build. The same happens
+for `test:linux` and artifacts from `build:linux`.
+
+The job `deploy` will download artifacts from all previous builds because of
+the [stage](#stages) precedence:
+
+```yaml
+build:osx:
+  stage: build
+  script: make build:osx
+  artifacts:
+    paths:
+    - binaries/
+
+build:linux:
+  stage: build
+  script: make build:linux
+  artifacts:
+    paths:
+    - binaries/
+
+test:osx:
+  stage: test
+  script: make test:osx
+  dependencies:
+  - build:osx
+
+test:linux:
+  stage: test
+  script: make test:linux
+  dependencies:
+  - build:linux
+
+deploy:
+  stage: deploy
+  script: make deploy
+```
+
 ### cache
 
-_**Note:** Introduced in GitLab Runner v0.7.0._
+>**Note:**
+Introduced in GitLab Runner v0.7.0.
 
 `cache` is used to specify list of files and directories which should be cached
 between builds. Below are some examples:
@@ -509,6 +636,155 @@ rspec:
 The cache is provided on best effort basis, so don't expect that cache will be
 always present. For implementation details please check GitLab Runner.
 
+## Hidden jobs
+
+>**Note:**
+Introduced in GitLab 8.6 and GitLab Runner v1.1.1.
+
+Jobs that start with a dot (`.`) will be not processed by GitLab CI. You can
+use this feature to ignore jobs, or use the
+[special YAML features](#special-yaml-features) and transform the hidden jobs
+into templates.
+
+In the following example, `.job_name` will be ignored:
+
+```yaml
+.job_name:
+  script:
+    - rake spec
+```
+
+## Special YAML features
+
+It's possible to use special YAML features like anchors (`&`), aliases (`*`)
+and map merging (`<<`), which will allow you to greatly reduce the complexity
+of `.gitlab-ci.yml`.
+
+Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/).
+
+### Anchors
+
+>**Note:**
+Introduced in GitLab 8.6 and GitLab Runner v1.1.1.
+
+YAML also has a handy feature called 'anchors', which let you easily duplicate
+content across your document. Anchors can be used to duplicate/inherit
+properties, and is a perfect example to be used with [hidden jobs](#hidden-jobs)
+to provide templates for your jobs.
+
+The following example uses anchors and map merging. It will create two jobs,
+`test1` and `test2`, that will inherit the parameters of `.job_template`, each
+having their own custom `script` defined:
+
+```yaml
+.job_template: &job_definition  # Hidden job that defines an anchor named 'job_definition'
+  image: ruby:2.1
+  services:
+    - postgres
+    - redis
+
+test1:
+  <<: *job_definition           # Merge the contents of the 'job_definition' alias
+  script:
+    - test1 project
+
+test2:
+  <<: *job_definition           # Merge the contents of the 'job_definition' alias
+  script:
+    - test2 project
+```
+
+`&` sets up the name of the anchor (`job_definition`), `<<` means "merge the
+given hash into the current one", and `*` includes the named anchor
+(`job_definition` again). The expanded version looks like this:
+
+```yaml
+.job_template:
+  image: ruby:2.1
+  services:
+    - postgres
+    - redis
+
+test1:
+  image: ruby:2.1
+  services:
+    - postgres
+    - redis
+  script:
+    - test1 project
+
+test2:
+  image: ruby:2.1
+  services:
+    - postgres
+    - redis
+  script:
+    - test2 project
+```
+
+Let's see another one example. This time we will use anchors to define two sets
+of services. This will create two jobs, `test:postgres` and `test:mysql`, that
+will share the `script` directive defined in `.job_template`, and the `services`
+directive defined in `.postgres_services` and `.mysql_services` respectively:
+
+```yaml
+.job_template: &job_definition
+  script:
+    - test project
+
+.postgres_services:
+  services: &postgres_definition
+    - postgres
+    - ruby
+
+.mysql_services:
+  services: &mysql_definition
+    - mysql
+    - ruby
+
+test:postgres:
+  << *job_definition
+  services: *postgres_definition
+
+test:mysql:
+  << *job_definition
+  services: *mysql_definition
+```
+
+The expanded version looks like this:
+
+```yaml
+.job_template:
+  script:
+    - test project
+
+.postgres_services:
+  services:
+    - postgres
+    - ruby
+
+.mysql_services:
+  services:
+    - mysql
+    - ruby
+
+test:postgres:
+  script:
+    - test project
+  services:
+    - postgres
+    - ruby
+
+test:mysql:
+  script:
+    - test project
+  services:
+    - mysql
+    - ruby
+```
+
+You can see that the hidden jobs are conveniently used as templates.
+
 ## Validate the .gitlab-ci.yml
 
 Each instance of GitLab CI has an embedded debug tool called Lint.

lib/ci/gitlab_ci_yaml_processor.rb

@@ -5,7 +5,9 @@ module Ci
     DEFAULT_STAGES = %w(build test deploy)
     DEFAULT_STAGE = 'test'
     ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables, :cache]
-    ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services, :allow_failure, :type, :stage, :when, :artifacts, :cache]
+    ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services,
+                        :allow_failure, :type, :stage, :when, :artifacts, :cache,
+                        :dependencies]
 
     attr_reader :before_script, :image, :services, :variables, :path, :cache
 
@@ -60,6 +62,7 @@ module Ci
 
       @jobs = {}
       @config.each do |key, job|
+        next if key.to_s.start_with?('.')
         stage = job[:stage] || job[:type] || DEFAULT_STAGE
         @jobs[key] = { stage: stage }.merge(job)
       end
@@ -81,6 +84,7 @@ module Ci
           services: job[:services] || @services,
           artifacts: job[:artifacts],
           cache: job[:cache] || @cache,
+          dependencies: job[:dependencies],
         }.compact
       }
     end
@@ -143,6 +147,7 @@ module Ci
       validate_job_stage!(name, job) if job[:stage]
       validate_job_cache!(name, job) if job[:cache]
       validate_job_artifacts!(name, job) if job[:artifacts]
+      validate_job_dependencies!(name, job) if job[:dependencies]
     end
 
     private
@@ -216,6 +221,10 @@ module Ci
     end
 
     def validate_job_artifacts!(name, job)
+      if job[:artifacts][:name] && !validate_string(job[:artifacts][:name])
+        raise ValidationError, "#{name} job: artifacts:name parameter should be a string"
+      end
+
       if job[:artifacts][:untracked] && !validate_boolean(job[:artifacts][:untracked])
         raise ValidationError, "#{name} job: artifacts:untracked parameter should be an boolean"
       end
@@ -225,6 +234,22 @@ module Ci
       end
     end
 
+    def validate_job_dependencies!(name, job)
+      if !validate_array_of_strings(job[:dependencies])
+        raise ValidationError, "#{name} job: dependencies parameter should be an array of strings"
+      end
+
+      stage_index = stages.index(job[:stage])
+
+      job[:dependencies].each do |dependency|
+        raise ValidationError, "#{name} job: undefined dependency: #{dependency}" unless @jobs[dependency]
+
+        unless stages.index(@jobs[dependency][:stage]) < stage_index
+          raise ValidationError, "#{name} job: dependency #{dependency} is not defined in prior stages"
+        end
+      end
+    end
+
     def validate_array_of_strings(values)
       values.is_a?(Array) && values.all? { |value| validate_string(value) }
     end

spec/lib/ci/gitlab_ci_yaml_processor_spec.rb

@@ -397,7 +397,7 @@ module Ci
                              services:      ["mysql"],
                              before_script: ["pwd"],
                              rspec:         {
-                               artifacts: { paths: ["logs/", "binaries/"], untracked: true },
+                               artifacts: { paths: ["logs/", "binaries/"], untracked: true, name: "custom_name" },
                                script: "rspec"
                              }
                            })
@@ -417,6 +417,7 @@ module Ci
             image: "ruby:2.1",
             services: ["mysql"],
             artifacts: {
+              name: "custom_name",
               paths: ["logs/", "binaries/"],
               untracked: true
             }
@@ -427,6 +428,73 @@ module Ci
       end
     end
 
+    describe "Dependencies" do
+      let(:config) do
+        {
+          build1: { stage: 'build', script: 'test' },
+          build2: { stage: 'build', script: 'test' },
+          test1: { stage: 'test', script: 'test', dependencies: dependencies },
+          test2: { stage: 'test', script: 'test' },
+          deploy: { stage: 'test', script: 'test' }
+        }
+      end
+
+      subject { GitlabCiYamlProcessor.new(YAML.dump(config)) }
+
+      context 'no dependencies' do
+        let(:dependencies) { }
+
+        it { expect { subject }.to_not raise_error }
+      end
+
+      context 'dependencies to builds' do
+        let(:dependencies) { [:build1, :build2] }
+
+        it { expect { subject }.to_not raise_error }
+      end
+
+      context 'undefined dependency' do
+        let(:dependencies) { [:undefined] }
+
+        it { expect { subject }.to raise_error(GitlabCiYamlProcessor::ValidationError, 'test1 job: undefined dependency: undefined') }
+      end
+
+      context 'dependencies to deploy' do
+        let(:dependencies) { [:deploy] }
+
+        it { expect { subject }.to raise_error(GitlabCiYamlProcessor::ValidationError, 'test1 job: dependency deploy is not defined in prior stages') }
+      end
+    end
+
+    describe "Hidden jobs" do
+      let(:config) do
+        YAML.dump({
+                    '.hidden_job' => { script: 'test' },
+                    'normal_job' => { script: 'test' }
+                  })
+      end
+
+      let(:config_processor) { GitlabCiYamlProcessor.new(config) }
+
+      subject { config_processor.builds_for_stage_and_ref("test", "master") }
+
+      it "doesn't create jobs that starts with dot" do
+        expect(subject.size).to eq(1)
+        expect(subject.first).to eq({
+          except: nil,
+          stage: "test",
+          stage_idx: 1,
+          name: :normal_job,
+          only: nil,
+          commands: "\ntest",
+          tag_list: [],
+          options: {},
+          when: "on_success",
+          allow_failure: false
+        })
+      end
+    end
+
     describe "YAML Alias/Anchor" do
       it "is correctly supported for jobs" do
         config = <<EOT
@@ -629,6 +697,13 @@ EOT
         end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: when parameter should be on_success, on_failure or always")
       end
 
+      it "returns errors if job artifacts:name is not an a string" do
+        config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", artifacts: { name: 1 } } })
+        expect do
+          GitlabCiYamlProcessor.new(config)
+        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: artifacts:name parameter should be a string")
+      end
+
       it "returns errors if job artifacts:untracked is not an array of strings" do
         config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", artifacts: { untracked: "string" } } })
         expect do
@@ -684,6 +759,13 @@ EOT
           GitlabCiYamlProcessor.new(config)
         end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: cache:paths parameter should be an array of strings")
       end
+
+      it "returns errors if job dependencies is not an array of strings" do
+        config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", dependencies: "string" } })
+        expect do
+          GitlabCiYamlProcessor.new(config)
+        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: dependencies parameter should be an array of strings")
+      end
     end
   end
 end