Merge branch 'feature/ci-job-env-variables' into 'master'

Add environment variables on a job level in CI

Make it possible to define environment variables on a job level.

Closes #14716

See merge request !3612

CHANGELOG

@@ -3,6 +3,7 @@ Please view this file on the master branch, on stable branches it's out of date.
 v 8.7.0 (unreleased)
   - Method instrumentation now uses Module#prepend instead of aliasing methods
   - Repository.clean_old_archives is now instrumented
+  - Add support for environment variables on a job level in CI configuration file
   - The Projects::HousekeepingService class has extra instrumentation
   - All service classes (those residing in app/services) are now instrumented
   - Developers can now add custom tags to transactions

app/models/ci/build.rb

@@ -365,11 +365,23 @@ module Ci
       self.update(erased_by: user, erased_at: Time.now)
     end
 
-    private
-
     def yaml_variables
+      global_yaml_variables + job_yaml_variables
+    end
+
+    def global_yaml_variables
+      if commit.config_processor
+        commit.config_processor.global_variables.map do |key, value|
+          { key: key, value: value, public: true }
+        end
+      else
+        []
+      end
+    end
+
+    def job_yaml_variables
       if commit.config_processor
-        commit.config_processor.variables.map do |key, value|
+        commit.config_processor.job_variables(name).map do |key, value|
           { key: key, value: value, public: true }
         end
       else

doc/ci/variables/README.md

 ## Variables
+
 When receiving a build from GitLab CI, the runner prepares the build environment.
 It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables**
 
 The variables can be overwritten. They take precedence over each other in this order:
+1. Trigger variables
 1. Secure variables
-1. YAML-defined variables
+1. YAML-defined job-level variables
+1. YAML-defined global variables
 1. Predefined variables
 
 For example, if you define:
-1. API_TOKEN=SECURE as Secure Variable
-1. API_TOKEN=YAML as YAML-defined variable
+1. `API_TOKEN=SECURE` as Secure Variable
+1. `API_TOKEN=YAML` as YAML-defined variable
 
-The API_TOKEN will take the Secure Variable value: `SECURE`.
+The `API_TOKEN` will take the Secure Variable value: `SECURE`.
 
 ### Predefined variables (Environment Variables)
 
@@ -70,15 +73,20 @@ These variables can be later used in all executed commands and scripts.
 
 The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them.
 
+Variables can be defined at a global level, but also at a job level.
+
 More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md).
 
 ### User-defined variables (Secure Variables)
 **This feature requires GitLab Runner 0.4.0 or higher**
 
-GitLab CI allows you to define per-project **Secure Variables** that are set in build environment. 
+GitLab CI allows you to define per-project **Secure Variables** that are set in
+the build environment.
 The secure variables are stored out of the repository (the `.gitlab-ci.yml`).
-The variables are securely passed to GitLab Runner and are available in build environment.
-It's desired method to use them for storing passwords, secret keys or whatever you want.
+The variables are securely passed to GitLab Runner and are available in the
+build environment.
+It's desired method to use them for storing passwords, secret keys or whatever
+you want.
 
 **The value of the variable can be shown in build log if explicitly asked to do so.**
 If your project is public or internal you can make the builds private.

doc/ci/yaml/README.md

@@ -23,6 +23,7 @@ If you want a quick introduction to GitLab CI, follow our
 - [Jobs](#jobs)
     - [script](#script)
     - [stage](#stage)
+    - [job variables](#job-variables)
     - [only and except](#only-and-except)
     - [tags](#tags)
     - [when](#when)
@@ -174,6 +175,8 @@ These variables can be later used in all executed commands and scripts.
 The YAML-defined variables are also set to all created service containers,
 thus allowing to fine tune them.
 
+Variables can be also defined on [job level](#job-variables).
+
 ### cache
 
 >**Note:**
@@ -324,6 +327,7 @@ job_name:
 | services      | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) |
 | stage         | no | Defines a build stage (default: `test`) |
 | type          | no | Alias for `stage` |
+| variables     | no | Define build variables on a job level |
 | only          | no | Defines a list of git refs for which build is created |
 | except        | no | Defines a list of git refs for which build is not created |
 | tags          | no | Defines a list of tags which are used to select Runner |
@@ -414,6 +418,18 @@ job:
 The above example will run `job` for all branches on `gitlab-org/gitlab-ce`,
 except master.
 
+### job variables
+
+It is possible to define build variables using a `variables` keyword on a job
+level. It works basically the same way as its global-level equivalent but
+allows you to define job-specific build variables.
+
+When the `variables` keyword is used on a job level, it overrides global YAML
+build variables and predefined variables.
+
+Build variables priority is defined in
+[variables documentation](../variables/README.md).
+
 ### tags
 
 `tags` is used to select specific Runners from the list of all Runners that are

lib/ci/gitlab_ci_yaml_processor.rb

@@ -7,9 +7,9 @@ module Ci
     ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables, :cache]
     ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services,
                         :allow_failure, :type, :stage, :when, :artifacts, :cache,
-                        :dependencies]
+                        :dependencies, :variables]
 
-    attr_reader :before_script, :image, :services, :variables, :path, :cache
+    attr_reader :before_script, :image, :services, :path, :cache
 
     def initialize(config, path = nil)
       @config = YAML.safe_load(config, [Symbol], [], true)
@@ -40,6 +40,17 @@ module Ci
       @stages || DEFAULT_STAGES
     end
 
+    def global_variables
+      @variables
+    end
+
+    def job_variables(name)
+      job = @jobs[name.to_sym]
+      return [] unless job
+
+      job.fetch(:variables, [])
+    end
+
     private
 
     def initial_parsing
@@ -115,7 +126,7 @@ module Ci
       end
 
       unless @variables.nil? || validate_variables(@variables)
-        raise ValidationError, "variables should be a map of key-valued strings"
+        raise ValidationError, "variables should be a map of key-value strings"
       end
 
       if @cache
@@ -145,6 +156,7 @@ module Ci
       validate_job_types!(name, job)
 
       validate_job_stage!(name, job) if job[:stage]
+      validate_job_variables!(name, job) if job[:variables]
       validate_job_cache!(name, job) if job[:cache]
       validate_job_artifacts!(name, job) if job[:artifacts]
       validate_job_dependencies!(name, job) if job[:dependencies]
@@ -206,6 +218,13 @@ module Ci
       end
     end
 
+    def validate_job_variables!(name, job)
+      unless validate_variables(job[:variables])
+        raise ValidationError,
+          "#{name} job: variables should be a map of key-value strings"
+      end
+    end
+
     def validate_job_cache!(name, job)
       if job[:cache][:key] && !validate_string(job[:cache][:key])
         raise ValidationError, "#{name} job: cache:key parameter should be a string"

spec/lib/ci/gitlab_ci_yaml_processor_spec.rb

@@ -345,20 +345,76 @@ module Ci
       end
     end
 
-    describe "Variables" do
-      it "returns variables when defined" do
-        variables = {
-          var1: "value1",
-          var2: "value2",
-        }
-        config = YAML.dump({
-                             variables: variables,
-                             before_script: ["pwd"],
-                             rspec: { script: "rspec" }
-                           })
+    describe 'Variables' do
+      context 'when global variables are defined' do
+        it 'returns global variables' do
+          variables = {
+            VAR1: 'value1',
+            VAR2: 'value2',
+          }
 
-        config_processor = GitlabCiYamlProcessor.new(config, path)
-        expect(config_processor.variables).to eq(variables)
+          config = YAML.dump({
+            variables: variables,
+            before_script: ['pwd'],
+            rspec: { script: 'rspec' }
+          })
+
+          config_processor = GitlabCiYamlProcessor.new(config, path)
+
+          expect(config_processor.global_variables).to eq(variables)
+        end
+      end
+
+      context 'when job variables are defined' do
+        context 'when syntax is correct' do
+          it 'returns job variables' do
+            variables =  {
+              KEY1: 'value1',
+              SOME_KEY_2: 'value2'
+            }
+
+            config =  YAML.dump(
+              { before_script: ['pwd'],
+                rspec: {
+                  variables: variables,
+                  script: 'rspec' }
+              })
+
+            config_processor = GitlabCiYamlProcessor.new(config, path)
+
+            expect(config_processor.job_variables(:rspec)).to eq variables
+          end
+        end
+
+        context 'when syntax is incorrect' do
+          it 'raises error' do
+            variables = [:KEY1, 'value1', :KEY2, 'value2']
+
+            config =  YAML.dump(
+              { before_script: ['pwd'],
+                rspec: {
+                  variables: variables,
+                  script: 'rspec' }
+              })
+
+            expect { GitlabCiYamlProcessor.new(config, path) }
+              .to raise_error(GitlabCiYamlProcessor::ValidationError,
+                               /job: variables should be a map/)
+          end
+        end
+      end
+
+      context 'when job variables are not defined' do
+        it 'returns empty array' do
+          config = YAML.dump({
+            before_script: ['pwd'],
+            rspec: { script: 'rspec' }
+          })
+
+          config_processor = GitlabCiYamlProcessor.new(config, path)
+
+          expect(config_processor.job_variables(:rspec)).to eq []
+        end
       end
     end
 
@@ -730,14 +786,14 @@ EOT
         config = YAML.dump({ variables: "test", rspec: { script: "test" } })
         expect do
           GitlabCiYamlProcessor.new(config, path)
-        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-valued strings")
+        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-value strings")
       end
 
-      it "returns errors if variables is not a map of key-valued strings" do
+      it "returns errors if variables is not a map of key-value strings" do
         config = YAML.dump({ variables: { test: false }, rspec: { script: "test" } })
         expect do
           GitlabCiYamlProcessor.new(config, path)
-        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-valued strings")
+        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-value strings")
       end
 
       it "returns errors if job when is not on_success, on_failure or always" do

spec/models/build_spec.rb

@@ -238,6 +238,22 @@ describe Ci::Build, models: true do
 
           it { is_expected.to eq(predefined_variables + predefined_trigger_variable + yaml_variables + secure_variables + trigger_variables) }
         end
+
+        context 'when job variables are defined' do
+          ##
+          # Job-level variables are defined in gitlab_ci.yml fixture
+          #
+          context 'when job variables are unique' do
+            let(:build) { create(:ci_build, name: 'staging') }
+
+            it 'includes job variables' do
+              expect(subject).to include(
+                { key: :KEY1, value: 'value1', public: true },
+                { key: :KEY2, value: 'value2', public: true }
+              )
+            end
+          end
+        end
       end
     end
   end

spec/support/gitlab_stubs/gitlab_ci.yml

@@ -4,7 +4,7 @@ services:
 
 before_script:
   - gem install bundler
-  - bundle install 
+  - bundle install
   - bundle exec rake db:create
 
 variables:
@@ -17,7 +17,7 @@ types:
 
 rspec:
   script: "rake spec"
-  tags: 
+  tags:
     - ruby
     - postgres
   only:
@@ -26,27 +26,32 @@ rspec:
 spinach:
   script: "rake spinach"
   allow_failure: true
-  tags: 
+  tags:
     - ruby
     - mysql
   except:
     - tags
 
 staging:
+  variables:
+    KEY1: value1
+    KEY2: value2
   script: "cap deploy stating"
   type: deploy
-  tags: 
+  tags:
     - ruby
     - mysql
   except:
     - stable
 
 production:
+  variables:
+    DB_NAME: mysql
   type: deploy
-  script: 
+  script:
     - cap deploy production
     - cap notify
-  tags: 
+  tags:
     - ruby
     - mysql
   only: