Commit 06952aaf authored by Grzegorz Bizon's avatar Grzegorz Bizon

Merge branch 'feature/ci-job-env-variables' into 'master'

Add environment variables on a job level in CI

Make it possible to define environment variables on a job level.

Closes #14716

See merge request !3612
parents 2b8fc138 1339fda1
...@@ -3,6 +3,7 @@ Please view this file on the master branch, on stable branches it's out of date. ...@@ -3,6 +3,7 @@ Please view this file on the master branch, on stable branches it's out of date.
v 8.7.0 (unreleased) v 8.7.0 (unreleased)
- Method instrumentation now uses Module#prepend instead of aliasing methods - Method instrumentation now uses Module#prepend instead of aliasing methods
- Repository.clean_old_archives is now instrumented - Repository.clean_old_archives is now instrumented
- Add support for environment variables on a job level in CI configuration file
- The Projects::HousekeepingService class has extra instrumentation - The Projects::HousekeepingService class has extra instrumentation
- All service classes (those residing in app/services) are now instrumented - All service classes (those residing in app/services) are now instrumented
- Developers can now add custom tags to transactions - Developers can now add custom tags to transactions
......
...@@ -365,11 +365,23 @@ module Ci ...@@ -365,11 +365,23 @@ module Ci
self.update(erased_by: user, erased_at: Time.now) self.update(erased_by: user, erased_at: Time.now)
end end
private
def yaml_variables def yaml_variables
global_yaml_variables + job_yaml_variables
end
def global_yaml_variables
if commit.config_processor
commit.config_processor.global_variables.map do |key, value|
{ key: key, value: value, public: true }
end
else
[]
end
end
def job_yaml_variables
if commit.config_processor if commit.config_processor
commit.config_processor.variables.map do |key, value| commit.config_processor.job_variables(name).map do |key, value|
{ key: key, value: value, public: true } { key: key, value: value, public: true }
end end
else else
......
## Variables ## Variables
When receiving a build from GitLab CI, the runner prepares the build environment. When receiving a build from GitLab CI, the runner prepares the build environment.
It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables** It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables**
The variables can be overwritten. They take precedence over each other in this order: The variables can be overwritten. They take precedence over each other in this order:
1. Trigger variables
1. Secure variables 1. Secure variables
1. YAML-defined variables 1. YAML-defined job-level variables
1. YAML-defined global variables
1. Predefined variables 1. Predefined variables
For example, if you define: For example, if you define:
1. API_TOKEN=SECURE as Secure Variable 1. `API_TOKEN=SECURE` as Secure Variable
1. API_TOKEN=YAML as YAML-defined variable 1. `API_TOKEN=YAML` as YAML-defined variable
The API_TOKEN will take the Secure Variable value: `SECURE`. The `API_TOKEN` will take the Secure Variable value: `SECURE`.
### Predefined variables (Environment Variables) ### Predefined variables (Environment Variables)
...@@ -70,15 +73,20 @@ These variables can be later used in all executed commands and scripts. ...@@ -70,15 +73,20 @@ These variables can be later used in all executed commands and scripts.
The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them.
Variables can be defined at a global level, but also at a job level.
More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md). More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md).
### User-defined variables (Secure Variables) ### User-defined variables (Secure Variables)
**This feature requires GitLab Runner 0.4.0 or higher** **This feature requires GitLab Runner 0.4.0 or higher**
GitLab CI allows you to define per-project **Secure Variables** that are set in build environment. GitLab CI allows you to define per-project **Secure Variables** that are set in
the build environment.
The secure variables are stored out of the repository (the `.gitlab-ci.yml`). The secure variables are stored out of the repository (the `.gitlab-ci.yml`).
The variables are securely passed to GitLab Runner and are available in build environment. The variables are securely passed to GitLab Runner and are available in the
It's desired method to use them for storing passwords, secret keys or whatever you want. build environment.
It's desired method to use them for storing passwords, secret keys or whatever
you want.
**The value of the variable can be shown in build log if explicitly asked to do so.** **The value of the variable can be shown in build log if explicitly asked to do so.**
If your project is public or internal you can make the builds private. If your project is public or internal you can make the builds private.
......
...@@ -23,6 +23,7 @@ If you want a quick introduction to GitLab CI, follow our ...@@ -23,6 +23,7 @@ If you want a quick introduction to GitLab CI, follow our
- [Jobs](#jobs) - [Jobs](#jobs)
- [script](#script) - [script](#script)
- [stage](#stage) - [stage](#stage)
- [job variables](#job-variables)
- [only and except](#only-and-except) - [only and except](#only-and-except)
- [tags](#tags) - [tags](#tags)
- [when](#when) - [when](#when)
...@@ -174,6 +175,8 @@ These variables can be later used in all executed commands and scripts. ...@@ -174,6 +175,8 @@ These variables can be later used in all executed commands and scripts.
The YAML-defined variables are also set to all created service containers, The YAML-defined variables are also set to all created service containers,
thus allowing to fine tune them. thus allowing to fine tune them.
Variables can be also defined on [job level](#job-variables).
### cache ### cache
>**Note:** >**Note:**
...@@ -324,6 +327,7 @@ job_name: ...@@ -324,6 +327,7 @@ job_name:
| services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | | services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) |
| stage | no | Defines a build stage (default: `test`) | | stage | no | Defines a build stage (default: `test`) |
| type | no | Alias for `stage` | | type | no | Alias for `stage` |
| variables | no | Define build variables on a job level |
| only | no | Defines a list of git refs for which build is created | | only | no | Defines a list of git refs for which build is created |
| except | no | Defines a list of git refs for which build is not created | | except | no | Defines a list of git refs for which build is not created |
| tags | no | Defines a list of tags which are used to select Runner | | tags | no | Defines a list of tags which are used to select Runner |
...@@ -414,6 +418,18 @@ job: ...@@ -414,6 +418,18 @@ job:
The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, The above example will run `job` for all branches on `gitlab-org/gitlab-ce`,
except master. except master.
### job variables
It is possible to define build variables using a `variables` keyword on a job
level. It works basically the same way as its global-level equivalent but
allows you to define job-specific build variables.
When the `variables` keyword is used on a job level, it overrides global YAML
build variables and predefined variables.
Build variables priority is defined in
[variables documentation](../variables/README.md).
### tags ### tags
`tags` is used to select specific Runners from the list of all Runners that are `tags` is used to select specific Runners from the list of all Runners that are
......
...@@ -7,9 +7,9 @@ module Ci ...@@ -7,9 +7,9 @@ module Ci
ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables, :cache] ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables, :cache]
ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services, ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services,
:allow_failure, :type, :stage, :when, :artifacts, :cache, :allow_failure, :type, :stage, :when, :artifacts, :cache,
:dependencies] :dependencies, :variables]
attr_reader :before_script, :image, :services, :variables, :path, :cache attr_reader :before_script, :image, :services, :path, :cache
def initialize(config, path = nil) def initialize(config, path = nil)
@config = YAML.safe_load(config, [Symbol], [], true) @config = YAML.safe_load(config, [Symbol], [], true)
...@@ -40,6 +40,17 @@ module Ci ...@@ -40,6 +40,17 @@ module Ci
@stages || DEFAULT_STAGES @stages || DEFAULT_STAGES
end end
def global_variables
@variables
end
def job_variables(name)
job = @jobs[name.to_sym]
return [] unless job
job.fetch(:variables, [])
end
private private
def initial_parsing def initial_parsing
...@@ -115,7 +126,7 @@ module Ci ...@@ -115,7 +126,7 @@ module Ci
end end
unless @variables.nil? || validate_variables(@variables) unless @variables.nil? || validate_variables(@variables)
raise ValidationError, "variables should be a map of key-valued strings" raise ValidationError, "variables should be a map of key-value strings"
end end
if @cache if @cache
...@@ -145,6 +156,7 @@ module Ci ...@@ -145,6 +156,7 @@ module Ci
validate_job_types!(name, job) validate_job_types!(name, job)
validate_job_stage!(name, job) if job[:stage] validate_job_stage!(name, job) if job[:stage]
validate_job_variables!(name, job) if job[:variables]
validate_job_cache!(name, job) if job[:cache] validate_job_cache!(name, job) if job[:cache]
validate_job_artifacts!(name, job) if job[:artifacts] validate_job_artifacts!(name, job) if job[:artifacts]
validate_job_dependencies!(name, job) if job[:dependencies] validate_job_dependencies!(name, job) if job[:dependencies]
...@@ -206,6 +218,13 @@ module Ci ...@@ -206,6 +218,13 @@ module Ci
end end
end end
def validate_job_variables!(name, job)
unless validate_variables(job[:variables])
raise ValidationError,
"#{name} job: variables should be a map of key-value strings"
end
end
def validate_job_cache!(name, job) def validate_job_cache!(name, job)
if job[:cache][:key] && !validate_string(job[:cache][:key]) if job[:cache][:key] && !validate_string(job[:cache][:key])
raise ValidationError, "#{name} job: cache:key parameter should be a string" raise ValidationError, "#{name} job: cache:key parameter should be a string"
......
...@@ -345,20 +345,76 @@ module Ci ...@@ -345,20 +345,76 @@ module Ci
end end
end end
describe "Variables" do describe 'Variables' do
it "returns variables when defined" do context 'when global variables are defined' do
it 'returns global variables' do
variables = { variables = {
var1: "value1", VAR1: 'value1',
var2: "value2", VAR2: 'value2',
} }
config = YAML.dump({ config = YAML.dump({
variables: variables, variables: variables,
before_script: ["pwd"], before_script: ['pwd'],
rspec: { script: "rspec" } rspec: { script: 'rspec' }
})
config_processor = GitlabCiYamlProcessor.new(config, path)
expect(config_processor.global_variables).to eq(variables)
end
end
context 'when job variables are defined' do
context 'when syntax is correct' do
it 'returns job variables' do
variables = {
KEY1: 'value1',
SOME_KEY_2: 'value2'
}
config = YAML.dump(
{ before_script: ['pwd'],
rspec: {
variables: variables,
script: 'rspec' }
})
config_processor = GitlabCiYamlProcessor.new(config, path)
expect(config_processor.job_variables(:rspec)).to eq variables
end
end
context 'when syntax is incorrect' do
it 'raises error' do
variables = [:KEY1, 'value1', :KEY2, 'value2']
config = YAML.dump(
{ before_script: ['pwd'],
rspec: {
variables: variables,
script: 'rspec' }
})
expect { GitlabCiYamlProcessor.new(config, path) }
.to raise_error(GitlabCiYamlProcessor::ValidationError,
/job: variables should be a map/)
end
end
end
context 'when job variables are not defined' do
it 'returns empty array' do
config = YAML.dump({
before_script: ['pwd'],
rspec: { script: 'rspec' }
}) })
config_processor = GitlabCiYamlProcessor.new(config, path) config_processor = GitlabCiYamlProcessor.new(config, path)
expect(config_processor.variables).to eq(variables)
expect(config_processor.job_variables(:rspec)).to eq []
end
end end
end end
...@@ -730,14 +786,14 @@ EOT ...@@ -730,14 +786,14 @@ EOT
config = YAML.dump({ variables: "test", rspec: { script: "test" } }) config = YAML.dump({ variables: "test", rspec: { script: "test" } })
expect do expect do
GitlabCiYamlProcessor.new(config, path) GitlabCiYamlProcessor.new(config, path)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-valued strings") end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-value strings")
end end
it "returns errors if variables is not a map of key-valued strings" do it "returns errors if variables is not a map of key-value strings" do
config = YAML.dump({ variables: { test: false }, rspec: { script: "test" } }) config = YAML.dump({ variables: { test: false }, rspec: { script: "test" } })
expect do expect do
GitlabCiYamlProcessor.new(config, path) GitlabCiYamlProcessor.new(config, path)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-valued strings") end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-value strings")
end end
it "returns errors if job when is not on_success, on_failure or always" do it "returns errors if job when is not on_success, on_failure or always" do
......
...@@ -238,6 +238,22 @@ describe Ci::Build, models: true do ...@@ -238,6 +238,22 @@ describe Ci::Build, models: true do
it { is_expected.to eq(predefined_variables + predefined_trigger_variable + yaml_variables + secure_variables + trigger_variables) } it { is_expected.to eq(predefined_variables + predefined_trigger_variable + yaml_variables + secure_variables + trigger_variables) }
end end
context 'when job variables are defined' do
##
# Job-level variables are defined in gitlab_ci.yml fixture
#
context 'when job variables are unique' do
let(:build) { create(:ci_build, name: 'staging') }
it 'includes job variables' do
expect(subject).to include(
{ key: :KEY1, value: 'value1', public: true },
{ key: :KEY2, value: 'value2', public: true }
)
end
end
end
end end
end end
end end
......
...@@ -33,6 +33,9 @@ spinach: ...@@ -33,6 +33,9 @@ spinach:
- tags - tags
staging: staging:
variables:
KEY1: value1
KEY2: value2
script: "cap deploy stating" script: "cap deploy stating"
type: deploy type: deploy
tags: tags:
...@@ -42,6 +45,8 @@ staging: ...@@ -42,6 +45,8 @@ staging:
- stable - stable
production: production:
variables:
DB_NAME: mysql
type: deploy type: deploy
script: script:
- cap deploy production - cap deploy production
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment