Commit fd074088 authored by Erick Banks's avatar Erick Banks Committed by Mark Lapierre

Add ability to constrain tests per pipeline

Adds ability to run tests only if the pipeline 'ci_project_name'
equals the value of the key 'only_in_pipeline'.
parent 144f4dfb
# Environment selection
Some tests are designed to be run against specific environments. We can specify
what environments to run tests against using the `only` metadata.
Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines).
We can specify what environments or pipelines to run tests against using the `only` metadata.
## Available switches
......@@ -11,6 +11,7 @@ what environments to run tests against using the `only` metadata.
| `subdomain` | Set the subdomain matcher | `Array` or `String` |
| `domain` | Set the domain matcher | `String` |
| `production` | Match against production | `Static` |
| `pipeline` | Match against a pipeline | `Array` or `Static`|
CAUTION: **Caution:**
You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously.
......@@ -19,7 +20,7 @@ can control the `tld` and `domain` independently.
## Examples
| Environment | Key | Matches (regex) |
| Environment or pipeline | Key | Matches (regex for environments, string matching for pipelines) |
| ---------------- | --- | --------------- |
| `any` | `` | `.+.com` |
| `gitlab.com` | `only: :production` | `gitlab.com` |
......@@ -27,18 +28,24 @@ can control the `tld` and `domain` independently.
| `gitlab.com and staging.gitlab.com` | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` |
| `dev.gitlab.org` | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` |
| `staging.gitlab.com & domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging|domain).+.com` |
| `nightly` | `only: { pipeline: :nightly }` | "nightly" |
| `nightly`, `canary` | `only_run_in_pipeline: [:nightly, :canary]` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) |
```ruby
RSpec.describe 'Area' do
it 'runs in any environment' do; end
it 'runs in any environment or pipeline' do; end
it 'runs only in production', only: :production do; end
it 'runs only in production environment', only: :production do; end
it 'runs only in staging', only: { subdomain: :staging } do; end
it 'runs only in staging environment', only: { subdomain: :staging } do; end
it 'runs in dev', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
it 'runs in prod and staging', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end
it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end
end
```
......@@ -46,6 +53,8 @@ NOTE: **Note:**
If the test has a `before` or `after`, you must add the `only` metadata
to the outer `RSpec.describe`.
If you want to run an `only: { :pipeline }` tagged test on your local GDK make sure either the `CI_PROJECT_NAME` environment variable is unset, or that the `CI_PROJECT_NAME` environment variable matches the specified pipeline in the `only: { :pipeline }` tag, or just delete the `only: { :pipeline }` tag.
## Quarantining a test for a specific environment
Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a
......
......@@ -11,7 +11,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run.
| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* |
| `:only` | The test is only to be run against specific environments. See [Environment selection](environment_selection.md) for more information. |
| `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. |
| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). |
| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
......
......@@ -24,7 +24,7 @@ module QA
SUPPORTED_FEATURES
end
def address_matches?(*options)
def context_matches?(*options)
return false unless Runtime::Scenario.attributes[:gitlab_address]
opts = {}
......@@ -33,29 +33,38 @@ module QA
uri = URI(Runtime::Scenario.gitlab_address)
if options.any?
options.each do |option|
opts[:domain] = 'gitlab' if option == :production
if option.is_a?(Hash) && !option[:subdomain].nil?
opts.merge!(option)
opts[:subdomain] = case option[:subdomain]
when Array
"(#{option[:subdomain].join("|")})."
when Regexp
option[:subdomain]
else
"(#{option[:subdomain]})."
end
end
options.each do |option|
opts[:domain] = 'gitlab' if option == :production
if option.is_a?(Hash) && !option[:pipeline].nil? && !ci_project_name.nil?
return pipeline_matches?(option[:pipeline])
elsif option.is_a?(Hash) && !option[:subdomain].nil?
opts.merge!(option)
opts[:subdomain] = case option[:subdomain]
when Array
"(#{option[:subdomain].join("|")})."
when Regexp
option[:subdomain]
else
"(#{option[:subdomain]})."
end
end
end
uri.host.match?(/^#{opts[:subdomain]}#{opts[:domain]}#{opts[:tld]}$/)
end
alias_method :dot_com?, :address_matches?
alias_method :dot_com?, :context_matches?
def pipeline_matches?(pipeline_to_run_in)
Array(pipeline_to_run_in).any? { |pipeline| pipeline.to_s.casecmp?(pipeline_from_project_name) }
end
def pipeline_from_project_name
ci_project_name.to_s.start_with?('gitlab-qa') ? 'master' : ci_project_name
end
def additional_repository_storage
ENV['QA_ADDITIONAL_REPOSITORY_STORAGE']
......
......@@ -20,7 +20,7 @@ module QA
Quarantine.skip_or_run_quarantined_tests_or_contexts(config.inclusion_filter.rules, example)
if example.metadata.key?(:only)
skip('Test is not compatible with this environment') unless Runtime::Env.address_matches?(example.metadata[:only])
skip('Test is not compatible with this environment or pipeline') unless Runtime::Env.context_matches?(example.metadata[:only])
end
end
end
......@@ -55,7 +55,7 @@ module QA
if quarantine_tag&.is_a?(Hash) && quarantine_tag&.key?(:only)
# If the :quarantine hash contains :only, we respect that.
# For instance `quarantine: { only: { subdomain: :staging } }` will only quarantine the test when it runs against staging.
return unless Runtime::Env.address_matches?(quarantine_tag[:only])
return unless Runtime::Env.context_matches?(quarantine_tag[:only])
end
skip(quarantine_message(quarantine_tag))
......
......@@ -341,7 +341,7 @@ RSpec.describe QA::Runtime::Env do
end
end
describe '.address_matches?' do
describe '.context_matches?' do
it 'returns true when url has .com' do
QA::Runtime::Scenario.define(:gitlab_address, "https://staging.gitlab.com")
......@@ -364,24 +364,24 @@ RSpec.describe QA::Runtime::Env do
it 'matches multiple subdomains' do
QA::Runtime::Scenario.define(:gitlab_address, "https://staging.gitlab.com")
expect(described_class.address_matches?(subdomain: [:release, :staging])).to be_truthy
expect(described_class.address_matches?(:production, subdomain: [:release, :staging])).to be_truthy
expect(described_class.context_matches?(subdomain: [:release, :staging])).to be_truthy
expect(described_class.context_matches?(:production, subdomain: [:release, :staging])).to be_truthy
end
it 'matches :production' do
QA::Runtime::Scenario.define(:gitlab_address, "https://gitlab.com/")
expect(described_class.address_matches?(:production)).to be_truthy
expect(described_class.context_matches?(:production)).to be_truthy
end
it 'doesnt match with mismatching switches' do
QA::Runtime::Scenario.define(:gitlab_address, 'https://gitlab.test')
aggregate_failures do
expect(described_class.address_matches?(tld: '.net')).to be_falsey
expect(described_class.address_matches?(:production)).to be_falsey
expect(described_class.address_matches?(subdomain: [:staging])).to be_falsey
expect(described_class.address_matches?(domain: 'example')).to be_falsey
expect(described_class.context_matches?(tld: '.net')).to be_falsey
expect(described_class.context_matches?(:production)).to be_falsey
expect(described_class.context_matches?(subdomain: [:staging])).to be_falsey
expect(described_class.context_matches?(domain: 'example')).to be_falsey
end
end
end
......@@ -389,7 +389,7 @@ RSpec.describe QA::Runtime::Env do
it 'returns false for mismatching' do
QA::Runtime::Scenario.define(:gitlab_address, "https://staging.gitlab.com")
expect(described_class.address_matches?(:production)).to be_falsey
expect(described_class.context_matches?(:production)).to be_falsey
end
end
end
......@@ -37,6 +37,8 @@ RSpec.configure do |c|
end
RSpec.describe QA::Specs::Helpers::Quarantine do
include Helpers::StubENV
describe '.skip_or_run_quarantined_contexts' do
context 'with no tag focused' do
before do
......@@ -312,7 +314,7 @@ RSpec.describe QA::Specs::Helpers::Quarantine do
end
end
describe 'running against specific environments' do
describe 'running against specific environments or pipelines' do
before do
QA::Runtime::Scenario.define(:gitlab_address, 'https://staging.gitlab.com')
described_class.configure_rspec
......@@ -400,5 +402,70 @@ RSpec.describe QA::Specs::Helpers::Quarantine do
expect(group.examples.first.execution_result.pending_message).to match(/[Tt]est.*not compatible.*environment/)
end
context 'with pipeline constraints' do
context 'without CI_PROJECT_NAME set' do
before do
stub_env('CI_PROJECT_NAME', nil)
described_class.configure_rspec
end
it 'runs on any pipeline' do
group = describe_successfully do
it('runs given a single named pipeline', only: { pipeline: :nightly } ) {}
it('runs given an array of pipelines', only: { pipeline: [:canary, :not_nightly] }) {}
end
aggregate_failures do
expect(group.examples[0].execution_result.status).to eq(:passed)
expect(group.examples[1].execution_result.status).to eq(:passed)
end
end
end
context 'when a pipeline triggered from master runs in gitlab-qa' do
before do
stub_env('CI_PROJECT_NAME', 'gitlab-qa')
described_class.configure_rspec
end
it 'runs on master pipelines' do
group = describe_successfully do
it('runs on master pipeline given a single pipeline', only: { pipeline: :master } ) {}
it('runs in master given an array of pipelines', only: { pipeline: [:canary, :master] }) {}
it('does not run in non-master pipelines', only: { pipeline: [:nightly, :not_nightly, :not_master] } ) {}
end
aggregate_failures do
expect(group.examples[0].execution_result.status).to eq(:passed)
expect(group.examples[1].execution_result.status).to eq(:passed)
expect(group.examples[2].execution_result.status).to eq(:pending)
end
end
end
context 'with CI_PROJECT_NAME set' do
before do
stub_env('CI_PROJECT_NAME', 'NIGHTLY')
described_class.configure_rspec
end
it 'runs on designated pipeline' do
group = describe_successfully do
it('runs on nightly', only: { pipeline: :nightly } ) {}
it('does not run in not_nightly', only: { pipeline: :not_nightly } ) {}
it('runs on nightly given an array', only: { pipeline: [:canary, :nightly] }) {}
it('does not run in not_nightly given an array', only: { pipeline: [:not_nightly, :canary] }) {}
end
aggregate_failures do
expect(group.examples[0].execution_result.status).to eq(:passed)
expect(group.examples[1].execution_result.status).to eq(:pending)
expect(group.examples[2].execution_result.status).to eq(:passed)
expect(group.examples[3].execution_result.status).to eq(:pending)
end
end
end
end
end
end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment