Commit 8c56c46e authored by Rémy Coutable's avatar Rémy Coutable Committed by Ramya Authappan

Allow to quarantine end-to-end tests per environment

This builds upon https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35668
and allows to quarantine end-to-end tests only when they run against
specific environments.
Signed-off-by: default avatarRémy Coutable <remy@rymai.me>
parent 655a14fc
......@@ -33,7 +33,7 @@ RSpec.describe 'Area' do
it 'runs in any environment' do; end
it 'runs only in production', only: :production do; end
it 'runs only in staging', only: { subdomain: :staging } do; end
it 'runs in dev', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
......@@ -45,3 +45,10 @@ end
NOTE: **Note:**
If the test has a `before` or `after`, you must add the `only` metadata
to the outer `RSpec.describe`.
## Quarantining a test for a specific environment
Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a
test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }`
hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash.
For instance, `quarantine: { only: { subdomain: :staging } }` will only quarantine the test when run against staging.
......@@ -10,7 +10,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* |
| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). |
| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
| `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. |
| `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. |
......
......@@ -20,7 +20,7 @@ module QA
Quarantine.skip_or_run_quarantined_tests_or_contexts(config.inclusion_filter.rules, example)
if example.metadata.key?(:only)
skip('Test is not compatible with this environment') unless Runtime::Env.dot_com?(example.metadata[:only])
skip('Test is not compatible with this environment') unless Runtime::Env.address_matches?(example.metadata[:only])
end
end
end
......@@ -50,21 +50,15 @@ module QA
skip("Only running tests tagged with :quarantine and any of #{included_filters.keys}") if should_skip_when_focused?(example.metadata, included_filters)
else
if example.metadata.key?(:quarantine)
quarantine_message = %w(In quarantine)
quarantine_tag = example.metadata[:quarantine]
if !!quarantine_tag
quarantine_message << case quarantine_tag
when String
": #{quarantine_tag}"
when Hash
": #{quarantine_tag[:issue]}"
else
''
end
if quarantine_tag&.is_a?(Hash) && quarantine_tag&.key?(:only)
# If the :quarantine hash contains :only, we respect that.
# For instance `quarantine: { only: { subdomain: :staging } }` will only quarantine the test when it runs against staging.
return unless Runtime::Env.address_matches?(quarantine_tag[:only])
end
skip(quarantine_message.join(' ').strip)
skip(quarantine_message(quarantine_tag))
end
end
end
......@@ -73,6 +67,20 @@ module QA
filter.reject { |key, _| key == :quarantine }
end
def quarantine_message(quarantine_tag)
quarantine_message = %w(In quarantine)
quarantine_message << case quarantine_tag
when String
": #{quarantine_tag}"
when Hash
quarantine_tag.key?(:issue) ? ": #{quarantine_tag[:issue]}" : ''
else
''
end
quarantine_message.join(' ').strip
end
# Checks if a test or context should be skipped.
#
# Returns true if
......
......@@ -124,7 +124,7 @@ describe QA::Specs::Helpers::Quarantine do
end
end
describe '.skip_or_run_quarantined_tests' do
describe '.skip_or_run_quarantined_tests_or_contexts' do
context 'with no tag focused' do
before do
described_class.configure_rspec
......@@ -148,6 +148,37 @@ describe QA::Specs::Helpers::Quarantine do
expect(group.examples.first.execution_result.status).to eq(:passed)
end
context 'with environment set' do
before do
QA::Runtime::Scenario.define(:gitlab_address, 'https://staging.gitlab.com')
described_class.configure_rspec
end
it 'is skipped when set on contexts or descriptions' do
group = describe_successfully 'Quarantined in staging', quarantine: { only: { subdomain: :staging } } do
it('runs in staging') {}
end
expect(group.examples.first.execution_result.status).to eq(:pending)
expect(group.examples.first.execution_result.pending_message)
.to eq('In quarantine')
end
it 'is skipped only in staging' do
group = describe_successfully do
it('skipped in staging', quarantine: { only: { subdomain: :staging } }) {}
it('runs in staging', quarantine: { only: :production }) {}
it('skipped in staging also', quarantine: { only: { subdomain: %i[release staging] } }) {}
it('runs in any env') {}
end
expect(group.examples[0].execution_result.status).to eq(:pending)
expect(group.examples[1].execution_result.status).to eq(:passed)
expect(group.examples[2].execution_result.status).to eq(:pending)
expect(group.examples[3].execution_result.status).to eq(:passed)
end
end
context 'quarantine message' do
shared_examples 'test with quarantine message' do |quarantine_tag|
it 'outputs the quarantine message' do
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment