Skip to content

G
gitlab-ce
Commit f9e527a1 authored by Dan Davison's avatar Dan Davison
Browse files
Options

Merge branch 'ml-update-e2e-test-context-selection-doc' into 'master' 

Update QA test execution context selection docs

See merge request gitlab-org/gitlab!61319
parents ad618fdc f9a7baf3
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 358 additions and 133 deletions
.gitlab/merge_request_templates/Quarantine End to End Test.md
View file @ f9e527a1
......@@ -21,7 +21,7 @@ the noise (due to constantly failing tests, flaky tests, and so on) so that new
- [ ] Quarantine test check-list
- [ ] Follow the [Quarantining Tests guide](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests).
- [ ] Confirm the test has a [`quarantine:` tag with the specified quarantine type](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantined-test-types).
- [ ] Note if the test should be [quarantined for a specific environment](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/environment_selection.html#quarantining-a-test-for-a-specific-environment).
- [ ] Note if the test should be [quarantined for a specific environment](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/execution_context_selection.html#quarantine-a-test-for-a-specific-environment).
- [ ] Dequarantine test check-list
- [ ] Follow the [Dequarantining Tests guide](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#dequarantining-tests).
- [ ] Confirm the test consistently passes on the target GitLab environment(s).
......
doc/development/testing_guide/end_to_end/environment_selection.md
View file @ f9e527a1
---
stage: none
group: unassigned
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
redirect_to: 'execution_context_selection.md'
---
# Environment selection
This file was moved to [another location](execution_context_selection.md).
Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines).
We can specify what environments or pipelines to run tests against using the `only` metadata.
## Available switches
| Switch | Function | Type |
| -------| ------- | ----- |
| `tld` | Set the top-level domain matcher | `String` |
| `subdomain` | Set the subdomain matcher | `Array` or `String` |
| `domain` | Set the domain matcher | `String` |
| `production` | Match against production | `Static` |
| `pipeline` | Match against a pipeline | `Array` or `Static`|
WARNING:
You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously.
These options are mutually exclusive. If you want to specify production, you
can control the `tld` and `domain` independently.
## Examples
| Environment or pipeline | Key | Matches (regex for environments, string matching for pipelines) |
| ---------------- | --- | --------------- |
| `any` | `` | `.+.com` |
| `gitlab.com` | `only: :production` | `gitlab.com` |
| `staging.gitlab.com` | `only: { subdomain: :staging }` | `(staging).+.com` |
| `gitlab.com and staging.gitlab.com` | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` |
| `dev.gitlab.org` | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` |
| `staging.gitlab.com & domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging|domain).+.com` |
| `nightly` | `only: { pipeline: :nightly }` | "nightly" |
| `nightly`, `canary` | `only: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) |
```ruby
RSpec.describe 'Area' do
it 'runs in any environment or pipeline' do; end
it 'runs only in production environment', only: :production do; end
it 'runs only in staging environment', only: { subdomain: :staging } do; end
it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end
it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end
end
```
If the test has a `before` or `after`, you must add the `only` metadata
to the outer `RSpec.describe`.
If you want to run an `only: { :pipeline }` tagged test on your local GDK make sure either the `CI_PROJECT_NAME` CI/CD variable is unset, or that the `CI_PROJECT_NAME` variable matches the specified pipeline in the `only: { :pipeline }` tag, or just delete the `only: { :pipeline }` tag.
## Quarantining a test for a specific environment
Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a
test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }`
hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash.
For instance, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against staging.
## Excluding a test from running in a particular job
Sometimes we need to skip a test in a particular job but allow it to run in other jobs of the same pipeline or environment.
We can do it with the help of `exclude` metadata.
Examples:
```ruby
RSpec.describe 'Excluding' do
it 'skips given a single named job', exclude: { job: 'ee:instance-image' } do; end
it 'skips given a single regex pattern', exclude: { job: '.*:instance-image' } do; end
it 'skips given an array of jobs', exclude: { job: %w[ee:instance-image qa-schedules-browser_ui-3_create] } do; end
it 'skips given an array of regex patterns', exclude: { job: %w[ee:.* qa-schedules-browser_ui.*] } do; end
it 'skips given a mix of strings and regex patterns', exclude: { job: %w[ee:instance-image qa-schedules-browser_ui.*] } do; end
end
```
<!-- This redirect file can be deleted after <2021-08-14>. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
doc/development/testing_guide/end_to_end/execution_context_selection.md 0 → 100644
View file @ f9e527a1
---
stage: none
group: unassigned
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Execution context selection
Some tests are designed to be run against specific environments, or in specific [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines) or jobs. We can specify the test execution context using the `only` and `except` metadata.
## Available switches
| Switch | Function | Type |
| -------| ------- | ----- |
| `tld` | Set the top-level domain matcher | `String` |
| `subdomain` | Set the subdomain matcher | `Array` or `String` |
| `domain` | Set the domain matcher | `String` |
| `production` | Match the production environment | `Static` |
| `pipeline` | Match a pipeline | `Array` or `Static`|
| `job` | Match a job | `Array` or `Static`|
WARNING:
You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously.
These options are mutually exclusive. If you want to specify production, you
can control the `tld` and `domain` independently.
## Examples
### Only
Run tests in only the specified context.
Matches use:
- Regex for environments.
- String matching for pipelines.
- Regex or string matching for jobs.
| Test execution context | Key | Matches |
| ---------------- | --- | --------------- |
| `gitlab.com` | `only: :production` | `gitlab.com` |
| `staging.gitlab.com` | `only: { subdomain: :staging }` | `(staging).+.com` |
| `gitlab.com and staging.gitlab.com` | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` |
| `dev.gitlab.org` | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` |
| `staging.gitlab.com and domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging\|domain).+.com` |
| The `nightly` pipeline | `only: { pipeline: :nightly }` | "nightly" |
| The `nightly` and `canary` pipelines | `only: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) |
| The `ee:instance` job | `only: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline |
| Any `quarantine` job | `only: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline |
```ruby
RSpec.describe 'Area' do
it 'runs in any environment or pipeline' do; end
it 'runs only in production environment', only: :production do; end
it 'runs only in staging environment', only: { subdomain: :staging } do; end
it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end
it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end
end
```
### Except
Run tests in their typical contexts _except_ as specified.
Matches use:
- Regex for environments.
- String matching for pipelines.
- Regex or string matching for jobs.
| Test execution context | Key | Matches |
| ---------------- | --- | --------------- |
| `gitlab.com` | `except: :production` | `gitlab.com` |
| `staging.gitlab.com` | `except: { subdomain: :staging }` | `(staging).+.com` |
| `gitlab.com and staging.gitlab.com` | `except: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` |
| `dev.gitlab.org` | `except: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` |
| `staging.gitlab.com and domain.gitlab.com` | `except: { subdomain: %i[staging domain] }` | `(staging\|domain).+.com` |
| The `nightly` pipeline | `except: { pipeline: :nightly }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) |
| The `nightly` and `canary` pipelines | `except: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) |
| The `ee:instance` job | `except: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline |
| Any `quarantine` job | `except: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline |
```ruby
RSpec.describe 'Area' do
it 'runs in any execution context except the production environment', except: :production do; end
it 'runs in any execution context except the staging environment', except: { subdomain: :staging } do; end
it 'runs in any execution context except the nightly pipeline', except: { pipeline: :nightly } do; end
it 'runs in any execution context except the ee:instance job', except: { job: 'ee:instance' } do; end
end
```
## Usage notes
If the test has a `before` or `after` block, you must add the `only` or `except` metadata to the outer `RSpec.describe` block.
To run a test tagged with `only` on your local GitLab instance, you can do one of the following:
- Make sure you **do not** have the `CI_PROJECT_NAME` or `CI_JOB_NAME` environment variables set.
- Set the appropriate variable to match the metadata. For example, if the metadata is `only: { pipeline: :nightly }` then set `CI_PROJECT_NAME=nightly`. If the metadata is `only: { job: 'ee:instance' }` then set `CI_JOB_NAME=ee:instance`.
- Temporarily remove the metadata.
To run a test tagged with `except` locally, you can either:
- Make sure you **do not** have the `CI_PROJECT_NAME` or `CI_JOB_NAME` environment variables set.
- Temporarily remove the metadata.
## Quarantine a test for a specific environment
Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a
test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }`
hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash.
For example, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against `staging`.
doc/development/testing_guide/end_to_end/index.md
View file @ f9e527a1
......@@ -212,6 +212,7 @@ Continued reading:
- [Testing with feature flags](feature_flags.md)
- [Flows](flows.md)
- [RSpec metadata/tags](rspec_metadata_tests.md)
- [Execution context selection](execution_context_selection.md)
## Where can I ask for help?
......
doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
View file @ f9e527a1
......@@ -14,7 +14,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| Tag | Description |
|-----|-------------|
| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
| `:exclude` | The test is excluded from running in a specific job. See [Environment selection](environment_selection.md#excluding-a-test-from-running-in-a-particular-job) for more information. |
| `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. |
| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. |
| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
| `:github` | The test requires a GitHub personal access token. |
......@@ -27,10 +27,10 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. |
| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. |
| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. |
| `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. |
| `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. |
| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). |
| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs in a specific context](execution_context_selection.md#quarantine-a-test-for-a-specific-environment). |
| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). |
| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. |
......
qa/qa/specs/features/browser_ui/2_plan/issue/create_issue_spec.rb
View file @ f9e527a1
......@@ -51,7 +51,7 @@ module QA
# The following example is excluded from running in `review-qa-smoke` job
# as it proved to be flaky when running against Review App
# See https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/11568#note_621999351
it 'comments on an issue with an attachment', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/1742', exclude: { job: 'review-qa-smoke' } do
it 'comments on an issue with an attachment', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/1742', except: { job: 'review-qa-smoke' } do
Page::Project::Issue::Show.perform do |show|
show.comment('See attached image for scale', attachment: file_to_attach)
......
qa/qa/specs/features/ee/browser_ui/1_manage/group/restrict_by_ip_address_spec.rb
View file @ f9e527a1
......@@ -3,7 +3,7 @@ require 'securerandom'
module QA
RSpec.describe 'Manage' do
describe 'Group access', :requires_admin, :skip_live_env, exclude: { job: 'review-qa-*' } do
describe 'Group access', :requires_admin, :skip_live_env, except: { job: 'review-qa-*' } do
include Runtime::IPAddress
before(:all) do
......
qa/qa/specs/helpers/context_selector.rb
View file @ f9e527a1
......@@ -13,15 +13,17 @@ module QA
config.before do |example|
if example.metadata.key?(:only)
skip('Test is not compatible with this environment or pipeline') unless ContextSelector.context_matches?(example.metadata[:only])
elsif example.metadata.key?(:exclude)
skip('Test is excluded in this job') if ContextSelector.exclude?(example.metadata[:exclude])
elsif example.metadata.key?(:except)
skip('Test is excluded in this job') if ContextSelector.except?(example.metadata[:except])
end
end
end
end
def exclude?(*options)
return false unless Runtime::Env.ci_job_name.present?
def except?(*options)
return false if Runtime::Env.ci_job_name.blank? && options.any? { |o| o.is_a?(Hash) && o[:job].present? }
return false if Runtime::Env.ci_project_name.blank? && options.any? { |o| o.is_a?(Hash) && o[:pipeline].present? }
return false if Runtime::Scenario.attributes[:gitlab_address].blank?
context_matches?(*options)
end
......@@ -40,10 +42,14 @@ module QA
next unless option.is_a?(Hash)
if option[:pipeline].present? && Runtime::Env.ci_project_name.present?
if option[:pipeline].present?
return true if Runtime::Env.ci_project_name.blank?
return pipeline_matches?(option[:pipeline])
elsif option[:job].present?
return true if Runtime::Env.ci_job_name.blank?
return job_matches?(option[:job])
elsif option[:subdomain].present?
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7