Merge branch 'ml-update-e2e-test-context-selection-doc' into 'master'

Update QA test execution context selection docs

See merge request gitlab-org/gitlab!61319

.gitlab/merge_request_templates/Quarantine End to End Test.md

@@ -21,7 +21,7 @@ the noise (due to constantly failing tests, flaky tests, and so on) so that new
 - [ ] Quarantine test check-list
   - [ ] Follow the [Quarantining Tests guide](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests).
   - [ ] Confirm the test has a [`quarantine:` tag with the specified quarantine type](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantined-test-types).
-  - [ ] Note if the test should be [quarantined for a specific environment](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/environment_selection.html#quarantining-a-test-for-a-specific-environment).
+  - [ ] Note if the test should be [quarantined for a specific environment](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/execution_context_selection.html#quarantine-a-test-for-a-specific-environment).
 - [ ] Dequarantine test check-list
   - [ ] Follow the [Dequarantining Tests guide](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#dequarantining-tests).
   - [ ] Confirm the test consistently passes on the target GitLab environment(s).

doc/development/testing_guide/end_to_end/environment_selection.md

 ---
-stage: none
-group: unassigned
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+redirect_to: 'execution_context_selection.md'
 ---
 
-# Environment selection
+This file was moved to [another location](execution_context_selection.md).
 
-Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines).
-We can specify what environments or pipelines to run tests against using the `only` metadata.
-
-## Available switches
-
-| Switch | Function | Type |
-| -------| ------- | ----- |
-| `tld`  | Set the top-level domain matcher | `String` |
-| `subdomain` | Set the subdomain matcher | `Array` or `String` |
-| `domain` | Set the domain matcher | `String` |
-| `production` | Match against production | `Static` |
-| `pipeline` | Match against a pipeline | `Array` or `Static`|
-
-WARNING:
-You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously.
-These options are mutually exclusive. If you want to specify production, you
-can control the `tld` and `domain` independently.
-
-## Examples
-
-| Environment or pipeline                  | Key | Matches (regex for environments, string matching for pipelines)            |
-| ----------------                         | --- | ---------------                                                            |
-| `any`                                    | ``  | `.+.com`                                                                   |
-| `gitlab.com`                             | `only: :production` | `gitlab.com`                                               |
-| `staging.gitlab.com`                     | `only: { subdomain: :staging }` | `(staging).+.com`                              |
-| `gitlab.com and staging.gitlab.com`      | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` |
-| `dev.gitlab.org`                         | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` |
-| `staging.gitlab.com & domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging|domain).+.com`             |
-| `nightly`                                | `only: { pipeline: :nightly }` | "nightly" |
-| `nightly`, `canary` | `only: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) |
-
-```ruby
-RSpec.describe 'Area' do
-  it 'runs in any environment or pipeline' do; end
-
-  it 'runs only in production environment', only: :production do; end
-
-  it 'runs only in staging environment', only: { subdomain: :staging } do; end
-
-  it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
-
-  it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
-
-  it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end
-
-  it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end
-end
-```
-
-If the test has a `before` or `after`, you must add the `only` metadata
-to the outer `RSpec.describe`.
-
-If you want to run an `only: { :pipeline }` tagged test on your local GDK make sure either the `CI_PROJECT_NAME` CI/CD variable is unset, or that the `CI_PROJECT_NAME` variable matches the specified pipeline in the `only: { :pipeline }` tag, or just delete the `only: { :pipeline }` tag.
-
-## Quarantining a test for a specific environment
-
-Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a
-test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }`
-hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash.
-For instance, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against staging.
-
-## Excluding a test from running in a particular job
-
-Sometimes we need to skip a test in a particular job but allow it to run in other jobs of the same pipeline or environment.
-We can do it with the help of `exclude` metadata.
-
-Examples:
-
-```ruby
-RSpec.describe 'Excluding' do
-  it 'skips given a single named job', exclude: { job: 'ee:instance-image' } do; end
-
-  it 'skips given a single regex pattern', exclude: { job: '.*:instance-image' } do; end
-
-  it 'skips given an array of jobs', exclude: { job: %w[ee:instance-image qa-schedules-browser_ui-3_create] } do; end
-
-  it 'skips given an array of regex patterns', exclude: { job: %w[ee:.* qa-schedules-browser_ui.*] } do; end
-
-  it 'skips given a mix of strings and regex patterns', exclude: { job: %w[ee:instance-image qa-schedules-browser_ui.*] } do; end
-end
-```
+<!-- This redirect file can be deleted after <2021-08-14>. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->

doc/development/testing_guide/end_to_end/execution_context_selection.md

+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Execution context selection
+
+Some tests are designed to be run against specific environments, or in specific [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines) or jobs. We can specify the test execution context using the `only` and `except` metadata.
+
+## Available switches
+
+| Switch | Function | Type |
+| -------| ------- | ----- |
+| `tld`  | Set the top-level domain matcher | `String` |
+| `subdomain` | Set the subdomain matcher | `Array` or `String` |
+| `domain` | Set the domain matcher | `String` |
+| `production` | Match the production environment | `Static` |
+| `pipeline` | Match a pipeline | `Array` or `Static`|
+| `job` | Match a job | `Array` or `Static`|
+
+WARNING:
+You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously.
+These options are mutually exclusive. If you want to specify production, you
+can control the `tld` and `domain` independently.
+
+## Examples
+
+### Only
+
+Run tests in only the specified context.
+
+Matches use:
+
+- Regex for environments.
+- String matching for pipelines.
+- Regex or string matching for jobs.
+
+| Test execution context                   | Key | Matches |
+| ----------------                         | --- | ---------------                                                            |
+| `gitlab.com`                             | `only: :production` | `gitlab.com`                                               |
+| `staging.gitlab.com`                     | `only: { subdomain: :staging }` | `(staging).+.com`                              |
+| `gitlab.com and staging.gitlab.com`      | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` |
+| `dev.gitlab.org`                         | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` |
+| `staging.gitlab.com and domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging\|domain).+.com`             |
+| The `nightly` pipeline                     | `only: { pipeline: :nightly }` | "nightly" |
+| The `nightly` and `canary` pipelines | `only: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) |
+| The `ee:instance` job | `only: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline |
+| Any `quarantine` job | `only: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline |
+
+```ruby
+RSpec.describe 'Area' do
+  it 'runs in any environment or pipeline' do; end
+  it 'runs only in production environment', only: :production do; end
+
+  it 'runs only in staging environment', only: { subdomain: :staging } do; end
+
+  it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
+
+  it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
+
+  it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end
+
+  it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end
+end
+```
+
+### Except
+
+Run tests in their typical contexts _except_ as specified.
+
+Matches use:
+
+- Regex for environments.
+- String matching for pipelines.
+- Regex or string matching for jobs.
+
+| Test execution context                   | Key | Matches |
+| ----------------                         | --- | ---------------                                                            |
+| `gitlab.com`                             | `except: :production` | `gitlab.com`                                               |
+| `staging.gitlab.com`                     | `except: { subdomain: :staging }` | `(staging).+.com`                              |
+| `gitlab.com and staging.gitlab.com`      | `except: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` |
+| `dev.gitlab.org`                         | `except: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` |
+| `staging.gitlab.com and domain.gitlab.com` | `except: { subdomain: %i[staging domain] }` | `(staging\|domain).+.com`             |
+| The `nightly` pipeline                     | `except: { pipeline: :nightly }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) |
+| The `nightly` and `canary` pipelines | `except: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) |
+| The `ee:instance` job | `except: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline |
+| Any `quarantine` job | `except: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline |
+
+```ruby
+RSpec.describe 'Area' do
+  it 'runs in any execution context except the production environment', except: :production do; end
+
+  it 'runs in any execution context except the staging environment', except: { subdomain: :staging } do; end
+
+  it 'runs in any execution context except the nightly pipeline', except: { pipeline: :nightly } do; end
+
+  it 'runs in any execution context except the ee:instance job', except: { job: 'ee:instance' } do; end
+end
+```
+
+## Usage notes
+
+If the test has a `before` or `after` block, you must add the `only` or `except` metadata to the outer `RSpec.describe` block.
+
+To run a test tagged with `only` on your local GitLab instance, you can do one of the following:
+
+- Make sure you **do not** have the `CI_PROJECT_NAME` or `CI_JOB_NAME` environment variables set.
+- Set the appropriate variable to match the metadata. For example, if the metadata is `only: { pipeline: :nightly }` then set `CI_PROJECT_NAME=nightly`. If the metadata is `only: { job: 'ee:instance' }` then set `CI_JOB_NAME=ee:instance`.
+- Temporarily remove the metadata.
+
+To run a test tagged with `except` locally, you can either:
+
+- Make sure you **do not** have the `CI_PROJECT_NAME` or `CI_JOB_NAME` environment variables set.
+- Temporarily remove the metadata.
+
+## Quarantine a test for a specific environment
+
+Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a
+test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }`
+hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash.
+For example, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against `staging`.

doc/development/testing_guide/end_to_end/index.md

@@ -212,6 +212,7 @@ Continued reading:
 - [Testing with feature flags](feature_flags.md)
 - [Flows](flows.md)
 - [RSpec metadata/tags](rspec_metadata_tests.md)
+- [Execution context selection](execution_context_selection.md)
 
 ## Where can I ask for help?
 

doc/development/testing_guide/end_to_end/rspec_metadata_tests.md

@@ -14,7 +14,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
 | Tag | Description |
 |-----|-------------|
 | `:elasticsearch`  | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
-| `:exclude`        | The test is excluded from running in a specific job. See [Environment selection](environment_selection.md#excluding-a-test-from-running-in-a-particular-job) for more information. |
+| `:except`         | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. |
 | `:geo`            | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. |
 | `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
 | `:github`         | The test requires a GitHub personal access token. |
@@ -27,10 +27,10 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
 | `:ldap_tls`       | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. |
 | `:mattermost`     | The test requires a GitLab Mattermost service on the GitLab instance. |
 | `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. |
-| `:only`           | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. |
+| `:only`           | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. |
 | `:orchestrated`   | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). |
 | `:packages`       | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. |
-| `:quarantine`     | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). |
+| `:quarantine`     | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs in a specific context](execution_context_selection.md#quarantine-a-test-for-a-specific-environment). |
 | `:relative_url`   | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). |
 | `:reliable`       | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
 | `:repository_storage` |  The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. |

qa/qa/specs/features/browser_ui/2_plan/issue/create_issue_spec.rb

@@ -51,7 +51,7 @@ module QA
         # The following example is excluded from running in `review-qa-smoke` job
         # as it proved to be flaky when running against Review App
         # See https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/11568#note_621999351
-        it 'comments on an issue with an attachment', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/1742', exclude: { job: 'review-qa-smoke' } do
+        it 'comments on an issue with an attachment', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/1742', except: { job: 'review-qa-smoke' } do
           Page::Project::Issue::Show.perform do |show|
             show.comment('See attached image for scale', attachment: file_to_attach)
 

qa/qa/specs/features/ee/browser_ui/1_manage/group/restrict_by_ip_address_spec.rb

@@ -3,7 +3,7 @@ require 'securerandom'
 
 module QA
   RSpec.describe 'Manage' do
-    describe 'Group access', :requires_admin, :skip_live_env, exclude: { job: 'review-qa-*' } do
+    describe 'Group access', :requires_admin, :skip_live_env, except: { job: 'review-qa-*' } do
       include Runtime::IPAddress
 
       before(:all) do

qa/qa/specs/helpers/context_selector.rb

@@ -13,15 +13,17 @@ module QA
             config.before do |example|
               if example.metadata.key?(:only)
                 skip('Test is not compatible with this environment or pipeline') unless ContextSelector.context_matches?(example.metadata[:only])
-              elsif example.metadata.key?(:exclude)
-                skip('Test is excluded in this job') if ContextSelector.exclude?(example.metadata[:exclude])
+              elsif example.metadata.key?(:except)
+                skip('Test is excluded in this job') if ContextSelector.except?(example.metadata[:except])
               end
             end
           end
         end
 
-        def exclude?(*options)
-          return false unless Runtime::Env.ci_job_name.present?
+        def except?(*options)
+          return false if Runtime::Env.ci_job_name.blank? && options.any? { |o| o.is_a?(Hash) && o[:job].present? }
+          return false if Runtime::Env.ci_project_name.blank? && options.any? { |o| o.is_a?(Hash) && o[:pipeline].present? }
+          return false if Runtime::Scenario.attributes[:gitlab_address].blank?
 
           context_matches?(*options)
         end
@@ -40,10 +42,14 @@ module QA
 
             next unless option.is_a?(Hash)
 
-            if option[:pipeline].present? && Runtime::Env.ci_project_name.present?
+            if option[:pipeline].present?
+              return true if Runtime::Env.ci_project_name.blank?
+
               return pipeline_matches?(option[:pipeline])
 
             elsif option[:job].present?
+              return true if Runtime::Env.ci_job_name.blank?
+
               return job_matches?(option[:job])
 
             elsif option[:subdomain].present?