Merge branch 'egb-only-nightly' into 'master'

Add ability to constrain tests per pipeline

See merge request gitlab-org/gitlab!43043

doc/development/testing_guide/end_to_end/environment_selection.md

 # Environment selection
 
-Some tests are designed to be run against specific environments. We can specify
-what environments to run tests against using the `only` metadata.
+Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines). 
+We can specify what environments or pipelines to run tests against using the `only` metadata.
 
 ## Available switches
 
@@ -11,6 +11,7 @@ what environments to run tests against using the `only` metadata.
 | `subdomain` | Set the subdomain matcher | `Array` or `String` |
 | `domain` | Set the domain matcher | `String` |
 | `production` | Match against production | `Static` |
+| `pipeline` | Match against a pipeline | `Array` or `Static`|
 
 CAUTION: **Caution:**
 You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously.
@@ -19,7 +20,7 @@ can control the `tld` and `domain` independently.
 
 ## Examples
 
-| Environment                              | Key | Matches (regex)                                                            |
+| Environment or pipeline                  | Key | Matches (regex for environments, string matching for pipelines)            |
 | ----------------                         | --- | ---------------                                                            |
 | `any`                                    | ``  | `.+.com`                                                                   |
 | `gitlab.com`                             | `only: :production` | `gitlab.com`                                               |
@@ -27,18 +28,24 @@ can control the `tld` and `domain` independently.
 | `gitlab.com and staging.gitlab.com`      | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` |
 | `dev.gitlab.org`                         | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` |
 | `staging.gitlab.com & domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging|domain).+.com`             |
+| `nightly`                                | `only: { pipeline: :nightly }` | "nightly" |
+| `nightly`, `canary` | `only_run_in_pipeline: [:nightly, :canary]` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) |
 
 ```ruby
 RSpec.describe 'Area' do
-  it 'runs in any environment' do; end
+  it 'runs in any environment or pipeline' do; end
 
-  it 'runs only in production', only: :production do; end
+  it 'runs only in production environment', only: :production do; end
 
-  it 'runs only in staging', only: { subdomain: :staging } do; end
+  it 'runs only in staging environment', only: { subdomain: :staging } do; end
 
-  it 'runs in dev', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
+  it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
 
-  it 'runs in prod and staging', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
+  it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
+  
+  it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end
+  
+  it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end
 end
 ```
 
@@ -46,6 +53,8 @@ NOTE: **Note:**
 If the test has a `before` or `after`, you must add the `only` metadata
 to the outer `RSpec.describe`.
 
+If you want to run an `only: { :pipeline }` tagged test on your local GDK make sure either the `CI_PROJECT_NAME` environment variable is unset, or that the `CI_PROJECT_NAME` environment variable matches the specified pipeline in the `only: { :pipeline }` tag, or just delete the `only: { :pipeline }` tag.
+
 ## Quarantining a test for a specific environment
 
 Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a

doc/development/testing_guide/end_to_end/rspec_metadata_tests.md

@@ -11,7 +11,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
 | `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
 | `:jira`           | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run.
 | `:kubernetes`     | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* |
-| `:only`           | The test is only to be run against specific environments. See [Environment selection](environment_selection.md) for more information. |
+| `:only`           | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. |
 | `:orchestrated`   | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). |
 | `:quarantine`     | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). |
 | `:reliable`       | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |

qa/qa/runtime/env.rb

@@ -24,7 +24,7 @@ module QA
         SUPPORTED_FEATURES
       end
 
-      def address_matches?(*options)
+      def context_matches?(*options)
         return false unless Runtime::Scenario.attributes[:gitlab_address]
 
         opts = {}
@@ -33,29 +33,38 @@ module QA
 
         uri = URI(Runtime::Scenario.gitlab_address)
 
-        if options.any?
-          options.each do |option|
-            opts[:domain] = 'gitlab' if option == :production
-
-            if option.is_a?(Hash) && !option[:subdomain].nil?
-              opts.merge!(option)
-
-              opts[:subdomain] = case option[:subdomain]
-                                 when Array
-                                   "(#{option[:subdomain].join("|")})."
-                                 when Regexp
-                                   option[:subdomain]
-                                 else
-                                   "(#{option[:subdomain]})."
-                                 end
-            end
+        options.each do |option|
+          opts[:domain] = 'gitlab' if option == :production
+
+          if option.is_a?(Hash) && !option[:pipeline].nil? && !ci_project_name.nil?
+            return pipeline_matches?(option[:pipeline])
+
+          elsif option.is_a?(Hash) && !option[:subdomain].nil?
+            opts.merge!(option)
+
+            opts[:subdomain] = case option[:subdomain]
+                               when Array
+                                 "(#{option[:subdomain].join("|")})."
+                               when Regexp
+                                 option[:subdomain]
+                               else
+                                 "(#{option[:subdomain]})."
+                               end
           end
         end
 
         uri.host.match?(/^#{opts[:subdomain]}#{opts[:domain]}#{opts[:tld]}$/)
       end
 
-      alias_method :dot_com?, :address_matches?
+      alias_method :dot_com?, :context_matches?
+
+      def pipeline_matches?(pipeline_to_run_in)
+        Array(pipeline_to_run_in).any? { |pipeline| pipeline.to_s.casecmp?(pipeline_from_project_name) }
+      end
+
+      def pipeline_from_project_name
+        ci_project_name.to_s.start_with?('gitlab-qa') ? 'master' : ci_project_name
+      end
 
       def additional_repository_storage
         ENV['QA_ADDITIONAL_REPOSITORY_STORAGE']

qa/qa/specs/helpers/quarantine.rb

@@ -20,7 +20,7 @@ module QA
               Quarantine.skip_or_run_quarantined_tests_or_contexts(config.inclusion_filter.rules, example)
 
               if example.metadata.key?(:only)
-                skip('Test is not compatible with this environment') unless Runtime::Env.address_matches?(example.metadata[:only])
+                skip('Test is not compatible with this environment or pipeline') unless Runtime::Env.context_matches?(example.metadata[:only])
               end
             end
           end
@@ -55,7 +55,7 @@ module QA
               if quarantine_tag&.is_a?(Hash) && quarantine_tag&.key?(:only)
                 # If the :quarantine hash contains :only, we respect that.
                 # For instance `quarantine: { only: { subdomain: :staging } }` will only quarantine the test when it runs against staging.
-                return unless Runtime::Env.address_matches?(quarantine_tag[:only])
+                return unless Runtime::Env.context_matches?(quarantine_tag[:only])
               end
 
               skip(quarantine_message(quarantine_tag))

qa/spec/runtime/env_spec.rb

@@ -341,7 +341,7 @@ RSpec.describe QA::Runtime::Env do
     end
   end
 
-  describe '.address_matches?' do
+  describe '.context_matches?' do
     it 'returns true when url has .com' do
       QA::Runtime::Scenario.define(:gitlab_address, "https://staging.gitlab.com")
 
@@ -364,24 +364,24 @@ RSpec.describe QA::Runtime::Env do
       it 'matches multiple subdomains' do
         QA::Runtime::Scenario.define(:gitlab_address, "https://staging.gitlab.com")
 
-        expect(described_class.address_matches?(subdomain: [:release, :staging])).to be_truthy
-        expect(described_class.address_matches?(:production, subdomain: [:release, :staging])).to be_truthy
+        expect(described_class.context_matches?(subdomain: [:release, :staging])).to be_truthy
+        expect(described_class.context_matches?(:production, subdomain: [:release, :staging])).to be_truthy
       end
 
       it 'matches :production' do
         QA::Runtime::Scenario.define(:gitlab_address, "https://gitlab.com/")
 
-        expect(described_class.address_matches?(:production)).to be_truthy
+        expect(described_class.context_matches?(:production)).to be_truthy
       end
 
       it 'doesnt match with mismatching switches' do
         QA::Runtime::Scenario.define(:gitlab_address, 'https://gitlab.test')
 
         aggregate_failures do
-          expect(described_class.address_matches?(tld: '.net')).to be_falsey
-          expect(described_class.address_matches?(:production)).to be_falsey
-          expect(described_class.address_matches?(subdomain: [:staging])).to be_falsey
-          expect(described_class.address_matches?(domain: 'example')).to be_falsey
+          expect(described_class.context_matches?(tld: '.net')).to be_falsey
+          expect(described_class.context_matches?(:production)).to be_falsey
+          expect(described_class.context_matches?(subdomain: [:staging])).to be_falsey
+          expect(described_class.context_matches?(domain: 'example')).to be_falsey
         end
       end
     end
@@ -389,7 +389,7 @@ RSpec.describe QA::Runtime::Env do
     it 'returns false for mismatching' do
       QA::Runtime::Scenario.define(:gitlab_address, "https://staging.gitlab.com")
 
-      expect(described_class.address_matches?(:production)).to be_falsey
+      expect(described_class.context_matches?(:production)).to be_falsey
     end
   end
 end

qa/spec/specs/helpers/quarantine_spec.rb

@@ -37,6 +37,8 @@ RSpec.configure do |c|
 end
 
 RSpec.describe QA::Specs::Helpers::Quarantine do
+  include Helpers::StubENV
+
   describe '.skip_or_run_quarantined_contexts' do
     context 'with no tag focused' do
       before do
@@ -312,7 +314,7 @@ RSpec.describe QA::Specs::Helpers::Quarantine do
     end
   end
 
-  describe 'running against specific environments' do
+  describe 'running against specific environments or pipelines' do
     before do
       QA::Runtime::Scenario.define(:gitlab_address, 'https://staging.gitlab.com')
       described_class.configure_rspec
@@ -400,5 +402,70 @@ RSpec.describe QA::Specs::Helpers::Quarantine do
 
       expect(group.examples.first.execution_result.pending_message).to match(/[Tt]est.*not compatible.*environment/)
     end
+
+    context 'with pipeline constraints' do
+      context 'without CI_PROJECT_NAME set' do
+        before do
+          stub_env('CI_PROJECT_NAME', nil)
+          described_class.configure_rspec
+        end
+
+        it 'runs on any pipeline' do
+          group = describe_successfully do
+            it('runs given a single named pipeline', only: { pipeline: :nightly } ) {}
+            it('runs given an array of pipelines', only: { pipeline: [:canary, :not_nightly] }) {}
+          end
+
+          aggregate_failures do
+            expect(group.examples[0].execution_result.status).to eq(:passed)
+            expect(group.examples[1].execution_result.status).to eq(:passed)
+          end
+        end
+      end
+
+      context 'when a pipeline triggered from master runs in gitlab-qa' do
+        before do
+          stub_env('CI_PROJECT_NAME', 'gitlab-qa')
+          described_class.configure_rspec
+        end
+
+        it 'runs on master pipelines' do
+          group = describe_successfully do
+            it('runs on master pipeline given a single pipeline', only: { pipeline: :master } ) {}
+            it('runs in master given an array of pipelines', only: { pipeline: [:canary, :master] }) {}
+            it('does not run in non-master pipelines', only: { pipeline: [:nightly, :not_nightly, :not_master] } ) {}
+          end
+
+          aggregate_failures do
+            expect(group.examples[0].execution_result.status).to eq(:passed)
+            expect(group.examples[1].execution_result.status).to eq(:passed)
+            expect(group.examples[2].execution_result.status).to eq(:pending)
+          end
+        end
+      end
+
+      context 'with CI_PROJECT_NAME set' do
+        before do
+          stub_env('CI_PROJECT_NAME', 'NIGHTLY')
+          described_class.configure_rspec
+        end
+
+        it 'runs on designated pipeline' do
+          group = describe_successfully do
+            it('runs on nightly', only: { pipeline: :nightly } ) {}
+            it('does not run in not_nightly', only: { pipeline: :not_nightly } ) {}
+            it('runs on nightly given an array', only: { pipeline: [:canary, :nightly] }) {}
+            it('does not run in not_nightly given an array', only: { pipeline: [:not_nightly, :canary] }) {}
+          end
+
+          aggregate_failures do
+            expect(group.examples[0].execution_result.status).to eq(:passed)
+            expect(group.examples[1].execution_result.status).to eq(:pending)
+            expect(group.examples[2].execution_result.status).to eq(:passed)
+            expect(group.examples[3].execution_result.status).to eq(:pending)
+          end
+        end
+      end
+    end
   end
 end