Merge branch 'ml-manage-reusable-resources-as-a-collection' into 'master'

Validate resource reuse after test suite

See merge request gitlab-org/gitlab!78247

doc/development/testing_guide/end_to_end/resources.md

@@ -431,11 +431,32 @@ module QA
       def validate_reuse_preconditions
         raise ResourceReuseError unless reused_name_valid?
       end
+
+      # Internally we identify an instance of a reusable resource by a unique value of `@reuse_as`, but in GitLab the
+      # resource has one or more attributes that must also be unique. This method lists those attributes and allows the
+      # test framework to check that each instance of a reusable resource has values that match the associated values
+      # in Gitlab.
+      def unique_identifiers
+        [:name, :path]
+      end
     end
   end
 end
 ```
 
+Reusable resources aren't removed immediately when `remove_via_api!` is called. Instead, they're removed after the test
+suite completes. To do so each class must be registered with `QA::Resource::ReusableCollection` in `qa/spec/spec_helper.rb`
+as in the example below. Registration allows `QA::Resource::ReusableCollection` to keep track of each instance of each
+registered class, and to delete them all in the `config.after(:suite)` hook.
+
+```ruby
+config.before(:suite) do |suite|
+  QA::Resource::ReusableCollection.register_resource_classes do |collection|
+    QA::Resource::ReusableProject.register(collection)
+  end
+end
+```
+
 Consider some examples of how a reusable resource is used:
 
 ```ruby
@@ -488,6 +509,65 @@ end
 # match the name specified when the project was first fabricated.
 ```
 
+### Validating reusable resources
+
+Reusable resources can speed up test suites by avoiding the cost of creating the same resource again and again. However,
+that can cause problems if a test makes changes to a resource that prevent it from being reused as expected by later
+tests. That can lead to order-dependent test failures that can be difficult to troubleshoot.
+
+For example, the default project created by `QA::Resource::ReusableProject` has `auto_devops_enabled` set to `false`
+(inherited from `QA::Resource::Project`). If a test reuses that project and enables Auto DevOps, subsequent tests that reuse
+the project will fail if they expect Auto DevOps to be disabled.
+
+We try to avoid that kind of trouble by validating reusable resources after a test suite. If the environment variable
+`QA_VALIDATE_RESOURCE_REUSE` is set to `true` the test framework will check each reusable resource to verify that none
+of the attributes they were created with have been changed. It does that by creating a new resource using the same
+attributes that were used to create the original resource. It then compares the new resource to the original and raises
+an error if any attributes don't match.
+
+#### Implementation
+
+When you implement a new type of reusable resource there are two `private` methods you must implement so the resource
+can be validated. They are:
+
+- `reference_resource`: creates a new instance of the resource that can be compared with the one that was used during the tests.
+- `unique_identifiers`: returns an array of attributes that allow the resource to be identified (e.g., name) and that are therefore
+expected to differ when comparing the reference resource with the resource reused in the tests.
+
+The following example shows the implementation of those two methods in `QA::Resource::ReusableProject`.
+
+```ruby
+# Creates a new project that can be compared to a reused project, using the attributes of the original.
+#
+# @return [QA::Resource] a new instance of Resource::ReusableProject that should be a copy of the original resource
+def reference_resource
+  # These are the attributes that the reused resource was created with
+  attributes = self.class.resources[reuse_as][:attributes]
+
+  # Two projects can't have the same path, and since we typically use the same value for the name and path, we assign
+  # a unique name and path to the reference resource.
+  name = "reference_resource_#{SecureRandom.hex(8)}_for_#{attributes.delete(:name)}"
+
+  Project.fabricate_via_api! do |project|
+    self.class.resources[reuse_as][:attributes].each do |attribute_name, attribute_value|
+      project.instance_variable_set("@#{attribute_name}", attribute_value) if attribute_value
+    end
+    project.name = name
+    project.path = name
+    project.path_with_namespace = "#{project.group.full_path}/#{project.name}"
+  end
+end
+
+# The attributes of the resource that should be the same whenever a test wants to reuse a project.
+#
+# @return [Array<Symbol>] the attribute names.
+def unique_identifiers
+  # As noted above, path must be unique, and since we typically use the same value for both, we treat name and path
+  # as unique. These attributes are ignored when we compare the reference and reused resources.
+  [:name, :path]
+end
+```
+
 ## Where to ask for help?
 
 If you need more information, ask for help on `#quality` channel on Slack

qa/qa/resource/base.rb

@@ -81,8 +81,6 @@ module QA
           Support::FabricationTracker.start_fabrication
           result = yield.tap do
             fabrication_time = Time.now - start
-            identifier = resource_identifier(resource)
-
             fabrication_http_method = if resource.api_fabrication_http_method == :get
                                         if include?(Reusable)
                                           "Retrieved for reuse"
@@ -96,7 +94,7 @@ module QA
             Support::FabricationTracker.save_fabrication(:"#{fabrication_method}_fabrication", fabrication_time)
             Tools::TestResourceDataProcessor.collect(
               resource: resource,
-              info: identifier,
+              info: resource.identifier,
               fabrication_method: fabrication_method,
               fabrication_time: fabrication_time
             )
@@ -104,7 +102,7 @@ module QA
             Runtime::Logger.debug do
               msg = ["==#{'=' * parents.size}>"]
               msg << "#{fabrication_http_method} a #{Rainbow(name).black.bg(:white)}"
-              msg << identifier
+              msg << resource.identifier
               msg << "as a dependency of #{parents.last}" if parents.any?
               msg << "via #{fabrication_method}"
               msg << "in #{fabrication_time} seconds"
@@ -117,26 +115,6 @@ module QA
           result
         end
 
-        # Unique resource identifier
-        #
-        # @param [QA::Resource::Base] resource
-        # @return [String]
-        def resource_identifier(resource)
-          if resource.respond_to?(:username) && resource.username
-            "with username '#{resource.username}'"
-          elsif resource.respond_to?(:full_path) && resource.full_path
-            "with full_path '#{resource.full_path}'"
-          elsif resource.respond_to?(:name) && resource.name
-            "with name '#{resource.name}'"
-          elsif resource.respond_to?(:id) && resource.id
-            "with id '#{resource.id}'"
-          elsif resource.respond_to?(:iid) && resource.iid
-            "with iid '#{resource.iid}'"
-          end
-        rescue QA::Resource::Base::NoValueError
-          nil
-        end
-
         # Define custom attribute
         #
         # @param [Symbol] name
@@ -220,6 +198,35 @@ module QA
         JSON.pretty_generate(comparable)
       end
 
+      def diff(other)
+        return if self == other
+
+        diff_values = self.comparable.to_a - other.comparable.to_a
+        diff_values.to_h
+      end
+
+      def identifier
+        if respond_to?(:username) && username
+          "with username '#{username}'"
+        elsif respond_to?(:full_path) && full_path
+          "with full_path '#{full_path}'"
+        elsif respond_to?(:name) && name
+          "with name '#{name}'"
+        elsif respond_to?(:id) && id
+          "with id '#{id}'"
+        elsif respond_to?(:iid) && iid
+          "with iid '#{iid}'"
+        end
+      rescue QA::Resource::Base::NoValueError
+        nil
+      end
+
+      def remove_via_api!
+        super
+
+        Runtime::Logger.debug(["Removed a #{self.class.name}", identifier].compact.join(' '))
+      end
+
       protected
 
       # Custom resource comparison logic using resource attributes from api_resource

qa/qa/resource/group.rb

@@ -3,12 +3,16 @@
 module QA
   module Resource
     class Group < GroupBase
-      attributes :require_two_factor_authentication, :description
+      attributes :require_two_factor_authentication, :description, :path
 
       attribute :full_path do
         determine_full_path
       end
 
+      attribute :name do
+        @name || path
+      end
+
       attribute :sandbox do
         Sandbox.fabricate_via_api! do |sandbox|
           sandbox.api_client = api_client
@@ -60,7 +64,7 @@ module QA
         {
           parent_id: sandbox.id,
           path: path,
-          name: path,
+          name: name || path,
           visibility: 'public',
           require_two_factor_authentication: @require_two_factor_authentication,
           avatar: avatar

qa/qa/resource/group_base.rb

@@ -7,6 +7,8 @@ module QA
     class GroupBase < Base
       include Members
 
+      MAX_NAME_LENGTH = 255
+
       attr_accessor :path, :avatar
 
       attributes :id,

qa/qa/resource/project.rb

@@ -17,6 +17,7 @@ module QA
 
       attributes :id,
                  :name,
+                 :path,
                  :add_name_uuid,
                  :description,
                  :runners_token,

qa/qa/resource/reusable.rb

@@ -4,8 +4,13 @@ module QA
   module Resource
     #
     # This module includes methods that allow resource classes to be reused safely. It should be prepended to a new
-    # reusable version of an existing resource class. See Resource::Project and ReusableResource::Project for an example
+    # reusable version of an existing resource class. See Resource::Project and ReusableResource::Project for an example.
+    # Reusable resource classes must also be registered with a resource collection that will manage cleanup.
     #
+    # @example Register a resource class with a collection
+    #   QA::Resource::ReusableCollection.register_resource_classes do |collection|
+    #     QA::Resource::ReusableProject.register(collection)
+    #   end
     module Reusable
       attr_accessor :reuse,
                     :reuse_as
@@ -16,7 +21,7 @@ module QA
         base.extend(ClassMethods)
       end
 
-      # Gets an existing resource if it exists and the parameters of the new specification of the resource are valid.
+      # Gets an existing resource if it exists and the specified attributes of the resource are valid.
       # Creates a new instance of the resource if it does not exist.
       #
       # @return [String] The URL of the resource.
@@ -27,35 +32,128 @@ module QA
       rescue Errors::ResourceNotFoundError
         super
       ensure
-        self.class.resources[reuse_as] = self
+        self.class.resources[reuse_as] ||= {
+          tests: Set.new,
+          resource: self
+        }
+
+        self.class.resources[reuse_as][:attributes] ||= all_attributes.each_with_object({}) do |attribute_name, attributes|
+          attributes[attribute_name] = instance_variable_get("@#{attribute_name}")
+        end
+        self.class.resources[reuse_as][:tests] << Runtime::Example.location
       end
 
-      # Including classes must confirm that the resource can be reused as defined. For example, a project can't be
-      # fabricated with a unique name.
+      # Overrides remove_via_api! to log a debug message stating that removal will happen after the suite completes.
       #
       # @return [nil]
+      def remove_via_api!
+        QA::Runtime::Logger.debug("#{self.class.name} - deferring removal until after suite")
+      end
+
+      # Object comparison
+      #
+      # @param [QA::Resource::Base] other
+      # @return [Boolean]
+      def ==(other)
+        self.class <= other.class && comparable == other.comparable
+      end
+
+      # Confirms that reuse of the resource did not change it in a way that breaks later reuse.
+      # For example, this should fail if a reusable resource should have a specific name, but the name has been changed.
+      def validate_reuse
+        QA::Runtime::Logger.debug(["Validating a #{self.class.name} that was reused as #{reuse_as}", identifier].compact.join(' '))
+
+        fresh_resource = reference_resource
+        diff = reuse_validation_diff(fresh_resource)
+
+        if diff.present?
+          raise ResourceReuseError, <<~ERROR
+            The reused #{self.class.name} resource does not have the attributes expected.
+            The following change was found: #{diff}"
+            The resource's web_url is #{web_url}.
+            It was used in these tests: #{self.class.resources[reuse_as][:tests].to_a.join(', ')}
+          ERROR
+        end
+
+      ensure
+        fresh_resource.remove_via_api!
+      end
+
+      private
+
+      # Creates a new resource that can be compared to a reused resource, using the post body of the original.
+      # Must be implemented by classes that include this module.
+      def reference_resource
+        return super if defined?(super)
+
+        raise NotImplementedError
+      end
+
+      # Confirms that the resource attributes specified in its fabricate_via_api! block will allow it to be reused.
+      #
+      # @return [nil] returns nil unless an error is raised
       def validate_reuse_preconditions
+        return unless self.class.resources.key?(reuse_as)
+
+        attributes = unique_identifiers.each_with_object({ proposed: {}, existing: {} }) do |id, attrs|
+          proposed = public_send(id)
+          existing = self.class.resources[reuse_as][:resource].public_send(id)
+
+          next if proposed == existing
+
+          attrs[:proposed][id] = proposed
+          attrs[:existing][id] = existing
+        end
+
+        unless attributes[:proposed].empty? && attributes[:existing].empty?
+          raise ResourceReuseError, "Reusable resources must use the same unique identifier(s). " \
+            "The #{self.class.name} to be reused as :#{reuse_as} has the identifier(s) #{attributes[:proposed]} " \
+            "but it should have #{attributes[:existing]}"
+        end
+      end
+
+      # Compares the attributes of the current reused resource with a reference instance.
+      #
+      # @return [Hash] any differences between the resources.
+      def reuse_validation_diff(other)
+        original, reference = prepare_reuse_validation_diff(other)
+
+        return if original == reference
+
+        diff_values = original.to_a - reference.to_a
+        diff_values.to_h
+      end
+
+      # Compares the current reusable resource to a reference instance, ignoring identifying unique attributes that
+      # had to be changed.
+      #
+      # @return [Hash, Hash] the current and reference resource attributes, respectively.
+      def prepare_reuse_validation_diff(other)
+        original = self.reload!.comparable
+        reference = other.reload!.comparable
+        unique_identifiers.each { |id| reference[id] = original[id] }
+        [original, reference]
+      end
+
+      # The attributes of the resource that should be the same whenever a test wants to reuse a resource. Must be
+      # implemented by classes that include this module.
+      #
+      # @return [Array<Symbol>] the attribute names.
+      def unique_identifiers
         return super if defined?(super)
 
         raise NotImplementedError
       end
 
       module ClassMethods
-        # Removes all created resources of this type.
-        #
-        # @return [Hash<Symbol, QA::Resource>] the resources that were to be removed.
-        def remove_all_via_api!
-          resources.each do |reuse_as, resource|
-            QA::Runtime::Logger.debug("#{self.name} - removing resource reused as :#{reuse_as}")
-            next QA::Runtime::Logger.debug("#{self.name} reused as :#{reuse_as} has already been removed.") unless resource.exists?
-
-            resource.method(:remove_via_api!).super_method.call
-          end
+        # Includes the resources created/reused by this class in the specified collection
+        def register(collection)
+          collection[self.name] = resources
         end
 
-        # The resources created by this resource class.
+        # The resources created/reused by this resource class.
         #
-        # @return [Hash<Symbol, QA::Resource>] the resources created by this resource class.
+        # @return [Hash<Symbol, Hash>] the resources created/reused by this resource class.
         def resources
           @resources ||= {}
         end

qa/qa/resource/reusable_collection.rb

+# frozen_string_literal: true
+
+require 'singleton'
+
+module QA
+  module Resource
+    #
+    # This singleton class collects all reusable resources used by tests and allows operations to be performed on them
+    # all. For example, verifying their state after tests have run and might have changed them.
+    #
+    class ReusableCollection
+      include Singleton
+
+      attr_accessor :resource_classes
+
+      def initialize
+        @resource_classes = {}
+      end
+
+      # Yields each resource in the collection.
+      #
+      # @yieldparam [Symbol] reuse_as the name that identifies the resource instance.
+      # @yieldparam [QA::Resource] reuse_instance the resource.
+      def each_resource
+        resource_classes.each_value do |reuse_instances|
+          reuse_instances.each do |reuse_as, reuse_instance|
+            yield reuse_as, reuse_instance[:resource]
+          end
+        end
+      end
+
+      class << self
+        # Removes all created resources that are included in the collection.
+        def remove_all_via_api!
+          instance.each_resource do |reuse_as, resource|
+            next QA::Runtime::Logger.debug("#{resource.class.name} reused as :#{reuse_as} has already been removed.") unless resource.exists?
+
+            resource.method(:remove_via_api!).super_method.call
+          end
+        end
+
+        # Validates the reuse of each resource as defined by the resource class of each resource in the collection.
+        def validate_resource_reuse
+          instance.each_resource { |_, resource| resource.validate_reuse }
+        end
+
+        # Yields the collection of resources to allow resource classes to register themselves with the collection.
+        #
+        # @yieldparam [Hash] resource_classes the resource classes in the collection.
+        def register_resource_classes
+          yield instance.resource_classes
+        end
+      end
+    end
+  end
+end

qa/qa/resource/reusable_group.rb

@@ -8,46 +8,35 @@ module QA
       def initialize
         super
 
-        @path = "reusable_group"
+        @name = @path = 'reusable_group'
         @description = "QA reusable group"
         @reuse_as = :default_group
       end
 
-      # Confirms that the group can be reused
-      #
-      # @return [nil] returns nil unless an error is raised
-      def validate_reuse_preconditions
-        unless reused_path_unique?
-          raise ResourceReuseError,
-            "Reusable groups must have the same name. The group reused as #{reuse_as} has the path '#{path}' but it should be '#{self.class.resources[reuse_as].path}'"
-        end
-      end
+      private
 
-      # Confirms that reuse of the resource did not change it in a way that breaks later reuse. This raises an error if
-      # the current group path doesn't match the original path.
-      def validate_reuse
-        reload!
-
-        if api_resource[:path] != @path
-          raise ResourceReuseError, "The group now has the path '#{api_resource[:path]}' but it should be '#{path}'"
-        end
-      end
-
-      # Checks if the group is being reused with the same path.
+      # Creates a new group that can be compared to a reused group, using the attributes of the original. Attributes that
+      # must be unique (path and name) are replaced with new unique values.
       #
-      # @return [Boolean] true if the group's path is different from another group with the same reuse symbol (reuse_as)
-      def reused_path_unique?
-        return true unless self.class.resources.key?(reuse_as)
-
-        self.class.resources[reuse_as].path == path
+      # @return [QA::Resource] a new instance of Resource::ReusableGroup that should be a copy of the original resource
+      def reference_resource
+        attributes = self.class.resources[reuse_as][:attributes]
+        name = "ref#{SecureRandom.hex(8)}_#{attributes.delete(:path)}"[0...MAX_NAME_LENGTH]
+
+        Group.fabricate_via_api! do |resource|
+          self.class.resources[reuse_as][:attributes].each do |attribute_name, attribute_value|
+            resource.instance_variable_set("@#{attribute_name}", attribute_value) if attribute_value
+          end
+          resource.path = name
+          resource.name = name
+        end
       end
 
-      # Overrides QA::Resource::Group#remove_via_api! to log a debug message stating that removal will happen after
-      # the suite completes rather than now.
+      # The attributes of the resource that should be the same whenever a test wants to reuse a group.
       #
-      # @return [nil]
-      def remove_via_api!
-        QA::Runtime::Logger.debug("#{self.class.name} - deferring removal until after suite")
+      # @return [Array<Symbol>] the attribute names.
+      def unique_identifiers
+        [:name, :path]
       end
     end
   end

qa/qa/resource/reusable_project.rb

@@ -15,36 +15,36 @@ module QA
         super
 
         @add_name_uuid = false
-        @name = "reusable_project"
+        @name = @path = 'reusable_project'
         @reuse_as = :default_project
         @initialize_with_readme = true
       end
 
-      # Confirms that the project can be reused
-      #
-      # @return [nil] returns nil unless an error is raised
-      def validate_reuse_preconditions
-        unless reused_name_unique?
-          raise ResourceReuseError,
-            "Reusable projects must have the same name. The project reused as #{reuse_as} has the name '#{name}' but it should be '#{self.class.resources[reuse_as].name}'"
-        end
-      end
+      private
 
-      # Checks if the project is being reused with the same name.
+      # Creates a new project that can be compared to a reused project, using the attributes of the original. Attributes
+      # that must be unique (path and name) are replaced with new unique values.
       #
-      # @return [Boolean] true if the project's name is different from another project with the same reuse symbol (reuse_as)
-      def reused_name_unique?
-        return true unless self.class.resources.key?(reuse_as)
-
-        self.class.resources[reuse_as].name == name
+      # @return [QA::Resource] a new instance of Resource::ReusableProject that should be a copy of the original resource
+      def reference_resource
+        attributes = self.class.resources[reuse_as][:attributes]
+        name = "reference_resource_#{SecureRandom.hex(8)}_for_#{attributes.delete(:name)}"
+
+        Project.fabricate_via_api! do |project|
+          self.class.resources[reuse_as][:attributes].each do |attribute_name, attribute_value|
+            project.instance_variable_set("@#{attribute_name}", attribute_value) if attribute_value
+          end
+          project.name = name
+          project.path = name
+          project.path_with_namespace = "#{project.group.full_path}/#{project.name}"
+        end
       end
 
-      # Overrides QA::Resource::Project#remove_via_api! to log a debug message stating that removal will happen after
-      # the suite completes rather than now.
+      # The attributes of the resource that should be the same whenever a test wants to reuse a project.
       #
-      # @return [nil]
-      def remove_via_api!
-        QA::Runtime::Logger.debug("#{self.class.name} - deferring removal until after suite")
+      # @return [Array<Symbol>] the attribute names.
+      def unique_identifiers
+        [:name, :path]
       end
     end
   end

qa/qa/runtime/env.rb

@@ -454,6 +454,10 @@ module QA
         enabled?(ENV['DISABLE_QUARANTINE'], default: false)
       end
 
+      def validate_resource_reuse?
+        enabled?(ENV['QA_VALIDATE_RESOURCE_REUSE'], default: false)
+      end
+
       private
 
       def remote_grid_credentials

qa/qa/runtime/example.rb

+# frozen_string_literal: true
+
+module QA
+  module Runtime
+    module Example
+      extend self
+
+      attr_accessor :current
+
+      def location
+        current.respond_to?(:location) ? current.location : 'unknown'
+      end
+    end
+  end
+end

qa/spec/resource/base_spec.rb

@@ -3,7 +3,7 @@
 RSpec.describe QA::Resource::Base do
   include QA::Support::Helpers::StubEnv
 
-  let(:resource) { spy('resource', username: 'qa') }
+  let(:resource) { spy('resource') }
   let(:location) { 'http://location' }
   let(:log_regex) { %r{==> Built a MyResource with username 'qa' via #{method} in [\d.\-e]+ seconds+} }
 
@@ -12,6 +12,36 @@ RSpec.describe QA::Resource::Base do
     allow(QA::Tools::TestResourceDataProcessor).to receive(:write_to_file)
   end
 
+  shared_context 'with simple resource' do
+    subject do
+      Class.new(QA::Resource::Base) do
+        def self.name
+          'MyResource'
+        end
+
+        attribute :test do
+          'block'
+        end
+
+        attribute :username do
+          'qa'
+        end
+
+        attribute :no_block
+
+        def fabricate!(*args)
+          'any'
+        end
+
+        def self.current_url
+          'http://stub'
+        end
+      end
+    end
+
+    let(:resource) { subject.new }
+  end
+
   shared_context 'with fabrication context' do
     subject do
       Class.new(described_class) do
@@ -61,23 +91,29 @@ RSpec.describe QA::Resource::Base do
   end
 
   describe '.fabricate_via_api!' do
-    include_context 'with fabrication context'
+    context 'when fabricating' do
+      include_context 'with fabrication context'
 
-    it_behaves_like 'fabrication method', :fabricate_via_api!
+      it_behaves_like 'fabrication method', :fabricate_via_api!
 
-    it 'instantiates the resource, calls resource method returns the resource' do
-      expect(resource).to receive(:fabricate_via_api!).and_return(location)
+      it 'instantiates the resource, calls resource method returns the resource' do
+        expect(resource).to receive(:fabricate_via_api!).and_return(location)
 
-      result = subject.fabricate_via_api!(resource: resource, parents: [])
+        result = subject.fabricate_via_api!(resource: resource, parents: [])
 
-      expect(result).to eq(resource)
+        expect(result).to eq(resource)
+      end
     end
 
     context "with debug log level" do
+      include_context 'with simple resource'
+
       let(:method) { 'api' }
 
       before do
         allow(QA::Runtime::Logger).to receive(:debug)
+        allow(resource).to receive(:api_support?).and_return(true)
+        allow(resource).to receive(:fabricate_via_api!)
       end
 
       it 'logs the resource and build method' do
@@ -93,27 +129,32 @@ RSpec.describe QA::Resource::Base do
   end
 
   describe '.fabricate_via_browser_ui!' do
-    include_context 'with fabrication context'
+    context 'when fabricating' do
+      include_context 'with fabrication context'
 
-    it_behaves_like 'fabrication method', :fabricate_via_browser_ui!, :fabricate!
+      it_behaves_like 'fabrication method', :fabricate_via_browser_ui!, :fabricate!
 
-    it 'instantiates the resource and calls resource method' do
-      subject.fabricate_via_browser_ui!('something', resource: resource, parents: [])
+      it 'instantiates the resource and calls resource method' do
+        subject.fabricate_via_browser_ui!('something', resource: resource, parents: [])
 
-      expect(resource).to have_received(:fabricate!).with('something')
-    end
+        expect(resource).to have_received(:fabricate!).with('something')
+      end
 
-    it 'returns fabrication resource' do
-      result = subject.fabricate_via_browser_ui!('something', resource: resource, parents: [])
+      it 'returns fabrication resource' do
+        result = subject.fabricate_via_browser_ui!('something', resource: resource, parents: [])
 
-      expect(result).to eq(resource)
+        expect(result).to eq(resource)
+      end
     end
 
     context "with debug log level" do
+      include_context 'with simple resource'
+
       let(:method) { 'browser_ui' }
 
       before do
         allow(QA::Runtime::Logger).to receive(:debug)
+        # allow(resource).to receive(:fabricate!)
       end
 
       it 'logs the resource and build method' do
@@ -128,28 +169,6 @@ RSpec.describe QA::Resource::Base do
     end
   end
 
-  shared_context 'with simple resource' do
-    subject do
-      Class.new(QA::Resource::Base) do
-        attribute :test do
-          'block'
-        end
-
-        attribute :no_block
-
-        def fabricate!
-          'any'
-        end
-
-        def self.current_url
-          'http://stub'
-        end
-      end
-    end
-
-    let(:resource) { subject.new }
-  end
-
   describe '.attribute' do
     include_context 'with simple resource'
 

qa/spec/resource/reusable_collection_spec.rb

+# frozen_string_literal: true
+
+RSpec.describe QA::Resource::ReusableCollection do
+  let(:reusable_resource_class) do
+    Class.new do
+      prepend QA::Resource::Reusable
+
+      attr_reader :removed
+
+      def self.name
+        'FooReusableResource'
+      end
+
+      def comparable
+        self.class.name
+      end
+
+      def remove_via_api!
+        @removed = true
+      end
+
+      def exists?() end
+    end
+  end
+
+  let(:another_reusable_resource_class) do
+    Class.new(reusable_resource_class) do
+      def self.name
+        'BarReusableResource'
+      end
+    end
+  end
+
+  let(:a_resource_instance) { reusable_resource_class.new }
+  let(:another_resource_instance) { another_reusable_resource_class.new }
+
+  it 'is a singleton class' do
+    expect { described_class.new }.to raise_error(NoMethodError)
+  end
+
+  subject(:collection) do
+    described_class.instance
+  end
+
+  before do
+    described_class.register_resource_classes do |c|
+      reusable_resource_class.register(c)
+      another_reusable_resource_class.register(c)
+    end
+
+    collection.resource_classes = {
+      'FooReusableResource' => {
+        reuse_as_identifier: {
+          resource: a_resource_instance
+        }
+      },
+      'BarReusableResource' => {
+        another_reuse_as_identifier: {
+          resource: another_resource_instance
+        }
+      }
+    }
+
+    allow(a_resource_instance).to receive(:validate_reuse)
+    allow(another_resource_instance).to receive(:validate_reuse)
+  end
+
+  after do
+    collection.resource_classes = {}
+  end
+
+  describe '#each_resource' do
+    it 'yields each resource and reuse_as identifier in the collection' do
+      expect { |blk| collection.each_resource(&blk) }
+        .to yield_successive_args(
+          [:reuse_as_identifier, a_resource_instance],
+          [:another_reuse_as_identifier, another_resource_instance]
+        )
+    end
+  end
+
+  describe '.remove_all_via_api!' do
+    before do
+      allow(a_resource_instance).to receive(:exists?).and_return(true)
+      allow(another_resource_instance).to receive(:exists?).and_return(true)
+    end
+
+    it 'removes each instance of each resource class' do
+      described_class.remove_all_via_api!
+
+      expect(a_resource_instance.removed).to be true
+      expect(another_resource_instance.removed).to be true
+    end
+  end
+
+  describe '.validate_resource_reuse' do
+    it 'validates each instance of each resource class' do
+      expect(a_resource_instance).to receive(:validate_reuse)
+      expect(another_resource_instance).to receive(:validate_reuse)
+
+      described_class.validate_resource_reuse
+    end
+  end
+
+  describe '.register_resource_classes' do
+    it 'yields the hash of resource classes in the collection' do
+      expect { |blk| described_class.register_resource_classes(&blk) }.to yield_with_args(collection.resource_classes)
+    end
+  end
+end

qa/spec/spec_helper.rb

@@ -28,8 +28,16 @@ RSpec.configure do |config|
   config.add_formatter QA::Support::Formatters::QuarantineFormatter
   config.add_formatter QA::Support::Formatters::TestStatsFormatter if QA::Runtime::Env.export_metrics?
 
+  config.before(:suite) do |suite|
+    QA::Resource::ReusableCollection.register_resource_classes do |collection|
+      QA::Resource::ReusableProject.register(collection)
+      QA::Resource::ReusableGroup.register(collection)
+    end
+  end
+
   config.prepend_before do |example|
     QA::Runtime::Logger.debug("\nStarting test: #{example.full_description}\n")
+    QA::Runtime::Example.current = example
 
     # Reset fabrication counters tracked in resource base
     Thread.current[:api_fabrication] = 0
@@ -66,11 +74,15 @@ RSpec.configure do |config|
   end
 
   config.after(:suite) do |suite|
-    # If any tests failed, leave the resources behind to help troubleshoot
-    QA::Resource::ReusableProject.remove_all_via_api! unless suite.reporter.failed_examples.present?
-
     # Write all test created resources to JSON file
     QA::Tools::TestResourceDataProcessor.write_to_file
+
+    # If requested, confirm that resources were used appropriately (e.g., not left with changes that interfere with
+    # further reuse)
+    QA::Resource::ReusableCollection.validate_resource_reuse if QA::Runtime::Env.validate_resource_reuse?
+
+    # If any tests failed, leave the resources behind to help troubleshoot, otherwise remove them.
+    QA::Resource::ReusableCollection.remove_all_via_api! unless suite.reporter.failed_examples.present?
   end
 
   config.append_after(:suite) do