Merge branch 'mwaw/add_database_sourced_aggregated_metric' into 'master'

Add database sourced aggregated metric [RUN ALL RSPEC] [RUN AS-IF-FOSS]

See merge request gitlab-org/gitlab!53854

doc/development/usage_ping/index.md

@@ -928,7 +928,9 @@ appear to be associated to any of the services running, because they all appear
 WARNING:
 This feature is intended solely for internal GitLab use.
 
-To add data for aggregated metrics into Usage Ping payload you should add corresponding definition in [`aggregated_metrics`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/aggregated_metrics/). Each aggregate definition includes following parts:
+To add data for aggregated metrics into Usage Ping payload you should add corresponding definition at [`lib/gitlab/usage_data_counters/aggregated_metrics/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/aggregated_metrics/) for metrics available at Community Edition and at [`ee/lib/gitlab/usage_data_counters/aggregated_metrics/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/usage_data_counters/aggregated_metrics/) for Enterprise Edition ones.
+ 
+Each aggregate definition includes following parts:
 
 - `name`: Unique name under which the aggregate metric is added to the Usage Ping payload.
 - `operator`: Operator that defines how the aggregated metric data is counted. Available operators are:

ee/lib/ee/gitlab/usage/metrics/aggregates/aggregate.rb

+# frozen_string_literal: true
+
+module EE
+  module Gitlab
+    module Usage
+      module Metrics
+        module Aggregates
+          module Aggregate
+            extend ActiveSupport::Concern
+            extend ::Gitlab::Utils::Override
+
+            EE_AGGREGATED_METRICS_PATH = Rails.root.join('ee/lib/gitlab/usage_data_counters/aggregated_metrics/*.yml')
+
+            override :initialize
+            def initialize(recorded_at)
+              super
+              @aggregated_metrics += load_metrics(EE_AGGREGATED_METRICS_PATH)
+            end
+          end
+        end
+      end
+    end
+  end
+end

ee/lib/gitlab/usage_data_counters/aggregated_metrics/common.yml

+#- name: unique name of aggregated metric
+#  operator: aggregation operator. Valid values are:
+#             - "OR": counts unique elements that were observed triggering any of following events
+#             - "AND": counts unique elements that were observed triggering all of following events
+#  events: list of events names to aggregate into metric. All events in this list must have the same 'redis_slot' and 'aggregation' attributes
+#           see from lib/gitlab/usage_data_counters/known_events/ for the list of valid events.
+#  source: defines which datasource will be used to locate events that should be included in aggregated metric. Valid values are:
+#          - database
+#          - redis
+#  time_frame: defines time frames for aggregated metrics:
+#          - 7d - last 7 days
+#          - 28d - last 28 days
+#          - all - all historical available data, this time frame is not available for redis source
+#  feature_flag: name of development feature flag that will be checked before metrics aggregation is performed.
+#                Corresponding feature flag should have `default_enabled` attribute set to `false`.
+#                This attribute is OPTIONAL and can be omitted, when `feature_flag` is missing no feature flag will be checked.
+- name: product_analytics_test_metrics_intersection_database_sourced
+  source: database
+  time_frame: [28d]
+  events: ['dependency_scanning_pipeline', 'container_scanning_pipeline']
+  operator: AND

ee/spec/lib/gitlab/usage/metrics/aggregates/aggregate_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+RSpec.describe Gitlab::Usage::Metrics::Aggregates::Aggregate, :clean_gitlab_redis_shared_state do
+  describe '.new' do
+    it 'loads aggregated metrics from both sources' do
+      expect(Dir).to receive(:[]).with(Gitlab::Usage::Metrics::Aggregates::AGGREGATED_METRICS_PATH).and_return([])
+      expect(Dir).to receive(:[]).with(described_class::EE_AGGREGATED_METRICS_PATH).and_return([])
+
+      described_class.new(Time.current.to_i)
+    end
+  end
+end

lib/gitlab/usage/metrics/aggregates/aggregate.rb

@@ -169,3 +169,5 @@ module Gitlab
     end
   end
 end
+
+Gitlab::Usage::Metrics::Aggregates::Aggregate.prepend_if_ee('EE::Gitlab::Usage::Metrics::Aggregates::Aggregate')

lib/gitlab/usage_data_counters/aggregated_metrics/common.yml

+# Aggregated metrics that include EE only event names within `events:` attribute have to be defined at ee/lib/gitlab/usage_data_counters/aggregated_metrics/common.yml
+# instead of this file.
 #- name: unique name of aggregated metric
 #  operator: aggregation operator. Valid values are:
 #             - "OR": counts unique elements that were observed triggering any of following events