Add metrics to monitor pipelines with policy jobs

doc/administration/monitoring/prometheus/gitlab_metrics.md

@@ -129,6 +129,7 @@ The following metrics are available:
 | `pipeline_graph_link_calculation_duration_seconds`               | Histogram   | 13.9    | Total time spent calculating links, in seconds                                                                        |                                                           |
 | `pipeline_graph_links_total`                                     | Histogram   | 13.9    | Number of links per graph                                                                                             |                                                           |
 | `pipeline_graph_links_per_job_ratio`                             | Histogram   | 13.9    | Ratio of links to job per graph                                                                                       |                                                           |
+| `gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds` | Histogram   | 13.12    | Time in seconds it takes to process Security Policies in CI/CD pipeline                                                                    |                                                           |
 
 ## Metrics controlled by a feature flag
 
@@ -254,7 +255,7 @@ The following metrics are available:
 |:--------------------------------- |:--------- |:------------------------------------------------------------- |:-------------------------------------- |:--------------------------------------------------------- |
 | `db_load_balancing_hosts`         | Gauge     | [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/13630)   | Current number of load balancing hosts | |
 | `sidekiq_load_balancing_count`    | Counter   | 13.11                                                         | Sidekiq jobs using load balancing with data consistency set to :sticky or :delayed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency`, `data_consistency`, `database_chosen` |
- 
+
 ## Database partitioning metrics **(PREMIUM SELF)**
 
 The following metrics are available:

ee/lib/gitlab/ci/config/security_orchestration_policies/processor.rb

@@ -9,20 +9,17 @@ module Gitlab
             @config = config
             @project = project
             @ref = ref
+            @start = Time.current
           end
 
           def perform
             return @config unless project&.feature_available?(:security_orchestration_policies)
             return @config unless security_orchestration_policy_configuration&.enabled?
 
-            @config
-              .deep_merge(on_demand_scans_template)
-          end
+            merged_config = @config.deep_merge(on_demand_scans_template)
+            observe_processing_duration(Time.current - @start)
 
-          def on_demand_scans_template
-            ::Security::SecurityOrchestrationPolicies::OnDemandScanPipelineConfigurationService
-              .new(project)
-              .execute(security_orchestration_policy_configuration.on_demand_scan_actions(@ref))
+            merged_config
           end
 
           private
@@ -30,6 +27,18 @@ module Gitlab
           attr_reader :project
 
           delegate :security_orchestration_policy_configuration, to: :project, allow_nil: true
+
+          def on_demand_scans_template
+            ::Security::SecurityOrchestrationPolicies::OnDemandScanPipelineConfigurationService
+              .new(project)
+              .execute(security_orchestration_policy_configuration.on_demand_scan_actions(@ref))
+          end
+
+          def observe_processing_duration(duration)
+            ::Gitlab::Ci::Pipeline::Metrics
+              .pipeline_security_orchestration_policy_processing_duration_histogram
+              .observe({}, duration.seconds)
+          end
         end
       end
     end

lib/gitlab/ci/pipeline/metrics.rb

@@ -13,6 +13,13 @@ module Gitlab
           ::Gitlab::Metrics.histogram(name, comment, labels, buckets)
         end
 
+        def self.pipeline_security_orchestration_policy_processing_duration_histogram
+          name = :gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds
+          comment = 'Pipeline security orchestration policy processing duration'
+
+          ::Gitlab::Metrics.histogram(name, comment)
+        end
+
         def self.pipeline_size_histogram
           name = :gitlab_ci_pipeline_size_builds
           comment = 'Pipeline size'