Merge branch 'add-network-policies-usage-data' into 'master'

Track usage data for Network Policies UI

See merge request gitlab-org/gitlab!64731

doc/development/usage_ping/dictionary.md

@@ -11712,7 +11712,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_alert_assigned_monthly`
 
-Count of an alert's assignments per month
+Count of unique users assigning an alert per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180533_incident_management_alert_assigned_monthly.yml)
 
@@ -11724,7 +11724,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_alert_assigned_weekly`
 
-Count of an alert's assignments per week
+Count of unique users assigning an alert per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180532_incident_management_alert_assigned_weekly.yml)
 
@@ -11736,7 +11736,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_alert_status_changed_monthly`
 
-Count of an alert's status changes per month
+Count of unique users changing alert's status changes per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180530_incident_management_alert_status_changed_monthly.yml)
 
@@ -11748,7 +11748,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_alert_status_changed_weekly`
 
-Count of an alert's status changes per week
+Count of unique users changing alert's status per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180528_incident_management_alert_status_changed_weekly.yml)
 
@@ -11760,7 +11760,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_alert_todo_monthly`
 
-Count of alerts added to the TODO list per month
+Count of unique users adding alerts to the TODO list per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180537_incident_management_alert_todo_monthly.yml)
 
@@ -11772,7 +11772,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_alert_todo_weekly`
 
-Count of alerts added to the TODO list per week
+Count of unique users adding alerts to the TODO list per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180535_incident_management_alert_todo_weekly.yml)
 
@@ -11784,7 +11784,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_assigned_monthly`
 
-Count of an incidents's assignments per month
+Count of users assigning incidents per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180552_incident_management_incident_assigned_monthly.yml)
 
@@ -11796,7 +11796,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_assigned_weekly`
 
-Count of an incidents's assignments per week
+Count of unique users assiging incidents per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180550_incident_management_incident_assigned_weekly.yml)
 
@@ -11808,7 +11808,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_change_confidential_monthly`
 
-Count of incidents changed to confidential per month
+Count of users changing incidents to confidential per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180618_incident_management_incident_change_confidential_monthly.yml)
 
@@ -11820,7 +11820,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_change_confidential_weekly`
 
-Count of incidents changed to confidential per week
+Count of unique users changing incidents to confidential per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180616_incident_management_incident_change_confidential_weekly.yml)
 
@@ -11832,7 +11832,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_closed_monthly`
 
-Count of incidents closed per month
+Count of users closing incidents per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180548_incident_management_incident_closed_monthly.yml)
 
@@ -11844,7 +11844,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_closed_weekly`
 
-Count of incidents closed per week
+Count of users closing incidents per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180546_incident_management_incident_closed_weekly.yml)
 
@@ -11856,7 +11856,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_comment_monthly`
 
-Count of comments per month on incidents
+Count of unique users adding comments per month on incidents
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180559_incident_management_incident_comment_monthly.yml)
 
@@ -11868,7 +11868,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_comment_weekly`
 
-Count of comments per week on incidents
+Count of unique users adding comments on incidents per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180558_incident_management_incident_comment_weekly.yml)
 
@@ -11880,7 +11880,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_created_monthly`
 
-Count of incidents created per month
+Count of unique users creating incidents per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180541_incident_management_incident_created_monthly.yml)
 
@@ -11892,7 +11892,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_created_weekly`
 
-Count of incidents created per week
+Count of unique users creating incidents  per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180539_incident_management_incident_created_weekly.yml)
 
@@ -11904,7 +11904,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_published_monthly`
 
-Count of published incidents per month
+Count of unique users that published incidents per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216180607_incident_management_incident_published_monthly.yml)
 
@@ -11916,7 +11916,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_published_weekly`
 
-Count of published incidents per week
+Count of unique users that published incidents per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216180605_incident_management_incident_published_weekly.yml)
 
@@ -11928,7 +11928,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_relate_monthly`
 
-Count of issues added per month that are related to an incident
+Count of unique users adding issues per month that are related to an incident
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180611_incident_management_incident_relate_monthly.yml)
 
@@ -11940,7 +11940,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_relate_weekly`
 
-Count of issues added per week that are related to an incident
+Count of unique users adding issues per that are related to an incident week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180609_incident_management_incident_relate_weekly.yml)
 
@@ -11952,7 +11952,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_reopened_monthly`
 
-Count of incidents reopened per month
+Count of unique users reopening incidents per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180545_incident_management_incident_reopened_monthly.yml)
 
@@ -11964,7 +11964,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_reopened_weekly`
 
-Count of incidents reopened per week
+Count of unique users reopening incidents per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180543_incident_management_incident_reopened_weekly.yml)
 
@@ -11976,7 +11976,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_todo_monthly`
 
-Count of incidents added to the TODO list per month
+Count of unique users adding incidents to the TODO list per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180556_incident_management_incident_todo_monthly.yml)
 
@@ -11988,7 +11988,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_todo_weekly`
 
-Count of incidents added to the TODO list per week
+Count of unique users adding incidents to the TODO list per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180554_incident_management_incident_todo_weekly.yml)
 
@@ -12000,7 +12000,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_unrelate_monthly`
 
-Count of issues removed per month that are related to an incident
+Count of users removing issues that are related to an incident per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180614_incident_management_incident_unrelate_monthly.yml)
 
@@ -12012,7 +12012,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_unrelate_weekly`
 
-Count of issues removed per week that are related to an incident
+Count of unique users removing issue that are related to an incident per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180612_incident_management_incident_unrelate_weekly.yml)
 
@@ -12024,7 +12024,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_zoom_meeting_monthly`
 
-Count of Zoom meetings about incidents per month
+Count of users creating Zoom meetings about incidents per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180603_incident_management_incident_zoom_meeting_monthly.yml)
 
@@ -12036,7 +12036,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_incident_zoom_meeting_weekly`
 
-Count of Zoom meetings about incidents per week
+Count of unique users creating Zoom meetings about incidents per week
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180601_incident_management_incident_zoom_meeting_weekly.yml)
 
@@ -12048,7 +12048,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_total_unique_counts_monthly`
 
-Count of total unique incidents per month
+Count of unique users performing events related with incidents per month
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180622_incident_management_total_unique_counts_monthly.yml)
 
@@ -12060,7 +12060,7 @@ Tiers: `free`, `premium`, `ultimate`
 
 ### `redis_hll_counters.incident_management.incident_management_total_unique_counts_weekly`
 
-Count of total unique incidents per week
+Count of unique users performing events related to the incident management
 
 [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180620_incident_management_total_unique_counts_weekly.yml)
 
@@ -12934,6 +12934,30 @@ Status: `data_available`
 
 Tiers: `free`, `premium`, `ultimate`
 
+### `redis_hll_counters.network_policies.clusters_using_network_policies_ui_monthly`
+
+Monthly number of distinct clusters with network policies using the network policies UI
+
+[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210623202402_clusters_using_network_policies_ui_monthly.yml)
+
+Group: `group::container security`
+
+Status: `implemented`
+
+Tiers: `ultimate`
+
+### `redis_hll_counters.network_policies.clusters_using_network_policies_ui_weekly`
+
+Weekly number of distinct clusters with network policies using the network policies UI
+
+[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210623202358_clusters_using_network_policies_ui_weekly.yml)
+
+Group: `group::container security`
+
+Status: `implemented`
+
+Tiers: `ultimate`
+
 ### `redis_hll_counters.pipeline_authoring.o_pipeline_authoring_unique_users_committing_ciconfigfile_monthly`
 
 Monthly unique user count doing commits which contains the CI config file

ee/app/services/network_policies/resources_service.rb

@@ -3,6 +3,7 @@
 module NetworkPolicies
   class ResourcesService
     include NetworkPolicies::Responses
+    include Gitlab::Utils::UsageData
 
     LIMIT = 100
 
@@ -26,6 +27,12 @@ module NetworkPolicies
 
     private
 
+    def track_usage_data_for_cluster(platform, policies)
+      return if policies.empty?
+
+      track_usage_event(:clusters_using_network_policies_ui, platform.cluster_id)
+    end
+
     def execute_per_environment(platform, namespace)
       policies = platform.kubeclient
         .get_network_policies(namespace: namespace)
@@ -33,6 +40,7 @@ module NetworkPolicies
       policies += platform.kubeclient
         .get_cilium_network_policies(namespace: namespace)
         .map { |resource| Gitlab::Kubernetes::CiliumNetworkPolicy.from_resource(resource) }
+      track_usage_data_for_cluster(platform, policies)
       [policies, nil]
     rescue Kubeclient::HttpError => e
       [policies, e]

ee/config/metrics/counts_28d/20210623202402_clusters_using_network_policies_ui_monthly.yml

+---
+key_path: redis_hll_counters.network_policies.clusters_using_network_policies_ui_monthly
+description: Monthly number of distinct clusters with network policies using the network policies UI
+product_section: sec
+product_stage: protect
+product_group: group::container security
+product_category: container_network_security
+value_type: number
+status: implemented
+milestone: "14.1"
+introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64731
+time_frame: 28d
+data_source: redis_hll
+data_category: Operational
+distribution:
+- ee
+tier:
+- ultimate

ee/config/metrics/counts_7d/20210623202358_clusters_using_network_policies_ui_weekly.yml

+---
+key_path: redis_hll_counters.network_policies.clusters_using_network_policies_ui_weekly
+description: Weekly number of distinct clusters with network policies using the network policies UI
+product_section: sec
+product_stage: protect
+product_group: group::container security
+product_category: container_network_security
+value_type: number
+status: implemented
+milestone: "14.1"
+introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64731
+time_frame: 7d
+data_source: redis_hll
+data_category: Operational
+distribution:
+- ee
+tier:
+- ultimate

ee/spec/services/network_policies/resources_service_spec.rb

@@ -9,7 +9,7 @@ RSpec.describe NetworkPolicies::ResourcesService do
   let(:project) { create(:project) }
   let(:cluster) { create(:cluster, :instance) }
   let!(:cluster_kubernetes_namespace) { create(:cluster_kubernetes_namespace, project: project, cluster: cluster, environment: environment, namespace: 'namespace') }
-  let(:platform) { double('Clusters::Platforms::Kubernetes', kubeclient: kubeclient) }
+  let(:platform) { double('Clusters::Platforms::Kubernetes', kubeclient: kubeclient, cluster_id: cluster.id) }
   let(:kubeclient) { double('Kubeclient::Client') }
   let(:policy) do
     Gitlab::Kubernetes::NetworkPolicy.new(
@@ -46,6 +46,23 @@ RSpec.describe NetworkPolicies::ResourcesService do
       expect(subject.payload.last.as_json).to eq(cilium_policy.as_json)
     end
 
+    it_behaves_like 'tracking unique hll events' do
+      subject(:request) { service.execute }
+
+      let(:target_id) { 'clusters_using_network_policies_ui' }
+      let(:expected_type) { instance_of(Integer) }
+
+      before do
+        allow(kubeclient).to receive(:get_network_policies)
+          .with(namespace: cluster_kubernetes_namespace.namespace)
+          .and_return [policy.generate]
+
+        allow(kubeclient).to receive(:get_cilium_network_policies)
+          .with(namespace: cluster_kubernetes_namespace.namespace)
+          .and_return [cilium_policy.generate]
+      end
+    end
+
     context 'without deployment_platform' do
       let(:platform) { nil }
 

lib/gitlab/usage_data_counters/known_events/common.yml

@@ -369,3 +369,8 @@
   category: testing
   aggregation: weekly
   feature_flag: users_expanding_widgets_usage_data
+# Container Security - Network Policies
+- name: clusters_using_network_policies_ui
+  redis_slot: network_policies
+  category: network_policies
+  aggregation: weekly

spec/lib/gitlab/usage_data_counters/hll_redis_counter_spec.rb

@@ -46,7 +46,8 @@ RSpec.describe Gitlab::UsageDataCounters::HLLRedisCounter, :clean_gitlab_redis_s
         'pipeline_authoring',
         'epics_usage',
         'epic_boards_usage',
-        'secure'
+        'secure',
+        'network_policies'
       )
     end
   end

spec/lib/gitlab/usage_data_spec.rb

@@ -1269,7 +1269,7 @@ RSpec.describe Gitlab::UsageData, :aggregate_failures do
 
     let(:categories) { ::Gitlab::UsageDataCounters::HLLRedisCounter.categories }
     let(:ineligible_total_categories) do
-      %w[source_code ci_secrets_management incident_management_alerts snippets terraform incident_management_oncall secure]
+      %w[source_code ci_secrets_management incident_management_alerts snippets terraform incident_management_oncall secure network_policies]
     end
 
     context 'with redis_hll_tracking feature enabled' do