Merge branch 'docs-prometheus-annotations' into 'master'

Docs: Add sample of Prometheus annotations and labels

See merge request gitlab-org/gitlab!42613

doc/user/project/integrations/prometheus.md

@@ -58,6 +58,43 @@ CPU and Memory consumption is monitored, but requires [naming conventions](prome
 
 The [NGINX Ingress](../clusters/index.md#installing-applications) that is deployed by GitLab to clusters, is automatically annotated for monitoring providing key response metrics: latency, throughput, and error rates.
 
+##### Example of Kubernetes service annotations and labels
+
+As an example, to activate Prometheus monitoring of a service:
+
+1. Add at least this annotation: `prometheus.io/scrape: 'true'`.
+1. Add two labels so GitLab can retrieve metrics dynamically for any environment:
+   - `application: ${CI_ENVIRONMENT_SLUG}`
+   - `release: ${CI_ENVIRONMENT_SLUG}`
+1. Create a dynamic PromQL query. For example, a query like
+   `temperature{application="{{ci_environment_slug}}",release="{{ci_environment_slug}}"}` to either:
+   - Add [custom metrics](../../../operations/metrics/index.md#adding-custom-metrics).
+   - Add [custom dashboards](../../../operations/metrics/dashboards/index.md).
+
+The following is a service definition to accomplish this:
+
+```yaml
+---
+# Service
+apiVersion: v1
+kind: Service
+metadata:
+  name: service-${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}
+  # === Prometheus annotations ===
+  annotations:
+    prometheus.io/scrape: 'true'
+  labels:
+    application: ${CI_ENVIRONMENT_SLUG}
+    release: ${CI_ENVIRONMENT_SLUG}
+  # === End of Prometheus ===
+spec:
+  selector:
+    app: ${CI_PROJECT_NAME}
+  ports:
+    - port: ${EXPOSED_PORT}
+      targetPort: ${CONTAINER_PORT}
+```
+
 ### Manual configuration of Prometheus
 
 #### Requirements