Commit 39e7e42d authored by Amy Qualls's avatar Amy Qualls

Remove information from the Prometheus page

Move more info from the Prometheus page to where it belongs in the
Operations section.
parent 08bc371d
...@@ -73,11 +73,11 @@ you should ...@@ -73,11 +73,11 @@ you should
## Taking action on Prometheus alerts **(ULTIMATE)** ## Taking action on Prometheus alerts **(ULTIMATE)**
You can [add a webhook](../../../operations/metrics/index.md#external-prometheus-instances) You can [add a webhook](../../../operations/metrics/alerts.md#external-prometheus-instances)
to the Prometheus configuration in order for GitLab to receive notifications of any alerts. to the Prometheus configuration in order for GitLab to receive notifications of any alerts.
Once the webhook is setup, you can Once the webhook is setup, you can
[take action on incoming alerts](../../../operations/metrics/index.md#taking-action-on-incidents-ultimate). [take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts-ultimate).
## Adding custom metrics to the self monitoring project ## Adding custom metrics to the self monitoring project
......
---
stage: Monitor
group: APM
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
# Set up alerts for Prometheus metrics
After [configuring metrics for your CI/CD environment](index.md), you can set up
alerting for Prometheus metrics depending on the location of your instances, and
[trigger actions from alerts](#trigger-actions-from-alerts-ultimate) to notify
your team when environment performance falls outside of the boundaries you set.
## Managed Prometheus instances
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6590) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2 for [custom metrics](index.md#adding-custom-metrics), and 11.3 for [library metrics](../../user/project/integrations/prometheus_library/metrics.md).
For managed Prometheus instances using auto configuration, you can
[configure alerts for metrics](index.md#adding-custom-metrics) directly in the
[metrics dashboard](index.md). To set an alert:
1. In your project, navigate to **{cloud-gear}** **Operations > Metrics**,
1. Identify the metric you want to create the alert for, and click the
**ellipsis** **{ellipsis_v}** icon in the top right corner of the metric.
1. Choose **Alerts**.
1. Set threshold and operator.
1. Click **Add** to save and activate the alert.
![Adding an alert](../../user/project/integrations/img/prometheus_alert.png)
To remove the alert, click back on the alert icon for the desired metric, and click **Delete**.
## External Prometheus instances
>- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9258) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.8.
>- [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) to [GitLab Core](https://about.gitlab.com/pricing/) in 12.10.
For manually configured Prometheus servers, GitLab provides a notify endpoint for
use with Prometheus webhooks. If you have manual configuration enabled, an
**Alerts** section is added to **{settings}** **Settings > Integrations > Prometheus**.
This section contains the **URL** and **Authorization Key** you will need. The
**Reset Key** button will invalidate the key and generate a new one.
![Prometheus service configuration of Alerts](../../user/project/integrations/img/prometheus_service_alerts.png)
To send GitLab alert notifications, copy the **URL** and **Authorization Key** into the
[`webhook_configs`](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config)
section of your Prometheus Alertmanager configuration:
```yaml
receivers:
name: gitlab
webhook_configs:
- http_config:
bearer_token: 9e1cbfcd546896a9ea8be557caf13a76
send_resolved: true
url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json
...
```
For GitLab to associate your alerts with an [environment](../../ci/environments/index.md),
you must configure a `gitlab_environment_name` label on the alerts you set up in
Prometheus. The value of this should match the name of your environment in GitLab.
NOTE: **Note:**
In GitLab versions 13.1 and greater, you can configure your manually configured
Prometheus server to use the
[Generic alerts integration](../../user/project/integrations/generic_alerts.md).
## Trigger actions from alerts **(ULTIMATE)**
>- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11.
>- [From GitLab Ultimate 12.5](https://gitlab.com/gitlab-org/gitlab/-/issues/13401), when GitLab receives a recovery alert, it will automatically close the associated issue.
Alerts can be used to trigger actions, like opening an issue automatically
(disabled by default since `13.1`). To configure the actions:
1. Navigate to your project's **{settings}** **Settings > Operations > Incidents**.
1. Enable the option to create issues.
1. Choose the [issue template](../../user/project/description_templates.md) to create the issue from.
1. Optionally, select whether to send an email notification to the developers of the project.
1. Click **Save changes**.
After enabling, GitLab automatically opens an issue when an alert is triggered containing
values extracted from [alert's payload](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config):
- Issue author: `GitLab Alert Bot`
- Issue title: Extract from `annotations/title`, `annotations/summary` or `labels/alertname`
- Alert `Summary`: A list of properties
- `starts_at`: Alert start time via `startsAt`
- `full_query`: Alert query extracted from `generatorURL`
- Optional list of attached annotations extracted from `annotations/*`
- Alert [GFM](../../user/markdown.md): GitLab Flavored Markdown from `annotations/gitlab_incident_markdown`
When GitLab receives a **Recovery Alert**, it closes the associated issue.
This action is recorded as a system message on the issue indicating that it
was closed automatically by the GitLab Alert bot.
To further customize the issue, you can add labels, mentions, or any other supported
[quick action](../../user/project/quick_actions.md) in the selected issue template,
which applies to all incidents. To limit quick actions or other information to
only specific types of alerts, use the `annotations/gitlab_incident_markdown` field.
Since [version 12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63373),
GitLab tags each incident issue with the `incident` label automatically. If the label
does not yet exist, it is also created automatically.
If the metric exceeds the threshold of the alert for over 5 minutes, GitLab sends
an email to all [Maintainers and Owners](../../user/permissions.md#project-members-permissions)
of the project.
...@@ -6,32 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w ...@@ -6,32 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Using the Metrics Dashboard # Using the Metrics Dashboard
## Select an environment
The **Environment** dropdown box above the dashboard displays the list of all [environments](../index.md).
It enables you to search as you type through all environments and select the one you're looking for.
![Monitoring Dashboard Environments](../../../user/project/integrations/img/prometheus_dashboard_environments_v12_8.png)
## Select a dashboard
The **dashboard** dropdown box above the dashboard displays the list of all dashboards available for the project.
It enables you to search as you type through all dashboards and select the one you're looking for.
![Monitoring Dashboard select](../../../user/project/integrations/img/prometheus_dashboard_select_v_13_0.png)
## Mark a dashboard as favorite
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214582) in GitLab 13.0.
When viewing a dashboard, click the empty **Star dashboard** **{star-o}** button to mark a
dashboard as a favorite. Starred dashboards display a solid star **{star}** button,
and appear at the top of the dashboard select list.
To remove dashboard from the favorites list, click the solid **Unstar Dashboard** **{star}** button.
![Monitoring Dashboard favorite state toggle](../../../user/project/integrations/img/toggle_metrics_user_starred_dashboard_v13_0.png)
## Manage the metrics dashboard settings ## Manage the metrics dashboard settings
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223204) in GitLab 13.2. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223204) in GitLab 13.2.
...@@ -57,7 +31,7 @@ The options are: ...@@ -57,7 +31,7 @@ The options are:
- [View logs](#view-logs-ultimate) - [View logs](#view-logs-ultimate)
- [Download CSV](#downloading-data-as-csv) - [Download CSV](#downloading-data-as-csv)
- [Copy link to chart](../embed.md#embedding-gitlab-managed-kubernetes-metrics) - [Copy link to chart](../embed.md#embedding-gitlab-managed-kubernetes-metrics)
- [Alerts](../index.md#setting-up-alerts-for-prometheus-metrics) - [Alerts](../alerts.md)
### View and edit the source file of a custom dashboard ### View and edit the source file of a custom dashboard
......
...@@ -76,7 +76,7 @@ Panels in a panel group are laid out in rows consisting of two panels per row. A ...@@ -76,7 +76,7 @@ Panels in a panel group are laid out in rows consisting of two panels per row. A
| Property | Type | Required | Description | | Property | Type | Required | Description |
| ------ | ------ | ------ | ------ | | ------ | ------ | ------ | ------ |
| `id` | string | no | Used for associating dashboard metrics with database records. Must be unique across dashboard configuration files. Required for [alerting](../index.md#setting-up-alerts-for-prometheus-metrics) (support not yet enabled, see [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/27980)). | | `id` | string | no | Used for associating dashboard metrics with database records. Must be unique across dashboard configuration files. Required for [alerting](../alerts.md) (support not yet enabled, see [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/27980)). |
| `unit` | string | yes | Defines the unit of the query's return data. | | `unit` | string | yes | Defines the unit of the query's return data. |
| `label` | string | no, but highly encouraged | Defines the legend-label for the query. Should be unique within the panel's metrics. Can contain time series labels as interpolated variables. | | `label` | string | no, but highly encouraged | Defines the legend-label for the query. Should be unique within the panel's metrics. Can contain time series labels as interpolated variables. |
| `query` | string | yes if `query_range` is not defined | Defines the Prometheus query to be used to populate the chart/panel. If defined, the `query` endpoint of the [Prometheus API](https://prometheus.io/docs/prometheus/latest/querying/api/) will be utilized. | | `query` | string | yes if `query_range` is not defined | Defines the Prometheus query to be used to populate the chart/panel. If defined, the `query` endpoint of the [Prometheus API](https://prometheus.io/docs/prometheus/latest/querying/api/) will be utilized. |
......
...@@ -58,7 +58,7 @@ It is also possible to embed either the default dashboard metrics or individual ...@@ -58,7 +58,7 @@ It is also possible to embed either the default dashboard metrics or individual
## Embedding metrics based on alerts in incident issues ## Embedding metrics based on alerts in incident issues
For [GitLab-managed alerting rules](index.md#setting-up-alerts-for-prometheus-metrics), the issue will include an embedded chart for the query corresponding to the alert. The chart displays an hour of data surrounding the starting point of the incident, 30 minutes before and after. For [GitLab-managed alerting rules](alerts.md), the issue will include an embedded chart for the query corresponding to the alert. The chart displays an hour of data surrounding the starting point of the incident, 30 minutes before and after.
For [manually configured Prometheus instances](../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus), a chart corresponding to the query can be included if these requirements are met: For [manually configured Prometheus instances](../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus), a chart corresponding to the query can be included if these requirements are met:
......
This diff is collapsed.
...@@ -25,7 +25,7 @@ to create issues when alerts are triggered: ...@@ -25,7 +25,7 @@ to create issues when alerts are triggered:
checkbox to create an issue based on your own checkbox to create an issue based on your own
[issue templates](../project/description_templates.md#creating-issue-templates). [issue templates](../project/description_templates.md#creating-issue-templates).
For more information, see For more information, see
[Taking Action on Incidents](../../operations/metrics/index.md#taking-action-on-incidents-ultimate) **(ULTIMATE)**. [Trigger actions from alerts](../../operations/metrics/alerts.md#trigger-actions-from-alerts-ultimate) **(ULTIMATE)**.
1. To create issues from alerts, select the template in the **Issue Template** 1. To create issues from alerts, select the template in the **Issue Template**
select box. select box.
1. To send [separate email notifications](#notify-developers-of-alerts) to users 1. To send [separate email notifications](#notify-developers-of-alerts) to users
...@@ -36,7 +36,7 @@ to create issues when alerts are triggered: ...@@ -36,7 +36,7 @@ to create issues when alerts are triggered:
Appropriately configured alerts include an Appropriately configured alerts include an
[embedded chart](../../operations/metrics/embed.md#embedding-metrics-based-on-alerts-in-incident-issues) [embedded chart](../../operations/metrics/embed.md#embedding-metrics-based-on-alerts-in-incident-issues)
for the query corresponding to the alert. You can also configure GitLab to for the query corresponding to the alert. You can also configure GitLab to
[close issues](../../operations/metrics/index.md#taking-action-on-incidents-ultimate) [close issues](../../operations/metrics/alerts.md#trigger-actions-from-alerts-ultimate)
when you receive notification that the alert is resolved. when you receive notification that the alert is resolved.
### Notify developers of alerts ### Notify developers of alerts
...@@ -53,8 +53,8 @@ To send separate email notifications to users with ...@@ -53,8 +53,8 @@ To send separate email notifications to users with
You can set up Prometheus alerts in: You can set up Prometheus alerts in:
- [GitLab-managed Prometheus](../../operations/metrics/index.md#setting-up-alerts-for-prometheus-metrics) installations. - [GitLab-managed Prometheus](../../operations/metrics/alerts.md) installations.
- [Self-managed Prometheus](../../operations/metrics/index.md#external-prometheus-instances) installations. - [Self-managed Prometheus](../../operations/metrics/alerts.md#external-prometheus-instances) installations.
Prometheus alerts are created by the special Alert Bot user. You can't remove this Prometheus alerts are created by the special Alert Bot user. You can't remove this
user, but it does not count toward your license limit. user, but it does not count toward your license limit.
......
...@@ -19,7 +19,9 @@ The entire payload will be posted in the issue discussion as a comment ...@@ -19,7 +19,9 @@ The entire payload will be posted in the issue discussion as a comment
authored by the GitLab Alert Bot. authored by the GitLab Alert Bot.
NOTE: **Note:** NOTE: **Note:**
In GitLab versions 13.1 and greater, you can configure [External Prometheus instances](../../../operations/metrics/index.md#external-prometheus-instances) to use this endpoint. In GitLab versions 13.1 and greater, you can configure
[External Prometheus instances](../../../operations/metrics/alerts.md#external-prometheus-instances)
to use this endpoint.
## Setting up generic alerts ## Setting up generic alerts
......
...@@ -19,7 +19,8 @@ There are two ways to set up Prometheus integration, depending on where your app ...@@ -19,7 +19,8 @@ There are two ways to set up Prometheus integration, depending on where your app
- For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes). - For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes).
- For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). - For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus).
Once enabled, GitLab will automatically detect metrics from known services in the [metric library](prometheus_library/index.md). You can also [add your own metrics](../../../operations/metrics/index.md#adding-custom-metrics). Once enabled, GitLab will automatically detect metrics from known services in the [metric library](prometheus_library/index.md). You can also [add your own metrics](../../../operations/metrics/index.md#adding-custom-metrics) and create
[custom dashboards](../../../operations/metrics/dashboards/index.md).
## Enabling Prometheus Integration ## Enabling Prometheus Integration
...@@ -43,25 +44,6 @@ Once you have a connected Kubernetes cluster, deploying a managed Prometheus is ...@@ -43,25 +44,6 @@ Once you have a connected Kubernetes cluster, deploying a managed Prometheus is
![Managed Prometheus Deploy](img/prometheus_deploy.png) ![Managed Prometheus Deploy](img/prometheus_deploy.png)
#### Getting metrics to display on the Metrics Dashboard
After completing the steps above, you will also need deployments in order to view the
**Operations > Metrics** page. Setting up [Auto DevOps](../../../topics/autodevops/index.md)
will help you to quickly create a deployment:
1. Navigate to your project's **Operations > Kubernetes** page, and ensure that,
in addition to "Prometheus", you also have "Runner" and "Ingress"
installed. Once "Ingress" is installed, copy its endpoint.
1. Navigate to your project's **Settings > CI/CD** page. In the Auto DevOps section,
select a deployment strategy and save your changes.
1. On the same page, in the Variables section, add a variable named `KUBE_INGRESS_BASE_DOMAIN`
with the value of the Ingress endpoint you have copied in the previous step. Leave the type
as "Variable".
1. Navigate to your project's **CI/CD > Pipelines** page, and run a pipeline on any branch.
1. When the pipeline has run successfully, graphs will be available on the **Operations > Metrics** page.
![Monitoring Dashboard](img/prometheus_monitoring_dashboard_v13_1.png)
#### About managed Prometheus deployments #### About managed Prometheus deployments
Prometheus is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/helm/charts/tree/master/stable/prometheus). Prometheus is only accessible within the cluster, with GitLab communicating through the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/). Prometheus is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/helm/charts/tree/master/stable/prometheus). Prometheus is only accessible within the cluster, with GitLab communicating through the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/).
......
...@@ -65,16 +65,16 @@ When GitLab-managed Prometheus is installed, the [Alerts list](#alert-management ...@@ -65,16 +65,16 @@ When GitLab-managed Prometheus is installed, the [Alerts list](#alert-management
is also enabled. is also enabled.
To populate the alerts with data, see To populate the alerts with data, see
[GitLab-Managed Prometheus instances](../../../operations/metrics/index.md#managed-prometheus-instances). [GitLab-Managed Prometheus instances](../../../operations/metrics/alerts.md#managed-prometheus-instances).
### Enable external Prometheus alerts ### Enable external Prometheus alerts
You can configure an externally-managed Prometheus instance to send alerts You can configure an externally-managed Prometheus instance to send alerts
to GitLab. To set up this configuration, see the [configuring Prometheus](../../../operations/metrics/index.md#external-prometheus-instances) documentation. Activating the external Prometheus to GitLab. To set up this configuration, see the [configuring Prometheus](../../../operations/metrics/alerts.md#external-prometheus-instances) documentation. Activating the external Prometheus
configuration also enables the [Alerts list](#alert-management-list). configuration also enables the [Alerts list](#alert-management-list).
To populate the alerts with data, see To populate the alerts with data, see
[External Prometheus instances](../../../operations/metrics/index.md#external-prometheus-instances). [External Prometheus instances](../../../operations/metrics/alerts.md#external-prometheus-instances).
## Alert Management severity ## Alert Management severity
...@@ -240,14 +240,14 @@ To view the metrics for an alert: ...@@ -240,14 +240,14 @@ To view the metrics for an alert:
For GitLab-managed Prometheus instances, metrics data is automatically available For GitLab-managed Prometheus instances, metrics data is automatically available
for the alert, making it easy to see surrounding behavior. See for the alert, making it easy to see surrounding behavior. See
[Managed Prometheus instances](../../../operations/metrics/index.md#managed-prometheus-instances) [Managed Prometheus instances](../../../operations/metrics/alerts.md#managed-prometheus-instances)
for information on setting up alerts. for information on setting up alerts.
For externally-managed Prometheus instances, you can configure your alerting rules to For externally-managed Prometheus instances, you can configure your alerting rules to
display a chart in the alert. See display a chart in the alert. See
[Embedding metrics based on alerts in incident issues](../../../operations/metrics/embed.md#embedding-metrics-based-on-alerts-in-incident-issues) [Embedding metrics based on alerts in incident issues](../../../operations/metrics/embed.md#embedding-metrics-based-on-alerts-in-incident-issues)
for information on how to appropriately configure your alerting rules. See for information on how to appropriately configure your alerting rules. See
[External Prometheus instances](../../../operations/metrics/index.md#external-prometheus-instances) [External Prometheus instances](../../../operations/metrics/alerts.md#external-prometheus-instances)
for information on setting up alerts for your self-managed Prometheus instance. for information on setting up alerts for your self-managed Prometheus instance.
## Use cases for assigning alerts ## Use cases for assigning alerts
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment