Commit 5e01cf72 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch '56473-update-nginx-docs' into 'master'

Add documentation for new NGINX Ingress metrics

Closes #56473

See merge request gitlab-org/gitlab-ce!24449
parents fc676141 27dccbfb
...@@ -10,7 +10,8 @@ Currently supported exporters are: ...@@ -10,7 +10,8 @@ Currently supported exporters are:
- [Kubernetes](kubernetes.md) - [Kubernetes](kubernetes.md)
- [NGINX](nginx.md) - [NGINX](nginx.md)
- [NGINX Ingress Controller](nginx_ingress.md) - [NGINX Ingress Controller 0.9.0-0.15.x](nginx_ingress_vts.md)
- [NGINX Ingress Controller 0.16.0+](nginx_ingress.md)
- [HAProxy](haproxy.md) - [HAProxy](haproxy.md)
- [Amazon Cloud Watch](cloudwatch.md) - [Amazon Cloud Watch](cloudwatch.md)
......
# Monitoring NGINX Ingress Controller # Monitoring NGINX Ingress Controller
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13438) in GitLab 9.5. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22133) in GitLab 11.7.
GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built in Prometheus metrics included in [version 0.9.0](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#09-beta1) and above of the ingress. NOTE: **Note:** NGINX Ingress versions prior to 0.16.0 offer an included [VTS Prometheus metrics exporter](nginx_ingress_vts.md), which exports metrics different than the built-in metrics.
GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built-in Prometheus metrics included starting with [version 0.16.0](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#0160).
## Requirements ## Requirements
...@@ -12,9 +14,9 @@ GitLab has support for automatically detecting and monitoring the Kubernetes NGI ...@@ -12,9 +14,9 @@ GitLab has support for automatically detecting and monitoring the Kubernetes NGI
| Name | Query | | Name | Query |
| ---- | ----- | | ---- | ----- |
| Throughput (req/sec) | sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code) | | Throughput (req/sec) | sum(label_replace(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m]), "status_code", "${1}xx", "status", "(.)..")) by (status_code) |
| Latency (ms) | avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}) | | Latency (ms) | sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_sum{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) / sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_count{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) * 1000 |
| HTTP Error Rate (%) | sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) / sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) * 100 | | HTTP Error Rate (%) | sum(rate(nginx_ingress_controller_requests{status=~"5.*",namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) / sum(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m])) * 100 |
## Configuring NGINX ingress monitoring ## Configuring NGINX ingress monitoring
...@@ -22,9 +24,9 @@ If you have deployed NGINX Ingress using GitLab's [Kubernetes cluster integratio ...@@ -22,9 +24,9 @@ If you have deployed NGINX Ingress using GitLab's [Kubernetes cluster integratio
For other deployments, there is [some configuration](#manually-setting-up-nginx-ingress-for-prometheus-monitoring) required depending on your installation: For other deployments, there is [some configuration](#manually-setting-up-nginx-ingress-for-prometheus-monitoring) required depending on your installation:
- NGINX Ingress should be version 0.9.0 or above, with metrics enabled - NGINX Ingress should be version 0.16.0 or above, with metrics enabled.
- NGINX Ingress should be annotated for Prometheus monitoring - NGINX Ingress should be annotated for Prometheus monitoring.
- Prometheus should be configured to monitor annotated pods - Prometheus should be configured to monitor annotated pods.
### About managed NGINX Ingress deployments ### About managed NGINX Ingress deployments
...@@ -32,9 +34,9 @@ NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [o ...@@ -32,9 +34,9 @@ NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [o
NGINX is configured for Prometheus monitoring, by setting: NGINX is configured for Prometheus monitoring, by setting:
- `enable-vts-status: "true"`, to export Prometheus metrics - `enable-vts-status: "true"`, to export Prometheus metrics.
- `prometheus.io/scrape: "true"`, to enable automatic discovery - `prometheus.io/scrape: "true"`, to enable automatic discovery.
- `prometheus.io/port: "10254"`, to specify the metrics port - `prometheus.io/port: "10254"`, to specify the metrics port.
When used in conjunction with the GitLab deployed Prometheus service, response metrics will be automatically collected. When used in conjunction with the GitLab deployed Prometheus service, response metrics will be automatically collected.
...@@ -51,6 +53,6 @@ Managing these settings depends on how NGINX ingress has been deployed. If you h ...@@ -51,6 +53,6 @@ Managing these settings depends on how NGINX ingress has been deployed. If you h
## Specifying the Environment label ## Specifying the Environment label
In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do this, GitLab will search for metrics with appropriate labels. In this case, the `upstream` label must be of the form `<KUBE_NAMESPACE>-<CI_ENVIRONMENT_SLUG>-*`. In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do this, GitLab will search for metrics with appropriate labels. In this case, the `ingress` label must `<CI_ENVIRONMENT_SLUG>`.
If you have used [Auto Deploy](../../../../topics/autodevops/index.md#auto-deploy) to deploy your app, this format will be used automatically and metrics will be detected with no action on your part. If you have used [Auto Deploy](../../../../topics/autodevops/index.md#auto-deploy) to deploy your app, this format will be used automatically and metrics will be detected with no action on your part.
# Monitoring NGINX Ingress Controller with VTS metrics
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13438) in GitLab 9.5.
NOTE: **Note:** [NGINX Ingress version 0.16](nginx_ingress.md) and above have built-in Prometheus metrics, which are different than the VTS based metrics.
GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the included VTS Prometheus metrics exporter in [version 0.9.0](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#09-beta1) through [0.15.x](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#0150).
## Requirements
[Prometheus integration](../prometheus.md) must be active.
## Metrics supported
| Name | Query |
| ---- | ----- |
| Throughput (req/sec) | sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code) |
| Latency (ms) | avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}) |
| HTTP Error Rate (%) | sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) / sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) * 100 |
## Configuring NGINX ingress monitoring
If you have deployed NGINX Ingress using GitLab's [Kubernetes cluster integration](../../clusters/index.md#installing-applications), it will [automatically be monitored](#about-managed-nginx-ingress-deployments) by Prometheus.
For other deployments, there is [some configuration](#manually-setting-up-nginx-ingress-for-prometheus-monitoring) required depending on your installation:
- NGINX Ingress should be version 0.9.0 or above, with metrics enabled.
- NGINX Ingress should be annotated for Prometheus monitoring.
- Prometheus should be configured to monitor annotated pods.
### About managed NGINX Ingress deployments
NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress). NGINX Ingress will be [externally reachable via the Load Balancer's IP](../../clusters/index.md#getting-the-external-ip-address).
NGINX is configured for Prometheus monitoring, by setting:
- `enable-vts-status: "true"`, to export Prometheus metrics.
- `prometheus.io/scrape: "true"`, to enable automatic discovery.
- `prometheus.io/port: "10254"`, to specify the metrics port.
When used in conjunction with the GitLab deployed Prometheus service, response metrics will be automatically collected.
### Manually setting up NGINX Ingress for Prometheus monitoring
Version 0.9.0 and above of [NGINX ingress](https://github.com/kubernetes/ingress-nginx) have built-in support for exporting Prometheus metrics. To enable, a ConfigMap setting must be passed: `enable-vts-status: "true"`. Once enabled, a Prometheus metrics endpoint will start running on port 10254.
Next, the ingress needs to be annotated for Prometheus monitoring. Two new annotations need to be added:
- `prometheus.io/scrape: "true"`
- `prometheus.io/port: "10254"`
Managing these settings depends on how NGINX ingress has been deployed. If you have deployed via the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress), metrics can be enabled with `controller.stats.enabled` along with the required annotations. Alternatively it is possible edit the NGINX ingress YML directly in the [Kubernetes dashboard](https://github.com/kubernetes/dashboard).
## Specifying the Environment label
In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do this, GitLab will search for metrics with appropriate labels. In this case, the `upstream` label must be of the form `<KUBE_NAMESPACE>-<CI_ENVIRONMENT_SLUG>-*`.
If you have used [Auto Deploy](../../../../topics/autodevops/index.md#auto-deploy) to deploy your app, this format will be used automatically and metrics will be detected with no action on your part.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment