Merge branch 'add-prometheus-external-docs' into 'master'

Add docs for setting up external Prometheus

Closes #40949 and #37255

See merge request gitlab-org/gitlab-ce!22831

doc/administration/monitoring/prometheus/index.md

-# GitLab Prometheus
+# Monitoring GitLab with Prometheus
 
 > **Notes:**
 > - Prometheus and the various exporters listed in this page are bundled in the
@@ -24,7 +24,7 @@ dashboard tool like [Grafana].
 
 ## Configuring Prometheus
 
->**Note:**
+NOTE: **Note:**
 For installations from source you'll have to install and configure it yourself.
 
 Prometheus and it's exporters are on by default, starting with GitLab 9.0.
@@ -43,11 +43,11 @@ To disable Prometheus and all of its exporters, as well as any added in the futu
     ```
 
 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
-   take effect
+   take effect.
 
-## Changing the port and address Prometheus listens on
+### Changing the port and address Prometheus listens on
 
->**Note:**
+NOTE: **Note:**
 The following change was added in [GitLab Omnibus 8.17][1261]. Although possible,
 it's not recommended to change the port Prometheus listens
 on as this might affect or conflict with other services running on the GitLab
@@ -77,6 +77,60 @@ To change the address/port that Prometheus listens on:
 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
    take effect
 
+### Using an external Prometheus server
+
+NOTE: **Note:**
+Prometheus and most exporters do not support authentication. We do not recommend exposing them outside the local network.
+
+A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for highly available deployments of GitLab with multiple nodes.
+
+To use an external Prometheus server:
+
+1. Edit `/etc/gitlab/gitlab.rb`.
+1. Disable the bundled Prometheus:
+
+    ```ruby
+    prometheus['enable'] = false
+    ```
+
+1. Set each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, for example:
+
+     ```ruby
+     gitlab_monitor['listen_address'] = '0.0.0.0'
+     gitlab_monitor['listen_port'] = '9168'
+     gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
+     node_exporter['listen_address'] = '0.0.0.0:9100'
+     redis_exporter['listen_address'] = '0.0.0.0:9121'
+     postgres_exporter['listen_address'] = '0.0.0.0:9187'
+     ```
+
+1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/).
+1. Add the Prometheus server IP address to the [monitoring IP whitelist](../ip_whitelist.html). For example:
+
+    ```ruby
+    gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1']
+    ```
+
+1. [Reconfigure GitLab][reconfigure] to apply the changes
+1. Edit the Prometheus server's configuration file.
+1. Add each node's exporters to the Prometheus server's
+   [scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E).
+   For example, a sample snippet using `static_configs`:
+
+    ```yaml
+    scrape_configs:
+    - job_name: 'gitlab_exporters'
+      static_configs:
+      - targets: ['1.1.1.1:9168', '1.1.1.1:9236', '1.1.1.1:9236', '1.1.1.1:9100', '1.1.1.1:9121', '1.1.1.1:9187']
+
+    - job_name: 'gitlab_metrics'
+      metrics_path: /-/metrics
+      static_configs:
+      - targets: ['1.1.1.1:443']
+    ```
+
+1. Restart the Prometheus server.
+
 ## Viewing performance metrics
 
 You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default.
@@ -86,7 +140,7 @@ If SSL has been enabled on your GitLab instance, you may not be able to access
 Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS][hsts]. We plan to
 [provide access via GitLab][multi-user-prometheus], but in the interim there are
 some workarounds: using a separate FQDN, using server IP, using a separate browser for Prometheus, resetting HSTS, or
-having [Nginx proxy it][nginx-custom-config].
+having [NGINX proxy it][nginx-custom-config].
 
 The performance data collected by Prometheus can be viewed directly in the
 Prometheus console or through a compatible dashboard tool.
@@ -102,26 +156,7 @@ Sample Prometheus queries:
 - **Data transmitted:** `rate(node_network_transmit_bytes_total{device!="lo"}[5m])`
 - **Data received:** `rate(node_network_receive_bytes_total{device!="lo"}[5m])`
 
-## Configuring Prometheus to monitor Kubernetes
-
-> Introduced in GitLab 9.0.
-> Pod monitoring introduced in GitLab 9.4.
-
-If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them.
-
-To disable the monitoring of Kubernetes:
-
-1. Edit `/etc/gitlab/gitlab.rb`
-1. Add or find and uncomment the following line and set it to `false`:
-
-    ```ruby
-    prometheus['monitor_kubernetes'] = false
-    ```
-
-1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
-   take effect
-
-## GitLab Prometheus metrics
+## GitLab metrics
 
 > Introduced in GitLab 9.3.
 
@@ -129,17 +164,10 @@ GitLab monitors its own internal service metrics, and makes them available at th
 
 [➔ Read more about the GitLab Metrics.](gitlab_metrics.md)
 
-## Prometheus exporters
+## Bundled software metrics
 
-There are a number of libraries and servers which help in exporting existing
-metrics from third-party systems as Prometheus metrics. This is useful for cases
-where it is not feasible to instrument a given system with Prometheus metrics
-directly (for example, HAProxy or Linux system stats). You can read more in the
-[Prometheus exporters and integrations upstream documentation][prom-exporters].
-
-While you can use any exporter you like with your GitLab installation, the
-following ones documented here are bundled in the Omnibus GitLab packages
-making it easy to configure and use.
+Many of the GitLab dependencies bundled in Omnibus GitLab are preconfigured to
+export Prometheus metrics.
 
 ### Node exporter
 
@@ -166,6 +194,25 @@ The GitLab monitor exporter allows you to measure various GitLab metrics, pulled
 
 [➔ Read more about the GitLab monitor exporter.](gitlab_monitor_exporter.md)
 
+## Configuring Prometheus to monitor Kubernetes
+
+> Introduced in GitLab 9.0.
+> Pod monitoring introduced in GitLab 9.4.
+
+If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][prometheus integration] to monitor them.
+
+To disable the monitoring of Kubernetes:
+
+1. Edit `/etc/gitlab/gitlab.rb`.
+1. Add or find and uncomment the following line and set it to `false`:
+
+    ```ruby
+    prometheus['monitor_kubernetes'] = false
+    ```
+
+1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
+   take effect.
+
 [grafana]: https://grafana.net
 [hsts]: https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
 [multi-user-prometheus]: https://gitlab.com/gitlab-org/multi-user-prometheus