Merge branch 'add-prometheus-external-docs' into 'master'

Add docs for setting up external Prometheus

Closes #40949 and #37255

See merge request gitlab-org/gitlab-ce!22831

doc/administration/monitoring/prometheus/index.md

-# GitLab Prometheus
+# Monitoring GitLab with Prometheus
 
 > **Notes:**
 > - Prometheus and the various exporters listed in this page are bundled in the
@@ -24,7 +24,7 @@ dashboard tool like [Grafana].
 
 ## Configuring Prometheus
 
->**Note:**
+NOTE: **Note:**
 For installations from source you'll have to install and configure it yourself.
 
 Prometheus and it's exporters are on by default, starting with GitLab 9.0.
@@ -43,17 +43,17 @@ To disable Prometheus and all of its exporters, as well as any added in the futu
     ```
 
 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
-   take effect
+   take effect.
 
-## Changing the port and address Prometheus listens on
+### Changing the port and address Prometheus listens on
 
->**Note:**
+NOTE: **Note:**
 The following change was added in [GitLab Omnibus 8.17][1261]. Although possible,
 it's not recommended to change the port Prometheus listens
 on as this might affect or conflict with other services running on the GitLab
 server. Proceed at your own risk.
 
-In order to access Prometheus from outside the GitLab server you will need to 
+In order to access Prometheus from outside the GitLab server you will need to
 set a FQDN or IP in `prometheus['listen_address']`.
 To change the address/port that Prometheus listens on:
 
@@ -77,6 +77,60 @@ To change the address/port that Prometheus listens on:
 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
    take effect
 
+### Using an external Prometheus server
+
+NOTE: **Note:**
+Prometheus and most exporters do not support authentication. We do not recommend exposing them outside the local network.
+
+A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for highly available deployments of GitLab with multiple nodes.
+
+To use an external Prometheus server:
+
+1. Edit `/etc/gitlab/gitlab.rb`.
+1. Disable the bundled Prometheus:
+
+    ```ruby
+    prometheus['enable'] = false
+    ```
+
+1. Set each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, for example:
+
+     ```ruby
+     gitlab_monitor['listen_address'] = '0.0.0.0'
+     gitlab_monitor['listen_port'] = '9168'
+     gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
+     node_exporter['listen_address'] = '0.0.0.0:9100'
+     redis_exporter['listen_address'] = '0.0.0.0:9121'
+     postgres_exporter['listen_address'] = '0.0.0.0:9187'
+     ```
+
+1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/).
+1. Add the Prometheus server IP address to the [monitoring IP whitelist](../ip_whitelist.html). For example:
+
+    ```ruby
+    gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1']
+    ```
+
+1. [Reconfigure GitLab][reconfigure] to apply the changes
+1. Edit the Prometheus server's configuration file.
+1. Add each node's exporters to the Prometheus server's
+   [scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E).
+   For example, a sample snippet using `static_configs`:
+
+    ```yaml
+    scrape_configs:
+    - job_name: 'gitlab_exporters'
+      static_configs:
+      - targets: ['1.1.1.1:9168', '1.1.1.1:9236', '1.1.1.1:9236', '1.1.1.1:9100', '1.1.1.1:9121', '1.1.1.1:9187']
+
+    - job_name: 'gitlab_metrics'
+      metrics_path: /-/metrics
+      static_configs:
+      - targets: ['1.1.1.1:443']
+    ```
+
+1. Restart the Prometheus server.
+
 ## Viewing performance metrics
 
 You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default.
@@ -86,7 +140,7 @@ If SSL has been enabled on your GitLab instance, you may not be able to access
 Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS][hsts]. We plan to
 [provide access via GitLab][multi-user-prometheus], but in the interim there are
 some workarounds: using a separate FQDN, using server IP, using a separate browser for Prometheus, resetting HSTS, or
-having [Nginx proxy it][nginx-custom-config].
+having [NGINX proxy it][nginx-custom-config].
 
 The performance data collected by Prometheus can be viewed directly in the
 Prometheus console or through a compatible dashboard tool.
@@ -102,26 +156,7 @@ Sample Prometheus queries:
 - **Data transmitted:** `rate(node_network_transmit_bytes_total{device!="lo"}[5m])`
 - **Data received:** `rate(node_network_receive_bytes_total{device!="lo"}[5m])`
 
-## Configuring Prometheus to monitor Kubernetes
-
-> Introduced in GitLab 9.0.
-> Pod monitoring introduced in GitLab 9.4.
-
-If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them.
-
-To disable the monitoring of Kubernetes:
-
-1. Edit `/etc/gitlab/gitlab.rb`
-1. Add or find and uncomment the following line and set it to `false`:
-
-    ```ruby
-    prometheus['monitor_kubernetes'] = false
-    ```
-
-1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
-   take effect
-
-## GitLab Prometheus metrics
+## GitLab metrics
 
 > Introduced in GitLab 9.3.
 
@@ -129,17 +164,10 @@ GitLab monitors its own internal service metrics, and makes them available at th
 
 [➔ Read more about the GitLab Metrics.](gitlab_metrics.md)
 
-## Prometheus exporters
-
-There are a number of libraries and servers which help in exporting existing
-metrics from third-party systems as Prometheus metrics. This is useful for cases
-where it is not feasible to instrument a given system with Prometheus metrics
-directly (for example, HAProxy or Linux system stats). You can read more in the
-[Prometheus exporters and integrations upstream documentation][prom-exporters].
+## Bundled software metrics
 
-While you can use any exporter you like with your GitLab installation, the
-following ones documented here are bundled in the Omnibus GitLab packages
-making it easy to configure and use.
+Many of the GitLab dependencies bundled in Omnibus GitLab are preconfigured to
+export Prometheus metrics.
 
 ### Node exporter
 
@@ -166,6 +194,25 @@ The GitLab monitor exporter allows you to measure various GitLab metrics, pulled
 
 [➔ Read more about the GitLab monitor exporter.](gitlab_monitor_exporter.md)
 
+## Configuring Prometheus to monitor Kubernetes
+
+> Introduced in GitLab 9.0.
+> Pod monitoring introduced in GitLab 9.4.
+
+If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][prometheus integration] to monitor them.
+
+To disable the monitoring of Kubernetes:
+
+1. Edit `/etc/gitlab/gitlab.rb`.
+1. Add or find and uncomment the following line and set it to `false`:
+
+    ```ruby
+    prometheus['monitor_kubernetes'] = false
+    ```
+
+1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
+   take effect.
+
 [grafana]: https://grafana.net
 [hsts]: https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
 [multi-user-prometheus]: https://gitlab.com/gitlab-org/multi-user-prometheus