Document praefect's separate_database_metrics setting

praefect['separate_database_metrics'] will put praefect metrics that run
queries on the database on a separate /db_metrics endpoint so that
customers can choose to configure a slower scrape interval. Also, with a
multiple Praefect setup, customers can configure only one Praefect node
to query the database.

Also add /db_metrics prometheus metrics to the gitaly/index.md.

doc/administration/gitaly/index.md

@@ -490,7 +490,15 @@ The following are useful queries for monitoring Gitaly:
 
 ### Monitor Gitaly Cluster
 
-To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics:
+To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. There are two separate metrics
+endpoints from which metrics can be scraped:
+
+- The default `/metrics` endpoint.
+- `/db_metrics`, which contains metrics that require database queries.
+
+#### Default Prometheus `/metrics` endpoint
+
+The following metrics are available from the `/metrics` endpoint:
 
 - `gitaly_praefect_read_distribution`, a counter to track [distribution of reads](#distributed-reads).
   It has two labels:
@@ -523,6 +531,16 @@ To monitor [strong consistency](#strong-consistency), you can use the following
 
 You can also monitor the [Praefect logs](../logs.md#praefect-logs).
 
+#### Database metrics `/db_metrics` endpoint
+
+The following metrics are available from the `/db_metrics` endpoint:
+
+- `gitaly_praefect_unavailable_repositories`, the number of repositories that have no healthy, up to date replicas.
+- `gitaly_praefect_read_only_repositories`, the number of repositories in read-only mode within a virtual storage.
+  This is an older metric that is still available for backwards compatibility reasons. `gitaly_praefect_unavailable_repositories`
+  is a more accurate.
+- `gitaly_praefect_replication_queue_depth`, the number of jobs in the replication queue.
+
 ## Recover from failure
 
 Gitaly Cluster can [recover from certain types of failure](recovery.md).

doc/administration/gitaly/praefect.md

@@ -418,10 +418,21 @@ On the **Praefect** node:
 
    ```ruby
    praefect['listen_addr'] = '0.0.0.0:2305'
+   ```
+
+1. Configure Prometheus metrics by editing
+   `/etc/gitlab/gitlab.rb`:
 
+   ```ruby
    # Enable Prometheus metrics access to Praefect. You must use firewalls
    # to restrict access to this address/port.
+   # The default metrics endpoint is /metrics
    praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
+
+   # Some metrics run queries against the database. Enabling separate database metrics allows
+   # these metrics to be collected when the metrics are
+   # scraped on a separate /db_metrics endpoint. 
+   praefect['separate_database_metrics'] = true
    ```
 
 1. Configure a strong `auth_token` for **Praefect** by editing