Add example configurations for PG HA

* Adds recommended example
* Adds minimal example

The recommended example is a 7-node configuration.  3x consul, 3x
database, 1x application.  Pgbouncer runs on the application node.

The minimal example is 4 nodes.  3x database, 1x application.  Pgbouncer
runs on the application node, we don't have a dedicated consul tier.

The style of these examples are copied from the existing redis examples.

Note, it is intended to supplement the preceding sections, it should not
be your only reference in setting up PG HA.

doc/administration/high_availability/database.md

@@ -539,6 +539,295 @@ gitlab-rake gitlab:db:configure
 At this point, your GitLab instance should be up and running. Verify you are
 able to login, and create issues and merge requests.
 
+### Example configuration
+
+Here we'll show you some fully expanded example configurations.
+
+#### Example recommended setup.  3 consul servers, 3 postgresql, 1 application node
+
+In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses.
+
+Here is a list and description of each machine and the assigned IP:
+
+* `10.6.0.11`: Consul 1
+* `10.6.0.12`: Consul 2
+* `10.6.0.13`: Consul 3
+* `10.6.0.21`: PostgreSQL master
+* `10.6.0.22`: PostgreSQL secondary
+* `10.6.0.23`: PostgreSQL secondary
+* `10.6.0.31`: GitLab application
+
+All passwords are set to `toomanysecrets`, please do not use this password or derived hashes.
+
+The external_url for GitLab is `http://gitlab.example.com`
+
+Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back.
+
+##### Example recommended setup for Consul servers
+
+In `/etc/gitlab/gitlab.rb`:
+
+```ruby
+# Disable all components except Consul
+bootstrap['enable'] = false
+gitlab_rails['auto_migrate'] = false
+gitaly['enable'] = false
+gitlab_workhorse['enable'] = false
+mailroom['enable'] = false
+nginx['enable'] = false
+postgresql['enable'] = false
+redis['enable'] = false
+sidekiq['enable'] = false
+prometheus['enable'] = false
+unicorn['enable'] = false
+
+consul['enable'] = true
+consul['configuration'] = {
+  server: true,
+  retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
+}
+```
+
+[Reconfigure Omnibus GitLab][reconfigure Gitlab] for the changes to take effect.
+
+##### Example recommended setup for PostgreSQL servers
+
+In `/etc/gitlab/gitlab.rb`:
+
+```ruby
+# Disable all components except PostgreSQL and Repmgr and Consul
+bootstrap['enable'] = false
+gitaly['enable'] = false
+mailroom['enable'] = false
+nginx['enable'] = false
+unicorn['enable'] = false
+sidekiq['enable'] = false
+redis['enable'] = false
+gitlab_workhorse['enable'] = false
+prometheus_monitoring['enable'] = false
+
+repmgr['enable'] = true
+postgresql['enable'] = true
+consul['enable'] = true
+
+# PostgreSQL configuration
+postgresql['listen_address'] = '0.0.0.0'
+postgresql['hot_standby'] = 'on'
+postgresql['wal_level'] = 'replica'
+postgresql['shared_preload_libraries'] = 'repmgr_funcs'
+
+# Disable automatic database migrations
+gitlab_rails['auto_migrate'] = false
+
+# Configure the consul agent
+consul['services'] = %w(postgresql)
+
+postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c'
+postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f'
+postgresql['max_wal_senders'] = 4
+
+postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16)
+repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16)
+
+consul['configuration'] = {
+  retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
+}
+```
+
+[Reconfigure Omnibus GitLab][reconfigure Gitlab] for the changes to take effect.
+
+##### Example recommended setup for application server
+
+In `/etc/gitlab/gitlab.rb`:
+
+```ruby
+external_url 'http://gitlab.example.com'
+
+gitlab_rails['db_host'] = '127.0.0.1'
+gitlab_rails['db_port'] = 6432
+gitlab_rails['db_password'] = 'toomanysecrets'
+gitlab_rails['auto_migrate'] = false
+
+postgresql['enable'] = false
+pgbouncer['enable'] = true
+consul['enable'] = true
+
+# Configure Pgbouncer
+pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
+
+# Configure Consul agent
+consul['watchers'] = %w(postgresql)
+
+pgbouncer['users'] = {
+  'gitlab-consul': {
+    password: '5e0e3263571e3704ad655076301d6ebe'
+  },
+  'pgbouncer': {
+    password: '771a8625958a529132abe6f1a4acb19c'
+  }
+}
+
+consul['configuration'] = {
+  retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
+}
+```
+
+[Reconfigure Omnibus GitLab][reconfigure Gitlab] for the changes to take effect.
+
+##### Example recommended setup manual steps
+
+After deploying the configuration follow these steps:
+
+1. On `10.6.0.21`, our primary database
+
+    Enable the `pg_trgm` extension
+
+    ```sh
+    gitlab-psql -d gitlabhq_production
+    ```
+
+    ```
+    CREATE EXTENSION pg_trgm;
+    ```
+
+1. On `10.6.0.22`, our first standby database
+
+    Make this node a standby of the primary
+
+    ```sh
+    gitlab-ctl repmgr standby setup 10.6.0.21
+    ```
+
+1. On `10.6.0.23`, our second standby database
+
+    Make this node a standby of the primary
+
+    ```sh
+    gitlab-ctl repmgr standby setup 10.6.0.21
+    ```
+
+1. On `10.6.0.31`, our application server
+
+    Set gitlab-consul's pgbouncer password to `toomanysecrets`
+
+    ```sh
+    gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul
+    ```
+
+    Run database migrations
+
+    ```sh
+    gitlab-rake gitlab:db:configure
+    ```
+
+#### Example minimal setup. 3 database servers, 1 application node
+
+This example is more minimal.  It moves the consul servers into the same nodes we use for the PostgreSQL servers.
+
+In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses.
+
+Here is a list and description of each machine and the assigned IP:
+
+* `10.6.0.21`: PostgreSQL master
+* `10.6.0.22`: PostgreSQL secondary
+* `10.6.0.23`: PostgreSQL secondary
+* `10.6.0.31`: GitLab application
+
+All passwords are set to `toomanysecrets`, please do not use this password or derived hashes.
+
+The external_url for GitLab is `http://gitlab.example.com`
+
+Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back.
+
+##### Example minimal configuration for database servers
+
+In `/etc/gitlab/gitlab.rb`:
+
+```ruby
+# Disable all components except PostgreSQL, Repmgr, and Consul
+bootstrap['enable'] = false
+gitaly['enable'] = false
+mailroom['enable'] = false
+nginx['enable'] = false
+unicorn['enable'] = false
+sidekiq['enable'] = false
+redis['enable'] = false
+gitlab_workhorse['enable'] = false
+prometheus_monitoring['enable'] = false
+
+repmgr['enable'] = true
+postgresql['enable'] = true
+consul['enable'] = true
+
+# PostgreSQL configuration
+postgresql['listen_address'] = '0.0.0.0'
+postgresql['hot_standby'] = 'on'
+postgresql['wal_level'] = 'replica'
+postgresql['shared_preload_libraries'] = 'repmgr_funcs'
+
+# Disable automatic database migrations
+gitlab_rails['auto_migrate'] = false
+
+# Configure the consul agent
+consul['services'] = %w(postgresql)
+
+postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c'
+postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f'
+postgresql['max_wal_senders'] = 4
+
+postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16)
+repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16)
+
+consul['configuration'] = {
+  server: true,
+  retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23)
+}
+```
+
+[Reconfigure Omnibus GitLab][reconfigure Gitlab] for the changes to take effect.
+
+##### Example minimal configuration for application server
+
+In `/etc/gitlab/gitlab.rb`:
+
+```ruby
+external_url 'http://gitlab.example.com'
+
+gitlab_rails['db_host'] = '127.0.0.1'
+gitlab_rails['db_port'] = 6432
+gitlab_rails['db_password'] = 'toomanysecrets'
+gitlab_rails['auto_migrate'] = false
+
+postgresql['enable'] = false
+pgbouncer['enable'] = true
+consul['enable'] = true
+
+# Configure Pgbouncer
+pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
+
+# Configure Consul agent
+consul['watchers'] = %w(postgresql)
+
+pgbouncer['users'] = {
+  'gitlab-consul': {
+    password: '5e0e3263571e3704ad655076301d6ebe'
+  },
+  'pgbouncer': {
+    password: '771a8625958a529132abe6f1a4acb19c
+  }
+}
+
+consul['configuration'] = {
+  retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23)
+}
+```
+
+[Reconfigure Omnibus GitLab][reconfigure Gitlab] for the changes to take effect.
+
+##### Example minimal setup manual steps
+
+The manual steps for this configuration are the same as for the [example recommended setup](#example_recommended_setup_manual_steps).
+
 ### Failover procedure
 
 By default, if the master database fails, `repmgrd` should promote one of the