Merge branch 'docs-repo-merge-8-admin-ha' into 'master'

Docs: Merge EE doc/administration/high_availability to CE

See merge request gitlab-org/gitlab-ce!27985

doc/administration/high_availability/README.md

-# High Availability
+# Scaling and High Availability
 
 GitLab supports several different types of clustering and high-availability.
 The solution you choose will be based on the level of scalability and
@@ -13,47 +13,173 @@ of Git, developers can still commit code locally even when GitLab is not
 available. However, some GitLab features such as the issue tracker and
 Continuous Integration are not available when GitLab is down.
 
-**Keep in mind that all Highly Available solutions come with a trade-off between
+**Keep in mind that all highly-available solutions come with a trade-off between
 cost/complexity and uptime**. The more uptime you want, the more complex the
 solution. And the more complex the solution, the more work is involved in
 setting up and maintaining it. High availability is not free and every HA
 solution should balance the costs against the benefits.
 
-## Architecture
+There are many options when choosing a highly-available GitLab architecture. We
+recommend engaging with GitLab Support to choose the best architecture for your
+use-case. This page contains some various options and guidelines based on
+experience with GitLab.com and Enterprise Edition on-premises customers.
 
-There are two kinds of setups:
+For a detailed insight into how GitLab scales and configures GitLab.com, you can
+watch [this 1 hour Q&A](https://www.youtube.com/watch?v=uCU8jdYzpac)
+with [John Northrup](https://gitlab.com/northrup), one of our infrastructure
+engineers, and live questions coming in from some of our customers.
 
-- active/active
-- active/passive
+## GitLab Components
 
-### Active/Active
+The following components need to be considered for a scaled or highly-available
+environment. In many cases components can be combined on the same nodes to reduce
+complexity.
 
-This architecture scales easily because all application servers handle
-user requests simultaneously. The database, Redis, and GitLab application are
-all deployed on separate servers. The configuration is **only** highly-available
-if the database, Redis and storage are also configured as such.
+- Unicorn/Workhorse - Web-requests (UI, API, Git over HTTP)
+- Sidekiq - Asynchronous/Background jobs
+- PostgreSQL - Database
+  - Consul - Database service discovery and health checks/failover
+  - PGBouncer - Database pool manager
+- Redis - Key/Value store (User sessions, cache, queue for Sidekiq)
+  - Sentinel - Redis health check/failover manager
+- Gitaly - Provides high-level RPC access to Git repositories
 
-Follow the steps below to configure an active/active setup:
+## Scalable Architecture Examples
+
+When an organization reaches a certain threshold it will be necessary to scale
+the GitLab instance. Still, true high availability may not be necessary. There
+are options for scaling GitLab instances relatively easily without incurring the
+infrastructure and maintenance costs of full high availability.
+
+### Basic Scaling
+
+This is the simplest form of scaling and will work for the majority of
+cases. Backend components such as PostgreSQL, Redis and storage are offloaded
+to their own nodes while the remaining GitLab components all run on 2 or more
+application nodes.
+
+This form of scaling also works well in a cloud environment when it is more
+cost-effective to deploy several small nodes rather than a single
+larger one.
+
+- 1 PostgreSQL node
+- 1 Redis node
+- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq)
+- 1 NFS/Gitaly storage server
+
+#### Installation Instructions
+
+Complete the following installation steps in order. A link at the end of each
+section will bring you back to the Scalable Architecture Examples section so
+you can continue with the next step.
+
+1. [PostgreSQL](./database.md#postgresql-in-a-scaled-environment)
+1. [Redis](./redis.md#redis-in-a-scaled-environment)
+1. [Gitaly](./gitaly.md) (recommended) or [NFS](./nfs.md)
+1. [GitLab application nodes](./gitlab.md)
+
+### Full Scaling
+
+For very large installations it may be necessary to further split components
+for maximum scalability. In a fully-scaled architecture the application node
+is split into separate Sidekiq and Unicorn/Workhorse nodes. One indication that
+this architecture is required is if Sidekiq queues begin to periodically increase
+in size, indicating that there is contention or not enough resources.
+
+- 1 PostgreSQL node
+- 1 Redis node
+- 2 or more GitLab application nodes (Unicorn, Workhorse)
+- 2 or more Sidekiq nodes
+- 2 or more NFS/Gitaly storage servers
+
+## High Availability Architecture Examples
+
+When organizations require scaling *and* high availability the following
+architectures can be utilized. As the introduction section at the top of this
+page mentions, there is a tradeoff between cost/complexity and uptime. Be sure
+this complexity is absolutely required before taking the step into full
+high availability.
+
+For all examples below, we recommend running Consul and Redis Sentinel on
+dedicated nodes. If Consul is running on PostgreSQL nodes or Sentinel on
+Redis nodes there is a potential that high resource usage by PostgreSQL or
+Redis could prevent communication between the other Consul and Sentinel nodes.
+This may lead to the other nodes believing a failure has occurred and automated
+failover is necessary. Isolating them from the services they monitor reduces
+the chances of split-brain.
+
+The examples below do not really address high availability of NFS. Some enterprises
+have access to NFS appliances that manage availability. This is the best case
+scenario. In the future, GitLab may offer a more user-friendly solution to
+[GitLab HA Storage](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2472).
+
+There are many options in between each of these examples. Work with GitLab Support
+to understand the best starting point for your workload and adapt from there.
+
+### Horizontal
+
+This is the simplest form of high availability and scaling. It requires the
+fewest number of individual servers (virtual or physical) but does have some
+trade-offs and limits.
+
+This architecture will work well for many GitLab customers. Larger customers
+may begin to notice certain events cause contention/high load - for example,
+cloning many large repositories with binary files, high API usage, a large
+number of enqueued Sidekiq jobs, etc. If this happens you should consider
+moving to a hybrid or fully distributed architecture depending on what is causing
+the contention.
+
+- 3 PostgreSQL nodes
+- 2 Redis nodes
+- 3 Consul/Sentinel nodes
+- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PGBouncer)
+- 1 NFS/Gitaly server
+
+![Horizontal architecture diagram](https://docs.gitlab.com/ee/administration/img/high_availability/horizontal.png)
+
+### Hybrid
+
+In this architecture, certain components are split on dedicated nodes so high
+resource usage of one component does not interfere with others. In larger
+environments this is a good architecture to consider if you foresee or do have
+contention due to certain workloads.
+
+- 3 PostgreSQL nodes
+- 2 Redis nodes
+- 3 Consul/Sentinel nodes
+- 2 or more Sidekiq nodes
+- 2 or more Web nodes (Unicorn, Workhorse, PGBouncer)
+- 1 or more NFS/Gitaly servers
+
+![Hybrid architecture diagram](https://docs.gitlab.com/ee/administration/img/high_availability/hybrid.png)
+
+### Fully Distributed
+
+This architecture scales to hundreds of thousands of users and projects and is
+the basis of the GitLab.com architecture. While this scales well it also comes
+with the added complexity of many more nodes to configure, manage and monitor.
+
+- 3 PostgreSQL nodes
+- 4 or more Redis nodes (2 separate clusters for persistent and cache data)
+- 3 Consul nodes
+- 3 Sentinel nodes
+- Multiple dedicated Sidekiq nodes (Split into real-time, best effort, ASAP,
+  CI Pipeline and Pull Mirror sets)
+- 2 or more Git nodes (Git over SSH/Git over HTTP)
+- 2 or more API nodes (All requests to `/api`)
+- 2 or more Web nodes (All other web requests)
+- 2 or more NFS/Gitaly servers
+
+![Fully Distributed architecture diagram](https://docs.gitlab.com/ee/administration/img/high_availability/fully-distributed.png)
+
+The following pages outline the steps necessary to configure each component
+separately:
 
 1. [Configure the database](database.md)
 1. [Configure Redis](redis.md)
    1. [Configure Redis for GitLab source installations](redis_source.md)
 1. [Configure NFS](nfs.md)
+   1. [NFS Client and Host setup](nfs_host_client_setup.md)
 1. [Configure the GitLab application servers](gitlab.md)
 1. [Configure the load balancers](load_balancer.md)
 
-### Active/Passive
-
-For pure high-availability/failover with no scaling you can use an
-active/passive configuration. This utilizes DRBD (Distributed Replicated
-Block Device) to keep all data in sync. DRBD requires a low latency link to
-remain in sync. It is not advisable to attempt to run DRBD between data centers
-or in different cloud availability zones.
-
-> **Note:** GitLab recommends against choosing this HA method because of the
-  complexity of managing DRBD and crafting automatic failover. This is
-  *compatible* with GitLab, but not officially *supported*. If you are 
-  an EE customer, support will help you with GitLab related problems, but if the
-  root cause is identified as DRBD, we will not troubleshoot further.
-
-Components/Servers Required: 2 servers/virtual machines (one active/one passive)

doc/administration/high_availability/alpha_database.md

+---
+redirect_to: 'database.md'
+---
+
+This documentation has been moved to the main
+[database documentation](database.md#configure_using_omnibus_for_high_availability).

doc/administration/high_availability/consul.md

+# Working with the bundled Consul service **[PREMIUM ONLY]**
+
+## Overview
+
+As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](http://consul.io) that can be managed through `/etc/gitlab/gitlab.rb`.
+
+A Consul cluster consists of multiple server agents, as well as client agents that run on other nodes which need to talk to the consul cluster.
+
+## Operations
+
+### Checking cluster membership
+
+To see which nodes are part of the cluster, run the following on any member in the cluster
+```
+# /opt/gitlab/embedded/bin/consul members
+Node            Address               Status  Type    Build  Protocol  DC
+consul-b        XX.XX.X.Y:8301        alive   server  0.9.0  2         gitlab_consul
+consul-c        XX.XX.X.Y:8301        alive   server  0.9.0  2         gitlab_consul
+consul-c        XX.XX.X.Y:8301        alive   server  0.9.0  2         gitlab_consul
+db-a            XX.XX.X.Y:8301        alive   client  0.9.0  2         gitlab_consul
+db-b            XX.XX.X.Y:8301        alive   client  0.9.0  2         gitlab_consul
+```
+
+Ideally all nodes will have a `Status` of `alive`.
+
+### Restarting the server cluster
+
+**Note**: This section only applies to server agents. It is safe to restart client agents whenever needed.
+
+If it is necessary to restart the server cluster, it is important to do this in a controlled fashion in order to maintain quorum. If quorum is lost, you will need to follow the consul [outage recovery](#outage-recovery) process to recover the cluster.
+
+To be safe, we recommend you only restart one server agent at a time to ensure the cluster remains intact.
+
+For larger clusters, it is possible to restart multiple agents at a time. See the [Consul consensus document](https://www.consul.io/docs/internals/consensus.html#deployment-table) for how many failures it can tolerate. This will be the number of simulateneous restarts it can sustain.
+
+## Troubleshooting
+
+### Consul server agents unable to communicate
+
+By default, the server agents will attempt to [bind](https://www.consul.io/docs/agent/options.html#_bind) to '0.0.0.0', but they will advertise the first private IP address on the node for other agents to communicate with them. If the other nodes cannot communicate with a node on this address, then the cluster will have a failed status.
+
+You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue:
+
+```
+2017-09-25_19:53:39.90821     2017/09/25 19:53:39 [WARN] raft: no known peers, aborting election
+2017-09-25_19:53:41.74356     2017/09/25 19:53:41 [ERR] agent: failed to sync remote state: No cluster leader
+```
+
+
+To fix this:
+
+1. Pick an address on each node that all of the other nodes can reach this node through.
+1. Update your `/etc/gitlab/gitlab.rb`
+
+    ```ruby
+    consul['configuration'] = {
+      ...
+      bind_addr: 'IP ADDRESS'
+    }
+    ```
+1. Run `gitlab-ctl reconfigure`
+
+If you still see the errors, you may have to [erase the consul database and reinitialize](#recreate-from-scratch) on the affected node.
+
+### Consul agents do not start - Multiple private IPs
+
+In the case that a node has multiple private IPs the agent be confused as to which of the private addresses to advertise, and then immediately exit on start.
+
+You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue:
+
+```
+2017-11-09_17:41:45.52876 ==> Starting Consul agent...
+2017-11-09_17:41:45.53057 ==> Error creating agent: Failed to get advertise address: Multiple private IPs found. Please configure one.
+```
+
+To fix this:
+
+1. Pick an address on the node that all of the other nodes can reach this node through.
+1. Update your `/etc/gitlab/gitlab.rb`
+
+    ```ruby
+    consul['configuration'] = {
+      ...
+      bind_addr: 'IP ADDRESS'
+    }
+    ```
+1. Run `gitlab-ctl reconfigure`
+
+### Outage recovery
+
+If you lost enough server agents in the cluster to break quorum, then the cluster is considered failed, and it will not function without manual intervenetion.
+
+#### Recreate from scratch
+By default, GitLab does not store anything in the consul cluster that cannot be recreated. To erase the consul database and reinitialize
+
+```
+# gitlab-ctl stop consul
+# rm -rf /var/opt/gitlab/consul/data
+# gitlab-ctl start consul
+```
+
+After this, the cluster should start back up, and the server agents rejoin. Shortly after that, the client agents should rejoin as well.
+
+#### Recover a failed cluster
+If you have taken advantage of consul to store other data, and want to restore the failed cluster, please follow the [Consul guide](https://www.consul.io/docs/guides/outage.html) to recover a failed cluster.

doc/administration/high_availability/gitaly.md

+# Configuring Gitaly for Scaled and High Availability
+
+Gitaly does not yet support full high availability. However, Gitaly is quite
+stable and is in use on GitLab.com. Scaled and highly available GitLab environments
+should consider using Gitaly on a separate node. 
+
+See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to 
+track plans and progress toward high availability support. 
+
+This document is relevant for [Scaled Architecture](./README.md#scalable-architecture-examples)
+environments and [High Availability Architecture](./README.md#high-availability-architecture-examples). 
+
+## Running Gitaly on its own server
+
+Starting with GitLab 11.4, Gitaly is a replacement for NFS except
+when the [Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer)
+is used.
+
+NOTE: **Note:** While Gitaly can be used as a replacement for NFS, we do not recommend using EFS as it may impact GitLab's performance. Please review the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for more details.
+
+NOTE: **Note:** Gitaly network traffic is unencrypted so we recommend a firewall to
+restrict access to your Gitaly server.
+
+The steps below are the minimum necessary to configure a Gitaly server with
+Omnibus:
+
+1. SSH into the Gitaly server.
+1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab
+   package you want using **steps 1 and 2** from the GitLab downloads page.
+     - Do not complete any other steps on the download page.
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
+
+    Gitaly must trigger some callbacks to GitLab via GitLab Shell. As a result,
+    the GitLab Shell secret must be the same between the other GitLab servers and
+    the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gitlab-secrets.json`
+    from an existing GitLab server to the Gitaly server. Without this shared secret,
+    Git operations in GitLab will result in an API error.
+
+    > **NOTE:** In most or all cases the storage paths below end in `repositories` which is
+    different than `path` in `git_data_dirs` of Omnibus installations. Check the
+    directory layout on your Gitaly server to be sure.
+
+    ```ruby
+    # Enable Gitaly
+    gitaly['enable'] = true
+
+    ## Disable all other services
+    sidekiq['enable'] = false
+    gitlab_workhorse['enable'] = false
+    unicorn['enable'] = false
+    postgresql['enable'] = false
+    nginx['enable'] = false
+    prometheus['enable'] = false
+    alertmanager['enable'] = false
+    pgbouncer_exporter['enable'] = false
+    redis_exporter['enable'] = false
+    gitlab_monitor['enable'] = false
+
+    # Prevent database connections during 'gitlab-ctl reconfigure'
+    gitlab_rails['rake_cache_clear'] = false
+    gitlab_rails['auto_migrate'] = false
+
+    # Configure the gitlab-shell API callback URL. Without this, `git push` will
+    # fail. This can be your 'front door' GitLab URL or an internal load
+    # balancer.
+    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
+
+    # Make Gitaly accept connections on all network interfaces. You must use
+    # firewalls to restrict access to this address/port.
+    gitaly['listen_addr'] = "0.0.0.0:8075"
+    gitaly['auth_token'] = 'abc123secret'
+
+    gitaly['storage'] = [
+      { 'name' => 'default', 'path' => '/mnt/gitlab/default/repositories' },
+      { 'name' => 'storage1', 'path' => '/mnt/gitlab/storage1/repositories' },
+    ]
+
+    # To use tls for gitaly you need to add
+    gitaly['tls_listen_addr'] = "0.0.0.0:9999"
+    gitaly['certificate_path'] = "path/to/cert.pem"
+    gitaly['key_path'] = "path/to/key.pem"
+    ```
+
+Again, reconfigure (Omnibus) or restart (source).
+
+Continue configuration of other components by going back to:
+
+- [Scaled Architectures](./README.md#scalable-architecture-examples)
+- [High Availability Architectures](./README.md#high-availability-architecture-examples)

doc/administration/high_availability/gitlab.md

-# Configuring GitLab for HA
-
-Assuming you have already configured a [database](database.md), [Redis](redis.md), and [NFS](nfs.md), you can
-configure the GitLab application server(s) now. Complete the steps below
-for each GitLab application server in your environment.
+# Configuring GitLab Scaling and High Availability
 
 > **Note:** There is some additional configuration near the bottom for
   additional GitLab application servers. It's important to read and understand

doc/administration/high_availability/nfs_host_client_setup.md

+# Configuring NFS for GitLab HA
+
+Setting up NFS for a GitLab HA setup allows all applications nodes in a cluster
+to share the same files and maintain data consistency. Application nodes in an HA
+setup act as clients while the NFS server plays host.
+
+> Note: The instructions provided in this documentation allow for setting a quick
+proof of concept but will leave NFS as potential single point of failure and
+therefore not recommended for use in production. Explore options such as [Pacemaker
+and Corosync](http://clusterlabs.org/) for highly available NFS in production.
+
+Below are instructions for setting up an application node(client) in an HA cluster
+to read from and write to a central NFS server(host).
+
+NOTE: **Note:**
+Using EFS may negatively impact performance. Please review the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for additional details.
+
+## NFS Server Setup
+
+> Follow the instructions below to set up and configure your NFS server.
+
+### Step 1 - Install NFS Server on Host
+
+Installing the nfs-kernel-server package allows you to share directories with the clients running the GitLab application.
+
+```sh
+apt-get update
+apt-get install nfs-kernel-server
+```
+
+### Step 2 - Export Host's Home Directory to Client
+
+In this setup we will share the home directory on the host with the client. Edit the exports file as below to share the host's home directory with the client. If you have multiple clients running GitLab you must enter the client IP addresses in line in the `/etc/exports` file.
+
+```text
+#/etc/exports for one client
+/home <client-ip-address>(rw,sync,no_root_squash,no_subtree_check)
+
+#/etc/exports for three clients
+/home <client-ip-address>(rw,sync,no_root_squash,no_subtree_check) <client-2-ip-address>(rw,sync,no_root_squash,no_subtree_check) <client-3-ip-address>(rw,sync,no_root_squash,no_subtree_check)
+```
+
+Restart the NFS server after making changes to the `exports` file for the changes
+to take effect.
+
+```sh
+systemctl restart nfs-kernel-server
+```
+
+NOTE: **Note:**
+You may need to update your server's firewall. See the [firewall section](#nfs-in-a-firewalled-environment) at the end of this guide.
+
+## Client/ GitLab application node Setup
+
+> Follow the instructions below to connect any GitLab rails application node running
+inside your HA environment to the NFS server configured above.
+
+### Step 1 - Install NFS Common on Client
+
+The nfs-common provides NFS functionality without installing server components which
+we don't need running on the application nodes.
+
+```sh
+apt-get update
+apt-get install nfs-common
+```
+
+### Step 2 - Create Mount Points on Client
+
+Create a directroy on the client that we can mount the shared directory from the host.
+Please note that if your mount point directory contains any files they will be hidden
+once the remote shares are mounted. An empty/new directory on the client is recommended
+for this purpose.
+
+```sh
+mkdir -p /nfs/home
+```
+
+Confirm that the mount point works by mounting it on the client and checking that
+it is mounted with the command below:
+
+```sh
+mount <host_ip_address>:/home
+df -h
+```
+
+### Step 3 - Set up Automatic Mounts on Boot
+
+Edit `/etc/fstab` on client as below to mount the remote shares automatically at boot.
+Note that GitLab requires advisory file locking, which is only supported natively in
+NFS version 4. NFSv3 also supports locking as long as Linux Kernel 2.6.5+ is used.
+We recommend using version 4 and do not specifically test NFSv3.
+
+```text
+#/etc/fstab
+165.227.159.85:/home       /nfs/home      nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
+```
+
+Reboot the client and confirm that the mount point is mounted automatically.
+
+### Step 4 - Set up GitLab to Use NFS mounts
+
+When using the default Omnibus configuration you will need to share 5 data locations
+between all GitLab cluster nodes. No other locations should be shared. Changing the
+default file locations in `gitlab.rb` on the client allows you to have one main mount
+point and have all the required locations as subdirectories to use the NFS mount for
+git-data.
+
+```text
+git_data_dirs({"default" => {"path" => "/nfs/home/var/opt/gitlab-data/git-data"}})
+gitlab_rails['uploads_directory'] = '/nfs/home/var/opt/gitlab-data/uploads'
+gitlab_rails['shared_path'] = '/nfs/home/var/opt/gitlab-data/shared'
+gitlab_ci['builds_directory'] = '/nfs/home/var/opt/gitlab-data/builds'
+```
+
+Save the changes in `gitlab.rb` and run `gitlab-ctl reconfigure`.
+
+## NFS in a Firewalled Environment
+
+If the traffic between your NFS server and NFS client(s) is subject to port filtering
+by a firewall, then you will need to reconfigure that firewall to allow NFS communication.
+
+[This guide from TDLP](http://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS)
+covers the basics of using NFS in a firewalled environment. Additionally, we encourage you to
+search for and review the specific documentation for your OS/distro and your firewall software.
+
+Example for Ubuntu:
+
+Check that NFS traffic from the client is allowed by the firewall on the host by running
+the command: `sudo ufw status`. If it's being blocked, then you can allow traffic from a specific
+client with the command below.
+
+```sh
+sudo ufw allow from <client-ip-address> to any port nfs
+```

doc/administration/high_availability/pgbouncer.md

+# Working with the bundle Pgbouncer service
+
+## Overview
+
+As part of its High Availability stack, GitLab Premium includes a bundled version of [Pgbouncer](https://pgbouncer.github.io/) that can be managed through `/etc/gitlab/gitlab.rb`.
+
+In a High Availability setup, Pgbouncer is used to seamlessly migrate database connections between servers in a failover scenario.
+
+Additionally, it can be used in a non-HA setup to pool connections, speeding up response time while reducing resource usage.
+
+It is recommended to run pgbouncer alongside the `gitlab-rails` service, or on its own dedicated node in a cluster.
+
+## Operations
+
+### Running Pgbouncer as part of an HA GitLab installation
+
+See our [HA documentation for PostgreSQL](database.md) for information on running pgbouncer as part of a HA setup
+
+### Running Pgbouncer as part of a non-HA GitLab installation
+
+1. Generate PGBOUNCER_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 pgbouncer`
+
+1. Generate SQL_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 gitlab`. We'll also need to enter the plaintext SQL_USER_PASSWORD later
+
+1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb`
+
+    ```ruby
+    postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH'
+    postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH'
+    postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on
+    postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node
+    ```
+
+1. Run `gitlab-ctl reconfigure`
+
+    **Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`.
+
+1. On the node you are running pgbouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb`
+
+    ```ruby
+    pgbouncer['enable'] = true
+    pgbouncer['databases'] = {
+      gitlabhq_production: {
+        host: 'DATABASE_HOST',
+        user: 'pgbouncer',
+        password: 'PGBOUNCER_USER_PASSWORD_HASH'
+      }
+    }
+    ```
+
+1. Run `gitlab-ctl reconfigure`
+
+1. On the node running unicorn, make sure the following is set in `/etc/gitlab/gitlab.rb`
+
+    ```ruby
+    gitlab_rails['db_host'] = 'PGBOUNCER_HOST'
+    gitlab_rails['db_port'] = '6432'
+    gitlab_rails['db_password'] = 'SQL_USER_PASSWORD'
+    ```
+
+1. Run `gitlab-ctl reconfigure`
+
+1. At this point, your instance should connect to the database through pgbouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section
+
+### Interacting with pgbouncer
+
+#### Administrative console
+
+As part of omnibus-gitlab, we provide a command `gitlab-ctl pgb-console` to automatically connect to the pgbouncer administrative console. Please see the [pgbouncer documentation](https://pgbouncer.github.io/usage.html#admin-console) for detailed instructions on how to interact with the console.
+
+To start a session, run
+
+```shell
+# gitlab-ctl pgb-console
+Password for user pgbouncer:
+psql (9.6.8, server 1.7.2/bouncer)
+Type "help" for help.
+
+pgbouncer=#
+```
+
+The password you will be prompted for is the PGBOUNCER_USER_PASSWORD
+
+To get some basic information about the instance, run
+```shell
+pgbouncer=# show databases; show clients; show servers;
+        name         |   host    | port |      database       | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections
+---------------------+-----------+------+---------------------+------------+-----------+--------------+-----------+-----------------+---------------------
+ gitlabhq_production | 127.0.0.1 | 5432 | gitlabhq_production |            |       100 |            5 |           |               0 |                   1
+ pgbouncer           |           | 6432 | pgbouncer           | pgbouncer  |         2 |            0 | statement |               0 |                   0
+(2 rows)
+
+ type |   user    |      database       | state  |   addr    | port  | local_addr | local_port |    connect_time     |    request_time     |    ptr    | link
+| remote_pid | tls
+------+-----------+---------------------+--------+-----------+-------+------------+------------+---------------------+---------------------+-----------+------
++------------+-----
+ C    | gitlab    | gitlabhq_production | active | 127.0.0.1 | 44590 | 127.0.0.1  |       6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12444c0 |
+|          0 |
+ C    | gitlab    | gitlabhq_production | active | 127.0.0.1 | 44592 | 127.0.0.1  |       6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12447c0 |
+|          0 |
+ C    | gitlab    | gitlabhq_production | active | 127.0.0.1 | 44594 | 127.0.0.1  |       6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x1244940 |
+|          0 |
+ C    | gitlab    | gitlabhq_production | active | 127.0.0.1 | 44706 | 127.0.0.1  |       6432 | 2018-04-24 22:14:22 | 2018-04-24 22:16:31 | 0x1244ac0 |
+|          0 |
+ C    | gitlab    | gitlabhq_production | active | 127.0.0.1 | 44708 | 127.0.0.1  |       6432 | 2018-04-24 22:14:22 | 2018-04-24 22:15:15 | 0x1244c40 |
+|          0 |
+ C    | gitlab    | gitlabhq_production | active | 127.0.0.1 | 44794 | 127.0.0.1  |       6432 | 2018-04-24 22:15:15 | 2018-04-24 22:15:15 | 0x1244dc0 |
+|          0 |
+ C    | gitlab    | gitlabhq_production | active | 127.0.0.1 | 44798 | 127.0.0.1  |       6432 | 2018-04-24 22:15:15 | 2018-04-24 22:16:31 | 0x1244f40 |
+|          0 |
+ C    | pgbouncer | pgbouncer           | active | 127.0.0.1 | 44660 | 127.0.0.1  |       6432 | 2018-04-24 22:13:51 | 2018-04-24 22:17:12 | 0x1244640 |
+|          0 |
+(8 rows)
+
+ type |  user  |      database       | state |   addr    | port | local_addr | local_port |    connect_time     |    request_time     |    ptr    | link | rem
+ote_pid | tls
+------+--------+---------------------+-------+-----------+------+------------+------------+---------------------+---------------------+-----------+------+----
+--------+-----
+ S    | gitlab | gitlabhq_production | idle  | 127.0.0.1 | 5432 | 127.0.0.1  |      35646 | 2018-04-24 22:15:15 | 2018-04-24 22:17:10 | 0x124dca0 |      |
+  19980 |
+(1 row)
+```
+
+## Troubleshooting
+
+In case you are experiencing any issues connecting through pgbouncer, the first place to check is always the logs:
+
+```shell
+# gitlab-ctl tail pgbouncer
+```
+
+Additionally, you can check the output from `show databases` in the [Administrative console](#administrative-console). In the output, you would expect to see values in the `host` field for the `gitlabhq_production` database. Additionally, `current_connections` should be greater than 1.