> Note: GitLab HA requires an Enterprise Edition Premium license
# Omnibus GitLab PostgreSQL High Availability
**Warning**
This functionality should be considered beta, use with caution.
**Warning**
> Available in [Omnibus GitLab Enterprise Edition Premium](https://about.gitlab.com/gitlab-ee/).
CAUTION: **Warning:**
This functionality should be considered **beta**, use with caution.
## Overview
GitLab supports multiple options for its database backend
* Using the Omnibus GitLab package to configure PG in HA setup (EEP only). This document contains directions for EEP users.
* Using GitLab with an [externally managed PostgreSQL service](../external_database.md). This could be a cloud provider, or your own service.
or for a non-HA option
* Using the Omnibus Gitlab CE/EES package with a [single PostgreSQL instance](http://docs.gitlab.com/omnibus/settings/database.html).
GitLab supports multiple options for its database backend:
## Configure Omnibus GitLab package database HA (Enterprise Edition Premium)
1. Using the Omnibus GitLab package to configure PG in HA setup (Enterprise Premium only).
1. Using GitLab with an [externally managed PostgreSQL service](../external_database.md).
This could be a cloud provider, your own service, or for a non-HA option.
1. Using the Omnibus Gitlab Community or Enterprise Starter Edition packages with
a [single PostgreSQL instance](http://docs.gitlab.com/omnibus/settings/database.html).
This document focuses on the first option.
## Preparation
The recommended configuration for a PostgreSQL HA setup requires:
### Preparation
- A minimum of three consul server nodes
- A minimum of two database nodes
- Each node will run the following services:
- PostgreSQL - The database itself
- repmgrd - A service to monitor, and handle failover in case of a master failure
- Consul - Used for service discovery, to alert other nodes when failover occurs
- At least one separate node for running the `pgbouncer` service.
## Required information
**Network information for all nodes**
- DNS names - By default, `repmgr` and `pgbouncer` use DNS to locate nodes
- IP address - PostgreSQL does not listen on any network interface by default.
It needs to know which IP address to listen on in order to use the network
interface. It can be set to `0.0.0.0` to listen on all interfaces. It cannot
be set to the loopack address `127.0.0.1`
- Network Address - PostgreSQL access is controlled based on the network source.
This can be in subnet (i.e. `192.168.0.0/255.255.255.0`) or CIDR (i.e.
`192.168.0.0/24`) form.
**User information for `pgbouncer` service**
- The service runs as the same user as the database, default of `gitlab-psql`
- The service will have a regular database user account generated for it
- Default username is `pgbouncer`. In the rest of the documentation we will
refer to this username as `PGBOUNCER_USERNAME`
- Password for `pgbouncer` service. In the rest of the documentation we will
refer to this password as `PGBOUNCER_PASSWORD`
- Password hash for `pgbouncer` service generated from the `pgbouncer` username
and password pair with:
```sh
echo -n 'PASSWORD+USERNAME' | md5sum
```
The recommended configuration for a PostgreSQL HA setup requires:
* A minimum of three consul server nodes
* A minimum of two database nodes
* Each node will run the following services
* postgresql -- The database itself
* repmgrd -- A service to monitor, and handle failover in case of a master failure
* consul -- Used for service discovery, to alert other nodes when failover occurs
* At least one separate node for running the `pgbouncer` service.
#### Required information
* Network information for all nodes
* DNS names -- By default, `repmgr` and `pgbouncer` use DNS to locate nodes
* IP address -- PostgreSQL does not listen on any network interface by default. It needs to know which IP address to listen on in order to use the network interface. It can be set to `0.0.0.0` to listen on all interfaces. It cannot be set to the loopack address 127.0.0.1
* Network Address -- PostgreSQL access is controlled based on the network source. This can be in subnet (i.e. 192.168.0.0/255.255.255.0) or CIDR (i.e. 192.168.0.0/24) form.
* User information for `pgbouncer` service
* The service runs as the same user as the database, default of `gitlab-psql`
* The service will have a regular database user account generated for it
* Default username is `pgbouncer`. In the rest of the documentation we will refer to this username as `PGBOUNCER_USERNAME`
* Password for `pgbouncer` service. In the rest of the documentation we will refer to this password as `PGBOUNCER_PASSWORD`
* Password hash for `pgbouncer` service
* This should be generated from `pgbouncer` username and password pair
* Generate the hash with:
``
$ echo -n 'PASSWORD+USERNAME' | md5sum
``
* In the rest of the documentation we will refer to this hash as `PGBOUNCER_PASSWORD_HASH`
* This password will be stored in the following locations
*`/etc/gitlab/gitlab.rb`: hashed, and in plain text
*`/var/opt/gitlab/pgbouncer/pg_auth`: hashed
* User information for the Repmgr service
* The service runs under the same system account as the database by default.
* The service requires a superuser database account be generated for it. This defaults to `gitlab_repmgr`
* User information for the Consul service
* The consul service runs under a dedicated system account by default, `gitlab-consul`. In the rest of the documentation we will refer to this username as `CONSUL_USERNAME`
* There will be a database user created with read only access to the repmgr database
* Password for the database user. In the rest of the documentation we will refer to this password as `CONSUL_DATABASE_PASSWORD`
* Password hash for `gitlab-consul` service
* This should be generated from `gitlab-consul` username and password pair
* Generate the hash with:
``
$ echo -n 'PASSWORD+USERNAME' | md5sum
``
* In the rest of the documentation we will refer to this hash as `CONSUL_PASSWORD_HASH`
* This password will be stored in the following locations
* When configuring PostgreSQL, we will set `max_wal_senders` to one more than this number. This is used to prevent replication from using up all of the available database connections.
### Installation
#### On each node
1. Download/install GitLab Omnibus using **steps 1 and 2** from
[GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
steps on the download page.
#### Configuration
Each node needs to be configured to run only the services it needs. Create an `/etc/gitlab/gitlab.rb` on each node which looks like the following, then run `gitlab-ctl reconfigure`
##### On each consul server node
```ruby
# Disable all components except Consul
bootstrap['enable'] = false
gitaly['enable'] = false
gitlab_workhorse['enable'] = false
mailroom['enable'] = false
nginx['enable'] = false
postgresql['enable'] = false
redis['enable'] = false
sidekiq['enable'] = false
unicorn['enable'] = false
consul['enable'] = true
# START user configuration
# Please set the real values as explained in Required Information section
#
consul['configuration'] = {
In the rest of the documentation we will refer to this hash as `PGBOUNCER_PASSWORD_HASH`
- This password will be stored in the following locations:
-`/etc/gitlab/gitlab.rb`: hashed, and in plain text
-`/var/opt/gitlab/pgbouncer/pg_auth`: hashed
**User information for the Repmgr service**
- The service runs under the same system account as the database by default.
- The service requires a superuser database account be generated for it. This
defaults to `gitlab_repmgr`
**User information for the Consul service**
- The consul service runs under a dedicated system account by default,
`gitlab-consul`. In the rest of the documentation we will refer to this
username as `CONSUL_USERNAME`
- There will be a database user created with read only access to the repmgr
database
- Password for the database user. In the rest of the documentation we will
refer to this password as `CONSUL_DATABASE_PASSWORD`
- Password hash for `gitlab-consul` service generated from the `gitlab-consul`
username and password pair with:
```sh
echo -n 'PASSWORD+USERNAME' | md5sum
```
In the rest of the documentation we will refer to this hash as `CONSUL_PASSWORD_HASH`
- This password will be stored in the following locations:
1. It may be necessary to manually run migrations.
```
# gitlab-rake db:migrate
1. It may be necessary to manually run migrations:
```sh
sudo gitlab-rake gitlab:db:configure
```
#### Server running
At this point, your GitLab instance should be up and running, verify you are able to login, and create issues and merge requests.
## Ensure GitLab is running
At this point, your GitLab instance should be up and running. Verify you are
able to login, and create issues and merge requests.
### Failover procedure
By default, if the master database fails, repmgrd should promote one of the standby nodes to master automatically, and consul will update pgbouncer with the new master.
## Failover procedure
By default, if the master database fails, `repmgrd` should promote one of the
standby nodes to master automatically, and consul will update pgbouncer with
the new master.
If you need to failover manually, you have two options:
1. Shutdown the current master database
```
# gitlab-ctl stop postgresql
```
The automated failover process will see this and failover to one of the standby nodes.
1. Manually failover
1. Ensure the old master node is not still active.
**Shutdown the current master database**
1. Login to the server that should become the new master and run the following
```
# gitlab-ctl repmgr standby promote
```
Run:
1. If there are any other standby servers in the cluster, have them follow the new master server
```sh
sudo gitlab-ctl stop postgresql
```
The automated failover process will see this and failover to one of the
standby nodes.
**Or perform a manual failover**
1. Ensure the old master node is not still active.
1. Login to the server that should become the new master and run:
```sh
sudo gitlab-ctl repmgr standby promote
```
# gitlab-ctl repmgr standby follow NEW_MASTER
1. If there are any other standby servers in the cluster, have them follow
the new master server:
```sh
sudo gitlab-ctl repmgr standby follow NEW_MASTER
```
### Restore procedure
If a node fails, it can be removed from the cluster, or added back as a standby after it has been restored to service.
## Restore procedure
* If you want to remove the node from the cluster, on any other node in the cluster, run:
If a node fails, it can be removed from the cluster, or added back as a standby
after it has been restored to service.
- If you want to remove the node from the cluster, on any other node in the
[^1]: **Warning**: When the server is brought back online, and before you switch it to a standby node, repmgr will report that there are two masters.
If there are any clients that are still attempting to write to the old master, this will cause a split, and the old master will need to be resynced from scratch by performing a `standby setup NEW_MASTER`.
1. On each pgbouncer node, edit `/etc/gitlab/gitlab.rb`:
1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for
the `gitlab` database user
1. [Reconfigure GitLab] for the changes to take effect
---
...
...
@@ -335,3 +518,5 @@ Read more on high-availability configuration:
1.[Configure NFS](nfs.md)
1.[Configure the GitLab application servers](gitlab.md)
1.[Configure the load balancers](load_balancer.md)