Commit fb0dff6e authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Move some things over

parent c43b5400
# Configuring Redis for GitLab HA # Configuring Redis for GitLab HA
High Availability with Redis is possible using a **Master** x **Slave** High Availability with [Redis] is possible using a **Master** x **Slave**
topology with **Sentinel** service to watch and automatically start topology with a [Redis Sentinel][sentinel] service to watch and automatically
failover proceedings. start failover proceedings.
You can choose to install and manage Redis and Sentinel yourself, use You can choose to install and manage Redis and Sentinel yourself, use
a hosted, managed cloud solution or you can use or you can use the one a hosted cloud solution or you can use the one that comes bundled with
that comes bundled with Omnibus GitLab packages. Omnibus GitLab packages.
> **Note:** Redis requires authentication for High Availability. See > **Notes:**
- Redis requires authentication for High Availability. See
[Redis Security](http://redis.io/topics/security) documentation for more [Redis Security](http://redis.io/topics/security) documentation for more
information. We recommend using a combination of a Redis password and tight information. We recommend using a combination of a Redis password and tight
firewall rules to secure your Redis service. firewall rules to secure your Redis service.
- You are highly encouraged to read the [Redis Sentinel][sentinel] documentation
before configuring Redis HA with GitLab to fully understand the topology and
architecture.
<!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents** **Table of Contents**
- [Using an external Redis server](#using-an-external-redis-server) - [Overview](#overview)
- [High Availability with Sentinel](#high-availability-with-sentinel) - [Available setups](#available-setups)
- [Prerequisites](#prerequisites) - [Using a non-Omnibus external Redis server](#using-a-non-omnibus-external-redis-server)
- [Redis setup](#redis-setup) - [High Availability with Sentinel](#high-availability-with-sentinel)
- [Sentinel setup](#sentinel-setup) - [Prerequisites](#prerequisites)
- [Recommended setup](#recommended-setup) - [Recommended setup](#recommended-setup)
- [Redis HA configuration](#redis-ha-configuration)
- [Redis setup](#redis-setup)
- [Sentinel setup](#sentinel-setup)
- [Configuring instances using Omnibus](#configuring-instances-using-omnibus) - [Configuring instances using Omnibus](#configuring-instances-using-omnibus)
- [Existing single-machine installation](#existing-single-machine-installation) - [Existing single-machine installation](#existing-single-machine-installation)
- [Configuring Master Redis instance](#configuring-master-redis-instance) - [Configuring Master Redis instance](#configuring-master-redis-instance)
- [Configuring Slave Redis instances](#configuring-slave-redis-instances) - [Configuring Slave Redis instances](#configuring-slave-redis-instances)
- [Configuring Sentinel instances](#configuring-sentinel-instances) - [Configuring Sentinel instances](#configuring-sentinel-instances)
- [Community Edition](#community-edition) - [Community Edition](#community-edition)
- [Enterprise Edition](#enterprise-edition) - [Enterprise Edition](#enterprise-edition)
- [GitLab setup](#gitlab-setup) - [GitLab setup](#gitlab-setup)
- [Example Configurations](#example-configurations) - [Minimal example configuration with 1 master, 2 slaves and 3 sentinels](#minimal-example-configuration-with-1-master-2-slaves-and-3-sentinels)
- [Configuration for Redis Master](#configuration-for-redis-master) - [Configuration for Redis Master](#configuration-for-redis-master)
- [Configuration for Redis Slave](#configuration-for-redis-slave) - [Configuration for Redis Slave](#configuration-for-redis-slave)
- [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only) - [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only)
- [Control running services](#control-running-services) - [Control running services](#control-running-services)
- [Troubleshooting](#troubleshooting) - [Troubleshooting](#troubleshooting)
- [Redis replication](#redis-replication) - [Redis replication](#redis-replication)
- [Sentinel](#sentinel) - [Sentinel](#sentinel)
- [Omnibus GitLab](#omnibus-gitlab) - [Omnibus GitLab](#omnibus-gitlab)
- [Changelog](#changelog) - [Changelog](#changelog)
- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) - [Experimental Redis Sentinel support](#experimental-redis-sentinel-support)
<!-- END doctoc generated TOC please keep comment here to allow auto update --> <!-- END doctoc generated TOC please keep comment here to allow auto update -->
## Using an external Redis server
## Overview
Before diving into the details of setting up Redis and Redis Sentinel for HA,
make sure you read this section to better understand the underline architecture.
### Available setups
Based on your infrastructure setup, there are multiple ways to setup Redis HA
with GitLab. Omnibus GitLab packages have Redis and Redis Sentinel bundled with
them to save you the hassle to install it yourself. Pick the one that suits your
needs.
- **Installations from source:** You need to install Redis and Redis Sentinel
yourself. Use the [Redis HA source install](redis_source.md) guide.
- **Omnibus Community Edition (CE):** Redis is bundled so you can use the
package with only the Redis service enabled (works for both master and slave
setups).
- **Omnibus Enterprise Edition (EE):** Both Redis and Redis Sentinel are bundled
in the Omnibus package so you can use only them to setup the whole Redis HA
infrastructure (master, slave and Sentinel).
Note that with the Omnibus packages (both CE and EE), you can also use an
[external Redis server](#using-a-non-omnibus-external-redis-server).
### Using a non-Omnibus external Redis server
If you're hosting GitLab on a cloud provider, you can optionally use a If you're hosting GitLab on a cloud provider, you can optionally use a
managed service for Redis. For example, AWS offers a managed ElastiCache service managed service for Redis. For example, AWS offers a managed ElastiCache service
that runs Redis. that runs Redis.
Managed services can provide High Availability using their own proprietary Managed services can provide High Availability using their own proprietary
technology and provide a transparent proxy, which means that GitLab doesn't technology and provide a transparent proxy (which means that GitLab doesn't
need any additional change, or will use Sentinel and manage it for you. need any additional change) or they will use Sentinel and manage it for you.
If your provider, uses Sentinel method, see [GitLab Setup](#gitlab-setup) If your provider, uses Sentinel, see [GitLab Setup](#gitlab-setup)
to understand where you need to provide the list of servers and credentials. to understand where you need to provide the list of servers and credentials.
If you want to setup Redis by yourself, without using Omnibus, you can If you want to setup Redis by yourself, without using Omnibus, you can
read our documentation: [Configuring Redis for GitLab HA (source install)](redis_source.md). read the documentation on [configuring Redis HA for source installs](redis_source.md).
## High Availability with Sentinel
> Since GitLab `8.11`, you can configure a list of Redis Sentinel servers that ### High Availability with Sentinel
will monitor a group of Redis servers to provide failover support.
> With GitLab `8.14`, we bundled Redis Sentinel as part of Omnibus package and >
improved the way you use and configure it. - Since GitLab `8.11`, you can configure a list of Redis Sentinel servers that
will monitor a group of Redis servers to provide failover support.
- With GitLab `8.14`, we bundled Redis Sentinel as part of Omnibus package and
improved the way you use and configure it.
High Availability with Redis requires a few things: High Availability with Redis requires a few things:
...@@ -77,12 +109,12 @@ High Availability with Redis requires a few things: ...@@ -77,12 +109,12 @@ High Availability with Redis requires a few things:
- Application support and visibility to all Sentinel and Redis instances - Application support and visibility to all Sentinel and Redis instances
Redis Sentinel can handle the most important tasks in a HA environment to help Redis Sentinel can handle the most important tasks in a HA environment to help
keep servers online with minimal to no downtime: keep servers online with minimal to no downtime. Redis Sentinel:
- Monitors **Master** and **Slaves** instances to see if they are available - Monitors **Master** and **Slaves** instances to see if they are available
- Promote a **Slave** to **Master** when the **Master** fails - Promotes a **Slave** to **Master** when the **Master** fails
- Demote a **Master** to **Slave** when failed **Master** comes back online (to prevent - Demotes a **Master** to **Slave** when failed **Master** comes back online
data-partitioning) (to prevent data-partitioning)
- Can be queried by clients to always connect to the current **Master** server - Can be queried by clients to always connect to the current **Master** server
When a **Master** fails to respond, it's the client's responsibility to handle When a **Master** fails to respond, it's the client's responsibility to handle
...@@ -93,24 +125,16 @@ the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as ...@@ -93,24 +125,16 @@ the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
failing to configure it correctly can lead to data loss, or can bring your failing to configure it correctly can lead to data loss, or can bring your
whole cluster down, invalidating the failover effort. whole cluster down, invalidating the failover effort.
This documentation will provide you with a minimal and a recommended topology
that can resist to some levels of failure. Usually the more Redis and Sentinel
instances you have provisioned, the better will be your availability.
The configuration consists of three parts:
- Setup Redis Master and Slave nodes
- Setup Sentinel nodes
- Setup GitLab
### Prerequisites ### Prerequisites
You need at least `3` independent machines: physical, or VMs running into You need at least `3` independent machines: physical, or VMs running into
distinct physical machines. They must be believed to fail in an distinct physical machines. It is essential that all master and Redis slaves
independent way. run in different machines. If you fail to provision the machines in that
specific way, any issue with the shared environment can bring your entire setup
down.
If you fail to provision the machines in that specific way, any issue with It is OK to run a Sentinel along with a master or slave Redis instance.
the shared environment can bring your entire setup down. No more than one though.
You also need to take in consideration the underlying network topology, You also need to take in consideration the underlying network topology,
making sure you have redundant connectivity between Redis / Sentinel and making sure you have redundant connectivity between Redis / Sentinel and
...@@ -119,6 +143,30 @@ failure. ...@@ -119,6 +143,30 @@ failure.
Read carefully how to configure the components below. Read carefully how to configure the components below.
### Recommended setup
For a minimal setup, you will install the Omnibus GitLab package in `3`
independent machines, both with **Redis** and **Sentinel**:
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
Make sure you've read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup)
before, to understand how and why the amount of nodes came from.
For a recommended setup, that can resist more failures, you will install
the Omnibus GitLab package in `5` independent machines, both with
**Redis** and **Sentinel**:
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
## Redis HA configuration
### Redis setup ### Redis setup
You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they
...@@ -156,14 +204,14 @@ Initial **Slave** nodes requires `redis['master']` defined to `false` and ...@@ -156,14 +204,14 @@ Initial **Slave** nodes requires `redis['master']` defined to `false` and
simplified configuration by enabling `redis_slave_role['enable']`, you simplified configuration by enabling `redis_slave_role['enable']`, you
just need to fill in the `redis['master_ip']`. just need to fill in the `redis['master_ip']`.
This values doesn't have to be changed again in `/etc/gitlab/gitlab.rb` after This values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
a failover, as the nodes will be managed by the Sentinels, and even after a a failover, as the nodes will be managed by the Sentinels, and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by `gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels. the same Sentinels.
### Sentinel setup ### Sentinel setup
Sentinels watches both other sentinels and Redis nodes. Whenever a Sentinel Sentinels watch both other sentinels and Redis nodes. Whenever a Sentinel
detects that a Redis node is not responding, it will announce that to the detects that a Redis node is not responding, it will announce that to the
other sentinels. You have to reach the **quorum**, the minimum amount of other sentinels. You have to reach the **quorum**, the minimum amount of
sentinels that agrees that a node is down, to be able to start a failover. sentinels that agrees that a node is down, to be able to start a failover.
...@@ -222,33 +270,12 @@ official documentation: ...@@ -222,33 +270,12 @@ official documentation:
the slaves will be reconfigured by the Sentinels anyway, but not with the slaves will be reconfigured by the Sentinels anyway, but not with
the exact parallel-syncs progression as specified. the exact parallel-syncs progression as specified.
### Recommended setup
For a minimal setup, you will install the Omnibus GitLab package in `3`
independent machines, both with **Redis** and **Sentinel**:
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
Make sure you've read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup)
before, to understand how and why the amount of nodes came from.
For a recommended setup, that can resist more failures, you will install
the Omnibus GitLab package in `5` independent machines, both with
**Redis** and **Sentinel**:
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
## Configuring instances using Omnibus ## Configuring instances using Omnibus
This is a summary of what are we going to do: This is a summary of what are we going to do:
1. Provision the required number of instances specified previously 1. Provision the required number of instances specified previously
- You can opt to install Redis and Sentinel in the same machine or each in - You can opt to install Redis and Sentinel in the same machine or each in
independent ones. independent ones.
- Don't install Redis and Sentinel in the same machines your GitLab instance - Don't install Redis and Sentinel in the same machines your GitLab instance
...@@ -257,7 +284,7 @@ This is a summary of what are we going to do: ...@@ -257,7 +284,7 @@ This is a summary of what are we going to do:
connection over Redis (`6379`) and Sentinel (`26379`) ports. connection over Redis (`6379`) and Sentinel (`26379`) ports.
- GitLab machines must be able to access these machines and with the same - GitLab machines must be able to access these machines and with the same
permissions. permissions.
- Protected them from indiscriminating access from external networks (Internet), - Protect them from access from external networks (Internet),
to harden the security. to harden the security.
1. Download/install Omnibus GitLab using **steps 1 and 2** from 1. Download/install Omnibus GitLab using **steps 1 and 2** from
...@@ -394,7 +421,7 @@ which ideally should not have Redis or Sentinels in the same machine for a HA se ...@@ -394,7 +421,7 @@ which ideally should not have Redis or Sentinels in the same machine for a HA se
See [example configuration](#configuration-for-gitlab) below. See [example configuration](#configuration-for-gitlab) below.
## Example Configurations ## Minimal example configuration with 1 master, 2 slaves and 3 sentinels
In this example we consider that all servers have an internal network In this example we consider that all servers have an internal network
interface with IPs in the `10.0.0.x` range, and that they can connect interface with IPs in the `10.0.0.x` range, and that they can connect
...@@ -836,3 +863,5 @@ Read more on high-availability configuration: ...@@ -836,3 +863,5 @@ Read more on high-availability configuration:
[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure [reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
[gh-531]: https://github.com/redis/redis-rb/issues/531 [gh-531]: https://github.com/redis/redis-rb/issues/531
[gh-534]: https://github.com/redis/redis-rb/issues/534 [gh-534]: https://github.com/redis/redis-rb/issues/534
[redis]: http://redis.io/
[sentinel]: http://redis.io/topics/sentinel
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment