Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
Boxiang Sun
gitlab-ce
Commits
b575b004
Commit
b575b004
authored
Nov 10, 2016
by
Achilleas Pipinellis
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Rearrange sections in Sentinel docs
[ci skip]
parent
fb0dff6e
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
181 additions
and
162 deletions
+181
-162
doc/administration/high_availability/redis.md
doc/administration/high_availability/redis.md
+181
-162
No files found.
doc/administration/high_availability/redis.md
View file @
b575b004
...
@@ -16,82 +16,67 @@ Omnibus GitLab packages.
...
@@ -16,82 +16,67 @@ Omnibus GitLab packages.
-
You are highly encouraged to read the
[
Redis Sentinel
][
sentinel
]
documentation
-
You are highly encouraged to read the
[
Redis Sentinel
][
sentinel
]
documentation
before configuring Redis HA with GitLab to fully understand the topology and
before configuring Redis HA with GitLab to fully understand the topology and
architecture.
architecture.
-
This is the documentation for the Omnibus packages. For installations from
source, follow the
[
Redis HA source install
](
redis_source.md
)
guide.
-
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the
Omnibus Community Edition and installations from source, follow the
[
Redis HA source install
](
redis_source.md
)
guide.
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**
**Table of Contents**
-
[
Overview
](
#overview
)
-
[
Overview
](
#overview
)
-
[
Available setups
](
#available-setups
)
-
[
Using a non-Omnibus external Redis server
](
#using-a-non-omnibus-external-redis-server
)
-
[
High Availability with Sentinel
](
#high-availability-with-sentinel
)
-
[
Prerequisites
](
#prerequisites
)
-
[
Prerequisites
](
#prerequisites
)
-
[
High Availability with Sentinel
](
#high-availability-with-sentinel
)
-
[
Recommended setup
](
#recommended-setup
)
-
[
Recommended setup
](
#recommended-setup
)
-
[
Available configuration setups
](
#available-configuration-setups
)
-
[
Using a non-Omnibus external Redis server
](
#using-a-non-omnibus-external-redis-server
)
-
[
Redis setup overview
](
#redis-setup-overview
)
-
[
Sentinel setup overview
](
#sentinel-setup-overview
)
-
[
Redis HA configuration
](
#redis-ha-configuration
)
-
[
Redis HA configuration
](
#redis-ha-configuration
)
-
[
Redis setup
](
#redis-setup
)
-
[
Configuring the Master Redis instance
](
#configuring-the-master-redis-instance
)
-
[
Sentinel setup
](
#sentinel-setup
)
-
[
Configuring the Slave Redis instances
](
#configuring-the-slave-redis-instances
)
-
[
Configuring instances using Omnibus
](
#configuring-instances-using-omnibus
)
-
[
Configuring the Sentinel instances
](
#configuring-the-sentinel-instances
)
-
[
Existing single-machine installation
](
#existing-single-machine-installation
)
-
[
Configuring the GitLab application
](
#configuring-the-gitlab-application
)
-
[
Configuring Master Redis instance
](
#configuring-master-redis-instance
)
-
[
Switching from an existing single-machine installation to Redis HA
](
#switching-from-an-existing-single-machine-installation-to-redis-ha
)
-
[
Configuring Slave Redis instances
](
#configuring-slave-redis-instances
)
-
[
Configuring Sentinel instances
](
#configuring-sentinel-instances
)
-
[
Community Edition
](
#community-edition
)
-
[
Enterprise Edition
](
#enterprise-edition
)
-
[
GitLab setup
](
#gitlab-setup
)
-
[
Minimal example configuration with 1 master, 2 slaves and 3 sentinels
](
#minimal-example-configuration-with-1-master-2-slaves-and-3-sentinels
)
-
[
Minimal example configuration with 1 master, 2 slaves and 3 sentinels
](
#minimal-example-configuration-with-1-master-2-slaves-and-3-sentinels
)
-
[
Configuration for Redis Master
](
#configuration-for-redis-master
)
-
[
Configuration for Redis master
](
#configuration-for-redis-master
)
-
[
Configuration for Redis Slave
](
#configuration-for-redis-slave
)
-
[
Configuration for Redis slaves
](
#configuration-for-redis-slaves
)
-
[
Configuration for Sentinel (EE only)
](
#configuration-for-sentinel-ee-only
)
-
[
Configuration for Sentinels
](
#configuration-for-sentinels
)
-
[
Advanced configuration
](
#advanced-configuration
)
-
[
Control running services
](
#control-running-services
)
-
[
Control running services
](
#control-running-services
)
-
[
Troubleshooting
](
#troubleshooting
)
-
[
Troubleshooting
](
#troubleshooting
)
-
[
Redis replication
](
#redis-replication
)
-
[
Troubleshooting Redis replication
](
#troubleshooting-redis-replication
)
-
[
Sentinel
](
#sentinel
)
-
[
Troubleshooting Sentinel
](
#troubleshooting-sentinel
)
-
[
Omnibus GitLab
](
#omnibus-gitlab
)
-
[
Changelog
](
#changelog
)
-
[
Changelog
](
#changelog
)
-
[
Experimental Redis Sentinel support
](
#experimental-redis-sentinel-support
)
-
[
Experimental Redis Sentinel support
](
#experimental-redis-sentinel-support
)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## Overview
## Overview
Before diving into the details of setting up Redis and Redis Sentinel for HA,
Before diving into the details of setting up Redis and Redis Sentinel for HA,
make sure you read this section to better understand the underline architecture.
make sure you read this Overview section to better understand how the components
are tied together.
### Available setups
Based on your infrastructure setup, there are multiple ways to setup Redis HA
with GitLab. Omnibus GitLab packages have Redis and Redis Sentinel bundled with
them to save you the hassle to install it yourself. Pick the one that suits your
needs.
-
**Installations from source:**
You need to install Redis and Redis Sentinel
yourself. Use the
[
Redis HA source install
](
redis_source.md
)
guide.
-
**Omnibus Community Edition (CE):**
Redis is bundled so you can use the
package with only the Redis service enabled (works for both master and slave
setups).
-
**Omnibus Enterprise Edition (EE):**
Both Redis and Redis Sentinel are bundled
in the Omnibus package so you can use only them to setup the whole Redis HA
infrastructure (master, slave and Sentinel).
Note that with the Omnibus packages (both CE and EE), you can also use an
[
external Redis server
](
#using-a-non-omnibus-external-redis-server
)
.
###
Using a non-Omnibus external Redis server
###
Prerequisites
If you're hosting GitLab on a cloud provider, you can optionally use a
You need at least
`3`
independent machines: physical, or VMs running into
managed service for Redis. For example, AWS offers a managed ElastiCache service
distinct physical machines. It is essential that all master and slaves Redis
that runs Redis.
instances run in different machines. If you fail to provision the machines in
that specific way, any issue with the shared environment can bring your entire
setup down.
Managed services can provide High Availability using their own proprietary
It is OK to run a Sentinel along with a master or slave Redis instance.
technology and provide a transparent proxy (which means that GitLab doesn't
No more than one Sentinel in the same machine though.
need any additional change) or they will use Sentinel and manage it for you.
If your provider, uses Sentinel, see
[
GitLab Setup
](
#gitlab-setup
)
You also need to take in consideration the underlying network topology,
to understand where you need to provide the list of servers and credentials.
making sure you have redundant connectivity between Redis / Sentinel and
GitLab instances, otherwise the networks will become a single point of
failure.
If you want to setup Redis by yourself, without using Omnibus, you can
Read carefully how to configure the components below.
read the documentation on
[
configuring Redis HA for source installs
](
redis_source.md
)
.
### High Availability with Sentinel
### High Availability with Sentinel
...
@@ -108,12 +93,12 @@ High Availability with Redis requires a few things:
...
@@ -108,12 +93,12 @@ High Availability with Redis requires a few things:
-
Multiple Sentinel instances
-
Multiple Sentinel instances
-
Application support and visibility to all Sentinel and Redis instances
-
Application support and visibility to all Sentinel and Redis instances
Redis Sentinel can handle the most important tasks in a HA environment
to help
Redis Sentinel can handle the most important tasks in a HA environment
and that's
keep servers online with minimal to no downtime. Redis Sentinel:
to help
keep servers online with minimal to no downtime. Redis Sentinel:
-
Monitors
**Master**
and
**Slaves**
instances to see if they are available
-
Monitors
**Master**
and
**Slaves**
instances to see if they are available
-
Promotes a
**Slave**
to
**Master**
when the
**Master**
fails
-
Promotes a
**Slave**
to
**Master**
when the
**Master**
fails
-
Demotes a
**Master**
to
**Slave**
when failed
**Master**
comes back online
-
Demotes a
**Master**
to
**Slave**
when
the
failed
**Master**
comes back online
(to prevent data-partitioning)
(to prevent data-partitioning)
-
Can be queried by clients to always connect to the current
**Master**
server
-
Can be queried by clients to always connect to the current
**Master**
server
...
@@ -122,41 +107,23 @@ timeout and reconnect (querying a **Sentinel** for a new **Master**).
...
@@ -122,41 +107,23 @@ timeout and reconnect (querying a **Sentinel** for a new **Master**).
To get a better understanding on how to correctly setup Sentinel, please read
To get a better understanding on how to correctly setup Sentinel, please read
the
[
Redis Sentinel documentation
](
http://redis.io/topics/sentinel
)
first, as
the
[
Redis Sentinel documentation
](
http://redis.io/topics/sentinel
)
first, as
failing to configure it correctly can lead to data loss
,
or can bring your
failing to configure it correctly can lead to data loss or can bring your
whole cluster down, invalidating the failover effort.
whole cluster down, invalidating the failover effort.
### Prerequisites
You need at least
`3`
independent machines: physical, or VMs running into
distinct physical machines. It is essential that all master and Redis slaves
run in different machines. If you fail to provision the machines in that
specific way, any issue with the shared environment can bring your entire setup
down.
It is OK to run a Sentinel along with a master or slave Redis instance.
No more than one though.
You also need to take in consideration the underlying network topology,
making sure you have redundant connectivity between Redis / Sentinel and
GitLab instances, otherwise the networks will become a single point of
failure.
Read carefully how to configure the components below.
### Recommended setup
### Recommended setup
For a minimal setup, you will install the Omnibus GitLab package in
`3`
For a minimal setup, you will install the Omnibus GitLab package in
`3`
independent
machines, both with
**Redis**
and
**Sentinel**
:
**independent**
machines, both with
**Redis**
and
**Sentinel**
:
-
Redis Master + Sentinel
-
Redis Master + Sentinel
-
Redis Slave + Sentinel
-
Redis Slave + Sentinel
-
Redis Slave + Sentinel
-
Redis Slave + Sentinel
Make sure you've read
[
Redis Setup
](
#redis-setup
)
and
[
Sentinel Setup
](
#sentinel-setup
)
If you are not sure or don't understand why and where the amount of nodes come
before, to understand how and why the amount of nodes came from
.
from, read
[
Redis Setup
](
#redis-setup
)
and
[
Sentinel Setup
](
#sentinel-setup
)
.
For a recommended setup
,
that can resist more failures, you will install
For a recommended setup that can resist more failures, you will install
the Omnibus GitLab package in
`5`
independent
machines, both with
the Omnibus GitLab package in
`5`
**independent**
machines, both with
**Redis**
and
**Sentinel**
:
**Redis**
and
**Sentinel**
:
-
Redis Master + Sentinel
-
Redis Master + Sentinel
...
@@ -165,9 +132,42 @@ the Omnibus GitLab package in `5` independent machines, both with
...
@@ -165,9 +132,42 @@ the Omnibus GitLab package in `5` independent machines, both with
-
Redis Slave + Sentinel
-
Redis Slave + Sentinel
-
Redis Slave + Sentinel
-
Redis Slave + Sentinel
## Redis HA configuration
### Available configuration setups
Based on your infrastructure setup and how you have installed GitLab, there are
multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or
Redis Sentinel bundled with them to save you the hassle to install them yourself.
Pick the one that suits your needs.
-
[
Installations from source
][
source
]
: You need to install Redis and Sentinel
yourself. Use the
[
Redis HA source install
](
redis_source.md
)
guide.
-
[
Omnibus package Community Edition (CE)
][
ce
]
: Redis is bundled, so you can use the
package with only the Redis service enabled (works for both master and slave
setups).
-
[
Omnibus package Enterprise Edition (EE)
][
ee
]
: Both Redis and Sentinel are bundled,
so you can use the EE package to setup the whole Redis HA infrastructure
(master, slave and Sentinel).
Note that if you have installed GitLab using the Omnibus packages (both CE and EE),
you can also use an
[
external Redis server
](
#using-a-non-omnibus-external-redis-server
)
.
### Using a non-Omnibus external Redis server
If you're hosting GitLab on a cloud provider, you can optionally use a
managed service for Redis. For example, AWS offers a managed ElastiCache service
that runs Redis.
Managed services can provide High Availability using their own proprietary
technology and provide a transparent proxy (which means that GitLab doesn't
need any additional change) or they will use Sentinel and manage it for you.
If your provider uses Sentinel, see
[
GitLab Setup
](
#gitlab-setup
)
to understand where you need to provide the list of servers and credentials.
If you want to setup Redis by yourself, without using Omnibus, you can
read the documentation on
[
configuring Redis HA for source installs
](
redis_source.md
)
.
### Redis setup
### Redis setup
overview
You must have at least
`3`
Redis servers:
`1`
Master,
`2`
Slaves, and they
You must have at least
`3`
Redis servers:
`1`
Master,
`2`
Slaves, and they
need to be each in a independent machine (see explanation above).
need to be each in a independent machine (see explanation above).
...
@@ -194,22 +194,7 @@ protect all Redis nodes and the Sentinels. They will all share the same
...
@@ -194,22 +194,7 @@ protect all Redis nodes and the Sentinels. They will all share the same
password, and all instances must be able to talk to
password, and all instances must be able to talk to
each other over the network.
each other over the network.
Redis nodes will need the same password defined in
`redis['password']`
and
### Sentinel setup overview
`redis['master_password']`
, no matter if
**Master**
or
**Slave**
. At any time
during a failover the Sentinels can reconfigure a node and change it's status
from
**Master**
to
**Slave**
and vice versa.
Initial
**Slave**
nodes requires
`redis['master']`
defined to
`false`
and
`redis['master_ip']`
pointing to the initial
**Master**
. If you use the
simplified configuration by enabling
`redis_slave_role['enable']`
, you
just need to fill in the
`redis['master_ip']`
.
This values don't have to be changed again in
`/etc/gitlab/gitlab.rb`
after
a failover, as the nodes will be managed by the Sentinels, and even after a
`gitlab-ctl reconfigure`
, they will get their configuration restored by
the same Sentinels.
### Sentinel setup
Sentinels watch both other sentinels and Redis nodes. Whenever a Sentinel
Sentinels watch both other sentinels and Redis nodes. Whenever a Sentinel
detects that a Redis node is not responding, it will announce that to the
detects that a Redis node is not responding, it will announce that to the
...
@@ -249,8 +234,11 @@ is not achieved (see the odd number of nodes requirement above). In that case,
...
@@ -249,8 +234,11 @@ is not achieved (see the odd number of nodes requirement above). In that case,
a new attempt will be made after the amount of time defined in
a new attempt will be made after the amount of time defined in
`sentinel['failover_timeout']`
(in milliseconds).
`sentinel['failover_timeout']`
(in milliseconds).
The
`failover_time`
variable have a lot of different usages, according to
>**Note:**
official documentation:
We will see where
`sentinel['failover_timeout']`
is defined later.
The
`failover_timeout`
variable has a lot of different use cases, according to
the official documentation:
-
The time needed to re-start a failover after a previous failover was
-
The time needed to re-start a failover after a previous failover was
already tried against the same master by a given Sentinel, is two
already tried against the same master by a given Sentinel, is two
...
@@ -270,18 +258,29 @@ official documentation:
...
@@ -270,18 +258,29 @@ official documentation:
the slaves will be reconfigured by the Sentinels anyway, but not with
the slaves will be reconfigured by the Sentinels anyway, but not with
the exact parallel-syncs progression as specified.
the exact parallel-syncs progression as specified.
## Configuring instances using Omnibus
## Redis HA configuration
This is the section where we install and setup the new Redis instances.
This is a summary of what are we going to do:
>**Notes:**
-
We assume that you install GitLab and all HA components from scratch. If you
already have it installed and running, read how to
[
switch from a single-machine installation to Redis HA
](
#switching-from-an-existing-single-machine-installation-to-redis-ha
)
.
-
Redis nodes (both master and slaves) will need the same password defined in
`redis['password']`
and
`redis['master_password']`
. At any time during a
failover the Sentinels can reconfigure a node and change its status
from master to slave and vice versa.
1.
Provision the required number of instances specified previously
A summary of what are we going to do:
1.
Provision the required number of instances specified previously:
-
You can opt to install Redis and Sentinel in the same machine or each in
-
You can opt to install Redis and Sentinel in the same machine or each in
independent ones.
independent ones.
-
Don't install Redis and Sentinel in the same machines your GitLab
instance
-
Don't install Redis and Sentinel in the same machines your GitLab
application
is running on.
is running on.
-
All machines must be able to talk to each other and accept incoming
-
All machines must be able to talk to each other and accept incoming
connection over Redis (
`6379`
) and Sentinel (
`26379`
) ports.
connections over Redis (
`6379`
) and Sentinel (
`26379`
) ports (unless you
change the default ports).
-
GitLab machines must be able to access these machines and with the same
-
GitLab machines must be able to access these machines and with the same
permissions.
permissions.
-
Protect them from access from external networks (Internet),
-
Protect them from access from external networks (Internet),
...
@@ -297,32 +296,10 @@ This is a summary of what are we going to do:
...
@@ -297,32 +296,10 @@ This is a summary of what are we going to do:
from running on upgrade. Only the primary GitLab application server should
from running on upgrade. Only the primary GitLab application server should
handle migrations.
handle migrations.
1.
Create/e
dit
`/etc/gitlab/gitlab.rb`
and make the changes based on the
1.
E
dit
`/etc/gitlab/gitlab.rb`
and make the changes based on the
[
Example Configurations
](
#example-configurations
)
.
[
Example Configurations
](
#example-configurations
)
.
### Existing single-machine installation
### Configuring the Master Redis instance
If you already have a single-machine GitLab install running, you will need to
replicate from this machine first, before de-activating the Redis instance
inside it.
Your single-machine install will be the initial
**Master**
, and the
`3`
others
should be configured as
**Slave**
pointing to this machine.
After replication catches up, you will need to stop services in the
single-machine install, to rotate the
**Master**
to one of the new nodes.
Make the required changes in configuration and restart the new nodes again.
To disable redis in the single install, edit
`/etc/gitlab/gitlab.rb`
:
```
ruby
redis
[
'enable'
]
=
false
```
If you fail to replicate first, you may loose data (unprocessed background jobs).
### Configuring Master Redis instance
You will need to configure the following in
`/etc/gitlab/gitlab.rb`
:
You will need to configure the following in
`/etc/gitlab/gitlab.rb`
:
...
@@ -341,9 +318,9 @@ You will need to configure the following in `/etc/gitlab/gitlab.rb`:
...
@@ -341,9 +318,9 @@ You will need to configure the following in `/etc/gitlab/gitlab.rb`:
1.
Set up a password authentication with
`redis['password']`
and
1.
Set up a password authentication with
`redis['password']`
and
`redis['master_password']`
(use the same password in all nodes).
`redis['master_password']`
(use the same password in all nodes).
Reconfigure Omnibus GitLab for the changes to take effect:
`sudo gitlab-ctl reconfigure`
1.
[
Reconfigure Omnibus GitLab
][
reocnfigure
]
for the changes to take effect.
### Configuring Slave Redis instances
### Configuring
the
Slave Redis instances
You will need to configure the following in
`/etc/gitlab/gitlab.rb`
:
You will need to configure the following in
`/etc/gitlab/gitlab.rb`
:
...
@@ -367,7 +344,22 @@ You will need to configure the following in `/etc/gitlab/gitlab.rb`:
...
@@ -367,7 +344,22 @@ You will need to configure the following in `/etc/gitlab/gitlab.rb`:
1.
Define
`redis['master_port']`
with the port of the
**Master**
Redis (default to
`6379`
).
1.
Define
`redis['master_port']`
with the port of the
**Master**
Redis (default to
`6379`
).
### Configuring Sentinel instances
Initial
**Slave**
nodes require
`redis['master']`
defined to
`false`
and
`redis['master_ip']`
pointing to the initial
**Master**
. If you use the
simplified configuration by enabling
`redis_slave_role['enable']`
, you
just need to fill in the
`redis['master_ip']`
.
This values don't have to be changed again in
`/etc/gitlab/gitlab.rb`
after
a failover, as the nodes will be managed by the Sentinels, and even after a
`gitlab-ctl reconfigure`
, they will get their configuration restored by
the same Sentinels.
### Configuring the Sentinel instances
>**Note:**
-
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the
Omnibus Community Edition and installations from source, follow the
[
Redis HA source install
](
redis_source.md
)
guide.
Now that the Redis servers are all set up, let's configure the Sentinel
Now that the Redis servers are all set up, let's configure the Sentinel
servers.
servers.
...
@@ -380,22 +372,12 @@ You must have at least `3` Redis Sentinel servers, and they need to
...
@@ -380,22 +372,12 @@ You must have at least `3` Redis Sentinel servers, and they need to
be each in a independent machine. You can configure them in the same
be each in a independent machine. You can configure them in the same
machines where you've configured the other Redis servers.
machines where you've configured the other Redis servers.
##### Community Edition
With GitLab Enterprise Edition, you can use the Omnibus package to setup multiple
machines with the Sentinel daemon.
With GitLab Community Edition, you need to install, configure, execute and
monitor Sentinel from source. Omnibus GitLab Community Edition package does
not support Sentinel configuration.
See
[
documentation for Source Install
](
redis_source.md
)
.
##### Enterprise Edition
With GitLab Enterprise Edition, you can use Omnibus package to setup multiple
machines with Sentinel daemon.
See
[
example configuration
](
#configuration-for-sentinel-ee-only
)
below.
See
[
example configuration
](
#configuration-for-sentinel-ee-only
)
below.
###
GitLab setup
###
Configuring the GitLab application
The final part is to inform the main GitLab application server of the Redis
The final part is to inform the main GitLab application server of the Redis
Sentinels servers and authentication credentials.
Sentinels servers and authentication credentials.
...
@@ -409,7 +391,7 @@ it needs to access at least one of the listed.
...
@@ -409,7 +391,7 @@ it needs to access at least one of the listed.
>**Note:**
>**Note:**
The following steps should be performed in the
[
GitLab application server
](
gitlab.md
)
The following steps should be performed in the
[
GitLab application server
](
gitlab.md
)
which ideally should not have Redis or Sentinels
in the same machine
for a HA setup.
which ideally should not have Redis or Sentinels
on it
for a HA setup.
1.
Edit
`/etc/gitlab/gitlab.rb`
and add/change the following lines:
1.
Edit
`/etc/gitlab/gitlab.rb`
and add/change the following lines:
...
@@ -417,10 +399,32 @@ which ideally should not have Redis or Sentinels in the same machine for a HA se
...
@@ -417,10 +399,32 @@ which ideally should not have Redis or Sentinels in the same machine for a HA se
-
`redis['master_password']`
- the same password you've defined before for Redis and Sentinels
-
`redis['master_password']`
- the same password you've defined before for Redis and Sentinels
-
`gitlab_rails['redis_sentinels']`
- a list of sentinels with
`host`
and
`port`
-
`gitlab_rails['redis_sentinels']`
- a list of sentinels with
`host`
and
`port`
1.
[Reconfigure
] GitLab
for the changes to take effect.
1.
[
Reconfigure
Omnibus GitLab
][
reconfigure
]
for the changes to take effect.
See
[
example configuration
](
#configuration-for-gitlab
)
below.
See
[
example configuration
](
#configuration-for-gitlab
)
below.
## Switching from an existing single-machine installation to Redis HA
If you already have a single-machine GitLab install running, you will need to
replicate from this machine first, before de-activating the Redis instance
inside it.
Your single-machine install will be the initial
**Master**
, and the
`3`
others
should be configured as
**Slave**
pointing to this machine.
After replication catches up, you will need to stop services in the
single-machine install, to rotate the
**Master**
to one of the new nodes.
Make the required changes in configuration and restart the new nodes again.
To disable redis in the single install, edit
`/etc/gitlab/gitlab.rb`
:
```
ruby
redis
[
'enable'
]
=
false
```
If you fail to replicate first, you may loose data (unprocessed background jobs).
## Minimal example configuration with 1 master, 2 slaves and 3 sentinels
## Minimal example configuration with 1 master, 2 slaves and 3 sentinels
In this example we consider that all servers have an internal network
In this example we consider that all servers have an internal network
...
@@ -428,7 +432,7 @@ interface with IPs in the `10.0.0.x` range, and that they can connect
...
@@ -428,7 +432,7 @@ interface with IPs in the `10.0.0.x` range, and that they can connect
to each other using these IPs.
to each other using these IPs.
In a real world usage, you would also setup firewall rules to prevent
In a real world usage, you would also setup firewall rules to prevent
unauthorized access from other machines
,
and block traffic from the
unauthorized access from other machines and block traffic from the
outside (Internet).
outside (Internet).
We will use the same
`3`
nodes with
**Redis**
+
**Sentinel**
topology
We will use the same
`3`
nodes with
**Redis**
+
**Sentinel**
topology
...
@@ -450,7 +454,7 @@ The same thing will happen with `sentinel.conf` that will be overridden after th
...
@@ -450,7 +454,7 @@ The same thing will happen with `sentinel.conf` that will be overridden after th
initial execution, after any new sentinel node starts watching the
**Master**
,
initial execution, after any new sentinel node starts watching the
**Master**
,
or a failover promotes a different
**Master**
node.
or a failover promotes a different
**Master**
node.
### Configuration for Redis
M
aster
### Configuration for Redis
m
aster
**Example configation for Redis Master:**
**Example configation for Redis Master:**
...
@@ -464,9 +468,10 @@ redis['port'] = 6379
...
@@ -464,9 +468,10 @@ redis['port'] = 6379
redis
[
'password'
]
=
'redis-password-goes-here'
redis
[
'password'
]
=
'redis-password-goes-here'
redis
[
'master_password'
]
=
'redis-password-goes-here'
redis
[
'master_password'
]
=
'redis-password-goes-here'
```
```
Reconfigure Omnibus GitLab for the changes to take effect:
`sudo gitlab-ctl reconfigure`
### Configuration for Redis Slave
[
Reconfigure Omnibus GitLab
][
reconfigure
]
for the changes to take effect.
### Configuration for Redis slaves
**Example configation for Slave 1:**
**Example configation for Slave 1:**
...
@@ -502,9 +507,14 @@ redis['master_ip'] = '10.0.0.1' # IP of master Redis server
...
@@ -502,9 +507,14 @@ redis['master_ip'] = '10.0.0.1' # IP of master Redis server
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
```
```
Reconfigure Omnibus GitLab for the changes to take effect:
`sudo gitlab-ctl reconfigure`
[
Reconfigure Omnibus GitLab
][
reconfigure
]
for the changes to take effect.
### Configuration for Sentinels
### Configuration for Sentinel (EE only)
>**Note:**
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the
Omnibus Community Edition and installations from source, follow the
[
Redis HA source install
](
redis_source.md
)
guide.
Please note that some of the variables are already configured previously
Please note that some of the variables are already configured previously
as they are required for Redis replication.
as they are required for Redis replication.
...
@@ -517,9 +527,9 @@ In `/etc/gitlab/gitlab.rb`:
...
@@ -517,9 +527,9 @@ In `/etc/gitlab/gitlab.rb`:
redis_sentinel_role
[
'enable'
]
=
true
redis_sentinel_role
[
'enable'
]
=
true
redis
[
'master_name'
]
=
'gitlab-redis'
# must be the same in every sentinel node
redis
[
'master_name'
]
=
'gitlab-redis'
# must be the same in every sentinel node
redis
[
'master_password'
]
=
'redis-password-goes-here'
# the same value defined in redis['password'] in the master instance
redis
[
'master_ip'
]
=
'10.0.0.1'
# ip of the initial master redis instance
redis
[
'master_ip'
]
=
'10.0.0.1'
# ip of the initial master redis instance
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
redis
[
'master_password'
]
=
'redis-password-goes-here'
# the same value defined in redis['password'] in the master instance
## Configure Sentinel
## Configure Sentinel
sentinel
[
'bind'
]
=
'10.0.0.1'
sentinel
[
'bind'
]
=
'10.0.0.1'
...
@@ -571,9 +581,9 @@ In `/etc/gitlab/gitlab.rb`:
...
@@ -571,9 +581,9 @@ In `/etc/gitlab/gitlab.rb`:
redis_sentinel_role
[
'enable'
]
=
true
redis_sentinel_role
[
'enable'
]
=
true
redis
[
'master_name'
]
=
'gitlab-redis'
# must be the same in every sentinel node
redis
[
'master_name'
]
=
'gitlab-redis'
# must be the same in every sentinel node
redis
[
'master_password'
]
=
'redis-password-goes-here'
# the same value defined in redis['password'] in the master instance
redis
[
'master_ip'
]
=
'10.0.0.1'
# ip of the initial master redis instance
redis
[
'master_ip'
]
=
'10.0.0.1'
# ip of the initial master redis instance
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
redis
[
'master_password'
]
=
'redis-password-goes-here'
# the same value defined in redis['password'] in the master instance
## Configure Sentinel
## Configure Sentinel
sentinel
[
'bind'
]
=
'10.0.0.2'
sentinel
[
'bind'
]
=
'10.0.0.2'
...
@@ -625,9 +635,9 @@ In `/etc/gitlab/gitlab.rb`:
...
@@ -625,9 +635,9 @@ In `/etc/gitlab/gitlab.rb`:
redis_sentinel_role
[
'enable'
]
=
true
redis_sentinel_role
[
'enable'
]
=
true
redis
[
'master_name'
]
=
'gitlab-redis'
# must be the same in every sentinel node
redis
[
'master_name'
]
=
'gitlab-redis'
# must be the same in every sentinel node
redis
[
'master_password'
]
=
'redis-password-goes-here'
# the same value defined in redis['password'] in the master instance
redis
[
'master_ip'
]
=
'10.0.0.1'
# ip of the initial master redis instance
redis
[
'master_ip'
]
=
'10.0.0.1'
# ip of the initial master redis instance
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
redis
[
'master_password'
]
=
'redis-password-goes-here'
# the same value defined in redis['password'] in the master instance
## Configure Sentinel
## Configure Sentinel
sentinel
[
'bind'
]
=
'10.0.0.3'
sentinel
[
'bind'
]
=
'10.0.0.3'
...
@@ -671,10 +681,15 @@ sentinel['quorum'] = 2
...
@@ -671,10 +681,15 @@ sentinel['quorum'] = 2
# sentinel['failover_timeout'] = 60000
# sentinel['failover_timeout'] = 60000
```
```
## Advanced configuration
Omnibus GitLab configures some things behind the curtains to make the sysadmins'
lives easier. If you want to know what happens underneath keep reading.
### Control running services
### Control running services
In the
example above we've used
`redis_sentinel_role`
and
`redis_master_role`
In the
previous example above we've used
`redis_sentinel_role`
and
which simplify the amount of configuration changes.
`redis_master_role`
which simplify the amount of configuration changes.
If you want more control, here is what each one sets for you automatically
If you want more control, here is what each one sets for you automatically
when enabled:
when enabled:
...
@@ -716,13 +731,15 @@ mailroom['enable'] = false
...
@@ -716,13 +731,15 @@ mailroom['enable'] = false
redis
[
'master'
]
=
false
redis
[
'master'
]
=
false
```
```
You can find the relevant attributes defined in
[
gitlab_rails.rb
][
omnifile
]
.
## Troubleshooting
## Troubleshooting
There are a lot of moving parts that needs to be taken care carefully
There are a lot of moving parts that needs to be taken care carefully
in order for the HA setup to work as expected.
in order for the HA setup to work as expected.
Before proceeding with the troubleshooting below, check your firewall
Before proceeding with the troubleshooting below, check your firewall
rules:
rules:
-
Redis machines
-
Redis machines
-
Accept TCP connection in
`6379`
-
Accept TCP connection in
`6379`
-
Connect to the other Redis machines via TCP in
`6379`
-
Connect to the other Redis machines via TCP in
`6379`
...
@@ -731,7 +748,7 @@ rules:
...
@@ -731,7 +748,7 @@ rules:
-
Connect to other Sentinel machines via TCP in
`26379`
-
Connect to other Sentinel machines via TCP in
`26379`
-
Connect to the Redis machines via TCP in
`6379`
-
Connect to the Redis machines via TCP in
`6379`
### Redis replication
###
Troubleshooting
Redis replication
You can check if everything is correct by connecting to each server using
You can check if everything is correct by connecting to each server using
`redis-cli`
application, and sending the
`INFO`
command.
`redis-cli`
application, and sending the
`INFO`
command.
...
@@ -781,9 +798,7 @@ repl_backlog_first_byte_offset:0
...
@@ -781,9 +798,7 @@ repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
repl_backlog_histlen:0
```
```
### Sentinel
### Troubleshooting Sentinel
#### Omnibus GitLab
If you get an error like:
`Redis::CannotConnectError: No sentinels available.`
,
If you get an error like:
`Redis::CannotConnectError: No sentinels available.`
,
there may be something wrong with your configuration files or it can be related
there may be something wrong with your configuration files or it can be related
...
@@ -865,3 +880,7 @@ Read more on high-availability configuration:
...
@@ -865,3 +880,7 @@ Read more on high-availability configuration:
[
gh-534
]:
https://github.com/redis/redis-rb/issues/534
[
gh-534
]:
https://github.com/redis/redis-rb/issues/534
[
redis
]:
http://redis.io/
[
redis
]:
http://redis.io/
[
sentinel
]:
http://redis.io/topics/sentinel
[
sentinel
]:
http://redis.io/topics/sentinel
[
omnifile
]:
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/82b7345b150f072c8673c79738ce893f92d0d652/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb#L134-159
[
source
]:
../../install/installation.md
[
ce
]:
https://about.gitlab.com/downloads
[
ee
]:
https://about.gitlab.com/downloads-ee
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment