Fixed some documentation and moved Source install specific to other file.

494c2785 · Gabriel Mazetto · 92e60372 · 494c2785 · 494c2785
Commit 494c2785 authored Nov 08, 2016 by Gabriel Mazetto
Showing with 300 additions and 121 deletions

doc/administration/high_availability/redis.md doc/administration/high_availability/redis.md +14 -121

doc/administration/high_availability/redis_source.md doc/administration/high_availability/redis_source.md +286 -0

No files found.
--- a/doc/administration/high_availability/redis.md
+++ b/doc/administration/high_availability/redis.md
@@ -19,7 +19,6 @@ that comes bundled with Omnibus GitLab packages.
  - [Prerequisites](#prerequisites)
  - [Redis setup](#redis-setup)
    - [Existing single-machine installation](#existing-single-machine-installation)
-    - [Installation from source](#installation-from-source)
    - [Omnibus packages](#omnibus-packages)
  - [Configuring Sentinel](#configuring-sentinel)
    - [How sentinel handles a failover](#how-sentinel-handles-a-failover)
@@ -31,7 +30,6 @@ that comes bundled with Omnibus GitLab packages.
  - [Redis replication](#redis-replication)
  - [Sentinel](#sentinel)
    - [Omnibus GitLab](#omnibus-gitlab)
-    - [Install from Source](#install-from-source)

 <!-- END doctoc generated TOC please keep comment here to allow auto update -->

@@ -161,58 +159,6 @@ To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`:
 redis['enable'] = false
 ```

-#### Installation from source
-
-**Configuring Master Redis instance**
-
-You need to make the following changes in `redis.conf`:
-
-1. Define a `bind` address pointing to a local IP that your other machines
-   can reach you. If you really need to bind to an external acessible IP, make
-   sure you add extra firewall rules to prevent unauthorized access:
-
-   ```conf
-   # By default, if no "bind" configuration directive is specified, Redis listens
-   # for connections from all the network interfaces available on the server.
-   # It is possible to listen to just one or multiple selected interfaces using
-   # the "bind" configuration directive, followed by one or more IP addresses.
-   #
-   # Examples:
-   #
-   # bind 192.168.1.100 10.0.0.1
-   # bind 127.0.0.1 ::1
-   bind 0.0.0.0 # This will bind to all interfaces
-   ```
-
-1. Define a `port` to force redis to listin on TCP so other machines can
-   connect to it:
-
-   ```conf
-   # Accept connections on the specified port, default is 6379 (IANA #815344).
-   # If port 0 is specified Redis will not listen on a TCP socket.
-   port 6379
-   ```
-
-1. Set up password authentication (use the same password in all nodes)
-
-    ```conf
-    requirepass "redis-password-goes-here"
-    masterauth "redis-password-goes-here"
-    ```
-
-1. Restart the Redis services for the changes to take effect.
-
-**Configuring Slave Redis instance**
-
-1. Follow same instructions from master, with the extra change in `redis.conf`:
-
-   ```conf
-   # IP and port of the master Redis server
-   slaveof 10.10.10.10 6379
-   ```
-
-1. Restart the Redis services for the changes to take effect.
-
 #### Omnibus packages

 You need to install the Omnibus GitLab package in `3` independent machines.
@@ -304,25 +250,16 @@ GitLab Enterprise Edition provides [automated way to setup and run](#sentinel-se
 #### Sentinel setup

 ##### Community Edition
+
 With GitLab Community Edition, you need to install, configure, execute and
 monitor Sentinel from source. Omnibus GitLab Community Edition package does
 not support Sentinel configuration.

-A minimal configuration file (`sentinel.conf`) should contain the following:
-
-```conf
-bind 0.0.0.0 # bind to all interfaces or change to a specific IP
-port 26379 # default sentinel port
-sentinel auth-pass gitlab-redis redis-password-goes-here
-sentinel monitor gitlab-redis 10.0.0.1 6379 2
-sentinel down-after-milliseconds gitlab-redis 10000
-sentinel config-epoch gitlab-redis 0
-sentinel leader-epoch gitlab-redis 0
-```
+See documentation for Source Install [here](redis_source.md).

 ##### Enterprise Edition

-To setup sentinel, you edit `/etc/gitlab/gitlab.rb` file:
+To setup sentinel, edit `/etc/gitlab/gitlab.rb` file:

 ```ruby

@@ -336,7 +273,7 @@ redis_master_role['enable'] = true

 ## Enabled Sentinel and Redis Slave services
 redis_sentinel_role['enable'] = true
-redis_master_role['enable'] = true
+redis_slave_role['enable'] = true

 ## Configure Redis
 redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
@@ -345,7 +282,7 @@ redis['master_port'] = 6379 # port of the initial master redis instance
 redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance

 ## Configure Sentinel
-sentinel['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
+# sentinel['bind'] = '0.0.0.0' # bind to all interfaces, uncomment to specify an IP and bind to a single one
 # sentinel['port'] = 26379 # uncomment to change default port

 ## Quorum must reflect the amount of voting sentinels it take to start a failover.
@@ -435,24 +372,16 @@ master and the new sentinels servers.

 ### GitLab setup

-You can enable or disable sentinel support at any time in new or existing
+You can enable or disable Sentinel support at any time in new or existing
 installations. From the GitLab application perspective, all it requires is
-the correct credentials for the master Redis and for all Sentinel nodes.
+the correct credentials for the Sentinel nodes.

-It doesn't require a list of all Sentinel nodes, as in case of a failure,
-the application will need to query only one of them.
+While it doesn't require a list of all Sentinel nodes, in case of a failure,
+it needs to access at one of listed ones.

 >**Note:**
-The following steps should be performed in the [GitLab application server](gitlab.md).
-
-**For source based installations**
-
-1. Edit `/home/git/gitlab/config/resque.yml` following the example in
-   `/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels
-   line, changing to the correct server credentials.
-1. Restart GitLab for the changes to take effect.
-
-**For Omnibus installations**
+The following steps should be performed in the [GitLab application server](gitlab.md)
+which ideally should not have Redis or Sentinels in the same machine for a HA setup.

 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:

@@ -466,7 +395,7 @@ The following steps should be performed in the [GitLab application server](gitla
    ]
    ```

-1. [Reconfigure] the GitLab for the changes to take effect.
+1. [Reconfigure] GitLab for the changes to take effect.

 ## Troubleshooting

@@ -548,42 +477,6 @@ The way the redis connector `redis-rb` works with sentinel is a bit
 non-intuitive. We try to hide the complexity in omnibus, but it still requires
 a few extra configs.

-#### Install from Source
-
-If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
-there may be something wrong with your configuration files or it can be related
-to [this issue][gh-531].
-
-It's a bit non-intuitive the way you have to config `resque.yml` and
-`sentinel.conf`, otherwise `redis-rb` will not work properly.
-
-The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`)
-**must** be used as the hostname in GitLab (`resque.yml` for source installations
-or `gitlab-rails['redis_*']` in Omnibus):
-
-```conf
-# sentinel.conf:
-sentinel monitor gitlab-redis 10.10.10.10 6379 2
-sentinel down-after-milliseconds gitlab-redis 10000
-sentinel config-epoch gitlab-redis 0
-sentinel leader-epoch gitlab-redis 0
-```
-
-```yaml
-# resque.yaml
-production:
-  url: redis://:myredispassword@gitlab-redis/
-  sentinels:
-    -
-      host: slave1.example.com # or use ip
-      port: 26380 # point to sentinel, not to redis port
-    -
-      host: slave2.exampl.com # or use ip
-      port: 26381 # point to sentinel, not to redis port
-```
-
-When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
-
 ---

 To make sure your configuration is correct:
@@ -611,8 +504,8 @@ To make sure your configuration is correct:
 1. To simulate a failover on master Redis, SSH into the Redis server and run:

    ```bash
-    # port must match your master redis port
-     redis-cli -h localhost -p 6379 DEBUG sleep 60
+    # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one
+     redis-cli -h localhost -p 6379 DEBUG sleep 20
    ```

 1. Then back in the Rails console from the first step, run:

--- a/doc/administration/high_availability/redis_source.md
+++ b/doc/administration/high_availability/redis_source.md
+# Configuring Redis for GitLab HA (Source Install)
+
+We highly recommend that you use Omnibus GitLab packages, as we can optimize
+required packages specifically for GitLab, and we will take care of upgrading
+to the latest supported version.
+
+If you are building packages for a specific distro, or trying to build some
+internal automation, you can check this documentation to learn about the
+minimal setup, required changes, etc.
+
+If you want to see the documentation for Omnibus GitLab Install, please [read it here](redis.md).
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**
+
+- [Configure your own Redis server](#configure-your-own-redis-server)
+  - [Configuring Master Redis instance](#configuring-master-redis-instance)
+  - [Configuring Slave Redis instances](#configuring-slave-redis-instances)
+  - [Configuring Redis Sentinel instances](#configuring-redis-sentinel-instances)
+- [GitLab setup](#gitlab-setup)
+- [Example configurations](#example-configurations)
+  - [Configuring Redis Master](#configuring-redis-master)
+  - [Configuring Redis Slaves](#configuring-redis-slaves)
+  - [Configuring Redis Sentinel](#configuring-redis-sentinel)
+- [Troubleshooting](#troubleshooting)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Configure your own Redis server
+
+Redis server must be configured to use TCP connection instead of socket,
+and since Redis `3.2`, you must define a password to receive external
+connections (`requirepass`).
+
+You will also need to define equal password for slave password definition
+(`masterauth`), in the same instance, if you are using Redis with Sentinel.
+
+To configure Redis to use TCP connection you need to define both
+`bind` and `port`. You can bind to all interfaces (`0.0.0.0`) or specify the
+ip of the desired interface (for ex. one from an internal network).
+
+
+### Configuring Master Redis instance
+
+You need to make the following changes in `redis.conf`:
+
+1. Define a `bind` address pointing to a local IP that your other machines
+   can reach you. If you really need to bind to an external acessible IP, make
+   sure you add extra firewall rules to prevent unauthorized access:
+
+1. Define a `port` to force redis to listen on TCP so other machines can
+   connect to it (default port is `6379`).
+
+1. Set up password authentication (use the same password in all nodes).
+   The password should be defined equal for both `requirepass` and `masterauth`
+   when setting up Redis to use with Sentinel.
+
+1. Restart the Redis services for the changes to take effect.
+
+See [example configuration](#configuring-redis-master) below.
+
+### Configuring Slave Redis instances
+
+1. Follow same instructions for Redis Master
+
+1. Define `slaveof` pointing to the Redis master instance with **IP** and **port**.
+
+1. Restart the Redis services for the changes to take effect.
+
+See [example configuration](#configuring-redis-slaves) below.
+
+### Configuring Redis Sentinel instances
+
+Sentinel is a special type of Redis server. It inherits most of the basic
+configuration options you can define in `redis.conf`, with specific ones
+starting with `sentinel` prefix.
+
+You will need to define the initial configs to enable connectivity:
+
+1. Define a `bind` address pointing to a local IP that your other machines
+   can reach you. If you really need to bind to an external acessible IP, make
+   sure you add extra firewall rules to prevent unauthorized access:
+
+1. Define a `port` to force sentinel to listen on TCP so other machines can
+   connect to it (default port is `26379`).
+
+And the sentinel specific ones:
+
+1. Define with `sentinel auth-pass` the same shared password you have
+   defined for both Redis **Master** and **Slaves** instances.
+
+1. Define with `sentinel monitor` the **IP** and **port** of the Redis
+   **Master** node, and the **quorum** required to start a failover.
+   If you need more information to understand about quorum, please
+   read the detailed explanation in the [HA documentation for Omnibus Installs](redis.md).
+
+1. Define with `sentinel down-after-milliseconds` the ammount in `ms` of time
+   that an unresponsive server will be considered down.
+
+1. Define a value for `sentinel failover_timeout` in `ms`. This has multiple
+   meanings:
+
+   * The time needed to re-start a failover after a previous failover was
+     already tried against the same master by a given Sentinel, is two
+     times the failover timeout.
+
+   * The time needed for a slave replicating to a wrong master according
+     to a Sentinel current configuration, to be forced to replicate
+     with the right master, is exactly the failover timeout (counting since
+     the moment a Sentinel detected the misconfiguration).
+
+   * The time needed to cancel a failover that is already in progress but
+     did not produced any configuration change (SLAVEOF NO ONE yet not
+     acknowledged by the promoted slave).
+
+   * The maximum time a failover in progress waits for all the slaves to be
+     reconfigured as slaves of the new master. However even after this time
+     the slaves will be reconfigured by the Sentinels anyway, but not with
+     the exact parallel-syncs progression as specified.
+
+See [example configuration](#configuring-redis-sentinel) below.
+
+## GitLab setup
+
+You can enable or disable Sentinel support at any time in new or existing
+installations. From the GitLab application perspective, all it requires is
+the correct credentials for the Sentinel nodes.
+
+While it doesn't require a list of all Sentinel nodes, in case of a failure,
+it needs to access at one of listed ones.
+
+>**Note:**
+The following steps should be performed in the [GitLab application server](gitlab.md)
+which ideally should not have Redis or Sentinels in the same machine for a HA setup.
+
+1. Edit `/home/git/gitlab/config/resque.yml` following the example in
+   `/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels
+   lines, pointing to the correct server credentials.
+
+1. Restart GitLab for the changes to take effect.
+
+## Example configurations
+
+In this example we consider that all servers have an internal network
+interface with IPs in the `10.0.0.x` range, and that they can connect
+to each other using these IPs.
+
+In a real world usage, you would also setup firewall rules to prevent
+unauthorized access from other machines, and block traffic from the
+outside (Internet).
+
+We will use the same `3` nodes with **Redis** + **Sentinel** topology
+discussed in the [Configuring Redis for GitLab HA](redis.md) documentation.
+
+Here is a list and description of each **machine** and the assined **ip**:
+
+* `10.0.0.1`: Redis Master + Sentinel 1
+* `10.0.0.2`: Redis Slave 1 + Sentinel 2
+* `10.0.0.2`: Redis Slave 2 + Sentinel 3
+
+Please note that after the initial configuration, if a failover is initiated
+by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
+will change permanently (including in `redis.conf`) from one node to the other,
+until a new failover is initiated again.
+
+The same thing will happen with `sentinel.conf` that will be overriten after the
+initial execution, after any new sentinel node starts watching the **Master**,
+or a failover promotes a different **Master** node.
+
+### Configuring Redis Master
+
+`redis.conf`:
+
+```conf
+bind 10.0.0.1
+port 6379
+requirepass redis-password-goes-here
+masterauth redis-password-goes-here
+```
+
+### Configuring Redis Slaves
+
+**Slave 1 - `redis.conf`:**
+
+```conf
+bind 10.0.0.2
+port 6379
+requirepass redis-password-goes-here
+masterauth redis-password-goes-here
+
+# IP and port of the master Redis server
+slaveof 10.0.0.1 6379
+```
+
+**Slave 2 - `redis.conf`:**
+
+```conf
+bind 10.0.0.3
+port 6379
+requirepass redis-password-goes-here
+masterauth redis-password-goes-here
+
+# IP and port of the master Redis server
+slaveof 10.0.0.1 6379
+```
+
+### Configuring Redis Sentinel
+
+For this example, **Sentinel 1** will be configured in the same machine as the
+**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the
+**Slave 1** and **Slave 2** respectively.
+
+Sentinel 1 - `sentinel.conf`
+
+```conf
+bind 10.0.0.1
+port 26379
+sentinel auth-pass gitlab-redis redis-password-goes-here
+sentinel monitor gitlab-redis 10.0.0.1 6379 2
+sentinel down-after-milliseconds gitlab-redis 10000
+sentinel failover_timeout 30000
+```
+
+Sentinel 2 - `sentinel.conf`
+
+```conf
+bind 10.0.0.2
+port 26379
+sentinel auth-pass gitlab-redis redis-password-goes-here
+sentinel monitor gitlab-redis 10.0.0.1 6379 2
+sentinel down-after-milliseconds gitlab-redis 10000
+sentinel failover_timeout 30000
+```
+
+Sentinel 3 - `sentinel.conf`
+
+```conf
+bind 10.0.0.3
+port 26379
+sentinel auth-pass gitlab-redis redis-password-goes-here
+sentinel monitor gitlab-redis 10.0.0.1 6379 2
+sentinel down-after-milliseconds gitlab-redis 10000
+sentinel failover_timeout 30000
+```
+
+## Troubleshooting
+
+We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained in the documentation for Omnibus
+Install. Here we will list only the things that are specific to a **Source** install.
+
+If you get an error in GitLab like: `Redis::CannotConnectError: No sentinels available.`,
+there may be something wrong with your configuration files or it can be related
+to [this issue][gh-531].
+
+It's a bit non-intuitive the way you have to config `resque.yml` and
+`sentinel.conf`, otherwise `redis-rb` will not work properly.
+
+The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`)
+**must** be used as the hostname in GitLab (`resque.yml`):
+
+```conf
+# sentinel.conf:
+sentinel monitor gitlab-redis 10.0.0.1 6379 2
+sentinel down-after-milliseconds gitlab-redis 10000
+sentinel config-epoch gitlab-redis 0
+sentinel leader-epoch gitlab-redis 0
+```
+
+```yaml
+# resque.yaml
+production:
+  url: redis://:myredispassword@gitlab-redis/
+  sentinels:
+    -
+      host: 10.0.0.1
+      port: 26379 # point to sentinel, not to redis port
+    -
+      host: 10.0.0.2
+      port: 26379 # point to sentinel, not to redis port
+    -
+      host: 10.0.0.3
+      port: 26379 # point to sentinel, not to redis port
+```
+
+When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)