Merge branch 'redis-move' into 'master'

Split and move Redis to a new location

Closes #216098

See merge request gitlab-org/gitlab!34637

doc/administration/geo/replication/multiple_servers.md

@@ -46,7 +46,7 @@ for PostgreSQL and Redis, it is not covered by this Geo multi-server documentati
 
 For more information about setting up a multi-server PostgreSQL cluster and Redis cluster using the omnibus package see the multi-server documentation for
 [PostgreSQL](../../postgresql/replication_and_failover.md) and
-[Redis](../../high_availability/redis.md), respectively.
+[Redis](../../redis/replication_and_failover.md), respectively.
 
 NOTE: **Note:**
 It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document.
@@ -95,7 +95,7 @@ application servers, and connections from the application servers to those
 services on the backend servers configured, during normal GitLab multi-server set up. See
 multi-server configuration documentation for
 [PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes)
-and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application).
+and [Redis](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application).
 
 ### Step 2: Configure the **primary** database
 
@@ -131,7 +131,7 @@ and are not related to Geo setup.
 Configure the following services, again using the non-Geo multi-server
 documentation:
 
-- [Configuring Redis for GitLab](../../high_availability/redis.md) for multiple servers.
+- [Configuring Redis for GitLab](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application) for multiple servers.
 - [Gitaly](../../high_availability/gitaly.md), which will store data that is
   synchronized from the **primary** node.
 

doc/administration/redis/index.md

+---
+type: index
+stage: Enablement
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
+# Configuring Redis for scaling
+
+Based on your infrastructure setup and how you have installed GitLab, there are
+multiple ways to configure Redis.
+
+You can choose to install and manage Redis and Sentinel yourself, use a hosted
+cloud solution, or you can use the ones that come bundled with the Omnibus GitLab
+packages so you only need to focus on configuration. Pick the one that suits your needs.
+
+## Redis replication and failover using Omnibus GitLab
+
+This setup is for when you have installed GitLab using the
+[Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee).
+
+Both Redis and Sentinel are bundled in the package, so you can it to set up the whole
+Redis infrastructure (primary, replica and sentinel).
+
+[> Read how to set up Redis replication and failover using Omnibus GitLab](replication_and_failover.md)
+
+## Redis replication and failover using the non-bundled Redis
+
+This setup is for when you have installed GitLab using the
+[Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE),
+or installed it [from source](../../install/installation.md), but you want to use
+your own external Redis and sentinel servers.
+
+[> Read how to set up Redis replication and failover using the non-bundled Redis](replication_and_failover_external.md)
+
+## Standalone Redis using Omnibus GitLab
+
+This setup is for when you have installed the
+[Omnibus GitLab **Community Edition** (CE) package](https://about.gitlab.com/install/?version=ce)
+to use the bundled Redis, so you can use the package with only the Redis service enabled.
+
+[> Read how to set up a standalone Redis instance using Omnibus GitLab](standalone.md)

doc/administration/redis/standalone.md

+---
+type: howto
+stage: Enablement
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
+# Standalone Redis using Omnibus GitLab **(CORE ONLY)**
+
+The Omnibus GitLab package can be used to configure a standalone Redis server.
+In this configuration, Redis is not scaled, and represents a single
+point of failure. However, in a scaled environment the objective is to allow
+the environment to handle more users or to increase throughput. Redis itself
+is generally stable and can handle many requests, so it is an acceptable
+trade off to have only a single instance. See the [reference architectures](../reference_architectures/index.md)
+page for an overview of GitLab scaling options.
+
+## Set up a standalone Redis instance
+
+The steps below are the minimum necessary to configure a Redis server with
+Omnibus GitLab:
+
+1. SSH into the Redis server.
+1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
+   package you want by using **steps 1 and 2** from the GitLab downloads page.
+   Do not complete any other steps on the download page.
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
+
+   ```ruby
+   ## Enable Redis
+   redis['enable'] = true
+
+   ## Disable all other services
+   sidekiq['enable'] = false
+   gitlab_workhorse['enable'] = false
+   puma['enable'] = false
+   postgresql['enable'] = false
+   nginx['enable'] = false
+   prometheus['enable'] = false
+   alertmanager['enable'] = false
+   pgbouncer_exporter['enable'] = false
+   gitlab_exporter['enable'] = false
+   gitaly['enable'] = false
+
+   redis['bind'] = '0.0.0.0'
+   redis['port'] = 6379
+   redis['password'] = 'SECRET_PASSWORD_HERE'
+
+   gitlab_rails['enable'] = false
+   ```
+
+1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+1. Note the Redis node's IP address or hostname, port, and
+   Redis password. These will be necessary when configuring the GitLab
+   application servers later.
+
+[Advanced configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
+are supported and can be added if needed.
+
+## Troubleshooting
+
+See the [Redis troubleshooting guide](troubleshooting.md).

doc/administration/redis/troubleshooting.md

+---
+type: reference
+stage: Enablement
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
+# Troubleshooting Redis
+
+There are a lot of moving parts that needs to be taken care carefully
+in order for the HA setup to work as expected.
+
+Before proceeding with the troubleshooting below, check your firewall rules:
+
+- Redis machines
+  - Accept TCP connection in `6379`
+  - Connect to the other Redis machines via TCP in `6379`
+- Sentinel machines
+  - Accept TCP connection in `26379`
+  - Connect to other Sentinel machines via TCP in `26379`
+  - Connect to the Redis machines via TCP in `6379`
+
+## Troubleshooting Redis replication
+
+You can check if everything is correct by connecting to each server using
+`redis-cli` application, and sending the `info replication` command as below.
+
+```shell
+/opt/gitlab/embedded/bin/redis-cli -h <redis-host-or-ip> -a '<redis-password>' info replication
+```
+
+When connected to a `Primary` Redis, you will see the number of connected
+`replicas`, and a list of each with connection details:
+
+```plaintext
+# Replication
+role:master
+connected_replicas:1
+replica0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1
+master_repl_offset:208037658
+repl_backlog_active:1
+repl_backlog_size:1048576
+repl_backlog_first_byte_offset:206989083
+repl_backlog_histlen:1048576
+```
+
+When it's a `replica`, you will see details of the primary connection and if
+its `up` or `down`:
+
+```plaintext
+# Replication
+role:replica
+master_host:10.133.1.58
+master_port:6379
+master_link_status:up
+master_last_io_seconds_ago:1
+master_sync_in_progress:0
+replica_repl_offset:208096498
+replica_priority:100
+replica_read_only:1
+connected_replicas:0
+master_repl_offset:0
+repl_backlog_active:0
+repl_backlog_size:1048576
+repl_backlog_first_byte_offset:0
+repl_backlog_histlen:0
+```
+
+## Troubleshooting Sentinel
+
+If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
+there may be something wrong with your configuration files or it can be related
+to [this issue](https://github.com/redis/redis-rb/issues/531).
+
+You must make sure you are defining the same value in `redis['master_name']`
+and `redis['master_pasword']` as you defined for your sentinel node.
+
+The way the Redis connector `redis-rb` works with sentinel is a bit
+non-intuitive. We try to hide the complexity in omnibus, but it still requires
+a few extra configurations.
+
+---
+
+To make sure your configuration is correct:
+
+1. SSH into your GitLab application server
+1. Enter the Rails console:
+
+   ```shell
+   # For Omnibus installations
+   sudo gitlab-rails console
+
+   # For source installations
+   sudo -u git rails console -e production
+   ```
+
+1. Run in the console:
+
+   ```ruby
+   redis = Redis.new(Gitlab::Redis::SharedState.params)
+   redis.info
+   ```
+
+   Keep this screen open and try to simulate a failover below.
+
+1. To simulate a failover on primary Redis, SSH into the Redis server and run:
+
+   ```shell
+   # port must match your primary redis port, and the sleep time must be a few seconds bigger than defined one
+    redis-cli -h localhost -p 6379 DEBUG sleep 20
+   ```
+
+1. Then back in the Rails console from the first step, run:
+
+   ```ruby
+   redis.info
+   ```
+
+   You should see a different port after a few seconds delay
+   (the failover/reconnect time).
+
+## Troubleshooting a non-bundled Redis with an installation from source
+
+If you get an error in GitLab like `Redis::CannotConnectError: No sentinels available.`,
+there may be something wrong with your configuration files or it can be related
+to [this upstream issue](https://github.com/redis/redis-rb/issues/531).
+
+You must make sure that `resque.yml` and `sentinel.conf` are configured correctly,
+otherwise `redis-rb` will not work properly.
+
+The `master-group-name` (`gitlab-redis`) defined in (`sentinel.conf`)
+**must** be used as the hostname in GitLab (`resque.yml`):
+
+```conf
+# sentinel.conf:
+sentinel monitor gitlab-redis 10.0.0.1 6379 2
+sentinel down-after-milliseconds gitlab-redis 10000
+sentinel config-epoch gitlab-redis 0
+sentinel leader-epoch gitlab-redis 0
+```
+
+```yaml
+# resque.yaml
+production:
+  url: redis://:myredispassword@gitlab-redis/
+  sentinels:
+    -
+      host: 10.0.0.1
+      port: 26379 # point to sentinel, not to redis port
+    -
+      host: 10.0.0.2
+      port: 26379 # point to sentinel, not to redis port
+    -
+      host: 10.0.0.3
+      port: 26379 # point to sentinel, not to redis port
+```
+
+When in doubt, read the [Redis Sentinel documentation](https://redis.io/topics/sentinel).

doc/administration/reference_architectures/10k_users.md

@@ -47,7 +47,7 @@ For a full list of reference architectures, see
    For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all
    classes and that Redis Sentinel is hosted alongside Consul.
    For larger architectures (10,000 users or more) we suggest running a separate
-   [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class
+   [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class
    and another for the Queues and Shared State classes respectively. We also recommend
    that you run the Redis Sentinel clusters separately for each Redis Cluster.
 

doc/administration/reference_architectures/1k_users.md

@@ -57,7 +57,7 @@ added performance and reliability at a reduced complexity cost.
    For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all
    classes and that Redis Sentinel is hosted alongside Consul.
    For larger architectures (10,000 users or more) we suggest running a separate
-   [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class
+   [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class
    and another for the Queues and Shared State classes respectively. We also recommend
    that you run the Redis Sentinel clusters separately for each Redis Cluster.
 

doc/administration/reference_architectures/25k_users.md

@@ -47,7 +47,7 @@ For a full list of reference architectures, see
    For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all
    classes and that Redis Sentinel is hosted alongside Consul.
    For larger architectures (10,000 users or more) we suggest running a separate
-   [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class
+   [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class
    and another for the Queues and Shared State classes respectively. We also recommend
    that you run the Redis Sentinel clusters separately for each Redis Cluster.
 

doc/administration/reference_architectures/3k_users.md

@@ -50,7 +50,7 @@ following the [2,000-user reference architecture](2k_users.md).
    For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all
    classes and that Redis Sentinel is hosted alongside Consul.
    For larger architectures (10,000 users or more) we suggest running a separate
-   [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class
+   [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class
    and another for the Queues and Shared State classes respectively. We also recommend
    that you run the Redis Sentinel clusters separately for each Redis Cluster.
 

doc/administration/reference_architectures/50k_users.md

@@ -47,7 +47,7 @@ For a full list of reference architectures, see
    For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all
    classes and that Redis Sentinel is hosted alongside Consul.
    For larger architectures (10,000 users or more) we suggest running a separate
-   [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class
+   [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class
    and another for the Queues and Shared State classes respectively. We also recommend
    that you run the Redis Sentinel clusters separately for each Redis Cluster.
 

doc/administration/reference_architectures/5k_users.md

@@ -44,7 +44,7 @@ For a full list of reference architectures, see
    For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all
    classes and that Redis Sentinel is hosted alongside Consul.
    For larger architectures (10,000 users or more) we suggest running a separate
-   [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class
+   [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class
    and another for the Queues and Shared State classes respectively. We also recommend
    that you run the Redis Sentinel clusters separately for each Redis Cluster.
 

doc/administration/reference_architectures/index.md

@@ -209,7 +209,7 @@ cluster with the Rails nodes broken down into a number of smaller Pods across th
    For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all
    classes and that Redis Sentinel is hosted alongside Consul.
    For larger architectures (10,000 users or more) we suggest running a separate
-   [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class
+   [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class
    and another for the Queues and Shared State classes respectively. We also recommend
    that you run the Redis Sentinel clusters separately for each Redis Cluster.
 

doc/install/aws/index.md

@@ -360,7 +360,7 @@ persistence and is used to store session data, temporary cache information, and
 
 1. Navigate back to the ElastiCache dashboard.
 1. Select **Redis** on the left menu and click **Create** to create a new
-   Redis cluster. Do not enable **Cluster Mode** as it is [not supported](../../administration/high_availability/redis.md#provide-your-own-redis-instance-core-only). Even without cluster mode on, you still get the
+   Redis cluster. Do not enable **Cluster Mode** as it is [not supported](../../administration/redis/replication_and_failover_external.md#requirements). Even without cluster mode on, you still get the
    chance to deploy Redis in multiple availability zones.
 1. In the settings section:
    1. Give the cluster a name (`gitlab-redis`) and a description.