Changes to HA docs to inc different architecture types

Update across various scaling/HA documentation to make the layout
more clear. This sets the stage/format for more changes in the
future.

doc/administration/high_availability/README.md

@@ -37,10 +37,10 @@ complexity.
 
 - Unicorn/Workhorse - Web-requests (UI, API, Git over HTTP)
 - Sidekiq - Asynchronous/Background jobs
-- [PostgreSQL](database.md) - Database
-  - [Consul](consul.md) - Database service discovery and health checks/failover
-  - [PGBouncer](pgbouncer.md) - Database pool manager
-- [Redis](redis.md) - Key/Value store (User sessions, cache, queue for Sidekiq)
+- PostgreSQL - Database
+  - Consul - Database service discovery and health checks/failover
+  - PGBouncer - Database pool manager
+- Redis - Key/Value store (User sessions, cache, queue for Sidekiq)
   - Sentinel - Redis health check/failover manager
 - Gitaly - Provides high-level RPC access to Git repositories
 
@@ -51,11 +51,6 @@ the GitLab instance. Still, true high availability may not be necessary. There
 are options for scaling GitLab instances relatively easily without incurring the
 infrastructure and maintenance costs of full high availability.
 
-GitLab recommends that an organization begin to explore scaling when they have
-around 1,000 active users. At this point increasing CPU cores and memory is
-not recommended as there are some components that may not handle increased
-load well on a single host.
-
 ### Basic Scaling
 
 This is the simplest form of scaling and will work for the majority of
@@ -72,6 +67,17 @@ larger one.
 - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq)
 - 1 NFS/Gitaly storage server
 
+#### Installation Instructions
+
+Complete the following installation steps in order. A link at the end of each 
+section will bring you back to the Scalable Architecture Examples section so 
+you can continue with the next step.
+
+1. [PostgreSQL](./database.md#postgresql-in-a-scaled-environment)
+1. [Redis](./redis.md#redis-in-a-scaled-environment)
+1. [Gitaly](./gitaly.md) (recommended) or [NFS](./nfs.md)
+1. [GitLab application nodes](./gitlab.md)
+
 ### Full Scaling
 
 For very large installations it may be necessary to further split components

doc/administration/high_availability/gitaly.md

+# Configuring Gitaly for Scaled and High Availability
+
+Gitaly does not yet support full high availability. However, Gitaly is quite
+stable and is in use on GitLab.com. Scaled and highly available GitLab environments
+should consider using Gitaly on a separate node. 
+
+See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to 
+track plans and progress toward high availability support. 
+
+This document is relevant for [Scaled Architecture](./README.md#scalable-architecture-examples)
+environments and [High Availability Architecture](./README.md#high-availability-architecture-examples). 
+
+## Running Gitaly on its own server
+
+Starting with GitLab 11.4, Gitaly is a replacement for NFS except
+when the [Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer)
+is used.
+
+NOTE **Note:** Gitaly network traffic is unencrypted so we recommend a firewall to
+restrict access to your Gitaly server.
+
+The steps below are the minimum necessary to configure a Gitaly server with
+Omnibus:
+
+1. SSH into the Gitaly server.
+1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab
+   package you want using **steps 1 and 2** from the GitLab downloads page.
+     - Do not complete any other steps on the download page.
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
+
+    Gitaly must trigger some callbacks to GitLab via GitLab Shell. As a result,
+    the GitLab Shell secret must be the same between the other GitLab servers and
+    the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gitlab-secrets.json`
+    from an existing GitLab server to the Gitaly server. Without this shared secret,
+    Git operations in GitLab will result in an API error.
+
+    > **NOTE:** In most or all cases the storage paths below end in `repositories` which is
+    different than `path` in `git_data_dirs` of Omnibus installations. Check the
+    directory layout on your Gitaly server to be sure.
+
+    ```ruby
+    # Enable Gitaly
+    gitaly['enable'] = true
+
+    ## Disable all other services
+    sidekiq['enable'] = false
+    gitlab_workhorse['enable'] = false
+    unicorn['enable'] = false
+    postgresql['enable'] = false
+    nginx['enable'] = false
+    prometheus['enable'] = false
+    alertmanager['enable'] = false
+    pgbouncer_exporter['enable'] = false
+    redis_exporter['enable'] = false
+    gitlab_monitor['enable'] = false
+    gitaly['enable'] = false 
+
+    # Prevent database connections during 'gitlab-ctl reconfigure'
+    gitlab_rails['rake_cache_clear'] = false
+    gitlab_rails['auto_migrate'] = false
+
+    # Configure the gitlab-shell API callback URL. Without this, `git push` will
+    # fail. This can be your 'front door' GitLab URL or an internal load
+    # balancer.
+    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
+
+    # Make Gitaly accept connections on all network interfaces. You must use
+    # firewalls to restrict access to this address/port.
+    gitaly['listen_addr'] = "0.0.0.0:8075"
+    gitaly['auth_token'] = 'abc123secret'
+
+    gitaly['storage'] = [
+      { 'name' => 'default', 'path' => '/mnt/gitlab/default/repositories' },
+      { 'name' => 'storage1', 'path' => '/mnt/gitlab/storage1/repositories' },
+    ]
+
+    # To use tls for gitaly you need to add
+    gitaly['tls_listen_addr'] = "0.0.0.0:9999"
+    gitaly['certificate_path'] = "path/to/cert.pem"
+    gitaly['key_path'] = "path/to/key.pem"
+    ```
+
+Again, reconfigure (Omnibus) or restart (source).
+
+Continue configuration of other components by going back to:
+
+- [Scaled Architectures](./README.md#scalable-architecture-examples)
+- [High Availability Architectures](./README.md#high-availability-architecture-examples)

doc/administration/high_availability/gitlab.md

-# Configuring GitLab for HA
-
-Assuming you have already configured a [database](database.md), [Redis](redis.md), and [NFS](nfs.md), you can
-configure the GitLab application server(s) now. Complete the steps below
-for each GitLab application server in your environment.
+# Configuring GitLab Scaling and High Availability
 
 > **Note:** There is some additional configuration near the bottom for
   additional GitLab application servers. It's important to read and understand

doc/administration/high_availability/redis.md

-# Configuring Redis for GitLab HA
+# Configuring Redis for Scaling and High Availability
 
-> Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11.
+## Provide your own Redis instance **[CORE ONLY]**
+
+The following are the requirements for providing your own Redis instance:
+
+- Redis version 2.8 or higher. Version 3.2 or higher is recommend as this is
+  what ships with the GitLab Omnibus package.
+- Standalone Redis or Redis high availability with Sentinel are supported. Redis
+  Cluster is not supported.
+- Managed Redis from cloud providers such as AWS Elasticache will work. If these
+  services support high availability, be sure it is not the Redis Cluster type.
+
+Note the Redis node's IP address or hostname, port, and password (if required).
+These will be necessary when configuring the GitLab application servers later.
+
+## Redis in a Scaled Environment
+
+This section is relevant for [Scaled Architecture](./README.md#scalable-architecture-examples)
+environments including [Basic Scaling](./README.md#basic-scaling) and
+[Full Scaling](./README.md#full-scaling).
+
+### Provide your own Redis instance **[CORE ONLY]**
+
+If you want to use your own deployed Redis instance(s), 
+see [Provide your own Redis instance](#provide-your-own-redis-instance) 
+for more details. However, you can use the GitLab Omnibus package to easily 
+deploy the bundled Redis.  
+
+### Standalone Redis using GitLab Omnibus **[CORE ONLY]**
+
+The GitLab Omnibus package can be used to configure a standalone Redis server.
+In this configuration Redis is not highly available, and represents a single
+point of failure. However, in a scaled environment the objective is to allow
+the environment to handle more users or to increase throughput. Redis itself
+is generally stable and can handle many requests so it is an acceptable
+trade off to have only a single instance. See [Scaling and High Availability](./README.md)
+for an overview of GitLab scaling and high availability options.
+
+The steps below are the minimum necessary to configure a Redis server with
+Omnibus:
+
+1. SSH into the Redis server.
+1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab
+   package you want using **steps 1 and 2** from the GitLab downloads page.
+     - Do not complete any other steps on the download page.
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
+
+    ```ruby
+    ## Enable Redis
+    redis['enable'] = true
+
+    ## Disable all other services
+    sidekiq['enable'] = false
+    gitlab_workhorse['enable'] = false
+    unicorn['enable'] = false
+    postgresql['enable'] = false
+    nginx['enable'] = false
+    prometheus['enable'] = false
+    alertmanager['enable'] = false
+    pgbouncer_exporter['enable'] = false
+    gitlab_monitor['enable'] = false
+    gitaly['enable'] = false
+ 
+    redis['bind'] = '0.0.0.0'
+    redis['port'] = '6379'
+    redis['password'] = 'SECRET_PASSWORD_HERE'
+ 
+    gitlab_rails['auto_migrate'] = false
+    ```
+
+1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
+1. Note the Redis node's IP address or hostname, port, and
+   Redis password. These will be necessary when configuring the GitLab
+   application servers later.
+
+Advanced configuration options are supported and can be added if
+needed.
+
+Continue configuration of other components by going
+[back to Scaled Architectures](./README.md#scalable-architecture-examples)
+
+## Redis with High Availability
+
+This section is relevant for [High Availability Architecture](./README.md#high-availability-architecture-examples)
+environments including [Horizontal](./README.md#horizontal),
+[Hybrid](./README.md#hybrid), and
+[Fully Distributed](./README.md#fully-distributed).
+
+### Provide your own Redis instance **[CORE ONLY]**
+
+If you want to use your own deployed Redis instance(s), 
+see [Provide your own Redis instance](#provide-your-own-redis-instance) 
+for more details. However, you can use the GitLab Omnibus package to easily 
+deploy the bundled Redis.  
+
+### High Availability with GitLab Omnibus **[PREMIUM ONLY]**
+
+> Experimental Redis Sentinel support was [introduced in GitLab 8.11][ce-1877].
 Starting with 8.14, Redis Sentinel is no longer experimental.
 If you've used it with versions `< 8.14` before, please check the updated
 documentation here.
@@ -52,8 +149,6 @@ failure.
 Make sure that you read this document once as a whole before configuring the
 components below.
 
-### High Availability with Sentinel
-
 > **Notes:**
 > - Starting with GitLab `8.11`, you can configure a list of Redis Sentinel
 >   servers that will monitor a group of Redis servers to provide failover support.