Merge branch 'jramsay-4399-clarify-dr-docs' into 'master'

Add detail to disaster recovery docs

See merge request gitlab-org/gitlab-ee!3845

changelogs/unreleased-ee/jramsay-4399-clarify-dr-docs.yml

+---
+title: Update disaster recovery documentation with detailed steps
+merge_request: 3845
+author:
+type: other

doc/gitlab-geo/disaster-recovery.md

 # GitLab Geo Disaster Recovery
 
-> **Note:**
-GitLab Geo Disaster Recovery is in **Alpha** development. Please don't
-use as your only Disaster Recovery strategy as you may lose data.
+> **Note:** Disaster Recovery is in **Alpha** development. Do not use this as
+> your only Disaster Recovery strategy as you may lose data.
 
 GitLab Geo replicates your database and your Git repositories. We will
 support and replicate more data in the future, that will enable you to
 fail-over with minimal effort, in a disaster situation.
 
-See [current limitations](README.md#current-limitations)
-for more information.
+See [current limitations](README.md#current-limitations) for more information.
 
+## Promoting a secondary geo replica
 
-## Promoting a secondary node
+> **Warning:** Disaster Recovery does not yet support systems with multiple
+> secondary geo replicas (e.g. one primary and two or more secondaries).
 
-We don't provide yet an automated way to promote a node and do fail-over,
-but you can do it manually if you have `root` access to the machine.
+We don't currently provide an automated way to promote a geo replica and do a
+fail-over, but you can do it manually if you have `root` access to the machine.
 
-For system with only one secondary and one primary node (2-node system):
+This process promotes a secondary geo replica to a primary in the least steps.
+It does not enable GitLab Geo on the newly promoted primary.
 
-1. Take down your primary node (or make sure it will not go up during this
-   process or you may lose data)
-1. Log-in to your secondary node with a user with `sudo` permission
-1. Modify the `gitlab.rb` to reflect its new status
-1. Run `sudo gitlab-ctl promote-to-primary-node`
+1. SSH into your **primary** and stop disable GitLab.
 
-To bring your old primary node back into use as a working secondary, you need to
-run `gitlab-ctl reconfigure` against the node and then follow the
-[setup instructions](README.md) again, as if for a secondary node, from step 3.
+    ```
+    sudo gitlab-ctl stop
+
+    ```
+
+    If you do not have SSH access to your primary take the machine offline.
+    Depending on the nature of your primary this may mean physically
+    disconnecting the machine, stopping a virtual server, reconfiguring load
+    balancers, or changing DNS records (see next step).
+
+    Preventing the original primary from coming online during this process is
+    necessary to ensure data isn't added to the original primary that will not
+    be replicated to the newly promoted primary.
+
+1. SSH in to your **secondary** and login as root:
+
+    ```
+    sudo -i
+    ```
+
+1. Optional: Update the primary domain's DNS record.
+    
+    Updating the DNS records for the primary domain to point to the secondary
+    will prevent the need to update all references to the primary domain to the
+    secondary domain, like changing Git remotes and API URLs.
+
+    After updating the primary domain's DNS records to point to the secondary,
+    edit `/etc/gitlab/gitlab.rb` on the the secondary to reflect the new URL:
+
+    ```
+    # Change the existing external_url configuration
+    external_url 'https://gitlab.example.com'
+    ```
+
+1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as primary.
+
+    Remove the following line:
+
+    ```
+    ## REMOVE THIS LINE
+    geo_secondary_role['enable'] = true
+    ```
+
+    A new secondary should not be added at this time. If you want to add a new
+    secondary, do this after you have completed the entire process of promoting
+    the secondary to the primary .
+
+1. Promote the secondary to primary. Execute:
+
+    ```
+    gitlab-ctl promote-to-primary-node
+    ```
+
+1. Verify you can connect to the newly promoted primary using the URL used
+   previously for the secondary.
+1. Success! The secondary has now been promoted to primary.
+
+    If you updated the DNS records for the primary domain, these changes may
+    not have yet propagated depending on the previous DNS records TTL.
+
+## Add secondary geo replicas to a promoted primary
+
+Promoting a secondary to primary using the process above does not enable
+GitLab Geo on the new primary.
+
+To bring a new secondary online, follow the [GitLab Geo setup instructions](
+README.md#setup-instructions).