Merge branch 'doc/geo-89' into 'master'

Geo documentations improvement for 8.9

Important changes:

* Added note that we don't support MySQL (I don't know if we should add to the limitations or not)
* Added related files to the TOC
* Added disaster recovery instructions (we don't support this, but there are manual steps if anyone need)
* Added troubleshooting instructions

From recent feedback:

We need to improve the SSH keys part of setup instructions, as it's unclear which ssh key you have to add at geo nodes screen and that you have to pre-authorize the connection once (add to known_hosts), so clones can actually work.

Fixes gitlab-org/gitlab-ee#344

See merge request !431

doc/gitlab-geo/README.md

@@ -7,13 +7,12 @@ We recommend you use with at least GitLab 8.6 EE.
 GitLab Geo allows you to replicate your GitLab instance to other geographical
 locations as a read-only fully operational version.
 
-<!-- START doctoc generated TOC please keep comment here to allow auto update -->
-<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
-
-
 - [Overview](#overview)
 - [Setup instructions](#setup-instructions)
+    - [Database Replication](database.md)
+    - [Configuration](configuration.md)
 - [Current limitations](#current-limitations)
+- [Disaster Recovery](disaster-recovery.md)
 - [Frequently Asked Questions](#frequently-asked-questions)
     - [Can I use Geo in a disaster recovery situation?](#can-i-use-geo-in-a-disaster-recovery-situation)
     - [What data is replicated to a secondary node?](#what-data-is-replicated-to-a-secondary-node)
@@ -54,7 +53,7 @@ Geo instances. Follow the steps below in the order that they appear:
 
 1. Install GitLab Enterprise Edition on the server that will serve as the
    secondary Geo node
-1. [Setup a database replication](./database.md) in `primary <-> secondary (read-only)` topology
+1. [Setup a database replication](database.md) in `primary <-> secondary (read-only)` topology
 1. [Configure GitLab](configuration.md) and set the primary and secondary nodes
 After you set up the database replication and configure the GitLab Geo nodes,
 there are a few things to consider:
@@ -69,6 +68,7 @@ there are a few things to consider:
 - You cannot push code to secondary nodes
 - Git LFS is not supported yet
 - Git Annex is not supported yet
+- Primary node has to be online for OAuth login to happen (existing sessions and git are not affected)
 
 ## Frequently Asked Questions
 
@@ -78,6 +78,9 @@ There are limitations to what we replicate (see Current limitations).
 In an extreme data-loss situation you can make a secondary Geo into your
 primary, but this is not officially supported yet.
 
+If you still want to proceed, see our step-by-step instructions on how to 
+manually [promote a secondary node](disaster-recovery.md) into primary. 
+
 ### What data is replicated to a secondary node?
 
 We currently replicate project repositories and the whole database. This

doc/gitlab-geo/configuration.md

@@ -56,8 +56,13 @@ primary node (**Admin Area > Geo Nodes**) when adding a new one:
 sudo -u git -H ssh-keygen
 ```
 
-The public key for Omnibus installations will be at `/var/opt/gitlab/.ssh/id_rsa.pub`,
-whereas for installation from source it will be at `/home/git/.ssh/id_rsa.pub`.
+Remember to add your primary node to the `known_hosts` file of your `git` user.
+
+You can find ssh key files and `know_hosts` at `/var/opt/gitlab/.ssh/` in
+Omnibus installations or at `/home/git/.ssh/` when following the source
+installation guide.
+
+
 
 If for any reason you generate the key using a different name from the default
 `id_rsa`, or you want to generate an extra key only for the repository
@@ -93,11 +98,11 @@ add any secondary servers as well**.
 
 In the following table you can see what all these settings mean:
 
-| Setting | Description |
-| ------- | ----------- |
-| Primary | This marks a Geo Node as primary. There can be only one primary, make sure that you first add the primary node and then all the others.
-| URL | Your instance's full URL, in the same way it is configured in `gitlab.yml` (source based installations) or `/etc/gitlab/gitlab.rb` (omnibus installations). |
-|Public Key | The SSH public key of the user that your GitLab instance runs on (unless changed, should be the user `git`). That means that you have to go in each Geo Node separately and create an SSH key pair. See the [SSH key creation](#create-ssh-key-pairs-for-geo-nodes) section.
+| Setting   | Description |
+| --------- | ----------- |
+| Primary   | This marks a Geo Node as primary. There can be only one primary, make sure that you first add the primary node and then all the others. |
+| URL       | Your instance's full URL, in the same way it is configured in `gitlab.yml` (source based installations) or `/etc/gitlab/gitlab.rb` (omnibus installations). |
+|Public Key | The SSH public key of the user that your GitLab instance runs on (unless changed, should be the user `git`). That means that you have to go in each Geo Node separately and create an SSH key pair. See the [SSH key creation](#create-ssh-key-pairs-for-geo-nodes) section. |
 
 First, add your primary node by providing its full URL and the public SSH key
 you created previously. Make sure to check the box 'This is a primary node'
@@ -147,3 +152,37 @@ gitlab-rake gitlab:shell:setup
 # For source installations
 sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production
 ```
+
+## Troubleshooting
+
+Setting up Geo requires careful attention to details and sometimes it's easy to
+miss a step.
+
+Here is a checklist of questions you should ask to try to detect where you have
+to fix (all commands and path locations are for Omnibus installs):
+
+- Is Postgres replication working?
+- Are my nodes pointing to the correct database instance?
+    - You should make sure your primary Geo node points to the instance with
+      writting permissions.
+    - Any secondary nodes should point only to read-only instances.
+- Can Geo detect my current node correctly?
+    - Geo uses your defined node from `Admin > Geo` screen, and tries to match
+      with the value defined in `/etc/gitlab/gitlab.rb` configuration file.
+      The relevant line looks like: `external_url "http://gitlab.example.com"`.
+    - To check if node on current machine is correctly detected type:
+      `sudo gitlab-rails runner "Gitlab::Geo.current_node"`,
+      expect something like: `#<GeoNode id: 2, schema: "https", host: "gitlab.example.com", port: 443, relative_url_root: "", primary: false, ...>`
+    - By running the command above, `primary` should be `true` when executed in
+      the primary node, and `false` on any secondary
+- Did I defined the correct SSH Key for the node?
+    - You must create an SSH Key for `git` user
+    - This key is the one you have to inform at `Admin > Geo`
+- Can primary node communicate with secondary node by HTTP/HTTPS ports?
+- Can secondary nodes communicate with primary node by HTTP/HTTPS/SSH ports?
+- Can secondary nodes execute a succesfull git clone using git user's own
+  SSH Key to primary node repository?
+
+> This list is an atempt to document all the moving parts that can go wrong.
+We are working into getting all this steps verified automatically in a
+rake task in the future. :)

doc/gitlab-geo/database.md

@@ -220,6 +220,4 @@ When prompted, enter the password you set up for the `gitlab_replicator` user.
 
 ## MySQL replication
 
-TODO
-
-[reconfigure gitlab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
+We don't support MySQL replication for GitLab Geo.

doc/gitlab-geo/disaster-recovery.md

+# GitLab Geo Disaster Recovery
+
+> **Note:**
+This is not officially supported yet, please don't use as your only
+Disaster Recovery strategy as you may lose data.
+
+GitLab Geo replicates your database and your Git repositories. We will
+support and replicate more data in the future, that will enable you to
+fail-over with minimal effort, in a disaster situation.
+
+See [current limitations](README.md#current-limitations)
+for more information.
+
+
+## Promoting a secondary node
+
+We don't provide yet an automated way to promote a node and do fail-over,
+but you can do it manually if you have `root` access to the machine.
+
+You must make the changes in the exact specific order:
+
+1. Take down your primary node (or make sure it will not go up during this
+   process or you may lose data)
+2. Wait for any database replication to finish
+3. Promote the Postgres in your secondary node as primary
+4. Log-in to your secondary node with a user with `sudo` permission
+5. Open the interactive rails console: `sudo gitlab-rails console` and execute:
+    * List your primary node and note down it's id:
+
+        ```ruby
+        Gitlab::Geo.primary_node
+        ```
+    * Turn your primary into a secondary:
+
+        ```ruby
+        Gitlab::Geo.primary_node.update(primary: false)
+        ```
+    * List your secondary nodes and note down the id of the one you want to promote:
+
+        ```ruby
+        Gitlab::Geo.secondary_nodes
+        ```
+    * To promote a node with id `2` execute:
+
+        ```ruby
+        GeoNode.find(2).update!(primary: true)
+        ```
+    * Now you have to cleanup your new promoted node by running:
+
+        ```ruby
+        Gitlab::Geo.primary_node.oauth_application.destroy!
+        Gitlab::Geo.primary_node.system_hook.destroy!
+        ```
+    * And refresh your old primary node to behave correctly as secondary (assuming id is `1`)
+
+        ```ruby
+        GeoNode.find(1).save!
+        ```
+    * To exit the interactive console, type: `exit`
+
+6. Rsync everything in `/var/opt/gitlab/gitlab-rails/uploads` and
+   `/var/opt/gitlab/gitlab-rails/shared` from your old node to the new one.