Skip to content

G
gitlab-ce
Commit 8f3fa21c authored by Nick Thomas's avatar Nick Thomas
Browse files
Options

Merge branch 'docs-geo-10-5-improvements' into 'master' 

Improve Geo documentation

Closes #4921, #4889, #4883, #4892, #4882, #4886, #4890, and #4482

See merge request gitlab-org/gitlab-ee!4585
parents 1fc7ad5f 68bb1d0d
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 406 additions and 247 deletions
doc/administration/geo/disaster_recovery/bring_primary_back.md
View file @ 8f3fa21c
...@@ -8,13 +8,11 @@ restore your original configuration. This process consists of two steps: ...@@ -8,13 +8,11 @@ restore your original configuration. This process consists of two steps:
## Configure the former primary to be a secondary ## Configure the former primary to be a secondary
Since the former primary will be out of sync with the current primary, the first Since the former primary will be out of sync with the current primary, the first step is
step is to bring the former primary up to date. There is one downside though, to bring the former primary up to date. Note, deletion of data stored on disk like
some uploads and repositories that have been deleted during an idle period of a repositories and uploads will not be replayed when bringing the former primary in back
primary node, will not be deleted from the disk but the overall sync will be into sync, which may result in increased disk usage.
much faster. As an alternative, you can set up a Alternatively, you can [setup a new secondary GitLab instance][setup-geo] to avoid this.
[GitLab instance from scratch](../replication/index.md#setup-instructions) to
workaround this downside.
To bring the former primary up to date: To bring the former primary up to date:
...@@ -25,24 +23,24 @@ To bring the former primary up to date: ...@@ -25,24 +23,24 @@ To bring the former primary up to date:
sudo gitlab-ctl start sudo gitlab-ctl start
``` ```
NOTE: **Note:** If you [disabled the primary permanently](index.md#step-2-permanently-disable-the-primary), NOTE: **Note:** If you [disabled primary permanently][disaster-recovery-disable-primary],
you need to undo those steps now. For Debian/Ubuntu you just need to run you need to undo those steps now. For Debian/Ubuntu you just need to run
`sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install `sudo systemctl enable gitlab-runsvdir`. For CentoOS 6, you need to install
the GitLab instance from scratch and setup it as a secondary node by the GitLab instance from scratch and setup it as a secondary node by
following the [setup instructions](../replication/index.md#setup-instructions). following [Setup instructions][setup-geo].
In this case you don't need to follow the next step. In this case you don't need to follow the next step.
1. [Setup database replication](../replication/database.md). Note that in this 1. [Setup database replication][database-replication]. Note that in this
case, primary refers to the current primary, and secondary refers to the case, primary refers to the current primary, and secondary refers to the
former primary. former primary.
If you have lost your original primary, follow the If you have lost your original primary, follow the
[setup instructions](../replication/index.md#setup-instructions) to set up a new secondary. [setup instructions][setup-geo] to set up a new secondary.
## Promote the secondary to primary ## Promote the secondary to primary
When the initial replication is complete and the primary and secondary are When the initial replication is complete and the primary and secondary are
closely in sync, you can do a [planned failover](planned_failover.md). closely in sync, you can do a [planned failover].
## Restore the secondary node ## Restore the secondary node
...@@ -50,3 +48,8 @@ If your objective is to have two nodes again, you need to bring your secondary ...@@ -50,3 +48,8 @@ If your objective is to have two nodes again, you need to bring your secondary
node back online as well by repeating the first step node back online as well by repeating the first step
([configure the former primary to be a secondary](#configure-the-former-primary-to-be-a-secondary)) ([configure the former primary to be a secondary](#configure-the-former-primary-to-be-a-secondary))
for the secondary node. for the secondary node.
[setup-geo]: ../replication/index.md#setup-instructions
[database-replication]: ../replication/database.md
[disaster-recovery-disable-primary]: index.md#step-2-permanently-disable-the-primary
[planned failover]: planned_failover.md
doc/administration/geo/disaster_recovery/index.md
View file @ 8f3fa21c
# Disaster Recovery # Disaster Recovery
Geo replicates your database and your Git repositories. We will Geo replicates your database, your Git repositories, and few other assets.
support and replicate more data in the future, that will enable you to We will support and replicate more data in the future, that will enable you to
failover with minimal effort, in a disaster situation. failover with minimal effort, in a disaster situation.
See [Geo current limitations](../replication/index.md#current-limitations) See [Geo current limitations][geo-limitations] for more information.
for more information.
CAUTION: **Warning:**
Disaster recovery for multi-secondary configurations is in **Alpha**.
For the latest updates, check the multi-secondary [Disaster Recovery epic][gitlab-org&65].
## Promoting secondary Geo replica in single-secondary configurations ## Promoting secondary Geo replica in single-secondary configurations
...@@ -19,7 +22,7 @@ immediately after following these instructions. ...@@ -19,7 +22,7 @@ immediately after following these instructions.
### Step 1. Allow replication to finish if possible ### Step 1. Allow replication to finish if possible
If the secondary is still replicating data from the primary, follow If the secondary is still replicating data from the primary, follow
[the planned failover docs](planned_failover.md) as closely as possible in [the planned failover docs][planned-failover] as closely as possible in
order to avoid unnecessary data loss. order to avoid unnecessary data loss.
### Step 2. Permanently disable the primary ### Step 2. Permanently disable the primary
...@@ -30,7 +33,7 @@ that has not been replicated to the secondary. This data should be treated ...@@ -30,7 +33,7 @@ that has not been replicated to the secondary. This data should be treated
as lost if you proceed. as lost if you proceed.
If an outage on your primary happens, you should do everything possible to If an outage on your primary happens, you should do everything possible to
avoid a split-brain situation where writes can occur to two different GitLab avoid a split-brain situation where writes can occur in two different GitLab
instances, complicating recovery efforts. So to prepare for the failover, we instances, complicating recovery efforts. So to prepare for the failover, we
must disable the primary. must disable the primary.
...@@ -46,21 +49,31 @@ must disable the primary. ...@@ -46,21 +49,31 @@ must disable the primary.
sudo systemctl disable gitlab-runsvdir sudo systemctl disable gitlab-runsvdir
``` ```
On some operating systems such as CentOS 6, an easy way to prevent GitLab > **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being
from being started if the machine reboots isn't available started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]).
(see [Omnibus issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3058)).
It may be safest to uninstall the GitLab package completely: It may be safest to uninstall the GitLab package completely:
```bash ```bash
yum remove gitlab-ee yum remove gitlab-ee
``` ```
> **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu
or any other distro based on the Upstart init system, you can prevent GitLab
from starting if the machine reboots by doing the following:
```bash
initctl stop gitlab-runsvvdir
echo 'manual' > /etc/init/gitlab-runsvdir.override
initctl reload-configuration
```
1. If you do not have SSH access to your primary, take the machine offline and 1. If you do not have SSH access to your primary, take the machine offline and
prevent it from rebooting by any means at your disposal. prevent it from rebooting by any means at your disposal.
Since there are many ways you may prefer to accomplish this, we will avoid a Since there are many ways you may prefer to accomplish this, we will avoid a
single recommendation. You may need to: single recommendation. You may need to:
- Reconfigure the load balancers - Reconfigure the load balancers
- Change DNS records (e.g., point the primary DNS record to the secondary node in order to stop usage of the primary) - Change DNS records (e.g., point the primary DNS record to the secondary
node in order to stop usage of the primary)
- Stop the virtual servers - Stop the virtual servers
- Block traffic through a firewall - Block traffic through a firewall
- Revoke object storage permissions from the primary - Revoke object storage permissions from the primary
...@@ -96,13 +109,29 @@ must disable the primary. ...@@ -96,13 +109,29 @@ must disable the primary.
previously for the secondary. previously for the secondary.
1. Success! The secondary has now been promoted to primary. 1. Success! The secondary has now been promoted to primary.
#### Promoting a node with HA
The `gitlab-ctl promote-to-primary-node` command cannot be used yet in conjunction with
High Availability or with multiple machines, as it can only perform changes on
a single one.
The command above does the following changes:
- Promotes the PostgreSQL secondary to primary
- Executes `gitlab-ctl reconfigure` to apply the changes in `/etc/gitlab/gitlab.rb`
- Runs `gitlab-rake geo:set_secondary_as_primary`
You need to make sure all the affected machines no longer have `geo_secondary_role['enable'] = true` in
`/etc/gitlab/gitlab.rb`, that you execute the database promotion on the required database nodes
and you execute the `gitlab-rake geo:set_secondary_as_primary` in a machine running the application server.
### Step 4. (Optional) Updating the primary domain DNS record ### Step 4. (Optional) Updating the primary domain DNS record
Updating the DNS records for the primary domain to point to the secondary Updating the DNS records for the primary domain to point to the secondary
will prevent the need to update all references to the primary domain to the will prevent the need to update all references to the primary domain to the
secondary domain, like changing Git remotes and API URLs. secondary domain, like changing Git remotes and API URLs.
1. SSH in to your **secondary** and login as root: 1. SSH into your **secondary** and login as root:
```bash ```bash
sudo -i sudo -i
...@@ -141,20 +170,17 @@ secondary domain, like changing Git remotes and API URLs. ...@@ -141,20 +170,17 @@ secondary domain, like changing Git remotes and API URLs.
Promoting a secondary to primary using the process above does not enable Promoting a secondary to primary using the process above does not enable
Geo on the new primary. Geo on the new primary.
To bring a new secondary online, follow the To bring a new secondary online, follow the [Geo setup instructions][setup-geo].
[Geo setup instructions](../replication/index.md#setup-instructions).
## Promoting secondary Geo replica in multi-secondary configurations ## Promoting secondary Geo replica in multi-secondary configurations
CAUTION: **Caution:** CAUTION: **Warning:**
Disaster Recovery for multi-secondary configurations is in Disaster Recovery for multi-secondary configurations is in **Alpha** development.
**Alpha** development. Do not use this as your only Disaster Recovery Do not use this as your only Disaster Recovery strategy as you may lose data.
strategy as you may lose data.
Disaster Recovery does not yet support systems with multiple Disaster Recovery does not yet support systems with multiple
secondary Geo replicas (e.g., one primary and two or more secondaries). We are secondary Geo replicas (e.g., one primary and two or more secondaries). We are
working on it, see [#4284](https://gitlab.com/gitlab-org/gitlab-ee/issues/4284) working on it, see [gitlab-org/gitlab-ee#4284] for details.
for details.
## Troubleshooting ## Troubleshooting
...@@ -168,6 +194,15 @@ after a failover. ...@@ -168,6 +194,15 @@ after a failover.
If you still have access to the old primary node, you can follow the If you still have access to the old primary node, you can follow the
instructions in the instructions in the
[Upgrading to GitLab 10.5](../replication/updating_the_geo_nodes.md#upgrading-to-gitlab-105) [Upgrading to GitLab 10.5][updating-geo]
section to resolve the error. Otherwise, the secret is lost and you'll need to section to resolve the error. Otherwise, the secret is lost and you'll need to
[reset two-factor authentication for all users](../../../security/two_factor_authentication.md#disabling-2fa-for-everyone). [reset two-factor authentication for all users][sec-tfa].
[gitlab-org&65]: https://gitlab.com/groups/gitlab-org/-/epics/65
[geo-limitations]: ../replication/index.md#current-limitations
[planned-failover]: planned_failover.md
[setup-geo]: ../replication/index.md#setup-instructions
[updating-geo]: ../replication/updating_the_geo_nodes.md#upgrading-to-gitlab-105
[sec-tfa]: ../../../security/two_factor_authentication.md#disabling-2fa-for-everyone
[gitlab-org/omnibus-gitlab#3058]: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3058
[gitlab-org/gitlab-ee#4284]: https://gitlab.com/gitlab-org/gitlab-ee/issues/4284
doc/administration/geo/disaster_recovery/planned_failover.md
View file @ 8f3fa21c
...@@ -4,7 +4,7 @@ A planned failover is similar to a disaster recovery scenario, except you are ab ...@@ -4,7 +4,7 @@ A planned failover is similar to a disaster recovery scenario, except you are ab
to notify users of the maintenance window, and allow data to finish replicating to to notify users of the maintenance window, and allow data to finish replicating to
secondaries. secondaries.
Please read this entire document as well as [Disaster Recovery](index.md) Please read this entire document as well as [Disaster Recovery][disaster-recovery]
before proceeding. before proceeding.
## Notify users of scheduled maintenance ## Notify users of scheduled maintenance
...@@ -13,9 +13,8 @@ On the primary, navigate to **Admin Area > Messages**, add a broadcast message. ...@@ -13,9 +13,8 @@ On the primary, navigate to **Admin Area > Messages**, add a broadcast message.
You can check under **Admin Area > Geo Nodes** to estimate how long it will You can check under **Admin Area > Geo Nodes** to estimate how long it will
take to finish syncing. An example message would be: take to finish syncing. An example message would be:
> > A scheduled maintenance will take place at XX:XX UTC. We expect it to take
A scheduled maintenance will take place at XX:XX UTC. We expect it to take less than 1 hour.
less than 1 hour.
On the secondary, you may need to clear the cache for the broadcast message On the secondary, you may need to clear the cache for the broadcast message
to show up. to show up.
...@@ -35,5 +34,7 @@ IP. ...@@ -35,5 +34,7 @@ IP.
## Promote the secondary ## Promote the secondary
Finally, follow the [Disaster Recovery docs](index.md) to promote the secondary Finally, follow the [Disaster Recovery docs][disaster-recovery] to promote the secondary
to a primary. to a primary.
[disaster-recovery]: index.md
doc/administration/geo/replication/configuration.md
View file @ 8f3fa21c
...@@ -3,14 +3,14 @@ ...@@ -3,14 +3,14 @@
>**Note:** >**Note:**
This is the documentation for the Omnibus GitLab packages. For installations This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the [**Geo nodes configuration for installations from source, follow the [**Geo nodes configuration for installations
from source**](configuration_source.md) guide. from source**][configuration-source] guide.
## Configuring a new secondary node ## Configuring a new secondary node
>**Note:** >**Note:**
This is the final step in setting up a secondary Geo node. Stages of the This is the final step in setting up a secondary Geo node. Stages of the
setup process must be completed in the documented order. setup process must be completed in the documented order.
Before attempting the steps in this stage, [complete all prior stages](index.md#using-omnibus-gitlab). Before attempting the steps in this stage, [complete all prior stages][setup-geo-omnibus].
The basic steps of configuring a secondary node are to replicate required The basic steps of configuring a secondary node are to replicate required
configurations between the primary and the secondaries; to configure a tracking configurations between the primary and the secondaries; to configure a tracking
...@@ -19,19 +19,18 @@ database on each secondary; and to start GitLab on the secondary node. ...@@ -19,19 +19,18 @@ database on each secondary; and to start GitLab on the secondary node.
You are encouraged to first read through all the steps before executing them You are encouraged to first read through all the steps before executing them
in your testing/production environment. in your testing/production environment.
>**Notes:** > **Notes:**
- **Do not** setup any custom authentication in the secondary nodes, this will be - **Do not** setup any custom authentication in the secondary nodes, this will be
handled by the primary node. handled by the primary node.
- **Do not** add anything in the secondaries Geo nodes admin area - Any change that requires access to the **Admin Area** needs to be done in the
(**Admin Area ➔ Geo Nodes**). This is handled solely by the primary node. primary node, as the secondary node is a read-only replica.
### Step 1. Manually replicate secret GitLab values ### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json` GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json`
file which *must* match between the primary and secondary nodes. Until there is file which *must* match between the primary and secondary nodes. Until there is
a means of automatically replicating these between nodes (see a means of automatically replicating these between nodes (see issue [gitlab-org/gitlab-ee#3789]),
[issue #3789](https://gitlab.com/gitlab-org/gitlab-ee/issues/3789)), they must they must be manually replicated to the secondary.
be manually replicated to the secondary.
1. SSH into the **primary** node, and execute the command below: 1. SSH into the **primary** node, and execute the command below:
...@@ -76,21 +75,12 @@ be manually replicated to the secondary. ...@@ -76,21 +75,12 @@ be manually replicated to the secondary.
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
Once reconfigured, the secondary will automatically start
replicating missing data from the primary in a process known as backfill.
Meanwhile, the primary node will start to notify the secondary of any changes, so
that the secondary can act on those notifications immediately.
Make sure the secondary instance is
running and accessible. You can login to the secondary node
with the same credentials as used in the primary.
### Step 2. Manually replicate primary SSH host keys ### Step 2. Manually replicate primary SSH host keys
GitLab integrates with the system-installed SSH daemon, designating a user GitLab integrates with the system-installed SSH daemon, designating a user
(typically named git) through which all access requests are handled. (typically named git) through which all access requests are handled.
In a [Disaster Recovery](../disaster_recovery/index.md) situation, GitLab system In a [Disaster Recovery] situation, GitLab system
administrators will promote a secondary Geo replica to a primary and they can administrators will promote a secondary Geo replica to a primary and they can
update the DNS records for the primary domain to point to the secondary to prevent update the DNS records for the primary domain to point to the secondary to prevent
the need to update all references to the primary domain to the secondary domain, the need to update all references to the primary domain to the secondary domain,
...@@ -140,13 +130,49 @@ keys must be manually replicated to the secondary node. ...@@ -140,13 +130,49 @@ keys must be manually replicated to the secondary node.
service ssh restart service ssh restart
``` ```
### Step 3. (Optional) Enabling hashed storage (from GitLab 10.0) ### Step 3. Add the secondary GitLab node
1. Visit the **primary** node's **Admin Area ➔ Geo Nodes**
(`/admin/geo_nodes`) in your browser.
1. Add the secondary node by providing its full URL. **Do NOT** check the box
'This is a primary node'.
1. Optionally, choose which namespaces should be replicated by the
secondary node. Leave blank to replicate all. Read more in
[selective replication](#selective-replication).
1. Click the **Add node** button.
1. SSH into your GitLab **secondary** server and restart the services:
```
gitlab-ctl restart
```
Check if there are any common issue with your Geo setup by running:
```
gitlab-rake gitlab:geo:check
```
1. SSH into your GitLab **primary** server and login as root to verify the
secondary is reachable or there are any common issue with your Geo setup:
>**Warning** ```
Hashed storage is in **Beta**. It is not considered production-ready. See gitlab-rake gitlab:geo:check
[Hashed Storage](../../repository_storage_types.md) for more detail, ```
and for the latest updates, check
[infrastructure issue #2821](https://gitlab.com/gitlab-com/infrastructure/issues/2821). Once added to the admin panel and restarted, the secondary will automatically start
replicating missing data from the primary in a process known as **backfill**.
Meanwhile, the primary node will start to notify the secondary of any changes, so
that the secondary can act on those notifications immediately.
Make sure the secondary instance is running and accessible.
You can login to the secondary node with the same credentials as used in the primary.
### Step 4. (Optional) Enabling hashed storage (from GitLab 10.0)
CAUTION: **Warning**:
Hashed storage is in **Beta**. It is not considered production-ready. See
[Hashed Storage] for more detail, and for the latest updates, check
infrastructure issue [gitlab-com/infrastructure#2821].
Using hashed storage significantly improves Geo replication - project and group Using hashed storage significantly improves Geo replication - project and group
renames no longer require synchronization between nodes. renames no longer require synchronization between nodes.
...@@ -157,24 +183,24 @@ renames no longer require synchronization between nodes. ...@@ -157,24 +183,24 @@ renames no longer require synchronization between nodes.
![](img/hashed_storage.png) ![](img/hashed_storage.png)
### Step 4. (Optional) Configuring the secondary to trust the primary ### Step 5. (Optional) Configuring the secondary to trust the primary
You can safely skip this step if your primary uses a CA-issued HTTPS certificate. You can safely skip this step if your primary uses a CA-issued HTTPS certificate.
If your primary is using a self-signed certificate for *HTTPS* support, you will If your primary is using a self-signed certificate for *HTTPS* support, you will
need to add that certificate to the secondary's trust store. Retrieve the need to add that certificate to the secondary's trust store. Retrieve the
certificate from the primary and follow certificate from the primary and follow
[these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html) [these instructions][omnibus-ssl]
on the secondary. on the secondary.
### Step 5. Enable Git access over HTTP/HTTPS ### Step 6. Enable Git access over HTTP/HTTPS
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area ➔ Settings** method to be enabled. Navigate to **Admin Area ➔ Settings**
(`/admin/application_settings`) on the primary node, and set (`/admin/application_settings`) on the primary node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. `Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 6. Verify proper functioning of the secondary node ### Step 7. Verify proper functioning of the secondary node
Congratulations! Your secondary geo node is now configured! Congratulations! Your secondary geo node is now configured!
...@@ -190,7 +216,7 @@ node's Geo Nodes dashboard in your browser. ...@@ -190,7 +216,7 @@ node's Geo Nodes dashboard in your browser.
![Geo dashboard](img/geo_node_dashboard.png) ![Geo dashboard](img/geo_node_dashboard.png)
If your installation isn't working properly, check the If your installation isn't working properly, check the
[troubleshooting document](troubleshooting.md). [troubleshooting document].
The two most obvious issues that can become apparent in the dashboard are: The two most obvious issues that can become apparent in the dashboard are:
...@@ -198,7 +224,7 @@ The two most obvious issues that can become apparent in the dashboard are: ...@@ -198,7 +224,7 @@ The two most obvious issues that can become apparent in the dashboard are:
1. Instance to instance notification not working. In that case, it can be 1. Instance to instance notification not working. In that case, it can be
something of the following: something of the following:
- You are using a custom certificate or custom CA (see the - You are using a custom certificate or custom CA (see the
[troubleshooting document](troubleshooting.md)) [troubleshooting document])
- The instance is firewalled (check your firewall rules) - The instance is firewalled (check your firewall rules)
Please note that disabling a secondary node will stop the sync process. Please note that disabling a secondary node will stop the sync process.
...@@ -206,7 +232,7 @@ Please note that disabling a secondary node will stop the sync process. ...@@ -206,7 +232,7 @@ Please note that disabling a secondary node will stop the sync process.
Please note that if `git_data_dirs` is customized on the primary for multiple Please note that if `git_data_dirs` is customized on the primary for multiple
repository shards you must duplicate the same configuration on the secondary. repository shards you must duplicate the same configuration on the secondary.
Point your users to the ["Using a Geo Server" guide](using_a_geo_server.md). Point your users to the ["Using a Geo Server" guide][using-geo].
Currently, this is what is synced: Currently, this is what is synced:
...@@ -245,3 +271,13 @@ See the [updating the Geo nodes document](updating_the_geo_nodes.md). ...@@ -245,3 +271,13 @@ See the [updating the Geo nodes document](updating_the_geo_nodes.md).
## Troubleshooting ## Troubleshooting
See the [troubleshooting document](troubleshooting.md). See the [troubleshooting document](troubleshooting.md).
[configuration-source]: configuration_source.md
[setup-geo-omnibus]: index.md#using-omnibus-gitlab
[Hashed Storage]: ../../repository_storage_types.md
[Disaster Recovery]: ../disaster_recovery/index.md
[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789
[gitlab-com/infrastructure#2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821
[omnibus-ssl]: https://docs.gitlab.com/omnibus/settings/ssl.html
[troubleshooting document]: troubleshooting.md
[using-geo]: using_a_geo_server.md
doc/administration/geo/replication/configuration_source.md
View file @ 8f3fa21c
...@@ -3,14 +3,14 @@ ...@@ -3,14 +3,14 @@
>**Note:** >**Note:**
This is the documentation for installations from source. For installations This is the documentation for installations from source. For installations
using the Omnibus GitLab packages, follow the using the Omnibus GitLab packages, follow the
[**Omnibus Geo nodes configuration**](configuration.md) guide. [**Omnibus Geo nodes configuration**][configuration] guide.
## Configuring a new secondary node ## Configuring a new secondary node
>**Note:** >**Note:**
This is the final step in setting up a secondary Geo node. Stages of the setup This is the final step in setting up a secondary Geo node. Stages of the setup
process must be completed in the documented order. Before attempting the steps process must be completed in the documented order. Before attempting the steps
in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source). in this stage, [complete all prior stages][setup-geo-source].
The basic steps of configuring a secondary node are to replicate required The basic steps of configuring a secondary node are to replicate required
configurations between the primary and the secondaries; to configure a tracking configurations between the primary and the secondaries; to configure a tracking
...@@ -30,8 +30,7 @@ in your testing/production environment. ...@@ -30,8 +30,7 @@ in your testing/production environment.
GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml` GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml`
file which *must* match between the primary and secondary nodes. Until there is file which *must* match between the primary and secondary nodes. Until there is
a means of automatically replicating these between nodes (see a means of automatically replicating these between nodes (see [gitlab-org/gitlab-ee#3789]), they must
[issue #3789](https://gitlab.com/gitlab-org/gitlab-ee/issues/3789)), they must
be manually replicated to the secondary. be manually replicated to the secondary.
1. SSH into the **primary** node, and execute the command below: 1. SSH into the **primary** node, and execute the command below:
...@@ -71,12 +70,6 @@ be manually replicated to the secondary. ...@@ -71,12 +70,6 @@ be manually replicated to the secondary.
chmod 0600 /home/git/gitlab/config/secrets.yml chmod 0600 /home/git/gitlab/config/secrets.yml
``` ```
1. Restart GitLab for the changes to take effect:
```bash
service gitlab restart
```
Once restarted, the secondary will automatically start replicating missing data Once restarted, the secondary will automatically start replicating missing data
from the primary in a process known as backfill. Meanwhile, the primary node from the primary in a process known as backfill. Meanwhile, the primary node
will start to notify the secondary of any changes, so that the secondary can will start to notify the secondary of any changes, so that the secondary can
...@@ -87,13 +80,50 @@ the secondary node with the same credentials as used in the primary. ...@@ -87,13 +80,50 @@ the secondary node with the same credentials as used in the primary.
### Step 2. Manually replicate primary SSH host keys ### Step 2. Manually replicate primary SSH host keys
Read [Manually replicate primary SSH host keys](configuration.md#step-2-manually-replicate-primary-ssh-host-keys) Read [Manually replicate primary SSH host keys][configuration-replicate-ssh]
### Step 3. (Optional) Enabling hashed storage (from GitLab 10.0) ### Step 3. Add the secondary GitLab node
Read [Enabling Hashed Storage](configuration.md#step-3-optional-enabling-hashed-storage-from-gitlab-10-0) 1. Visit the **primary** node's **Admin Area ➔ Geo Nodes**
(`/admin/geo_nodes`) in your browser.
1. Add the secondary node by providing its full URL. **Do NOT** check the box
'This is a primary node'.
1. Optionally, choose which namespaces should be replicated by the
secondary node. Leave blank to replicate all. Read more in
[selective replication](#selective-replication).
1. Click the **Add node** button.
1. SSH into your GitLab **secondary** server and restart the services:
### Step 4. (Optional) Configuring the secondary to trust the primary ```bash
service gitlab restart
```
Check if there are any common issue with your Geo setup by running:
```bash
bundle exec rake gitlab:geo:check
```
1. SSH into your GitLab **primary** server and login as root to verify the
secondary is reachable or there are any common issue with your Geo setup:
```bash
bundle exec rake gitlab:geo:check
```
Once reconfigured, the secondary will automatically start
replicating missing data from the primary in a process known as backfill.
Meanwhile, the primary node will start to notify the secondary of any changes, so
that the secondary can act on those notifications immediately.
Make sure the secondary instance is running and accessible.
You can login to the secondary node with the same credentials as used in the primary.
### Step 4. (Optional) Enabling hashed storage (from GitLab 10.0)
Read [Enabling Hashed Storage][configuration-hashed-storage]
### Step 5. (Optional) Configuring the secondary to trust the primary
You can safely skip this step if your primary uses a CA-issued HTTPS certificate. You can safely skip this step if your primary uses a CA-issued HTTPS certificate.
...@@ -109,22 +139,31 @@ cp primary.geo.example.com.crt /usr/local/share/ca-certificates ...@@ -109,22 +139,31 @@ cp primary.geo.example.com.crt /usr/local/share/ca-certificates
update-ca-certificates update-ca-certificates
``` ```
### Step 5. Enable Git access over HTTP/HTTPS ### Step 6. Enable Git access over HTTP/HTTPS
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area ➔ Settings** method to be enabled. Navigate to **Admin Area ➔ Settings**
(`/admin/application_settings`) on the primary node, and set (`/admin/application_settings`) on the primary node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. `Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 6. Verify proper functioning of the secondary node ### Step 7. Verify proper functioning of the secondary node
Read [Verify proper functioning of the secondary node](configuration.md#step-6-verify-proper-functioning-of-the-secondary-node). Read [Verify proper functioning of the secondary node][configuration-verify-node].
## Selective synchronization ## Selective synchronization
Read [Selective synchronization](configuration.md#selective-synchronization). Read [Selective synchronization][configuration-selective-replication].
## Troubleshooting ## Troubleshooting
Read the [troubleshooting document](troubleshooting.md). Read the [troubleshooting document][troubleshooting].
[setup-geo-source]: index.md#using-gitlab-installed-from-source
[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789
[configuration]: configuration.md
[configuration-hashed-storage]: configuration.md#step-4-optional-enabling-hashed-storage-from-gitlab-10-0
[configuration-replicate-ssh]: configuration.md#step-2-manually-replicate-primary-ssh-host-keys
[configuration-selective-replication]: configuration.md#selective-synchronization
[configuration-verify-node]: configuration.md#step-7-verify-proper-functioning-of-the-secondary-node
[troubleshooting]: troubleshooting.md
doc/administration/geo/replication/database_source.md
View file @ 8f3fa21c
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
>**Note:** >**Note:**
This is the documentation for installations from source. For installations This is the documentation for installations from source. For installations
using the Omnibus GitLab packages, follow the using the Omnibus GitLab packages, follow the
[**database replication for Omnibus GitLab**](database.md) guide. [**database replication for Omnibus GitLab**][database] guide.
>**Note:** >**Note:**
The stages of the setup process must be completed in the documented order. The stages of the setup process must be completed in the documented order.
...@@ -26,8 +26,7 @@ connect to secondary database servers (which are read-only too). ...@@ -26,8 +26,7 @@ connect to secondary database servers (which are read-only too).
In many databases documentation you will see "primary" being referenced as "master" In many databases documentation you will see "primary" being referenced as "master"
and "secondary" as either "slave" or "standby" server (read-only). and "secondary" as either "slave" or "standby" server (read-only).
We recommend using [PostgreSQL replication We recommend using [PostgreSQL replication slots][replication-slots-article]
slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75)
to ensure the primary retains all the data necessary for the secondaries to to ensure the primary retains all the data necessary for the secondaries to
recover. See below for more details. recover. See below for more details.
...@@ -60,8 +59,12 @@ The following guide assumes that: ...@@ -60,8 +59,12 @@ The following guide assumes that:
1. Create a [replication user] named `gitlab_replicator`: 1. Create a [replication user] named `gitlab_replicator`:
```bash ```sql
sudo -u postgres psql -c "CREATE USER gitlab_replicator REPLICATION ENCRYPTED PASSWORD 'thepassword';" --- Create a new user 'replicator'
CREATE USER gitlab_replicator;
--- Set/change a password and grants replication privilege
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD 'replicationpasswordhere';
``` ```
1. Make sure your the `gitlab` database user has a password defined 1. Make sure your the `gitlab` database user has a password defined
...@@ -149,12 +152,11 @@ The following guide assumes that: ...@@ -149,12 +152,11 @@ The following guide assumes that:
The `listen_address` option opens PostgreSQL up to external connections The `listen_address` option opens PostgreSQL up to external connections
with the interface corresponding to the given IP. See [the PostgreSQL with the interface corresponding to the given IP. See [the PostgreSQL
documentation](https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html) documentation][pg-docs-runtime-conn] for more details.
for more details.
You may also want to edit the `wal_keep_segments` and `max_wal_senders` to You may also want to edit the `wal_keep_segments` and `max_wal_senders` to
match your database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html) match your database replication requirements. Consult the
for more information. [PostgreSQL - Replication documentation][pg-docs-runtime-replication] for more information.
1. Set the access control on the primary to allow TCP connections using the 1. Set the access control on the primary to allow TCP connections using the
server's public IP and set the connection from the secondary to require a server's public IP and set the connection from the secondary to require a
...@@ -162,8 +164,7 @@ The following guide assumes that: ...@@ -162,8 +164,7 @@ The following guide assumes that:
`/etc/postgresql/9.x/main/pg_hba.conf`): `/etc/postgresql/9.x/main/pg_hba.conf`):
```bash ```bash
host all all 127.0.0.1/32 trust host all all 1.2.3.4/32 md5
host all all 1.2.3.4/32 trust
host replication gitlab_replicator 5.6.7.8/32 md5 host replication gitlab_replicator 5.6.7.8/32 md5
``` ```
...@@ -173,8 +174,7 @@ The following guide assumes that: ...@@ -173,8 +174,7 @@ The following guide assumes that:
address: address:
```bash ```bash
host all all 127.0.0.1/32 trust host all all 1.2.3.4/32 md5
host all all 1.2.3.4/32 trust
host replication gitlab_replicator 5.6.7.8/32 md5 host replication gitlab_replicator 5.6.7.8/32 md5
host replication gitlab_replicator 11.22.33.44/32 md5 host replication gitlab_replicator 11.22.33.44/32 md5
``` ```
...@@ -200,13 +200,9 @@ The following guide assumes that: ...@@ -200,13 +200,9 @@ The following guide assumes that:
`netstat -plnt` to make sure that PostgreSQL is listening to the server's `netstat -plnt` to make sure that PostgreSQL is listening to the server's
public IP. public IP.
### Step 2. Add the secondary GitLab node ### Step 2. Configure the secondary server
Follow the steps in ["add the secondary GitLab node"](database.md#step-2-add-the-secondary-gitlab-node).
### Step 3. Configure the secondary server Follow the first steps in ["configure the secondary server"][database-replication],
Follow the first steps in ["configure the secondary server"](database.md#step-3-configure-the-secondary-server),
but note that since you are installing from source, the username and but note that since you are installing from source, the username and
group listed as `gitlab-psql` in those steps should be replaced by `postgres` group listed as `gitlab-psql` in those steps should be replaced by `postgres`
instead. After completing the "Test that the `gitlab-psql` user can connect to instead. After completing the "Test that the `gitlab-psql` user can connect to
...@@ -293,7 +289,7 @@ node. ...@@ -293,7 +289,7 @@ node.
And edit the content of `database_geo.yml` and to add `fdw: true` to And edit the content of `database_geo.yml` and to add `fdw: true` to
the `production:` block. the `production:` block.
### Step 4. Initiate the replication process ### Step 3. Initiate the replication process
Below we provide a script that connects the database on the secondary node to Below we provide a script that connects the database on the secondary node to
the database on the primary node, replicates the database, and creates the the database on the primary node, replicates the database, and creates the
...@@ -315,7 +311,7 @@ data before running `pg_basebackup`. ...@@ -315,7 +311,7 @@ data before running `pg_basebackup`.
1. Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the 1. Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
embedded paths if necessary: embedded paths if necessary:
```bash ```
#!/bin/bash #!/bin/bash
PORT="5432" PORT="5432"
...@@ -332,7 +328,8 @@ data before running `pg_basebackup`. ...@@ -332,7 +328,8 @@ data before running `pg_basebackup`.
read SSLMODE read SSLMODE
echo Stopping PostgreSQL and all GitLab services echo Stopping PostgreSQL and all GitLab services
gitlab-ctl stop sudo service gitlab stop
sudo service postgresql stop
echo Backing up postgresql.conf echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
...@@ -356,8 +353,8 @@ data before running `pg_basebackup`. ...@@ -356,8 +353,8 @@ data before running `pg_basebackup`.
echo Restoring postgresql.conf echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Starting PostgreSQL and all GitLab services echo Starting PostgreSQL
gitlab-ctl start sudo service postgresql start
``` ```
1. Run it with: 1. Run it with:
...@@ -375,7 +372,7 @@ data before running `pg_basebackup`. ...@@ -375,7 +372,7 @@ data before running `pg_basebackup`.
**not** safe over the public Internet! **not** safe over the public Internet!
You can read more details about each `sslmode` in the You can read more details about each `sslmode` in the
[PostgreSQL documentation](https://www.postgresql.org/docs/9.6/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION); [PostgreSQL documentation][pg-docs-ssl];
the instructions above are carefully written to ensure protection against the instructions above are carefully written to ensure protection against
both passive eavesdroppers and active "man-in-the-middle" attackers. both passive eavesdroppers and active "man-in-the-middle" attackers.
...@@ -389,7 +386,14 @@ MySQL replication is not supported for Geo. ...@@ -389,7 +386,14 @@ MySQL replication is not supported for Geo.
Read the [troubleshooting document](troubleshooting.md). Read the [troubleshooting document](troubleshooting.md).
[replication-slots-article]: https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75
[pgback]: http://www.postgresql.org/docs/9.6/static/app-pgbasebackup.html [pgback]: http://www.postgresql.org/docs/9.6/static/app-pgbasebackup.html
[replication user]:https://wiki.postgresql.org/wiki/Streaming_Replication [replication user]:https://wiki.postgresql.org/wiki/Streaming_Replication
[FDW]: https://www.postgresql.org/docs/9.6/static/postgres-fdw.html [FDW]: https://www.postgresql.org/docs/9.6/static/postgres-fdw.html
[toc]: index.md#using-gitlab-installed-from-source [toc]: index.md#using-gitlab-installed-from-source
[database]: database.md
[add-geo-node]: configuration.md#step-3-add-the-secondary-gitlab-node
[database-replication]: database.md#step-2-configure-the-secondary-server
[pg-docs-ssl]: https://www.postgresql.org/docs/9.6/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION
[pg-docs-runtime-conn]: https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html
[pg-docs-runtime-replication]: https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html
doc/administration/geo/replication/docker_registry.md
View file @ 8f3fa21c
# Docker Registry for a secondary node # Docker Registry for a secondary node
You can setup a [Docker Registry](https://docs.docker.com/registry/) on your You can setup a [Docker Registry] on your
secondary Geo node that mirrors the one on the primary Geo node. secondary Geo node that mirrors the one on the primary Geo node.
## Storage support ## Storage support
CAUTION: **Warning:** CAUTION: **Warning:**
If you use [local storage](../../container_registry.md#container-registry-storage-driver) If you use [local storage][registry-storage]
for the Container Registry you **cannot** replicate it to the secondary Geo node. for the Container Registry you **cannot** replicate it to the secondary Geo node.
Docker Registry currently supports a few types of storages. If you choose a Docker Registry currently supports a few types of storages. If you choose a
distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker
Registry on a primary Geo node, you can use the same storage for a secondary Registry on a primary Geo node, you can use the same storage for a secondary
Docker Registry as well. For more information, read the Docker Registry as well. For more information, read the
[Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) [Load balancing considerations][registry-load-balancing]
when deploying the Registry, and how to setup the storage driver for GitLab's when deploying the Registry, and how to setup the storage driver for GitLab's
integrated [Container Registry](../../container_registry.md#container-registry-storage-driver). integrated [Container Registry][registry-storage].
[ee]: https://about.gitlab.com/products/ [ee]: https://about.gitlab.com/products/
[Docker Registry]: https://docs.docker.com/registry/
[registry-storage]: ../../container_registry.md#container-registry-storage-driver
[registry-load-balancing]: https://docs.docker.com/registry/deploying/#load-balancing-considerations
doc/administration/geo/replication/high_availability.md
View file @ 8f3fa21c
...@@ -8,7 +8,7 @@ described, it is possible to adapt these instructions to your needs. ...@@ -8,7 +8,7 @@ described, it is possible to adapt these instructions to your needs.
![Geo HA Diagram](../../img/high_availability/geo-ha-diagram.png) ![Geo HA Diagram](../../img/high_availability/geo-ha-diagram.png)
_[diagram source - gitlab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ _[diagram source - gitlab employees only][diagram-source]_
The topology above assumes that the primary and secondary Geo clusters The topology above assumes that the primary and secondary Geo clusters
are located in two separate locations, on their own virtual network are located in two separate locations, on their own virtual network
...@@ -21,7 +21,7 @@ The only external way to access the two Geo deployments is by HTTPS at ...@@ -21,7 +21,7 @@ The only external way to access the two Geo deployments is by HTTPS at
`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. `gitlab.us.example.com` and `gitlab.eu.example.com` in the example above.
> **Note:** The primary and secondary Geo deployments must be able to > **Note:** The primary and secondary Geo deployments must be able to
> communicate to each other over HTTPS. communicate to each other over HTTPS.
## Redis and PostgreSQL High Availability ## Redis and PostgreSQL High Availability
...@@ -33,8 +33,7 @@ The two services will instead be configured such that ...@@ -33,8 +33,7 @@ The two services will instead be configured such that
they will each run on a single machine. they will each run on a single machine.
For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for
[PostgreSQL](../../high_availability/database.md) and [PostgreSQL][postgresql-ha] and [Redis][redis-ha], respectively.
[Redis](../../high_availability/redis.md), respectively.
From these instructions you will need the following for the examples below: From these instructions you will need the following for the examples below:
* `gitlab_rails['db_password']` for the PostgreSQL "DB password" * `gitlab_rails['db_password']` for the PostgreSQL "DB password"
...@@ -53,18 +52,18 @@ Make sure you have GitLab EE installed using the ...@@ -53,18 +52,18 @@ Make sure you have GitLab EE installed using the
On the **primary** backend servers configure the following services: On the **primary** backend servers configure the following services:
* [Redis](../../high_availability/redis.md) for high availability. * [Redis][redis-ha] for high availability.
* [NFS Server](../../high_availability/nfs.md) for repository, LFS, and upload storage. * [NFS Server][nfs-ha] for repository, LFS, and upload storage.
* [PostgreSQL](../../high_availability/database.md) for high availability. * [PostgreSQL][postgresql-ha] for high availability.
On the **secondary** backend servers configure the following services: On the **secondary** backend servers configure the following services:
* [Redis](../../high_availability/redis.md) for high availability. * [Redis][redis-ha] for high availability.
* [NFS Server](../../high_availability/nfs.md) which will store data that is synchronized from the Geo primary. * [NFS Server][nfs-ha] which will store data that is synchronized from the Geo primary.
### Step 2: Configure the Postgres services on the Geo Secondary ### Step 2: Configure the Postgres services on the Geo Secondary
1. Configure the [secondary Geo PostgreSQL database](database.md) 1. Configure the [secondary Geo PostgreSQL database][database]
as a read-only secondary of the primary Geo PostgreSQL database. as a read-only secondary of the primary Geo PostgreSQL database.
1. Configure the Geo tracking database on the secondary server, to do this modify `/etc/gitlab/gitlab.rb`: 1. Configure the Geo tracking database on the secondary server, to do this modify `/etc/gitlab/gitlab.rb`:
...@@ -82,7 +81,7 @@ On the **secondary** backend servers configure the following services: ...@@ -82,7 +81,7 @@ On the **secondary** backend servers configure the following services:
NOTE: **Note:** NOTE: **Note:**
Be sure that other non-postgresql services are disabled by setting `enable` to `false` in Be sure that other non-postgresql services are disabled by setting `enable` to `false` in
the [gitlab.rb configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template). the [gitlab.rb configuration][gitlab-rb-template].
After making these changes be sure to run `sudo gitlab-ctl reconfigure` so that they take effect. After making these changes be sure to run `sudo gitlab-ctl reconfigure` so that they take effect.
...@@ -91,7 +90,7 @@ After making these changes be sure to run `sudo gitlab-ctl reconfigure` so that ...@@ -91,7 +90,7 @@ After making these changes be sure to run `sudo gitlab-ctl reconfigure` so that
In this topology there will need to be a load balancers at each geographical location In this topology there will need to be a load balancers at each geographical location
to route traffic to the application servers. to route traffic to the application servers.
See the [Load Balancer for GitLab HA](../../high_availability/load_balancer.md) See the [Load Balancer for GitLab HA][load-balancer-ha]
documentation for more information. documentation for more information.
### Step 4: Configure the Geo Frontend Application Servers ### Step 4: Configure the Geo Frontend Application Servers
...@@ -130,7 +129,8 @@ the addresses of the remote endpoints for PostgreSQL and Redis will need to be s ...@@ -130,7 +129,8 @@ the addresses of the remote endpoints for PostgreSQL and Redis will need to be s
On the secondary the remote endpoint for the PostgreSQL Geo database will On the secondary the remote endpoint for the PostgreSQL Geo database will
be specified. be specified.
1. Edit `/etc/gitlab/gitlab.rb` and ensure the following to disable PostgreSQL and Redis from running locally. Configure the secondary to connect to the Geo tracking database. 1. Edit `/etc/gitlab/gitlab.rb` and ensure the following to disable PostgreSQL and
Redis from running locally. Configure the secondary to connect to the Geo tracking database.
```ruby ```ruby
...@@ -164,7 +164,7 @@ be specified. ...@@ -164,7 +164,7 @@ be specified.
``` ```
After making these changes [Reconfigure GitLab][] so that they take effect. After making these changes [Reconfigure GitLab][gitlab-reconfigure] so that they take effect.
On the primary the following GitLab frontend services will be enabled: On the primary the following GitLab frontend services will be enabled:
...@@ -192,5 +192,11 @@ On the secondary the following GitLab frontend services will be enabled: ...@@ -192,5 +192,11 @@ On the secondary the following GitLab frontend services will be enabled:
Verify these services by running `sudo gitlab-ctl status` on the frontend Verify these services by running `sudo gitlab-ctl status` on the frontend
application servers. application servers.
[reconfigure GitLab]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure [diagram-source]: https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit
[restart GitLab]: ../../restart_gitlab.md#omnibus-gitlab-restart [gitlab-reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure
[redis-ha]: ../../high_availability/redis.md
[postgresql-ha]: ../../high_availability/database.md
[nfs-ha]: ../../high_availability/nfs.md
[load-balancer-ha]: ../../high_availability/load_balancer.md
[database]: database.md
[gitlab-rb-template]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template
doc/administration/geo/replication/index.md
View file @ 8f3fa21c
...@@ -7,7 +7,7 @@ ...@@ -7,7 +7,7 @@
basic Geo features, or latest version for a better experience. basic Geo features, or latest version for a better experience.
- You should make sure that all nodes run the same GitLab version. - You should make sure that all nodes run the same GitLab version.
- Geo requires PostgreSQL 9.6 and Git 2.9 in addition to GitLab's usual - Geo requires PostgreSQL 9.6 and Git 2.9 in addition to GitLab's usual
[minimum requirements](../../../install/requirements.md) [minimum requirements][install-requirements]
- Using Geo in combination with High Availability is considered **GA** in GitLab Enterprise Edition 10.4 - Using Geo in combination with High Availability is considered **GA** in GitLab Enterprise Edition 10.4
>**Note:** >**Note:**
...@@ -51,14 +51,13 @@ to reading any data available in the GitLab web interface (see [current limitati ...@@ -51,14 +51,13 @@ to reading any data available in the GitLab web interface (see [current limitati
improving speed for distributed teams improving speed for distributed teams
- Helps reducing the loading time for automated tasks, - Helps reducing the loading time for automated tasks,
custom integrations and internal workflows custom integrations and internal workflows
- Quickly failover to a Geo secondary in a - Quickly failover to a Geo secondary in a [Disaster Recovery][disaster-recovery] scenario
[Disaster Recovery](../disaster_recovery/index.md) scenario - Allows [planned failover] to a Geo secondary
- Allows [planned failover](../disaster_recovery/planned_failover.md) to a Geo secondary
## Architecture ## Architecture
The following diagram illustrates the underlying architecture of Geo The following diagram illustrates the underlying architecture of Geo
([source diagram](https://docs.google.com/drawings/d/1Abw0P_H0Ew1-2Lj_xPDRWP87clGIke-1fil7_KQqrtE/edit)). ([source diagram]).
![Geo architecture](img/geo_architecture.png) ![Geo architecture](img/geo_architecture.png)
...@@ -88,7 +87,7 @@ current version of OpenSSH: ...@@ -88,7 +87,7 @@ current version of OpenSSH:
Note that CentOS 6 and 7.0 ship with an old version of OpenSSH that do not Note that CentOS 6 and 7.0 ship with an old version of OpenSSH that do not
support a feature that Geo requires. See the [documentation on Geo SSH support a feature that Geo requires. See the [documentation on Geo SSH
access](../../operations/fast_ssh_key_lookup.md) for more details. access][fast-ssh-lookup] for more details.
### LDAP ### LDAP
...@@ -100,7 +99,7 @@ tokens will still work. ...@@ -100,7 +99,7 @@ tokens will still work.
Check with your LDAP provider for instructions on on how to set up Check with your LDAP provider for instructions on on how to set up
replication. For example, OpenLDAP provides [these replication. For example, OpenLDAP provides [these
instructions](https://www.openldap.org/doc/admin24/replication.html). instructions][ldap-replication].
### Geo Tracking Database ### Geo Tracking Database
...@@ -143,18 +142,16 @@ If you installed GitLab using the Omnibus packages (highly recommended): ...@@ -143,18 +142,16 @@ If you installed GitLab using the Omnibus packages (highly recommended):
1. [Install GitLab Enterprise Edition][install-ee] on the server that will serve 1. [Install GitLab Enterprise Edition][install-ee] on the server that will serve
as the **secondary** Geo node. Do not create an account or login to the new as the **secondary** Geo node. Do not create an account or login to the new
secondary node. secondary node.
1. [Upload the GitLab License](../../../user/admin_area/license.md) on the **primary** 1. [Upload the GitLab License][upload-license] on the **primary**
Geo node to unlock Geo. Geo node to unlock Geo.
1. [Setup the database replication](database.md) (`primary (read-write) <-> 1. [Setup the database replication][database] (`primary (read-write) <->
secondary (read-only)` topology). secondary (read-only)` topology).
1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md), 1. [Configure fast lookup of authorized SSH keys in the database][fast-ssh-lookup],
this step is required and needs to be done on both the primary AND secondary nodes. this step is required and needs to be done on both the primary AND secondary nodes.
1. [Configure GitLab](configuration.md) to set the primary and secondary nodes. 1. [Configure GitLab][configuration] to set the primary and secondary nodes.
1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) 1. Optional: [Configure a secondary LDAP server][config-ldap]
for the secondary. See [notes on LDAP](#ldap). for the secondary. See [notes on LDAP](#ldap).
1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). 1. [Follow the "Using a Geo Server" guide][using-geo].
[install-ee]: https://about.gitlab.com/downloads-ee/ "GitLab Enterprise Edition Omnibus packages downloads page"
### Using GitLab installed from source ### Using GitLab installed from source
...@@ -163,50 +160,60 @@ If you installed GitLab from source: ...@@ -163,50 +160,60 @@ If you installed GitLab from source:
1. [Install GitLab Enterprise Edition][install-ee-source] on the server that 1. [Install GitLab Enterprise Edition][install-ee-source] on the server that
will serve as the **secondary** Geo node. Do not create an account or login will serve as the **secondary** Geo node. Do not create an account or login
to the new secondary node. to the new secondary node.
1. [Upload the GitLab License](../../../user/admin_area/license.md) on the **primary** 1. [Upload the GitLab License][upload-license] on the **primary**
Geo node to unlock Geo. Geo node to unlock Geo.
1. [Setup the database replication](database_source.md) (`primary (read-write) 1. [Setup the database replication][database-source] (`primary (read-write)
<-> secondary (read-only)` topology). <-> secondary (read-only)` topology).
1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md), 1. [Configure fast lookup of authorized SSH keys in the database][fast-ssh-lookup],
do this step for both primary AND secondary nodes. do this step for both primary AND secondary nodes.
1. [Configure GitLab](configuration_source.md) to set the primary and secondary 1. [Configure GitLab][configuration-source] to set the primary and secondary
nodes. nodes.
1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). 1. [Follow the "Using a Geo Server" guide][using-geo].
[install-ee-source]: https://docs.gitlab.com/ee/install/installation.html "GitLab Enterprise Edition installation from source"
## Configuring Geo ## Configuring Geo
Read through the [Geo configuration](configuration.md) documentation. Read through the [Geo configuration][configuration] documentation.
## Updating the Geo nodes ## Updating the Geo nodes
Read how to [update your Geo nodes to the latest GitLab version](updating_the_geo_nodes.md). Read how to [update your Geo nodes to the latest GitLab version][updating-geo].
## Configuring Geo HA ## Configuring Geo HA
Read through the [Geo High Availability documentation](high_availability.md). Read through the [Geo High Availability documentation][ha].
## Configuring Geo with Object storage ## Configuring Geo with Object storage
When you have object storage enabled, please consult the When you have object storage enabled, please consult the
[Geo with Object Storage](object_storage.md) documentation. [Geo with Object Storage][object-storage] documentation.
## Replicating the Container Registry ## Disaster Recovery
Read how to [replicate the Container Registry](docker_registry.md). Read through the [Disaster Recovery documentation][disaster-recovery] how to use Geo to mitigate data-loss and
restore services in a disaster scenario.
### Replicating the Container Registry
Read how to [replicate the Container Registry][docker-registry].
## Current limitations ## Current limitations
- You cannot push code to secondary nodes, see [3912](https://gitlab.com/gitlab-org/gitlab-ee/issues/3912) for details. > **IMPORTANT**: This list of limitations tracks only the latest version. If you are in an older version,
extra limitations may be in place.
- You cannot push code to secondary nodes, see [gitlab-org/gitlab-ee#3912] for details.
- The primary node has to be online for OAuth login to happen (existing sessions and Git are not affected) - The primary node has to be online for OAuth login to happen (existing sessions and Git are not affected)
- It works for repos, wikis, issues, and merge requests, but it does not work for job logs, artifacts, GitLab Pages, and Docker images of the Container - It works for repos, wikis, issues, merge requests, file attachments, artifacts and job logs but it does not work for,
Registry (by default, but you can configure it separately, see [replicate the Container Registry](docker_registry.md) for details) GitLab Pages, and Docker images of the Container Registry (by default, but you can configure it separately,
- The installation takes multiple manual steps that together can take about an hour depending on circumstances; we are working on improving this experience, see [#2978](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2978) for details. see [replicate the Container Registry][docker-registry] for details).
- The installation takes multiple manual steps that together can take about an hour depending on circumstances; we are
working on improving this experience, see [gitlab-org/omnibus-gitlab#2978] for details.
- Real-time updates of issues/merge requests (e.g. via long polling) doesn't work on the secondary
- Broadcast messages set on the primary won't be seen on the secondary without a cache flush (e.g. gitlab-rake cache:clear)
## Frequently Asked Questions ## Frequently Asked Questions
Read more in the [Geo FAQ](faq.md). Read more in the [Geo FAQ][faq].
## Log files ## Log files
...@@ -225,16 +232,39 @@ This message shows that Geo detected that a repository update was needed for pro ...@@ -225,16 +232,39 @@ This message shows that Geo detected that a repository update was needed for pro
## Security of Geo ## Security of Geo
Read the [security review](security_review.md) page. Read the [security review][security-review] page.
## Tuning Geo ## Tuning Geo
Read the [Geo tuning](tuning.md) documentation. Read the [Geo tuning][tunning] documentation.
## Troubleshooting ## Troubleshooting
Read the [troubleshooting document](troubleshooting.md). Read the [troubleshooting document][troubleshooting].
[ee]: https://about.gitlab.com/products/ "GitLab Enterprise Edition landing page" [ee]: https://about.gitlab.com/products/ "GitLab Enterprise Edition landing page"
[install-requirements]: ../../../install/requirements.md
[install-ee]: https://about.gitlab.com/downloads-ee/ "GitLab Enterprise Edition Omnibus packages downloads page" [install-ee]: https://about.gitlab.com/downloads-ee/ "GitLab Enterprise Edition Omnibus packages downloads page"
[install-ee-source]: https://docs.gitlab.com/ee/install/installation.html "GitLab Enterprise Edition installation from source" [install-ee-source]: https://docs.gitlab.com/ee/install/installation.html "GitLab Enterprise Edition installation from source"
[disaster-recovery]: ../disaster_recovery/index.md
[planned failover]: ../disaster_recovery/planned_failover.md
[fast-ssh-lookup]: ../../operations/fast_ssh_key_lookup.md
[upload-license]: ../../../user/admin_area/license.md
[database]: database.md
[database-source]: database_source.md
[configuration]: configuration.md
[configuration-source]: configuration_source.md
[config-ldap]: ../../auth/ldap.md
[using-geo]: using_a_geo_server.md
[updating-geo]: updating_the_geo_nodes.md
[ha]: high_availability.md
[object-storage]: object_storage.md
[docker-registry]: docker_registry.md
[faq]: faq.md
[security-review]: security_review.md
[tunning]: tuning.md
[troubleshooting]: troubleshooting.md
[source diagram]: https://docs.google.com/drawings/d/1Abw0P_H0Ew1-2Lj_xPDRWP87clGIke-1fil7_KQqrtE/edit
[ldap-replication]: https://www.openldap.org/doc/admin24/replication.html
[gitlab-org/gitlab-ee#3912]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3912
[gitlab-org/omnibus-gitlab#2978]: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2978
doc/administration/geo/replication/troubleshooting.md
View file @ 8f3fa21c
...@@ -66,7 +66,7 @@ be set on the primary database. In GitLab 9.4, we have made this setting ...@@ -66,7 +66,7 @@ be set on the primary database. In GitLab 9.4, we have made this setting
default to 1. You may need to increase this value if you have more Geo default to 1. You may need to increase this value if you have more Geo
secondary nodes. Be sure to restart PostgreSQL for this to take secondary nodes. Be sure to restart PostgreSQL for this to take
effect. See the [PostgreSQL replication effect. See the [PostgreSQL replication
setup](database.md#postgresql-replication) guide for more details. setup][database-pg-replication] guide for more details.
#### How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"? #### How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"?
...@@ -76,8 +76,8 @@ process](database.md) on the secondary. ...@@ -76,8 +76,8 @@ process](database.md) on the secondary.
#### How do I fix the message, "Command exceeded allowed execution time" when setting up replication? #### How do I fix the message, "Command exceeded allowed execution time" when setting up replication?
This may happen while [initiating the replication process](database.md#step-4-initiate-the-replication-process) on the Geo secondary, and indicates that your This may happen while [initiating the replication process][database-start-replication] on the Geo secondary,
initial dataset is too large to be replicated in the default timeout (30 minutes). and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes).
Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for
`--backup-timeout`: `--backup-timeout`:
...@@ -91,8 +91,8 @@ the default thirty minutes. Adjust as required for your installation. ...@@ -91,8 +91,8 @@ the default thirty minutes. Adjust as required for your installation.
#### How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device" #### How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device"
Determine if you have any unused replication slots in the primary database. This can cause large amounts of log data to build up in `pg_xlog`. Determine if you have any unused replication slots in the primary database. This can cause large amounts of
Removing the unused slots can reduce the amount of space used in the `pg_xlog`. log data to build up in `pg_xlog`. Removing the unused slots can reduce the amount of space used in the `pg_xlog`.
1. Start a PostgreSQL console session: 1. Start a PostgreSQL console session:
...@@ -100,7 +100,8 @@ Removing the unused slots can reduce the amount of space used in the `pg_xlog`. ...@@ -100,7 +100,8 @@ Removing the unused slots can reduce the amount of space used in the `pg_xlog`.
sudo gitlab-psql gitlabhq_production sudo gitlab-psql gitlabhq_production
``` ```
Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. > Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires
superuser permissions.
2. View your replication slots with 2. View your replication slots with
...@@ -111,9 +112,10 @@ Removing the unused slots can reduce the amount of space used in the `pg_xlog`. ...@@ -111,9 +112,10 @@ Removing the unused slots can reduce the amount of space used in the `pg_xlog`.
Slots where `active` is `f` are not active. Slots where `active` is `f` are not active.
- When this slot should be active, because you have a secondary configured using that slot, - When this slot should be active, because you have a secondary configured using that slot,
log in to that secondary and check the PostgreSQL logs why the replication is not running. log in to that secondary and check the PostgreSQL logs why the replication is not running.
- If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the PostgreSQL console session: - If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the
PostgreSQL console session:
```sql ```sql
SELECT pg_drop_replication_slot('name_of_extra_slot'); SELECT pg_drop_replication_slot('name_of_extra_slot');
...@@ -139,3 +141,6 @@ sudo gitlab-ctl reconfigure ...@@ -139,3 +141,6 @@ sudo gitlab-ctl reconfigure
This will increase the timeout to three hours (10800 seconds). Choose a time This will increase the timeout to three hours (10800 seconds). Choose a time
long enough to accomodate a full clone of your largest repositories. long enough to accomodate a full clone of your largest repositories.
[database-start-replication]: database.md#step-3-initiate-the-replication-process
[database-pg-replication]: database.md#postgresql-replication
doc/administration/geo/replication/tuning.md
View file @ 8f3fa21c
...@@ -13,5 +13,7 @@ but this may not lead to a more downloads in parallel unless the number of ...@@ -13,5 +13,7 @@ but this may not lead to a more downloads in parallel unless the number of
available Sidekiq threads is also increased. For example, if repository sync available Sidekiq threads is also increased. For example, if repository sync
capacity is increased from 25 to 50, you may also want to increase the number capacity is increased from 25 to 50, you may also want to increase the number
of Sidekiq threads from 25 to 50. See the [Sidekiq concurrency of Sidekiq threads from 25 to 50. See the [Sidekiq concurrency
documentation](../../operations/extra_sidekiq_processes.html#concurrency) documentation][sidekiq-concurrency]
for more details. for more details.
[sidekiq-concurrency]: ../../operations/extra_sidekiq_processes.html#concurrency
doc/administration/geo/replication/updating_the_geo_nodes.md
View file @ 8f3fa21c
...@@ -34,7 +34,7 @@ If you do not perform this step, you may find that two-factor authentication ...@@ -34,7 +34,7 @@ If you do not perform this step, you may find that two-factor authentication
To prevent SSH requests to the newly promoted primary node from failing To prevent SSH requests to the newly promoted primary node from failing
due to SSH host key mismatch when updating the primary domain's DNS record due to SSH host key mismatch when updating the primary domain's DNS record
you should perform the step to [Manually replicate primary SSH host keys](configuration.md#step-2-manually-replicate-primary-ssh-host-keys) in each you should perform the step to [Manually replicate primary SSH host keys][configuration-replicate-ssh] in each
secondary node. secondary node.
## Upgrading to GitLab 10.4 ## Upgrading to GitLab 10.4
...@@ -49,7 +49,7 @@ In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, ...@@ -49,7 +49,7 @@ In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3,
support is removed entirely. All installations will switch to the HTTP/HTTPS support is removed entirely. All installations will switch to the HTTP/HTTPS
cloning method instead. Before upgrading, ensure that all your Geo nodes are cloning method instead. Before upgrading, ensure that all your Geo nodes are
configured to use this method and that it works for your installation. In configured to use this method and that it works for your installation. In
particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-5-enable-git-access-over-http-https). particular, ensure that [Git access over HTTP/HTTPS is enabled][configuration-git-over-http].
Synchronizing repositories over the public Internet using HTTP is insecure, so Synchronizing repositories over the public Internet using HTTP is insecure, so
you should ensure that you have HTTPS configured before upgrading. Note that you should ensure that you have HTTPS configured before upgrading. Note that
...@@ -63,7 +63,7 @@ Support for TLS-secured PostgreSQL replication has been added. If you are ...@@ -63,7 +63,7 @@ Support for TLS-secured PostgreSQL replication has been added. If you are
currently using PostgreSQL replication across the open internet without an currently using PostgreSQL replication across the open internet without an
external means of securing the connection (e.g., a site-to-site VPN), then you external means of securing the connection (e.g., a site-to-site VPN), then you
should immediately reconfigure your primary and secondary PostgreSQL instances should immediately reconfigure your primary and secondary PostgreSQL instances
according to the [updated instructions](#database.md). according to the [updated instructions][database].
If you *are* securing the connections externally and wish to continue doing so, If you *are* securing the connections externally and wish to continue doing so,
ensure you include the new option `--sslmode=prefer` in future invocations of ensure you include the new option `--sslmode=prefer` in future invocations of
...@@ -96,10 +96,9 @@ secondary if ever promoted to a primary: ...@@ -96,10 +96,9 @@ secondary if ever promoted to a primary:
### Hashed Storage ### Hashed Storage
>**Warning** CAUTION: **Warning:**
Hashed storage is in **Alpha**. It is considered experimental and not Hashed storage is in **Alpha**. It is considered experimental and not
production-ready. See [Hashed production-ready. See [Hashed Storage] for more detail.
Storage](../../repository_storage_types.md) for more detail.
If you previously enabled Hashed Storage and migrated all your existing If you previously enabled Hashed Storage and migrated all your existing
projects to Hashed Storage, disabling hashed storage will not migrate projects projects to Hashed Storage, disabling hashed storage will not migrate projects
...@@ -108,19 +107,17 @@ migrated we recommend leaving Hashed Storage enabled. ...@@ -108,19 +107,17 @@ migrated we recommend leaving Hashed Storage enabled.
## Upgrading to GitLab 10.1 ## Upgrading to GitLab 10.1
>**Warning** CAUTION: **Warning:**
Hashed storage is in **Alpha**. It is considered experimental and not Hashed storage is in **Alpha**. It is considered experimental and not
production-ready. See [Hashed production-ready. See [Hashed Storage] for more detail.
Storage](../../repository_storage_types.md) for more detail.
[Hashed storage](../../repository_storage_types.md) was introduced [Hashed storage] was introduced in GitLab 10.0, and a [migration path][hashed-migration]
in GitLab 10.0, and a [migration path](../../raketasks/storage.md)
for existing repositories was added in GitLab 10.1. for existing repositories was added in GitLab 10.1.
## Upgrading to GitLab 10.0 ## Upgrading to GitLab 10.0
Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via
the database](../../operations/fast_ssh_key_lookup.md) to avoid having to maintain consistency of the the database][ssh-fast-lookup] to avoid having to maintain consistency of the
`authorized_keys` file for SSH access. Failing to do this will prevent users `authorized_keys` file for SSH access. Failing to do this will prevent users
from being able to clone via SSH. from being able to clone via SSH.
...@@ -158,10 +155,12 @@ instructions below. ...@@ -158,10 +155,12 @@ instructions below.
When in doubt, it does not hurt to do a resync. The easiest way to do this in When in doubt, it does not hurt to do a resync. The easiest way to do this in
Omnibus is the following: Omnibus is the following:
1. Install GitLab on the primary server 1. Make sure you have Omnibus GitLab on the primary server
1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the primary database. 1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the primary database.
1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`
1. Make sure `postgresql['max_replication_slots']` matches the number of secondary Geo Nodes locations
1. Install GitLab on the secondary server. 1. Install GitLab on the secondary server.
1. Re-run the [database replication process](database.md#step-3-initiate-the-replication-process). 1. Re-run the [database replication process][database-replication].
## Special update notes for 9.0.x ## Special update notes for 9.0.x
...@@ -255,7 +254,7 @@ is prepended with the relevant node for better clarity: ...@@ -255,7 +254,7 @@ is prepended with the relevant node for better clarity:
``` ```
1. **[secondary]** Create the `replica.sh` script as described in the 1. **[secondary]** Create the `replica.sh` script as described in the
[database configuration document](database.md#step-3-initiate-the-replication-process). [database configuration document][database-source-replication].
1. **[secondary]** Run the recovery script using the credentials from the 1. **[secondary]** Run the recovery script using the credentials from the
previous step: previous step:
...@@ -315,3 +314,11 @@ and it is required since 10.0. ...@@ -315,3 +314,11 @@ and it is required since 10.0.
1. Repeat this step for every secondary node 1. Repeat this step for every secondary node
[update]: ../../../update/README.md [update]: ../../../update/README.md
[database]: database.md
[database-replication]: database.md#step-3-initiate-the-replication-process
[database-source-replication]: database_source.md#step-3-initiate-the-replication-process
[Hashed Storage]: ../../repository_storage_types.md
[hashed-migration]: ../../raketasks/storage.md
[ssh-fast-lookup]: ../../operations/fast_ssh_key_lookup.md
[configuration-replicate-ssh]: configuration.md#step-2-manually-replicate-primary-ssh-host-keys
[configuration-git-over-http]: configuration.md#step-6-enable-git-access-over-http-https
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7