Praefect recovers from a failing primary Gitaly node by promoting a healthy secondary as the new primary. To minimize data loss, Praefect elects the secondary with the least unreplicated writes from the primary. There can still be some unreplicated writes, leading to data loss.
Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the
new primary.
To minimize data loss, Gitaly Cluster:
- Switches repositories that are outdated on the new primary to [read-only mode](#read-only-mode).
- Elects the secondary with the least unreplicated writes from the primary to be the new primary.
Because there can still be some unreplicated writes, [data loss can occur](#check-for-data-loss).
### Read-only mode
If the primary is not fully up to date, Praefect switches:
> - Introduced in GitLab 13.0 as [generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga).
> - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred.
> - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date.
When Gitaly Cluster switches to a new primary, repositories enter read-only mode if they are out of
date. This can happen after failing over to an outdated secondary. Read-only mode eases data
recovery efforts by preventing writes that may conflict with the unreplicated writes on other nodes.
To enable writes again, an administrator can:
- The virtual storage to read-only mode in GitLab 13.2 and earlier.
- The affected repositories to read-only mode in
[GitLab 13.3](https://gitlab.com/gitlab-org/gitaly/-/issues/2862) and later.
1.[Check](#check-for-data-loss) for data loss.
1. Attempt to [recover](#recover-missing-data) missing data.
1. Either [enable writes](#enable-writes-or-accept-data-loss) in the virtual storage or
[accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of
GitLab.
This can happen after failing over to an outdated secondary. Read-only mode eases data
recovery efforts by preventing writes that may conflict with the unreplicated writes.
### Check for data loss
To re-enable the repositories for writes, the administrator can attempt to
[recover the missing data](#recover-missing-data) from an up-to-date replica. If
recovering the data is not possible, the repository can be enabled for writes again by
[accepting the data loss](#accept-data-loss).
The Praefect `dataloss` sub-command identifies replicas that are likely to be outdated. This is
useful for identifying potential data loss after a failover. The following parameters are
available:
### Checking for data loss
-`-virtual-storage` that specifies which virtual storage to check. The default behavior is to
display outdated replicas of read-only repositories as they generally require administrator
action.
- In GitLab 13.3 and later, `-partially-replicated` that specifies whether to display a list of
[outdated replicas of writable repositories](#outdated-replicas-of-writable-repositories).
The Praefect `dataloss` sub-command identifies replicas that are likely to be outdated. This is useful for identifying potential data loss after a failover.
NOTE: **Note:**
`dataloss` is still in beta and the output format is subject to change.
To check for outdated replicas of read-only repositories, run:
-`-virtual-storage` specifies which virtual storage to check. Every configured virtual storage is checked if none is specified.
-`-writable` specifies whether to report outdated replicas of writable repositories. Repository is writable if the primary has the latest changes. Secondaries might be temporarily outdated while they are waiting to replicate the latest changes. To reduce the noise, default behavior is to display outdated replicas of read-only repositories as they generally require administrator action.
Every configured virtual storage is checked if none is specified:
The number of potentially unapplied changes to repositories are listed for each replica. Listed repositories might have the latest changes but it is not guaranteed. Only outdated replicas of read-only repositories are listed by default.
The number of potentially unapplied changes to repositories is listed for each replica. Listed
repositories might have the latest changes but it is not guaranteed. Only outdated replicas of
read-only repositories are listed by default. For example:
```shell
Virtual storage: default
...
...
@@ -1085,7 +1108,29 @@ Virtual storage: default
gitaly-3 is behind by 2 changes or less
```
Set the `-writable` flag also list the outdated replicas of writable repositories.
A confirmation is printed out when every repository is writable. For example:
```shell
Virtual storage: default
Primary: gitaly-1
All repositories are writable!
```
#### Outdated replicas of writable repositories
> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3019) in GitLab 13.3.
To also list information for outdated replicas of writable repositories, use the
`-partially-replicated` parameter.
A repository is writable if the primary has the latest changes. Secondaries might be temporarily
outdated while they are waiting to replicate the latest changes.
A confirmation is printed out when every repository is writable.
```shell
Virtual storage: default
Primary: gitaly-1
All repositories are writable!
```
With the `-writable` flag set, a confirmation is printed out if every replica is fully up to date.
With the `-partially-replicated` flag set, a confirmation is printed out if every replica is fully up to date.
For example:
```shell
Virtual storage: default
...
...
@@ -1114,17 +1152,15 @@ Virtual storage: default
All repositories are up to date!
```
NOTE: **Note:**
`dataloss` is still in beta and the output format is subject to change.
### Checking repository checksums
### Check repository checksums
To check a project's repository checksums across on all Gitaly nodes, run the
[replicas Rake task](../raketasks/praefect.md#replica-checksums) on the main GitLab node.
### Recover missing data
The Praefect `reconcile` sub-command can be used to recover unreplicated changes from another replica. The source must be on a later generation than the target storage.
The Praefect `reconcile` sub-command can be used to recover unreplicated changes from another replica.
The source must be on a later version than the target storage.
