Skip to content

G
gitlab-ce
Commit 238f768d authored by Russell Dickenson's avatar Russell Dickenson
Browse files
Options

Merge branch 'eread/move-concepts-to-main-page' into 'master' 

Move some conceptual material to main page and other minor improvements

See merge request gitlab-org/gitlab!66137
parents f9a26ae7 faa607b2
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 140 additions and 126 deletions
doc/administration/gitaly/faq.md
View file @ 238f768d
......@@ -25,7 +25,7 @@ The following table outlines the major differences between Gitaly Cluster and Ge
| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for |
|:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------------------|:-----------------------------------------|:------------------------|
| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](praefect.md#strong-consistency) | Data storage in Git |
| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git |
| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance |
For more information, see:
......@@ -40,7 +40,7 @@ Yes! For more information, see [Migrate to Gitaly Cluster](praefect.md#migrate-t
## What are some repository storage recommendations?
The size of the required storage can vary between instances and depends on the set
[replication factor](praefect.md#replication-factor). You might want to include implementing
[replication factor](index.md#replication-factor). You might want to include implementing
repository storage redundancy.
For a replication factor:
......
doc/administration/gitaly/index.md
View file @ 238f768d
This diff is collapsed.
doc/administration/gitaly/praefect.md
View file @ 238f768d
......@@ -24,7 +24,7 @@ NOTE:
Upgrade instructions for Omnibus GitLab installations
[are available](https://docs.gitlab.com/omnibus/update/#gitaly-cluster).
## Requirements for configuring a Gitaly Cluster
## Requirements
The minimum recommended configuration for a Gitaly Cluster requires:
......@@ -33,14 +33,33 @@ The minimum recommended configuration for a Gitaly Cluster requires:
- 3 Praefect nodes
- 3 Gitaly nodes (1 primary, 2 secondary)
See the [design
document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md)
See the [design document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md)
for implementation details.
NOTE:
If not set in GitLab, feature flags are read as false from the console and Praefect uses their
default value. The default value depends on the GitLab version.
### Network connectivity
Gitaly Cluster [components](index.md#components) need to communicate with each other over many
routes. Your firewall rules must allow the following for Gitaly Cluster to function properly:
| From | To | Default port / TLS port |
|:-----------------------|:------------------------|:------------------------|
| GitLab | Praefect load balancer | `2305` / `3305` |
| Praefect load balancer | Praefect | `2305` / `3305` |
| Praefect | Gitaly | `8075` / `9999` |
| Gitaly | GitLab (internal API) | `80` / `443` |
| Gitaly | Praefect load balancer | `2305` / `3305` |
| Gitaly | Praefect | `2305` / `3305` |
| Gitaly | Gitaly | `8075` / `9999` |
NOTE:
Gitaly does not directly connect to Praefect. However, requests from Gitaly to the Praefect
load balancer may still be blocked unless firewalls on the Praefect nodes allow traffic from
the Gitaly nodes.
## Setup Instructions
If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus GitLab package
......@@ -129,7 +148,7 @@ The following options are available:
- For non-Geo installations, either:
- Use one of the documented [PostgreSQL setups](../postgresql/index.md).
- Use your own third-party database setup. This will require [manual setup](#manual-database-setup).
- Use your own third-party database setup. This requires [manual setup](#manual-database-setup).
- For Geo instances, either:
- Set up a separate [PostgreSQL instance](https://www.postgresql.org/docs/11/high-availability.html).
- Use a cloud-managed PostgreSQL service. AWS
......@@ -457,7 +476,7 @@ On the **Praefect** node:
In [GitLab 13.8 and earlier](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4988),
Gitaly nodes were configured directly under the virtual storage, and not under the `nodes` key.
1. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2013) in GitLab 13.1 and later, enable [distribution of reads](#distributed-reads).
1. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2013) in GitLab 13.1 and later, enable [distribution of reads](index.md#distributed-reads).
1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure
Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure):
......@@ -1053,75 +1072,9 @@ To get started quickly:
Congratulations! You've configured an observable fault-tolerant Praefect
cluster.
## Network connectivity requirements
## Configure strong consistency
Gitaly Cluster components need to communicate with each other over many routes.
Your firewall rules must allow the following for Gitaly Cluster to function properly:
| From | To | Default port / TLS port |
|:-----------------------|:------------------------|:------------------------|
| GitLab | Praefect load balancer | `2305` / `3305` |
| Praefect load balancer | Praefect | `2305` / `3305` |
| Praefect | Gitaly | `8075` / `9999` |
| Gitaly | GitLab (internal API) | `80` / `443` |
| Gitaly | Praefect load balancer | `2305` / `3305` |
| Gitaly | Praefect | `2305` / `3305` |
| Gitaly | Gitaly | `8075` / `9999` |
NOTE:
Gitaly does not directly connect to Praefect. However, requests from Gitaly to the Praefect
load balancer may still be blocked unless firewalls on the Praefect nodes allow traffic from
the Gitaly nodes.
## Distributed reads
> - Introduced in GitLab 13.1 in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) with feature flag `gitaly_distributed_reads` set to disabled.
> - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3.
> - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5.
> - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8.
> - [Feature flag removed](https://gitlab.com/gitlab-org/gitaly/-/issues/3383) in GitLab 13.11.
Praefect supports distribution of read operations across Gitaly nodes that are
configured for the virtual node.
All RPCs marked with `ACCESSOR` option like
[GetBlob](https://gitlab.com/gitlab-org/gitaly/-/blob/v12.10.6/proto/blob.proto#L16)
are redirected to an up to date and healthy Gitaly node.
_Up to date_ in this context means that:
- There is no replication operations scheduled for this node.
- The last replication operation is in _completed_ state.
If there is no such nodes, or any other error occurs during node selection, the primary
node is chosen to serve the request.
To track distribution of read operations, you can use the `gitaly_praefect_read_distribution`
Prometheus counter metric. It has two labels:
- `virtual_storage`.
- `storage`.
They reflect configuration defined for this instance of Praefect.
## Strong consistency
> - Introduced in GitLab 13.1 in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), disabled by default.
> - Entered [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in GitLab 13.2, disabled by default.
> - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled.
> - From GitLab 13.4, enabled by default.
> - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency.
> - From GitLab 13.6, primary-wins voting strategy and `gitaly_reference_transactions_primary_wins` feature flag were removed from the source code.
Praefect guarantees eventual consistency by replicating all writes to secondary nodes
after the write to the primary Gitaly node has happened.
Praefect can instead provide strong consistency by creating a transaction and writing
changes to all Gitaly nodes at once.
If enabled, transactions are only available for a subset of RPCs. For more
information, see the [strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189).
To enable strong consistency:
To enable [strong consistency](index.md#strong-consistency):
- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency.
- In GitLab 13.4 and later, the strong consistency voting strategy has been improved and enabled by default.
......@@ -1155,14 +1108,7 @@ To monitor strong consistency, you can use the following Prometheus metrics:
- `gitaly_hook_transaction_voting_delay_seconds`: Client-side delay introduced
by waiting for the transaction to be committed.
## Replication factor
Replication factor is the number of copies Praefect maintains of a given repository. A higher
replication factor offers better redundancy and distribution of read workload, but also results
in a higher storage cost. By default, Praefect replicates repositories to every storage in a
virtual storage.
### Configure replication factor
## Configure replication factor
WARNING:
Configurable replication factors require [repository-specific primary nodes](#repository-specific-primary-nodes) to be used.
......
doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
View file @ 238f768d
......@@ -16,7 +16,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
| `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. |
| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. |
| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
| `:github` | The test requires a GitHub personal access token. |
| `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
| `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7