Commit 41f446a0 authored by Amy Qualls's avatar Amy Qualls

Merge branch 'eread/remove-future-tense-from-gitaly-docs' into 'master'

Remove future tense from Gitaly documentation

See merge request gitlab-org/gitlab!47982
parents f5379925 39d11976
...@@ -434,8 +434,8 @@ server (with `gitaly_address`) unless you setup with special ...@@ -434,8 +434,8 @@ server (with `gitaly_address`) unless you setup with special
``` ```
NOTE: **Note:** NOTE: **Note:**
`/some/local/path` should be set to a local folder that exists, however no data will be stored in `/some/local/path` should be set to a local folder that exists, however no data is stored in
this folder. This will no longer be necessary after this folder. This requirement is scheduled to be removed when
[this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved.
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
...@@ -461,7 +461,7 @@ GitLab can reside on the same server as one of many Gitaly servers, but doesn't ...@@ -461,7 +461,7 @@ GitLab can reside on the same server as one of many Gitaly servers, but doesn't
configuration that mixes local and remote configuration. The following setup is incorrect, because: configuration that mixes local and remote configuration. The following setup is incorrect, because:
- All addresses must be reachable from the other Gitaly servers. - All addresses must be reachable from the other Gitaly servers.
- `storage1` will be assigned a Unix socket for `gitaly_address` which is - `storage1` is assigned a Unix socket for `gitaly_address` which is
invalid for some of the Gitaly servers. invalid for some of the Gitaly servers.
```ruby ```ruby
...@@ -493,7 +493,7 @@ gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" ...@@ -493,7 +493,7 @@ gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
``` ```
`path` can only be included for storage shards on the local Gitaly server. `path` can only be included for storage shards on the local Gitaly server.
If it's excluded, default Git storage directory will be used for that storage shard. If it's excluded, default Git storage directory is used for that storage shard.
### Disable Gitaly where not required (optional) ### Disable Gitaly where not required (optional)
...@@ -590,7 +590,7 @@ To configure Gitaly with TLS: ...@@ -590,7 +590,7 @@ To configure Gitaly with TLS:
``` ```
1. Copy all Gitaly server certificates (or their certificate authority) to 1. Copy all Gitaly server certificates (or their certificate authority) to
`/etc/gitlab/trusted-certs` so that Gitaly servers will trust the certificate when calling into themselves `/etc/gitlab/trusted-certs` so that Gitaly servers trust the certificate when calling into themselves
or other Gitaly servers: or other Gitaly servers:
```shell ```shell
...@@ -647,8 +647,8 @@ To configure Gitaly with TLS: ...@@ -647,8 +647,8 @@ To configure Gitaly with TLS:
``` ```
NOTE: **Note:** NOTE: **Note:**
`/some/local/path` should be set to a local folder that exists, however no data will be stored `/some/local/path` should be set to a local folder that exists, however no data is stored
in this folder. This will no longer be necessary after in this folder. This requirement is scheduled to be removed when
[Gitaly issue #1282](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. [Gitaly issue #1282](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved.
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
...@@ -668,7 +668,7 @@ To configure Gitaly with TLS: ...@@ -668,7 +668,7 @@ To configure Gitaly with TLS:
``` ```
1. Copy all Gitaly server certificates (or their certificate authority) to the system trusted 1. Copy all Gitaly server certificates (or their certificate authority) to the system trusted
certificates folder so Gitaly server will trust the certificate when calling into itself or other Gitaly certificates folder so Gitaly server trusts the certificate when calling into itself or other Gitaly
servers. servers.
```shell ```shell
...@@ -806,7 +806,7 @@ You can observe the behavior of this queue using the Gitaly logs and Prometheus: ...@@ -806,7 +806,7 @@ You can observe the behavior of this queue using the Gitaly logs and Prometheus:
NOTE: **Note:** NOTE: **Note:**
Though the name of the Prometheus metric contains `rate_limiting`, it is a concurrency limiter, not Though the name of the Prometheus metric contains `rate_limiting`, it is a concurrency limiter, not
a rate limiter. If a Gitaly client makes 1000 requests in a row very quickly, concurrency will not a rate limiter. If a Gitaly client makes 1000 requests in a row very quickly, concurrency does not
exceed 1 and the concurrency limiter has no effect. exceed 1 and the concurrency limiter has no effect.
## Rotate Gitaly authentication token ## Rotate Gitaly authentication token
...@@ -835,7 +835,7 @@ behavior of your GitLab installation using Prometheus. Use the following Prometh ...@@ -835,7 +835,7 @@ behavior of your GitLab installation using Prometheus. Use the following Prometh
sum(rate(gitaly_authentications_total[5m])) by (enforced, status) sum(rate(gitaly_authentications_total[5m])) by (enforced, status)
``` ```
In a system where authentication is configured correctly and where you have live traffic, you will In a system where authentication is configured correctly and where you have live traffic, you
see something like this: see something like this:
```prometheus ```prometheus
...@@ -891,7 +891,7 @@ To update to a new Gitaly authentication token, on each Gitaly client **and** Gi ...@@ -891,7 +891,7 @@ To update to a new Gitaly authentication token, on each Gitaly client **and** Gi
``` ```
If you run your [Prometheus query](#verify-authentication-monitoring) while this change is If you run your [Prometheus query](#verify-authentication-monitoring) while this change is
being rolled out, you will see non-zero values for the `enforced="false",status="denied"` counter. being rolled out, you see non-zero values for the `enforced="false",status="denied"` counter.
### Ensure there are no authentication failures ### Ensure there are no authentication failures
...@@ -1003,7 +1003,7 @@ When GitLab calls a function that has a "Rugged patch", it performs two checks: ...@@ -1003,7 +1003,7 @@ When GitLab calls a function that has a "Rugged patch", it performs two checks:
- Is the feature flag for this patch set in the database? If so, the feature flag setting controls - Is the feature flag for this patch set in the database? If so, the feature flag setting controls
GitLab's use of "Rugged patch" code. GitLab's use of "Rugged patch" code.
- If the feature flag is not set, GitLab tries accessing the filesystem underneath the - If the feature flag is not set, GitLab tries accessing the filesystem underneath the
Gitaly server directly. If it can, it will use the "Rugged patch": Gitaly server directly. If it can, it uses the "Rugged patch":
- If using Unicorn. - If using Unicorn.
- If using Puma and [thread count](../../install/requirements.md#puma-threads) is set - If using Puma and [thread count](../../install/requirements.md#puma-threads) is set
to `1`. to `1`.
...@@ -1076,7 +1076,7 @@ gitaly-debug -h ...@@ -1076,7 +1076,7 @@ gitaly-debug -h
remote: GitLab: 401 Unauthorized remote: GitLab: 401 Unauthorized
``` ```
You will need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab You need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab
app nodes). app nodes).
### Client side gRPC logs ### Client side gRPC logs
......
...@@ -133,7 +133,7 @@ GitLab](https://about.gitlab.com/install/). ...@@ -133,7 +133,7 @@ GitLab](https://about.gitlab.com/install/).
- 3 Gitaly nodes (high CPU, high memory, fast storage) - 3 Gitaly nodes (high CPU, high memory, fast storage)
- 1 GitLab server - 1 GitLab server
You will need the IP/host address for each node. You need the IP/host address for each node.
1. `LOAD_BALANCER_SERVER_ADDRESS`: the IP/host address of the load balancer 1. `LOAD_BALANCER_SERVER_ADDRESS`: the IP/host address of the load balancer
1. `POSTGRESQL_SERVER_ADDRESS`: the IP/host address of the PostgreSQL server 1. `POSTGRESQL_SERVER_ADDRESS`: the IP/host address of the PostgreSQL server
...@@ -149,7 +149,7 @@ If you are using Google Cloud Platform, SoftLayer, or any other vendor that prov ...@@ -149,7 +149,7 @@ If you are using Google Cloud Platform, SoftLayer, or any other vendor that prov
The communication between components is secured with different secrets, which The communication between components is secured with different secrets, which
are described below. Before you begin, generate a unique secret for each, and are described below. Before you begin, generate a unique secret for each, and
make note of it. This will make it easy to replace these placeholder tokens make note of it. This makes it easy to replace these placeholder tokens
with secure tokens as you complete the setup process. with secure tokens as you complete the setup process.
1. `GITLAB_SHELL_SECRET_TOKEN`: this is used by Git hooks to make callback HTTP 1. `GITLAB_SHELL_SECRET_TOKEN`: this is used by Git hooks to make callback HTTP
...@@ -164,7 +164,7 @@ with secure tokens as you complete the setup process. ...@@ -164,7 +164,7 @@ with secure tokens as you complete the setup process.
1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to 1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to
PostgreSQL. PostgreSQL.
We will note in the instructions below where these secrets are required. We note in the instructions below where these secrets are required.
### PostgreSQL ### PostgreSQL
...@@ -184,13 +184,13 @@ failure. For greater fault tolerance, the following options are available: ...@@ -184,13 +184,13 @@ failure. For greater fault tolerance, the following options are available:
- Use a cloud-managed PostgreSQL service. AWS - Use a cloud-managed PostgreSQL service. AWS
[Relational Database Service](https://aws.amazon.com/rds/) is recommended. [Relational Database Service](https://aws.amazon.com/rds/) is recommended.
To complete this section you will need: To complete this section you need:
- 1 Praefect node - 1 Praefect node
- 1 PostgreSQL server (PostgreSQL 11 or newer) - 1 PostgreSQL server (PostgreSQL 11 or newer)
- An SQL user with permissions to create databases - An SQL user with permissions to create databases
During this section, we will configure the PostgreSQL server, from the Praefect During this section, we configure the PostgreSQL server, from the Praefect
node, using `psql` which is installed by Omnibus GitLab. node, using `psql` which is installed by Omnibus GitLab.
1. SSH into the **Praefect** node and login as root: 1. SSH into the **Praefect** node and login as root:
...@@ -207,7 +207,7 @@ node, using `psql` which is installed by Omnibus GitLab. ...@@ -207,7 +207,7 @@ node, using `psql` which is installed by Omnibus GitLab.
/opt/gitlab/embedded/bin/psql -U postgres -d template1 -h POSTGRESQL_SERVER_ADDRESS /opt/gitlab/embedded/bin/psql -U postgres -d template1 -h POSTGRESQL_SERVER_ADDRESS
``` ```
Create a new user `praefect` which will be used by Praefect. Replace Create a new user `praefect` to be used by Praefect. Replace
`PRAEFECT_SQL_PASSWORD` with the strong password you generated in the `PRAEFECT_SQL_PASSWORD` with the strong password you generated in the
preparation step. preparation step.
...@@ -281,11 +281,10 @@ PostgreSQL instances. Otherwise you should change the configuration parameter ...@@ -281,11 +281,10 @@ PostgreSQL instances. Otherwise you should change the configuration parameter
NOTE: **Note:** NOTE: **Note:**
If there are multiple Praefect nodes, complete these steps for **each** node. If there are multiple Praefect nodes, complete these steps for **each** node.
To complete this section you will need: To complete this section you need a [configured PostgreSQL server](#postgresql), including:
- [Configured PostgreSQL server](#postgresql), including: - IP/host address (`POSTGRESQL_SERVER_ADDRESS`)
- IP/host address (`POSTGRESQL_SERVER_ADDRESS`) - Password (`PRAEFECT_SQL_PASSWORD`)
- password (`PRAEFECT_SQL_PASSWORD`)
Praefect should be run on a dedicated node. Do not run Praefect on the Praefect should be run on a dedicated node. Do not run Praefect on the
application server, or a Gitaly node. application server, or a Gitaly node.
...@@ -331,8 +330,8 @@ application server, or a Gitaly node. ...@@ -331,8 +330,8 @@ application server, or a Gitaly node.
``` ```
1. Configure a strong `auth_token` for **Praefect** by editing 1. Configure a strong `auth_token` for **Praefect** by editing
`/etc/gitlab/gitlab.rb`. This will be needed by clients outside the cluster `/etc/gitlab/gitlab.rb`. This is needed by clients outside the cluster
(like GitLab Shell) to communicate with the Praefect cluster : (like GitLab Shell) to communicate with the Praefect cluster:
```ruby ```ruby
praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN' praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN'
...@@ -341,7 +340,7 @@ application server, or a Gitaly node. ...@@ -341,7 +340,7 @@ application server, or a Gitaly node.
1. Configure **Praefect** to connect to the PostgreSQL database by editing 1. Configure **Praefect** to connect to the PostgreSQL database by editing
`/etc/gitlab/gitlab.rb`. `/etc/gitlab/gitlab.rb`.
You will need to replace `POSTGRESQL_SERVER_ADDRESS` with the IP/host address You need to replace `POSTGRESQL_SERVER_ADDRESS` with the IP/host address
of the database, and `PRAEFECT_SQL_PASSWORD` with the strong password set of the database, and `PRAEFECT_SQL_PASSWORD` with the strong password set
above. above.
...@@ -364,7 +363,7 @@ application server, or a Gitaly node. ...@@ -364,7 +363,7 @@ application server, or a Gitaly node.
# praefect['database_sslrootcert'] = '/path/to/rootcert' # praefect['database_sslrootcert'] = '/path/to/rootcert'
``` ```
By default Praefect will refuse to make an unencrypted connection to By default, Praefect refuses to make an unencrypted connection to
PostgreSQL. You can override this by uncommenting the following line: PostgreSQL. You can override this by uncommenting the following line:
```ruby ```ruby
...@@ -377,7 +376,7 @@ application server, or a Gitaly node. ...@@ -377,7 +376,7 @@ application server, or a Gitaly node.
The virtual storage's name must match the configured storage name in GitLab The virtual storage's name must match the configured storage name in GitLab
configuration. In a later step, we configure the storage name as `default` configuration. In a later step, we configure the storage name as `default`
so we use `default` here as well. This cluster has three Gitaly nodes `gitaly-1`, so we use `default` here as well. This cluster has three Gitaly nodes `gitaly-1`,
`gitaly-2`, and `gitaly-3`, which will be replicas of each other. `gitaly-2`, and `gitaly-3`, which are intended to be replicas of each other.
CAUTION: **Caution:** CAUTION: **Caution:**
If you have data on an already existing storage called If you have data on an already existing storage called
...@@ -385,7 +384,7 @@ application server, or a Gitaly node. ...@@ -385,7 +384,7 @@ application server, or a Gitaly node.
[migrate the data to the Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster) [migrate the data to the Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster)
afterwards. afterwards.
Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which will be used by Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which is used by
Praefect when communicating with Gitaly nodes in the cluster. This token is Praefect when communicating with Gitaly nodes in the cluster. This token is
distinct from the `PRAEFECT_EXTERNAL_TOKEN`. distinct from the `PRAEFECT_EXTERNAL_TOKEN`.
...@@ -555,12 +554,12 @@ To configure Praefect with TLS: ...@@ -555,12 +554,12 @@ To configure Praefect with TLS:
NOTE: **Note:** NOTE: **Note:**
`/some/local/path` should be set to a local folder that exists, however no `/some/local/path` should be set to a local folder that exists, however no
data will be stored in this folder. This will no longer be necessary after data is stored in this folder. This requirement is scheduled to be removed when
[this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved.
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
1. Copy all Praefect server certificates, or their certificate authority, to the system 1. Copy all Praefect server certificates, or their certificate authority, to the system
trusted certificates on each Gitaly server so the Praefect server will trust the trusted certificates on each Gitaly server so the Praefect server trusts the
certificate when called by Gitaly servers: certificate when called by Gitaly servers:
```shell ```shell
...@@ -585,7 +584,7 @@ To configure Praefect with TLS: ...@@ -585,7 +584,7 @@ To configure Praefect with TLS:
NOTE: **Note:** NOTE: **Note:**
Complete these steps for **each** Gitaly node. Complete these steps for **each** Gitaly node.
To complete this section you will need: To complete this section you need:
- [Configured Praefect node](#praefect) - [Configured Praefect node](#praefect)
- 3 (or more) servers, with GitLab installed, to be configured as Gitaly nodes. - 3 (or more) servers, with GitLab installed, to be configured as Gitaly nodes.
...@@ -595,19 +594,19 @@ Every Gitaly server assigned to the Praefect cluster needs to be configured. The ...@@ -595,19 +594,19 @@ Every Gitaly server assigned to the Praefect cluster needs to be configured. The
configuration is the same as a normal [standalone Gitaly server](index.md), configuration is the same as a normal [standalone Gitaly server](index.md),
except: except:
- the storage names are exposed to Praefect, not GitLab - The storage names are exposed to Praefect, not GitLab
- the secret token is shared with Praefect, not GitLab - The secret token is shared with Praefect, not GitLab
The configuration of all Gitaly nodes in the Praefect cluster can be identical, The configuration of all Gitaly nodes in the Praefect cluster can be identical,
because we rely on Praefect to route operations correctly. because we rely on Praefect to route operations correctly.
Particular attention should be shown to: Particular attention should be shown to:
- the `gitaly['auth_token']` configured in this section must match the `token` - The `gitaly['auth_token']` configured in this section must match the `token`
value under `praefect['virtual_storages']` on the Praefect node. This was set value under `praefect['virtual_storages']` on the Praefect node. This was set
in the [previous section](#praefect). This document uses the placeholder in the [previous section](#praefect). This document uses the placeholder
`PRAEFECT_INTERNAL_TOKEN` throughout. `PRAEFECT_INTERNAL_TOKEN` throughout.
- the storage names in `git_data_dirs` configured in this section must match the - The storage names in `git_data_dirs` configured in this section must match the
storage names under `praefect['virtual_storages']` on the Praefect node. This storage names under `praefect['virtual_storages']` on the Praefect node. This
was set in the [previous section](#praefect). This document uses `gitaly-1`, was set in the [previous section](#praefect). This document uses `gitaly-1`,
`gitaly-2`, and `gitaly-3` as Gitaly storage names. `gitaly-2`, and `gitaly-3` as Gitaly storage names.
...@@ -659,8 +658,8 @@ documentation](index.md#configure-gitaly-servers). ...@@ -659,8 +658,8 @@ documentation](index.md#configure-gitaly-servers).
``` ```
1. Configure a strong `auth_token` for **Gitaly** by editing 1. Configure a strong `auth_token` for **Gitaly** by editing
`/etc/gitlab/gitlab.rb`. This will be needed by clients to communicate with `/etc/gitlab/gitlab.rb`. This is needed by clients to communicate with
this Gitaly nodes. Typically, this token will be the same for all Gitaly this Gitaly nodes. Typically, this token is the same for all Gitaly
nodes. nodes.
```ruby ```ruby
...@@ -754,7 +753,7 @@ We hope that if you’re managing HA systems like GitLab, you have a load balanc ...@@ -754,7 +753,7 @@ We hope that if you’re managing HA systems like GitLab, you have a load balanc
of choice already. Some examples include [HAProxy](https://www.haproxy.org/) of choice already. Some examples include [HAProxy](https://www.haproxy.org/)
(open-source), [Google Internal Load Balancer](https://cloud.google.com/load-balancing/docs/internal/), (open-source), [Google Internal Load Balancer](https://cloud.google.com/load-balancing/docs/internal/),
[AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/), F5 [AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/), F5
Big-IP LTM, and Citrix Net Scaler. This documentation will outline what ports Big-IP LTM, and Citrix Net Scaler. This documentation outlines what ports
and protocols you need configure. and protocols you need configure.
| LB Port | Backend Port | Protocol | | LB Port | Backend Port | Protocol |
...@@ -763,7 +762,7 @@ and protocols you need configure. ...@@ -763,7 +762,7 @@ and protocols you need configure.
### GitLab ### GitLab
To complete this section you will need: To complete this section you need:
- [Configured Praefect node](#praefect) - [Configured Praefect node](#praefect)
- [Configured Gitaly nodes](#gitaly) - [Configured Gitaly nodes](#gitaly)
...@@ -787,15 +786,15 @@ Particular attention should be shown to: ...@@ -787,15 +786,15 @@ Particular attention should be shown to:
1. Configure the `external_url` so that files could be served by GitLab 1. Configure the `external_url` so that files could be served by GitLab
by proper endpoint access by editing `/etc/gitlab/gitlab.rb`: by proper endpoint access by editing `/etc/gitlab/gitlab.rb`:
You will need to replace `GITLAB_SERVER_URL` with the real external facing You need to replace `GITLAB_SERVER_URL` with the real external facing
URL on which current GitLab instance is serving: URL on which current GitLab instance is serving:
```ruby ```ruby
external_url 'GITLAB_SERVER_URL' external_url 'GITLAB_SERVER_URL'
``` ```
1. Disable the default Gitaly service running on the GitLab host. It won't be needed 1. Disable the default Gitaly service running on the GitLab host. It isn't needed
as GitLab will connect to the configured cluster. because GitLab connects to the configured cluster.
CAUTION: **Caution:** CAUTION: **Caution:**
If you have existing data stored on the default Gitaly storage, If you have existing data stored on the default Gitaly storage,
...@@ -809,7 +808,7 @@ Particular attention should be shown to: ...@@ -809,7 +808,7 @@ Particular attention should be shown to:
1. Add the Praefect cluster as a storage location by editing 1. Add the Praefect cluster as a storage location by editing
`/etc/gitlab/gitlab.rb`. `/etc/gitlab/gitlab.rb`.
You will need to replace: You need to replace:
- `LOAD_BALANCER_SERVER_ADDRESS` with the IP address or hostname of the load - `LOAD_BALANCER_SERVER_ADDRESS` with the IP address or hostname of the load
balancer. balancer.
...@@ -828,7 +827,7 @@ Particular attention should be shown to: ...@@ -828,7 +827,7 @@ Particular attention should be shown to:
nodes during a `git push` are properly authenticated by editing nodes during a `git push` are properly authenticated by editing
`/etc/gitlab/gitlab.rb`: `/etc/gitlab/gitlab.rb`:
You will need to replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret. You need to replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret.
```ruby ```ruby
gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN'
...@@ -837,7 +836,7 @@ Particular attention should be shown to: ...@@ -837,7 +836,7 @@ Particular attention should be shown to:
1. Add Prometheus monitoring settings by editing `/etc/gitlab/gitlab.rb`. If Prometheus 1. Add Prometheus monitoring settings by editing `/etc/gitlab/gitlab.rb`. If Prometheus
is enabled on a different node, make edits on that node instead. is enabled on a different node, make edits on that node instead.
You will need to replace: You need to replace:
- `PRAEFECT_HOST` with the IP address or hostname of the Praefect node - `PRAEFECT_HOST` with the IP address or hostname of the Praefect node
- `GITALY_HOST` with the IP address or hostname of each Gitaly node - `GITALY_HOST` with the IP address or hostname of each Gitaly node
...@@ -922,7 +921,7 @@ To get started quickly: ...@@ -922,7 +921,7 @@ To get started quickly:
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
1. Set the Grafana admin password. This command will prompt you to enter a new 1. Set the Grafana admin password. This command prompts you to enter a new
password: password:
```shell ```shell
...@@ -966,7 +965,7 @@ _Up to date_ in this context means that: ...@@ -966,7 +965,7 @@ _Up to date_ in this context means that:
- The last replication operation is in _completed_ state. - The last replication operation is in _completed_ state.
If there is no such nodes, or any other error occurs during node selection, the primary If there is no such nodes, or any other error occurs during node selection, the primary
node will be chosen to serve the request. node is chosen to serve the request.
To track distribution of read operations, you can use the `gitaly_praefect_read_distribution` To track distribution of read operations, you can use the `gitaly_praefect_read_distribution`
Prometheus counter metric. It has two labels: Prometheus counter metric. It has two labels:
...@@ -1040,9 +1039,9 @@ current primary node is found to be unhealthy. ...@@ -1040,9 +1039,9 @@ current primary node is found to be unhealthy.
- **PostgreSQL (recommended):** Enabled by default, and equivalent to: - **PostgreSQL (recommended):** Enabled by default, and equivalent to:
`praefect['failover_election_strategy'] = sql`. This configuration `praefect['failover_election_strategy'] = sql`. This configuration
option will allow multiple Praefect nodes to coordinate via the option allows multiple Praefect nodes to coordinate via the
PostgreSQL database to elect a primary Gitaly node. This configuration PostgreSQL database to elect a primary Gitaly node. This configuration
will cause Praefect nodes to elect a new primary, monitor its health, causes Praefect nodes to elect a new primary, monitor its health,
and elect a new primary if the current one has not been reachable in and elect a new primary if the current one has not been reachable in
10 seconds by a majority of the Praefect nodes. 10 seconds by a majority of the Praefect nodes.
- **Memory:** Enabled by setting `praefect['failover_election_strategy'] = 'local'` - **Memory:** Enabled by setting `praefect['failover_election_strategy'] = 'local'`
...@@ -1051,8 +1050,7 @@ current primary node is found to be unhealthy. ...@@ -1051,8 +1050,7 @@ current primary node is found to be unhealthy.
be elected. **Do not use with multiple Praefect nodes!** Using with multiple be elected. **Do not use with multiple Praefect nodes!** Using with multiple
Praefect nodes is likely to result in a split brain. Praefect nodes is likely to result in a split brain.
It is likely that we will implement support for Consul, and a cloud native We are likely to implement support for Consul, and a cloud native, strategy in the future.
strategy in the future.
## Primary Node Failure ## Primary Node Failure
......
...@@ -75,7 +75,7 @@ the `gitaly_authentications_total` metric. ...@@ -75,7 +75,7 @@ the `gitaly_authentications_total` metric.
### TLS ### TLS
Gitaly supports TLS encryption. You will need to bring your own certificates as Gitaly supports TLS encryption. You need to bring your own certificates as
this isn't provided automatically. this isn't provided automatically.
| Name | Type | Required | Description | | Name | Type | Required | Description |
...@@ -128,7 +128,7 @@ The following values can be set in the `[git]` section of the configuration file ...@@ -128,7 +128,7 @@ The following values can be set in the `[git]` section of the configuration file
| Name | Type | Required | Description | | Name | Type | Required | Description |
| ---- | ---- | -------- | ----------- | | ---- | ---- | -------- | ----------- |
| `bin_path` | string | no | Path to Git binary. If not set, will be resolved using `PATH`. | | `bin_path` | string | no | Path to Git binary. If not set, is resolved using `PATH`. |
| `catfile_cache_size` | integer | no | Maximum number of cached [cat-file processes](#cat-file-cache). Default is `100`. | | `catfile_cache_size` | integer | no | Maximum number of cached [cat-file processes](#cat-file-cache). Default is `100`. |
#### `cat-file` cache #### `cat-file` cache
...@@ -192,8 +192,8 @@ For historical reasons ...@@ -192,8 +192,8 @@ For historical reasons
[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) contains [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) contains
the Git hooks that allow GitLab to validate and react to Git pushes. the Git hooks that allow GitLab to validate and react to Git pushes.
Because Gitaly "owns" Git pushes, GitLab Shell must therefore be Because Gitaly "owns" Git pushes, GitLab Shell must therefore be
installed alongside Gitaly. This will be [simplified in the installed alongside Gitaly. We plan to
future](https://gitlab.com/gitlab-org/gitaly/-/issues/1226). [simplify this](https://gitlab.com/gitlab-org/gitaly/-/issues/1226).
| Name | Type | Required | Description | | Name | Type | Required | Description |
| ---- | ---- | -------- | ----------- | | ---- | ---- | -------- | ----------- |
...@@ -238,9 +238,11 @@ The following values configure logging in Gitaly under the `[logging]` section. ...@@ -238,9 +238,11 @@ The following values configure logging in Gitaly under the `[logging]` section.
While the main Gitaly application logs go to `stdout`, there are some extra log While the main Gitaly application logs go to `stdout`, there are some extra log
files that go to a configured directory, like the GitLab Shell logs. files that go to a configured directory, like the GitLab Shell logs.
GitLab Shell does not support `panic` or `trace` level logs. `panic` will fall GitLab Shell does not support `panic` or `trace` level logs:
back to `error`, while `trace` will fall back to `debug`. Any other invalid log
levels will default to `info`. - `panic` falls back to `error`.
- `trace` falls back to `debug`.
- Any other invalid log levels default to `info`.
Example: Example:
......
...@@ -50,7 +50,7 @@ storage2: ...@@ -50,7 +50,7 @@ storage2:
> structure than Omnibus. In `gitlab.yml` you indicate the path for the > structure than Omnibus. In `gitlab.yml` you indicate the path for the
> repositories, for example `/home/git/repositories`, while in Omnibus you > repositories, for example `/home/git/repositories`, while in Omnibus you
> indicate `git_data_dirs`, which for the example above would be `/home/git`. > indicate `git_data_dirs`, which for the example above would be `/home/git`.
> Then, Omnibus will create a `repositories` directory under that path to use with > Then, Omnibus creates a `repositories` directory under that path to use with
> `gitlab.yml`. > `gitlab.yml`.
> >
> This little detail matters because while restoring a backup, the current > This little detail matters because while restoring a backup, the current
...@@ -90,8 +90,7 @@ This example uses NFS. We do not recommend using EFS for storage as it may impac ...@@ -90,8 +90,7 @@ This example uses NFS. We do not recommend using EFS for storage as it may impac
1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. 1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
NOTE: **Note:** NOTE: **Note:**
The [`gitlab_shell: repos_path` entry](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457) in `gitlab.yml` will be We plan to replace [`gitlab_shell: repos_path` entry](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457) in `gitlab.yml` with `repositories: storages`. If you
deprecated and replaced by `repositories: storages` in the future, so if you
are upgrading from a version prior to 8.10, make sure to add the configuration are upgrading from a version prior to 8.10, make sure to add the configuration
as described in the step above. After you make the changes and confirm they are as described in the step above. After you make the changes and confirm they are
working, you can remove the `repos_path` line. working, you can remove the `repos_path` line.
...@@ -112,16 +111,15 @@ working, you can remove the `repos_path` line. ...@@ -112,16 +111,15 @@ working, you can remove the `repos_path` line.
Note that Omnibus stores the repositories in a `repositories` subdirectory Note that Omnibus stores the repositories in a `repositories` subdirectory
of the `git-data` directory. of the `git-data` directory.
## Choose where new repositories will be stored ## Choose where new repositories are stored
Once you set the multiple storage paths, you can choose where new repositories Once you set the multiple storage paths, you can choose where new repositories
will be stored under **Admin Area > Settings > Repository > are stored in the Admin Area under **Settings > Repository > Repository storage > Storage nodes for new repositories**.
Repository storage > Storage nodes for new repositories**.
Each storage can be assigned a weight from 0-100. When a new project is created, these Each storage can be assigned a weight from 0-100. When a new project is created, these
weights are used to determine the storage location the repository will be created on. weights are used to determine the storage location the repository is created on.
![Choose repository storage path in Admin Area](img/repository_storages_admin_ui_v13_1.png) ![Choose repository storage path in Admin Area](img/repository_storages_admin_ui_v13_1.png)
Beginning with GitLab 8.13.4, multiple paths can be chosen. New repositories Beginning with GitLab 8.13.4, multiple paths can be chosen. New repositories
will be randomly placed on one of the selected paths. are randomly placed on one of the selected paths.
...@@ -19,7 +19,7 @@ locations that can be: ...@@ -19,7 +19,7 @@ locations that can be:
- Accessed via [Gitaly](gitaly/index.md) on its own machine. - Accessed via [Gitaly](gitaly/index.md) on its own machine.
In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})`
configuration hash. The storage layouts discussed here will apply to any shard configuration hash. The storage layouts discussed here apply to any shard
defined in it. defined in it.
The `default` repository shard that is available in any installations The `default` repository shard that is available in any installations
...@@ -30,22 +30,22 @@ Anything discussed below is expected to be part of that folder. ...@@ -30,22 +30,22 @@ Anything discussed below is expected to be part of that folder.
NOTE: **Note:** NOTE: **Note:**
In GitLab 13.0, hashed storage is enabled by default and the legacy storage is In GitLab 13.0, hashed storage is enabled by default and the legacy storage is
deprecated. Support for legacy storage will be removed in GitLab 14.0. deprecated. Support for legacy storage is scheduled to be removed in GitLab 14.0.
If you haven't migrated yet, check the If you haven't migrated yet, check the
[migration instructions](raketasks/storage.md#migrate-to-hashed-storage). [migration instructions](raketasks/storage.md#migrate-to-hashed-storage).
The option to choose between hashed and legacy storage in the admin area has The option to choose between hashed and legacy storage in the admin area has
been disabled. been disabled.
Hashed storage is the storage behavior we rolled out with 10.0. Instead Hashed storage is the storage behavior we rolled out with 10.0. Instead
of coupling project URL and the folder structure where the repository will be of coupling project URL and the folder structure where the repository is
stored on disk, we are coupling a hash, based on the project's ID. This makes stored on disk, we are coupling a hash, based on the project's ID. This makes
the folder structure immutable, and therefore eliminates any requirement to the folder structure immutable, and therefore eliminates any requirement to
synchronize state from URLs to disk structure. This means that renaming a group, synchronize state from URLs to disk structure. This means that renaming a group,
user, or project will cost only the database transaction, and will take effect user, or project costs only the database transaction, and takes effect
immediately. immediately.
The hash also helps to spread the repositories more evenly on the disk, so the The hash also helps to spread the repositories more evenly on the disk, so the
top-level directory will contain less folders than the total amount of top-level top-level directory contains fewer folders than the total number of top-level
namespaces. namespaces.
The hash format is based on the hexadecimal representation of SHA256: The hash format is based on the hexadecimal representation of SHA256:
...@@ -64,7 +64,7 @@ by another folder with the next 2 characters. They are both stored in a special ...@@ -64,7 +64,7 @@ by another folder with the next 2 characters. They are both stored in a special
### Translating hashed storage paths ### Translating hashed storage paths
Troubleshooting problems with the Git repositories, adding hooks, and other Troubleshooting problems with the Git repositories, adding hooks, and other
tasks will require you translate between the human readable project name tasks requires you translate between the human readable project name
and the hashed storage path. and the hashed storage path.
#### From project name to hashed path #### From project name to hashed path
...@@ -102,7 +102,7 @@ To translate from a hashed storage path to a project name: ...@@ -102,7 +102,7 @@ To translate from a hashed storage path to a project name:
ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project
``` ```
The quoted string in that command is the directory tree you'll find on your The quoted string in that command is the directory tree you can find on your
GitLab server. For example, on a default Omnibus installation this would be GitLab server. For example, on a default Omnibus installation this would be
`/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`
with `.git` from the end of the directory name removed. with `.git` from the end of the directory name removed.
...@@ -135,7 +135,7 @@ when housekeeping is run on the source project. ...@@ -135,7 +135,7 @@ when housekeeping is run on the source project.
### Hashed storage coverage migration ### Hashed storage coverage migration
Files stored in an S3 compatible endpoint will not have the downsides Files stored in an S3-compatible endpoint do not have the downsides
mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`, mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`,
which is true for CI Cache and LFS Objects. which is true for CI Cache and LFS Objects.
...@@ -183,7 +183,7 @@ NOTE: **Deprecated:** ...@@ -183,7 +183,7 @@ NOTE: **Deprecated:**
In GitLab 13.0, hashed storage is enabled by default and the legacy storage is In GitLab 13.0, hashed storage is enabled by default and the legacy storage is
deprecated. If you haven't migrated yet, check the deprecated. If you haven't migrated yet, check the
[migration instructions](raketasks/storage.md#migrate-to-hashed-storage). [migration instructions](raketasks/storage.md#migrate-to-hashed-storage).
Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab Support for legacy storage is scheduled to be removed in GitLab 14.0. If you're on GitLab
13.0 and later, switching new projects to legacy storage is not possible. 13.0 and later, switching new projects to legacy storage is not possible.
The option to choose between hashed and legacy storage in the admin area has The option to choose between hashed and legacy storage in the admin area has
been disabled. been disabled.
...@@ -199,7 +199,7 @@ easy for Administrators to find where the repository is stored. ...@@ -199,7 +199,7 @@ easy for Administrators to find where the repository is stored.
On the other hand this has some drawbacks: On the other hand this has some drawbacks:
Storage location will concentrate huge amount of top-level namespaces. The Storage location concentrates a huge number of top-level namespaces. The
impact can be reduced by the introduction of impact can be reduced by the introduction of
[multiple storage paths](repository_storage_paths.md). [multiple storage paths](repository_storage_paths.md).
...@@ -209,6 +209,6 @@ an old removed or renamed project sharing the same URL. This means that ...@@ -209,6 +209,6 @@ an old removed or renamed project sharing the same URL. This means that
`mygroup/myproject` from your backup may not be the same original project that `mygroup/myproject` from your backup may not be the same original project that
is at that same URL today. is at that same URL today.
Any change in the URL will need to be reflected on disk (when groups / users or Any change in the URL needs to be reflected on disk (when groups / users or
projects are renamed). This can add a lot of load in big installations, projects are renamed). This can add a lot of load in big installations,
especially if using any type of network based filesystem. especially if using any type of network based filesystem.
...@@ -331,7 +331,7 @@ listed in the descriptions of the relevant settings. ...@@ -331,7 +331,7 @@ listed in the descriptions of the relevant settings.
| `receive_max_input_size` | integer | no | Maximum push size (MB). | | `receive_max_input_size` | integer | no | Maximum push size (MB). |
| `repository_checks_enabled` | boolean | no | GitLab will periodically run `git fsck` in all project and wiki repositories to look for silent disk corruption issues. | | `repository_checks_enabled` | boolean | no | GitLab will periodically run `git fsck` in all project and wiki repositories to look for silent disk corruption issues. |
| `repository_size_limit` | integer | no | **(PREMIUM)** Size limit per repository (MB) | | `repository_size_limit` | integer | no | **(PREMIUM)** Size limit per repository (MB) |
| `repository_storages_weighted` | hash of strings to integers | no | (GitLab 13.1 and later) Hash of names of taken from `gitlab.yml` to [weights](../administration/repository_storage_paths.md#choose-where-new-repositories-will-be-stored). New projects are created in one of these stores, chosen by a weighted random selection. | | `repository_storages_weighted` | hash of strings to integers | no | (GitLab 13.1 and later) Hash of names of taken from `gitlab.yml` to [weights](../administration/repository_storage_paths.md#choose-where-new-repositories-are-stored). New projects are created in one of these stores, chosen by a weighted random selection. |
| `repository_storages` | array of strings | no | (GitLab 13.0 and earlier) List of names of enabled storage paths, taken from `gitlab.yml`. New projects are created in one of these stores, chosen at random. | | `repository_storages` | array of strings | no | (GitLab 13.0 and earlier) List of names of enabled storage paths, taken from `gitlab.yml`. New projects are created in one of these stores, chosen at random. |
| `require_admin_approval_after_user_signup` | boolean | no | When enabled, any user that signs up for an account using the registration form is placed under a **Pending approval** state and has to be explicitly [approved](../user/admin_area/approving_users.md) by an administrator. | | `require_admin_approval_after_user_signup` | boolean | no | When enabled, any user that signs up for an account using the registration form is placed under a **Pending approval** state and has to be explicitly [approved](../user/admin_area/approving_users.md) by an administrator. |
| `require_two_factor_authentication` | boolean | no | (**If enabled, requires:** `two_factor_grace_period`) Require all users to set up Two-factor authentication. | | `require_two_factor_authentication` | boolean | no | (**If enabled, requires:** `two_factor_grace_period`) Require all users to set up Two-factor authentication. |
......
...@@ -84,7 +84,7 @@ If your test-suite is failing with Gitaly issues, as a first step, try running: ...@@ -84,7 +84,7 @@ If your test-suite is failing with Gitaly issues, as a first step, try running:
rm -rf tmp/tests/gitaly rm -rf tmp/tests/gitaly
``` ```
During RSpec tests, the Gitaly instance will write logs to `gitlab/log/gitaly-test.log`. During RSpec tests, the Gitaly instance writes logs to `gitlab/log/gitaly-test.log`.
## Legacy Rugged code ## Legacy Rugged code
...@@ -124,23 +124,23 @@ Most of this code exists in the `lib/gitlab/git/rugged_impl` directory. ...@@ -124,23 +124,23 @@ Most of this code exists in the `lib/gitlab/git/rugged_impl` directory.
NOTE: **Note:** NOTE: **Note:**
You should NOT need to add or modify code related to You should NOT need to add or modify code related to
Rugged unless explicitly discussed with the [Gitaly Rugged unless explicitly discussed with the [Gitaly
Team](https://gitlab.com/groups/gl-gitaly/group_members). This code will Team](https://gitlab.com/groups/gl-gitaly/group_members). This code does
NOT work on GitLab.com or other GitLab instances that do not use NFS. NOT work on GitLab.com or other GitLab instances that do not use NFS.
## `TooManyInvocationsError` errors ## `TooManyInvocationsError` errors
During development and testing, you may experience `Gitlab::GitalyClient::TooManyInvocationsError` failures. During development and testing, you may experience `Gitlab::GitalyClient::TooManyInvocationsError` failures.
The `GitalyClient` will attempt to block against potential n+1 issues by raising this error The `GitalyClient` attempts to block against potential n+1 issues by raising this error
when Gitaly is called more than 30 times in a single Rails request or Sidekiq execution. when Gitaly is called more than 30 times in a single Rails request or Sidekiq execution.
As a temporary measure, export `GITALY_DISABLE_REQUEST_LIMITS=1` to suppress the error. This will disable the n+1 detection As a temporary measure, export `GITALY_DISABLE_REQUEST_LIMITS=1` to suppress the error. This disables the n+1 detection
in your development environment. in your development environment.
Please raise an issue in the GitLab CE or EE repositories to report the issue. Include the labels ~Gitaly Please raise an issue in the GitLab CE or EE repositories to report the issue. Include the labels ~Gitaly
~performance ~"technical debt". Please ensure that the issue contains the full stack trace and error message of the ~performance ~"technical debt". Please ensure that the issue contains the full stack trace and error message of the
`TooManyInvocationsError`. Also include any known failing tests if possible. `TooManyInvocationsError`. Also include any known failing tests if possible.
Isolate the source of the n+1 problem. This will normally be a loop that results in Gitaly being called for each Isolate the source of the n+1 problem. This is normally a loop that results in Gitaly being called for each
element in an array. If you are unable to isolate the problem, please contact a member element in an array. If you are unable to isolate the problem, please contact a member
of the [Gitaly Team](https://gitlab.com/groups/gl-gitaly/group_members) for assistance. of the [Gitaly Team](https://gitlab.com/groups/gl-gitaly/group_members) for assistance.
...@@ -154,7 +154,7 @@ Gitlab::GitalyClient.allow_n_plus_1_calls do ...@@ -154,7 +154,7 @@ Gitlab::GitalyClient.allow_n_plus_1_calls do
end end
``` ```
Once the code is wrapped in this block, this code-path will be excluded from n+1 detection. Once the code is wrapped in this block, this code path is excluded from n+1 detection.
## Request counts ## Request counts
...@@ -184,12 +184,12 @@ branches and SHA to use a custom commit in <https://gitlab.com/gitlab-org/gitaly ...@@ -184,12 +184,12 @@ branches and SHA to use a custom commit in <https://gitlab.com/gitlab-org/gitaly
NOTE: **Note:** NOTE: **Note:**
With the introduction of auto-deploy for Gitaly, the format of With the introduction of auto-deploy for Gitaly, the format of
`GITALY_SERVER_VERSION` was aligned with Omnibus syntax. `GITALY_SERVER_VERSION` was aligned with Omnibus syntax.
It no longer supports `=revision`, it will evaluate the file content as a Git It no longer supports `=revision`, it evaluates the file content as a Git
reference (branch or SHA), only if it matches a semver it will prepend a `v`. reference (branch or SHA). Only if it matches a semver does it prepend a `v`.
If you want to run tests locally against a modified version of Gitaly you If you want to run tests locally against a modified version of Gitaly you
can replace `tmp/tests/gitaly` with a symlink. This is much faster can replace `tmp/tests/gitaly` with a symlink. This is much faster
because if will avoid a Gitaly re-install each time you run `rspec`. because it avoids a Gitaly re-install each time you run `rspec`.
```shell ```shell
rm -rf tmp/tests/gitaly rm -rf tmp/tests/gitaly
...@@ -197,12 +197,12 @@ ln -s /path/to/gitaly tmp/tests/gitaly ...@@ -197,12 +197,12 @@ ln -s /path/to/gitaly tmp/tests/gitaly
``` ```
Make sure you run `make` in your local Gitaly directory before running Make sure you run `make` in your local Gitaly directory before running
tests. Otherwise, Gitaly will fail to boot. tests. Otherwise, Gitaly fails to boot.
If you make changes to your local Gitaly in between test runs you need If you make changes to your local Gitaly in between test runs you need
to manually run `make` again. to manually run `make` again.
Note that CI tests will not use your locally modified version of Note that CI tests do not use your locally modified version of
Gitaly. To use a custom Gitaly version in CI you need to update Gitaly. To use a custom Gitaly version in CI you need to update
GITALY_SERVER_VERSION as described at the beginning of this paragraph. GITALY_SERVER_VERSION as described at the beginning of this paragraph.
......
...@@ -87,7 +87,7 @@ download all the advertised refs. ...@@ -87,7 +87,7 @@ download all the advertised refs.
git push origin --force 'refs/heads/*' git push origin --force 'refs/heads/*'
``` ```
[Protected branches](../protected_branches.md) will cause this to fail. To proceed, you must [Protected branches](../protected_branches.md) cause this to fail. To proceed, you must
remove branch protection, push, and then re-enable protected branches. remove branch protection, push, and then re-enable protected branches.
1. To remove large files from tagged releases, force push your changes to all tags on GitLab: 1. To remove large files from tagged releases, force push your changes to all tags on GitLab:
...@@ -96,7 +96,7 @@ download all the advertised refs. ...@@ -96,7 +96,7 @@ download all the advertised refs.
git push origin --force 'refs/tags/*' git push origin --force 'refs/tags/*'
``` ```
[Protected tags](../protected_tags.md) will cause this to fail. To proceed, you must remove tag [Protected tags](../protected_tags.md) cause this to fail. To proceed, you must remove tag
protection, push, and then re-enable protected tags. protection, push, and then re-enable protected tags.
1. To prevent dead links to commits that no longer exist, push the `refs/replace` created by `git filter-repo`. 1. To prevent dead links to commits that no longer exist, push the `refs/replace` created by `git filter-repo`.
...@@ -131,7 +131,7 @@ To purge files from GitLab storage: ...@@ -131,7 +131,7 @@ To purge files from GitLab storage:
tar xzf project-backup.tar.gz tar xzf project-backup.tar.gz
``` ```
This will contain a `project.bundle` file, which was created by This contains a `project.bundle` file, which was created by
[`git bundle`](https://git-scm.com/docs/git-bundle). [`git bundle`](https://git-scm.com/docs/git-bundle).
1. Clone a fresh copy of the repository from the bundle: 1. Clone a fresh copy of the repository from the bundle:
...@@ -141,12 +141,12 @@ To purge files from GitLab storage: ...@@ -141,12 +141,12 @@ To purge files from GitLab storage:
``` ```
1. Using `git filter-repo`, purge any files from the history of your repository. Because we are 1. Using `git filter-repo`, purge any files from the history of your repository. Because we are
trying to remove internal refs, we will rely on the `commit-map` produced by each run to tell us trying to remove internal refs, we rely on the `commit-map` produced by each run to tell us
which internal refs to remove. which internal refs to remove.
NOTE: **Note:** NOTE: **Note:**
`git filter-repo` creates a new `commit-map` file every run, and overwrite the `commit-map` from `git filter-repo` creates a new `commit-map` file every run, and overwrite the `commit-map` from
the previous run. You will need this file from **every** run. Do the next step every time you run the previous run. You need this file from **every** run. Do the next step every time you run
`git filter-repo`. `git filter-repo`.
To purge all large files, the `--strip-blobs-bigger-than` option can be used: To purge all large files, the `--strip-blobs-bigger-than` option can be used:
...@@ -178,7 +178,7 @@ To purge files from GitLab storage: ...@@ -178,7 +178,7 @@ To purge files from GitLab storage:
git push origin --force 'refs/heads/*' git push origin --force 'refs/heads/*'
``` ```
[Protected branches](../protected_branches.md) will cause this to fail. To proceed, you must [Protected branches](../protected_branches.md) cause this to fail. To proceed, you must
remove branch protection, push, and then re-enable protected branches. remove branch protection, push, and then re-enable protected branches.
1. To remove large files from tagged releases, force push your changes to all tags on GitLab: 1. To remove large files from tagged releases, force push your changes to all tags on GitLab:
...@@ -187,7 +187,7 @@ To purge files from GitLab storage: ...@@ -187,7 +187,7 @@ To purge files from GitLab storage:
git push origin --force 'refs/tags/*' git push origin --force 'refs/tags/*'
``` ```
[Protected tags](../protected_tags.md) will cause this to fail. To proceed, you must remove tag [Protected tags](../protected_tags.md) cause this to fail. To proceed, you must remove tag
protection, push, and then re-enable protected tags. protection, push, and then re-enable protected tags.
1. To prevent dead links to commits that no longer exist, push the `refs/replace` created by `git filter-repo`. 1. To prevent dead links to commits that no longer exist, push the `refs/replace` created by `git filter-repo`.
...@@ -205,12 +205,12 @@ To purge files from GitLab storage: ...@@ -205,12 +205,12 @@ To purge files from GitLab storage:
NOTE: **Note:** NOTE: **Note:**
Safely cleaning the repository requires it to be made read-only for the duration Safely cleaning the repository requires it to be made read-only for the duration
of the operation. This happens automatically, but submitting the cleanup request of the operation. This happens automatically, but submitting the cleanup request
will fail if any writes are ongoing, so cancel any outstanding `git push` fails if any writes are ongoing, so cancel any outstanding `git push`
operations before continuing. operations before continuing.
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19376) in GitLab 11.6. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19376) in GitLab 11.6.
Repository cleanup allows you to upload a text file of objects and GitLab will remove internal Git Repository cleanup allows you to upload a text file of objects and GitLab removes internal Git
references to these objects. You can use references to these objects. You can use
[`git filter-repo`](https://github.com/newren/git-filter-repo) to produce a list of objects (in a [`git filter-repo`](https://github.com/newren/git-filter-repo) to produce a list of objects (in a
`commit-map` file) that can be used with repository cleanup. `commit-map` file) that can be used with repository cleanup.
...@@ -230,25 +230,25 @@ To clean up a repository: ...@@ -230,25 +230,25 @@ To clean up a repository:
1. Click **Start cleanup**. 1. Click **Start cleanup**.
This will: This:
- Remove any internal Git references to old commits. - Removes any internal Git references to old commits.
- Run `git gc` against the repository to remove unreferenced objects. Repacking your repository will temporarily - Runs `git gc` against the repository to remove unreferenced objects. Repacking your repository temporarily
cause the size of your repository to increase significantly, because the old pack files are not removed until the causes the size of your repository to increase significantly, because the old pack files are not removed until the
new pack files have been created. new pack files have been created.
- Unlink any unused LFS objects currently attached to your project, freeing up storage space. - Unlinks any unused LFS objects currently attached to your project, freeing up storage space.
- Recalculate the size of your repository on disk. - Recalculates the size of your repository on disk.
You will receive an email notification with the recalculated repository size after the cleanup has completed. GitLab sends an email notification with the recalculated repository size after the cleanup has completed.
When using repository cleanup, note: When using repository cleanup, note:
- Project statistics are cached. You may need to wait 5-10 minutes to see a reduction in storage utilization. - Project statistics are cached. You may need to wait 5-10 minutes to see a reduction in storage utilization.
- Housekeeping prunes loose objects older than 2 weeks. This means objects added in the last 2 weeks - Housekeeping prunes loose objects older than 2 weeks. This means objects added in the last 2 weeks
will not be removed immediately. If you have access to the are not be removed immediately. If you have access to the
[Gitaly](../../../administration/gitaly/index.md) server, you may run `git gc --prune=now` to [Gitaly](../../../administration/gitaly/index.md) server, you may run `git gc --prune=now` to
prune all loose objects immediately. prune all loose objects immediately.
- This process will remove some copies of the rewritten commits from GitLab's cache and database, - This process removes some copies of the rewritten commits from GitLab's cache and database,
but there are still numerous gaps in coverage and some of the copies may persist indefinitely. but there are still numerous gaps in coverage and some of the copies may persist indefinitely.
[Clearing the instance cache](../../../administration/raketasks/maintenance.md#clear-redis-cache) [Clearing the instance cache](../../../administration/raketasks/maintenance.md#clear-redis-cache)
may help to remove some of them, but it should not be depended on for security purposes! may help to remove some of them, but it should not be depended on for security purposes!
...@@ -290,7 +290,7 @@ history. We recommend the open-source community-maintained tool ...@@ -290,7 +290,7 @@ history. We recommend the open-source community-maintained tool
[`git filter-repo`](https://github.com/newren/git-filter-repo). [`git filter-repo`](https://github.com/newren/git-filter-repo).
NOTE: **Note:** NOTE: **Note:**
Until `git gc` runs on the GitLab side, the "removed" commits and blobs will still exist. You also Until `git gc` runs on the GitLab side, the "removed" commits and blobs still exist. You also
must be able to push the rewritten history to GitLab, which may be impossible if you've already must be able to push the rewritten history to GitLab, which may be impossible if you've already
exceeded the maximum size limit. exceeded the maximum size limit.
...@@ -304,7 +304,7 @@ increased, your only option is to: ...@@ -304,7 +304,7 @@ increased, your only option is to:
CAUTION: **Caution:** CAUTION: **Caution:**
This process is not suitable for removing sensitive data like password or keys from your repository. This process is not suitable for removing sensitive data like password or keys from your repository.
Information about commits, including file content, is cached in the database, and will remain Information about commits, including file content, is cached in the database, and remain
visible even after they have been removed from the repository. visible even after they have been removed from the repository.
## Troubleshooting ## Troubleshooting
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment