Merge branch 'patch-176' into 'master'

Improve container registry migrate to object storage details

See merge request gitlab-org/gitlab!35932

doc/administration/geo/replication/docker_registry.md

@@ -18,7 +18,7 @@ Registry on the **primary** node, you can use the same storage for a **secondary
 Docker Registry as well. For more information, read the
 [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations)
 when deploying the Registry, and how to set up the storage driver for GitLab's
-integrated [Container Registry](../../packages/container_registry.md#container-registry-storage-driver).
+integrated [Container Registry](../../packages/container_registry.md#use-object-storage).
 
 ## Replicating Docker Registry
 

doc/administration/object_storage.md

@@ -422,7 +422,7 @@ supported by consolidated configuration form, refer to the following guides:
 | [Job artifacts](job_artifacts.md#using-object-storage) and [incremental logging](job_logs.md#new-incremental-logging-architecture) | Yes |
 | [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes |
 | [Uploads](uploads.md#using-object-storage-core-only) | Yes |
-| [Container Registry](packages/container_registry.md#container-registry-storage-driver) (optional feature) | No |
+| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | No |
 | [Merge request diffs](merge_request_diffs.md#using-object-storage) | Yes |
 | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No |
 | [Packages](packages/index.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes |

doc/administration/packages/container_registry.md

@@ -76,7 +76,7 @@ where:
 | `port`    | The port under which the external Registry domain will listen on. |
 | `api_url` | The internal API URL under which the Registry is exposed to. It defaults to `http://localhost:5000`. |
 | `key`     | The private key location that is a pair of Registry's `rootcertbundle`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). |
-| `path`    | This should be the same directory like specified in Registry's `rootdirectory`. Read the [storage configuration documentation](https://docs.docker.com/registry/configuration/#storage). This path needs to be readable by the GitLab user, the web-server user and the Registry user. Read more in [#container-registry-storage-path](#container-registry-storage-path). |
+| `path`    | This should be the same directory like specified in Registry's `rootdirectory`. Read the [storage configuration documentation](https://docs.docker.com/registry/configuration/#storage). This path needs to be readable by the GitLab user, the web-server user and the Registry user. Read more in [#configure-storage-for-the-container-registry](#configure-storage-for-the-container-registry). |
 | `issuer`  | This should be the same value as configured in Registry's `issuer`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). |
 
 NOTE: **Note:**
@@ -313,11 +313,28 @@ the Container Registry by themselves, follow the steps below.
 
 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect.
 
-## Container Registry storage path
+## Configure storage for the Container Registry
 
-NOTE: **Note:**
-For configuring storage in the cloud instead of the filesystem, see the
-[storage driver configuration](#container-registry-storage-driver).
+You can configure the Container Registry to use various storage backends by
+configuring a storage driver. By default the GitLab Container Registry
+is configured to use the [filesystem driver](#use-filesystem)
+configuration.
+
+The different supported drivers are:
+
+| Driver     | Description                         |
+|------------|-------------------------------------|
+| filesystem | Uses a path on the local filesystem |
+| Azure      | Microsoft Azure Blob Storage        |
+| gcs        | Google Cloud Storage                |
+| s3         | Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct [S3 Permission Scopes](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). |
+| swift      | OpenStack Swift Object Storage      |
+| oss        | Aliyun OSS                          |
+
+Read more about the individual driver's configuration options in the
+[Docker Registry docs](https://docs.docker.com/registry/configuration/#storage).
+
+### Use filesystem
 
 If you want to store your images on the filesystem, you can change the storage
 path for the Container Registry, follow the steps below.
@@ -327,7 +344,7 @@ This path is accessible to:
 - The user running the Container Registry daemon.
 - The user running GitLab.
 
-CAUTION: **Warning** You should confirm that all GitLab, Registry and web server users
+CAUTION: **Warning:** You should confirm that all GitLab, Registry and web server users
 have access to this directory.
 
 **Omnibus GitLab installations**
@@ -358,26 +375,10 @@ The default location where images are stored in source installations, is
 
 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect.
 
-### Container Registry storage driver
+### Use object storage
 
-You can configure the Container Registry to use a different storage backend by
-configuring a different storage driver. By default the GitLab Container Registry
-is configured to use the filesystem driver, which makes use of [storage path](#container-registry-storage-path)
-configuration.
-
-The different supported drivers are:
-
-| Driver     | Description                         |
-|------------|-------------------------------------|
-| filesystem | Uses a path on the local filesystem |
-| Azure      | Microsoft Azure Blob Storage        |
-| gcs        | Google Cloud Storage                |
-| s3         | Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct [S3 Permission Scopes](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). |
-| swift      | OpenStack Swift Object Storage      |
-| oss        | Aliyun OSS                          |
-
-Read more about the individual driver's configuration options in the
-[Docker Registry docs](https://docs.docker.com/registry/configuration/#storage).
+If you want to store your images on object storage, you can change the storage
+driver for the Container Registry.
 
 [Read more about using object storage with GitLab](../object_storage.md).
 
@@ -435,21 +436,43 @@ storage:
 NOTE: **Note:**
 `your-s3-bucket` should only be the name of a bucket that exists, and can't include subdirectories.
 
-**Migrate without downtime**
+#### Migrate to object storage without downtime
+
+To migrate storage without stopping the Container Registry, set the Container Registry
+to read-only mode. On large instances, this may require the Container Registry
+to be in read-only mode for a while. During this time,
+you can pull from the Container Registry, but you cannot push.
+
+1. Optional: To reduce the amount of data to be migrated, run the [garbage collection tool without downtime](#performing-garbage-collection-without-downtime).
+1. Copy initial data to your S3 bucket, for example with the AWS CLI [`cp`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html)
+   or [`sync`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html)
+   command. Make sure to keep the `docker` folder as the top-level folder inside the bucket.
+
+   ```shell
+   aws s3 sync registry s3://mybucket
+   ```
+
+1. For the changes to take effect,
+   [put the Container Registry in `read-only` mode](#performing-garbage-collection-without-downtime) and
+   [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Sync any changes since the initial data load to your S3 bucket and delete files that exist in the destination bucket but not in the source:
+
+   ```shell
+   aws s3 sync registry s3://mybucket --delete
+   ```
 
-To migrate the data to AWS S3 without downtime:
+   DANGER: **Danger:**
+   The `--delete` flag will delete files that exist in the destination but not in the source.
+   Make sure not to swap the source and destination, or you will delete all data in the Registry.
 
-1. To reduce the amount of data to be migrated, run the [garbage collection tool without downtime](#performing-garbage-collection-without-downtime). Part of this process sets the registry to `read-only`.
-1. Copy the data to your AWS S3 bucket, for example with [AWS CLI's `cp`](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html) command.
 1. Configure your registry to use the S3 bucket for storage.
-1. Put the registry back to `read-write`.
-1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+1. For the changes to take effect, set the Registry back to `read-write` mode and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
 ### Disable redirect for storage driver
 
 By default, users accessing a registry configured with a remote backend are redirected to the default backend for the storage driver. For example, registries can be configured using the `s3` storage driver, which redirects requests to a remote S3 bucket to alleviate load on the GitLab server.
 
-However, this behavior is undesirable for registries used by internal hosts that usually can't access public servers. To disable redirects, set the `disable` flag to true as follows. This makes all traffic to always go through the Registry service. This results in improved security (less surface attack as the storage backend is not publicly accessible), but worse performance (all traffic is redirected via the service).
+However, this behavior is undesirable for registries used by internal hosts that usually can't access public servers. To disable redirects and [proxy download](../object_storage.md#proxy-download), set the `disable` flag to true as follows. This makes all traffic always go through the Registry service. This results in improved security (less surface attack as the storage backend is not publicly accessible), but worse performance (all traffic is redirected via the service).
 
 **Omnibus GitLab installations**
 
@@ -779,13 +802,15 @@ that you have backed up all registry data.
 
 > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/764) in GitLab 8.8.
 
-You can perform a garbage collection without stopping the Container Registry by setting
-it into a read-only mode and by not using the built-in command. During this time,
+You can perform garbage collection without stopping the Container Registry by putting
+it in read-only mode and by not using the built-in command. On large instances
+this could require Container Registry to be in read-only mode for a while.
+During this time,
 you will be able to pull from the Container Registry, but you will not be able to
 push.
 
 NOTE: **Note:**
-By default, the [registry storage path](#container-registry-storage-path)
+By default, the [registry storage path](#configure-storage-for-the-container-registry)
 is `/var/opt/gitlab/gitlab-rails/shared/registry`.
 
 To enable the read-only mode:

doc/administration/reference_architectures/2k_users.md

@@ -816,7 +816,7 @@ on the features you intend to use:
 1. [Object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage).
 1. [Object storage for uploads](../uploads.md#using-object-storage-core-only).
 1. [Object storage for merge request diffs](../merge_request_diffs.md#using-object-storage).
-1. [Object storage for Container Registry](../packages/container_registry.md#container-registry-storage-driver) (optional feature).
+1. [Object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature).
 1. [Object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature).
 1. [Object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)**
 1. [Object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)**