Skip to content

G
gitlab-ce
Commit d6faf65d authored by Craig Norris's avatar Craig Norris Committed by Achilleas Pipinellis
Browse files
Options

Tier badge and table cleanup for Object storage 

General style updates for Object storage doc page.
parent 9728d78a
Hide whitespace changes
Inline Side-by-side
Showing with 87 additions and 87 deletions
doc/administration/object_storage.md
View file @ d6faf65d
...@@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ...@@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
type: reference type: reference
--- ---
# Object Storage # Object storage **(FREE SELF)**
GitLab supports using an object storage service for holding numerous types of data. GitLab supports using an object storage service for holding numerous types of data.
It's recommended over NFS and It's recommended over NFS and
...@@ -29,7 +29,7 @@ GitLab has been tested on a number of object storage providers: ...@@ -29,7 +29,7 @@ GitLab has been tested on a number of object storage providers:
- Dell EMC ECS: Prior to GitLab 13.3, there is a [known bug in GitLab Workhorse that prevents - Dell EMC ECS: Prior to GitLab 13.3, there is a [known bug in GitLab Workhorse that prevents
HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806). HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806).
Be sure to upgrade to GitLab v13.3.0 or above if you use S3 storage with this hardware. Be sure to upgrade to GitLab 13.3.0 or above if you use S3 storage with this hardware.
- Ceph S3 prior to [Kraken 11.0.2](https://ceph.com/releases/kraken-11-0-2-released/) does not support the [Upload Copy Part API](https://gitlab.com/gitlab-org/gitlab/-/issues/300604). You may need to [disable multi-threaded copying](#multi-threaded-copying). - Ceph S3 prior to [Kraken 11.0.2](https://ceph.com/releases/kraken-11-0-2-released/) does not support the [Upload Copy Part API](https://gitlab.com/gitlab-org/gitlab/-/issues/300604). You may need to [disable multi-threaded copying](#multi-threaded-copying).
...@@ -47,7 +47,7 @@ For more information on the differences and to transition from one form to anoth ...@@ -47,7 +47,7 @@ For more information on the differences and to transition from one form to anoth
### Consolidated object storage configuration ### Consolidated object storage configuration
> Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368). > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368) in GitLab 13.2.
Using the consolidated object storage configuration has a number of advantages: Using the consolidated object storage configuration has a number of advantages:
...@@ -209,13 +209,13 @@ gitlab_rails['object_store']['connection'] = { ...@@ -209,13 +209,13 @@ gitlab_rails['object_store']['connection'] = {
} }
``` ```
| Setting | Description | | Setting | Description |
|---------|-------------| |-------------------|-----------------------------------|
| `enabled` | Enable/disable object storage | | `enabled` | Enable or disable object storage. |
| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | | `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. |
| `connection` | Various [connection options](#connection-settings) described below | | `connection` | Various [connection options](#connection-settings) described below. |
| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3 | | `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3. |
| `objects` | [Object-specific configuration](#object-specific-configuration) | `objects` | [Object-specific configuration](#object-specific-configuration). |
### Connection settings ### Connection settings
...@@ -226,27 +226,27 @@ in the `connection` setting. ...@@ -226,27 +226,27 @@ in the `connection` setting.
The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws): The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws):
| Setting | Description | Default | | Setting | Description | Default |
|---------|-------------|---------| |---------------------------------|------------------------------------|---------|
| `provider` | Always `AWS` for compatible hosts | `AWS` | | `provider` | Always `AWS` for compatible hosts. | `AWS` |
| `aws_access_key_id` | AWS credentials, or compatible | | | `aws_access_key_id` | AWS credentials, or compatible. | |
| `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible. | |
| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | | `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` |
| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | | `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` |
| `region` | AWS region. | | | `region` | AWS region. | |
| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | | `host` | S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` |
| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | | `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) |
| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false` | | `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false`. |
| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys | `false` | `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` |
#### Oracle Cloud S3 connection settings #### Oracle Cloud S3 connection settings
Note that Oracle Cloud S3 must be sure to use the following settings: Note that Oracle Cloud S3 must be sure to use the following settings:
| Setting | Value | | Setting | Value |
|---------|-------| |---------------------------------|---------|
| `enable_signature_v4_streaming` | `false` | | `enable_signature_v4_streaming` | `false` |
| `path_style` | `true` | | `path_style` | `true` |
If `enable_signature_v4_streaming` is set to `true`, you may see the If `enable_signature_v4_streaming` is set to `true`, you may see the
following error in `production.log`: following error in `production.log`:
...@@ -259,13 +259,13 @@ STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported ...@@ -259,13 +259,13 @@ STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported
Here are the valid connection parameters for GCS: Here are the valid connection parameters for GCS:
| Setting | Description | example | | Setting | Description | Example |
|---------|-------------|---------| |------------------------------|-------------------|---------|
| `provider` | The provider name | `Google` | | `provider` | Provider name. | `Google` |
| `google_project` | GCP project name | `gcp-project-12345` | | `google_project` | GCP project name. | `gcp-project-12345` |
| `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` | | `google_client_email` | Email address of the service account. | `foo@gcp-project-12345.iam.gserviceaccount.com` |
| `google_json_key_location` | The JSON key path | `/path/to/gcp-project-12345-abcde.json` | | `google_json_key_location` | JSON key path. | `/path/to/gcp-project-12345-abcde.json` |
| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | | `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | |
The service account must have permission to access the bucket. Learn more The service account must have permission to access the bucket. Learn more
in Google's in Google's
...@@ -328,12 +328,12 @@ The following are the valid connection parameters for Azure. Read the ...@@ -328,12 +328,12 @@ The following are the valid connection parameters for Azure. Read the
[Azure Blob storage documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) [Azure Blob storage documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction)
to learn more. to learn more.
| Setting | Description | Example | | Setting | Description | Example |
|---------|-------------|---------| |------------------------------|----------------|-----------|
| `provider` | Provider name | `AzureRM` | | `provider` | Provider name. | `AzureRM` |
| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage | `azuretest` | | `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage. | `azuretest` |
| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | | `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` |
| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | | `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` |
##### Azure example (consolidated form) ##### Azure example (consolidated form)
...@@ -382,15 +382,15 @@ consolidated form, see the [S3 settings](#s3-compatible-connection-settings). ...@@ -382,15 +382,15 @@ consolidated form, see the [S3 settings](#s3-compatible-connection-settings).
Here are the valid connection settings for the Swift API, provided by Here are the valid connection settings for the Swift API, provided by
[fog-openstack](https://github.com/fog/fog-openstack): [fog-openstack](https://github.com/fog/fog-openstack):
| Setting | Description | Default | | Setting | Description | Default |
|---------|-------------|---------| |--------------------------|----------------------|---------|
| `provider` | Always `OpenStack` for compatible hosts | `OpenStack` | | `provider` | Always `OpenStack` for compatible hosts. | `OpenStack` |
| `openstack_username` | OpenStack username | | | `openstack_username` | OpenStack username. | |
| `openstack_api_key` | OpenStack API key | | | `openstack_api_key` | OpenStack API key. | |
| `openstack_temp_url_key` | OpenStack key for generating temporary URLs | | | `openstack_temp_url_key` | OpenStack key for generating temporary URLs | |
| `openstack_auth_url` | OpenStack authentication endpoint | | | `openstack_auth_url` | OpenStack authentication endpoint | |
| `openstack_region` | OpenStack region | | | `openstack_region` | OpenStack region. | |
| `openstack_tenant` | OpenStack tenant ID | | `openstack_tenant` | OpenStack tenant ID. | |
#### Rackspace Cloud Files #### Rackspace Cloud Files
...@@ -400,13 +400,13 @@ Rackspace Cloud, provided by [fog-rackspace](https://github.com/fog/fog-rackspac ...@@ -400,13 +400,13 @@ Rackspace Cloud, provided by [fog-rackspace](https://github.com/fog/fog-rackspac
This isn't compatible with the consolidated object storage form. This isn't compatible with the consolidated object storage form.
Rackspace Cloud is supported only with the storage-specific form. Rackspace Cloud is supported only with the storage-specific form.
| Setting | Description | example | | Setting | Description | Example |
|---------|-------------|---------| |--------------------------|----------------|-------------|
| `provider` | The provider name | `Rackspace` | | `provider` | Provider name. | `Rackspace` |
| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | | `rackspace_username` | Username of the Rackspace account with access to the container. | `joe.smith` |
| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | | `rackspace_api_key` | API key of the Rackspace account with access to the container. | `ABC123DEF456ABC123DEF456ABC123DE` |
| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | | `rackspace_region` | Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/). | `iad` |
| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | | `rackspace_temp_url_key` | Private key you set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` |
Regardless of whether the container has public access enabled or disabled, Fog Regardless of whether the container has public access enabled or disabled, Fog
uses the TempURL method to grant access to LFS objects. If you see error uses the TempURL method to grant access to LFS objects. If you see error
...@@ -465,24 +465,24 @@ gitlab_rails['object_store']['objects']['pages']['bucket'] = 'pages' ...@@ -465,24 +465,24 @@ gitlab_rails['object_store']['objects']['pages']['bucket'] = 'pages'
This is the list of valid `objects` that can be used: This is the list of valid `objects` that can be used:
| Type | Description | | Type | Description |
|--------------------|---------------| |--------------------|----------------------------------------------------------------------------|
| `artifacts` | [CI artifacts](job_artifacts.md) | | `artifacts` | [CI artifacts](job_artifacts.md) |
| `external_diffs` | [Merge request diffs](merge_request_diffs.md) | | `external_diffs` | [Merge request diffs](merge_request_diffs.md) |
| `uploads` | [User uploads](uploads.md) | | `uploads` | [User uploads](uploads.md) |
| `lfs` | [Git Large File Storage objects](lfs/index.md) | | `lfs` | [Git Large File Storage objects](lfs/index.md) |
| `packages` | [Project packages (e.g. PyPI, Maven, NuGet, etc.)](packages/index.md) | | `packages` | [Project packages (for example, PyPI, Maven, or NuGet)](packages/index.md) |
| `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) | | `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) |
| `terraform_state` | [Terraform state files](terraform_state.md) | | `terraform_state` | [Terraform state files](terraform_state.md) |
| `pages` | [GitLab Pages](pages/index.md) | | `pages` | [GitLab Pages](pages/index.md) |
Within each object type, three parameters can be defined: Within each object type, three parameters can be defined:
| Setting | Required? | Description | | Setting | Required? | Description |
|------------------|-----------|-------------| |------------------|------------------------|-------------------------------------|
| `bucket` | Yes | The bucket name for the object storage. | | `bucket` | **{check-circle}** Yes | Bucket name for the object storage. |
| `enabled` | No | Overrides the common parameter | | `enabled` | **{dotted-circle}** No | Overrides the common parameter. |
| `proxy_download` | No | Overrides the common parameter | | `proxy_download` | **{dotted-circle}** No | Overrides the common parameter. |
#### Selectively disabling object storage #### Selectively disabling object storage
...@@ -542,21 +542,21 @@ original configuration (for example, `artifacts_object_store_enabled`, or ...@@ -542,21 +542,21 @@ original configuration (for example, `artifacts_object_store_enabled`, or
For configuring object storage in GitLab 13.1 and earlier, or for storage types not For configuring object storage in GitLab 13.1 and earlier, or for storage types not
supported by consolidated configuration form, refer to the following guides: supported by consolidated configuration form, refer to the following guides:
|Object storage type|Supported by consolidated configuration?| | Object storage type | Supported by consolidated configuration? |
|-------------------|----------------------------------------| |---------------------|------------------------------------------|
| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No |
| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | Yes | | [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes |
| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes |
| [Uploads](uploads.md#using-object-storage) | Yes | | [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes |
| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | No | | [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No |
| [Merge request diffs](merge_request_diffs.md#using-object-storage) | Yes | | [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes |
| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| **{dotted-circle}** No |
| [Packages](packages/index.md#using-object-storage) (optional feature) | Yes | | [Packages](packages/index.md#using-object-storage) (optional feature) | **{check-circle}** Yes |
| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | Yes | | [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | **{check-circle}** Yes |
| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | | [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | **{dotted-circle}** No |
| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No |
| [Terraform state files](terraform_state.md#using-object-storage) | Yes | | [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes |
| [GitLab Pages content](pages/index.md#using-object-storage) | Yes | | [GitLab Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes |
### Other alternatives to file system storage ### Other alternatives to file system storage
...@@ -709,10 +709,10 @@ only encrypted objects are uploaded](https://aws.amazon.com/premiumsupport/knowl ...@@ -709,10 +709,10 @@ only encrypted objects are uploaded](https://aws.amazon.com/premiumsupport/knowl
To do this, you must configure GitLab to send the proper encryption headers To do this, you must configure GitLab to send the proper encryption headers
in the `storage_options` configuration section: in the `storage_options` configuration section:
| Setting | Description | | Setting | Description |
|-------------------------------------|-------------| |-------------------------------------|------------------------------------------|
| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`) | | `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`). |
| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) | | `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). |
As with the case for default encryption, these options only work when As with the case for default encryption, these options only work when
the Workhorse S3 client is enabled. One of the following two conditions the Workhorse S3 client is enabled. One of the following two conditions
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7