Add Azure Blob Storage support
This merge request introduces a client for Azure Blob Storage in Workhorse. Currently customers wanting to use Azure Blob Storage have to set up a Minio Gateway (https://docs.gitlab.com/charts/advanced/external-object-storage/azure-minio-gateway.html), which isn't ideal because it requires customers to maintain their own proxy server for Azure. We have a number of customers who want native support for Azure Blob Storage. Unlike AWS and Google, Azure needs to use an Azure client inside Workhorse to support direct uploads. Using standard HTTP transfers with pre-signed URLs with the Azure Put Blob API (https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob) doesn't work because Azure doesn't support chunked transfer encoding. However, Azure does support uploading files in segments via the Put Block and Put Block List API (https://docs.microsoft.com/en-us/rest/api/storageservices/put-block), but this requires an Azure client that can speak this API. Instead of embedding the Microsoft Azure client directly, we use the Go Cloud Development Kit (https://godoc.org/gocloud.dev/blob) to make it easier to add other object storage providers later. For example, GitLab Rails might return this JSON payload in the `/internal/uploads/authorize` call: ```json { "UseWorkhorseClient":true, "ObjectStorage":{ "Provider":"AzureRM", "GoCloudConfig":{ "URL":"azblob://test-bucket" } } } ``` The `azblob` scheme is managed by the Go Cloud `URLMux` (https://godoc.org/gocloud.dev/blob#URLMux). Converting our existing S3 client with Go Cloud should be done later (https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/275). This changes requires https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38882 to work. Omnibus configuration changes are in https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4505. Part of https://gitlab.com/gitlab-org/gitlab/-/issues/25877
Showing
Please register or sign in to comment