• Stan Hu's avatar
    Consolidate object storage config in one place · 95870df2
    Stan Hu authored
    Currently each object type has its own section in gitlab.yml. For
    example:
    
    ```
      artifacts:
        path: tmp/tests/artifacts
        enabled: true
        object_store:
          enabled: false
          remote_directory: artifacts # The bucket name
          background_upload: false
          connection:
            provider: AWS
            aws_access_key_id: AWS_ACCESS_KEY_ID
            aws_secret_access_key: AWS_SECRET_ACCESS_KEY
            region: us-east-1
    ```
    
    External diffs, LFS, uploads, packages, etc. all have similar
    independent configuration object storage sections. While this redundancy
    makes it possible to configure each bucket with different providers or
    credentials, this causes a configuration explosion that makes GitLab
    hard to manage.
    
    This change preserves the legacy format but adds a new `gitlab.yml`
    section that enforces a single, common object storage provider for all
    object storage types. This will make it possible for the S3 client in
    Workhorse to operate with one credential and simplify the configuration
    for the end user. An example config:
    
    ```
      object_store:
        enabled: true
        connection:
          provider: AWS
          aws_access_key_id: AWS_ACCESS_KEY_ID
          aws_secret_access_key: AWS_SECRET_ACCESS_KEY
          region: us-east-1
        proxy_download: true
        objects:
          artifacts:
            bucket: artifacts
            proxy_download: false
          external_diffs:
            bucket: external-diffs
          lfs:
            bucket: lfs-objects
          uploads:
            bucket: uploads
          packages:
            bucket: packages
          dependency_proxy:
            bucket: dependency_proxy
    ```
    
    Note that:
    
    1. The consolidated config only gets used if `object_store` is NOT
       defined within the types themselves.
    2. A bucket needs to be defined for each object type.
    3. Only bucket, enabled, and proxy_download can be overridden from the
    common configuration.
    
    Consolidating support for a single bucket for all types is a larger and
    more involved change.
    
    First step of https://gitlab.com/gitlab-org/gitlab/-/issues/23345
    95870df2
gitlab.yml.example 56.1 KB