Commit 4c768b1b authored by Michael Kozono's avatar Michael Kozono

Merge branch '270449-geo-uploads-to-ssf-no-verification' into 'master'

Geo: Migrate Uploads replication to SSF (no verification)

See merge request gitlab-org/gitlab!68278
parents 0b219734 c6597b43
......@@ -268,6 +268,10 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `destroyed_job_artifacts_count_total` | Counter | 13.6 | Number of destroyed expired job artifacts | |
| `destroyed_pipeline_artifacts_count_total` | Counter | 13.8 | Number of destroyed expired pipeline artifacts | |
| `gitlab_optimistic_locking_retries` | Histogram | 13.10 | Number of retry attempts to execute optimistic retry lock | |
| `geo_uploads` | Gauge | 14.1 | Number of uploads on primary | `url` |
| `geo_uploads_synced` | Gauge | 14.1 | Number of uploads synced on secondary | `url` |
| `geo_uploads_failed` | Gauge | 14.1 | Number of syncable uploads failed to sync on secondary | `url` |
| `geo_uploads_registry` | Gauge | 14.1 | Number of uploads in the registry | `url` |
## Database load balancing metrics **(PREMIUM SELF)**
......
......@@ -453,6 +453,11 @@ Example response:
"pipeline_artifacts_verification_failed_count": null,
"pipeline_artifacts_synced_in_percentage": "0.00%",
"pipeline_artifacts_verified_in_percentage": "0.00%",
"uploads_count": 5,
"uploads_synced_count": null,
"uploads_failed_count": 0,
"uploads_registry_count": null,
"uploads_synced_in_percentage": "0.00%",
},
{
"geo_node_id": 2,
......@@ -595,6 +600,11 @@ Example response:
"pipeline_artifacts_verification_failed_count": 0,
"pipeline_artifacts_synced_in_percentage": "100.00%",
"pipeline_artifacts_verified_in_percentage": "100.00%",
"uploads_count": 5,
"uploads_synced_count": null,
"uploads_failed_count": 0,
"uploads_registry_count": null,
"uploads_synced_in_percentage": "0.00%",
}
]
```
......@@ -734,6 +744,11 @@ Example response:
"pipeline_artifacts_verification_failed_count": 0,
"pipeline_artifacts_synced_in_percentage": "100.00%",
"pipeline_artifacts_verified_in_percentage": "100.00%",
"uploads_count": 5,
"uploads_synced_count": null,
"uploads_failed_count": 0,
"uploads_registry_count": null,
"uploads_synced_in_percentage": "0.00%",
}
```
......
......@@ -7443,6 +7443,29 @@ The edge type for [`TreeEntry`](#treeentry).
| <a id="treeentryedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. |
| <a id="treeentryedgenode"></a>`node` | [`TreeEntry`](#treeentry) | The item at the end of the edge. |
#### `UploadRegistryConnection`
The connection type for [`UploadRegistry`](#uploadregistry).
##### Fields
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="uploadregistryconnectionedges"></a>`edges` | [`[UploadRegistryEdge]`](#uploadregistryedge) | A list of edges. |
| <a id="uploadregistryconnectionnodes"></a>`nodes` | [`[UploadRegistry]`](#uploadregistry) | A list of nodes. |
| <a id="uploadregistryconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. |
#### `UploadRegistryEdge`
The edge type for [`UploadRegistry`](#uploadregistry).
##### Fields
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="uploadregistryedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. |
| <a id="uploadregistryedgenode"></a>`node` | [`UploadRegistry`](#uploadregistry) | The item at the end of the edge. |
#### `UsageTrendsMeasurementConnection`
The connection type for [`UsageTrendsMeasurement`](#usagetrendsmeasurement).
......@@ -9951,6 +9974,22 @@ four standard [pagination arguments](#connection-pagination-arguments):
| ---- | ---- | ----------- |
| <a id="geonodeterraformstateversionregistriesids"></a>`ids` | [`[ID!]`](#id) | Filters registries by their ID. |
##### `GeoNode.uploadRegistries`
Find Upload registries on this Geo node Available only when feature flag `geo_upload_replication` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice.
Returns [`UploadRegistryConnection`](#uploadregistryconnection).
This field returns a [connection](#connections). It accepts the
four standard [pagination arguments](#connection-pagination-arguments):
`before: String`, `after: String`, `first: Int`, `last: Int`.
###### Arguments
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="geonodeuploadregistriesids"></a>`ids` | [`[ID!]`](#id) | Filters registries by their ID. |
### `GrafanaIntegration`
#### Fields
......@@ -14347,6 +14386,23 @@ Represents a directory.
| <a id="treeentrywebpath"></a>`webPath` | [`String`](#string) | Web path for the tree entry (directory). |
| <a id="treeentryweburl"></a>`webUrl` | [`String`](#string) | Web URL for the tree entry (directory). |
### `UploadRegistry`
Represents the Geo replication and verification state of an upload.
#### Fields
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="uploadregistrycreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp when the UploadRegistry was created. |
| <a id="uploadregistryfileid"></a>`fileId` | [`ID!`](#id) | ID of the Upload. |
| <a id="uploadregistryid"></a>`id` | [`ID!`](#id) | ID of the UploadRegistry. |
| <a id="uploadregistrylastsyncfailure"></a>`lastSyncFailure` | [`String`](#string) | Error message during sync of the UploadRegistry. |
| <a id="uploadregistrylastsyncedat"></a>`lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the UploadRegistry. |
| <a id="uploadregistryretryat"></a>`retryAt` | [`Time`](#time) | Timestamp after which the UploadRegistry should be resynced. |
| <a id="uploadregistryretrycount"></a>`retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the UploadRegistry. |
| <a id="uploadregistrystate"></a>`state` | [`RegistryState`](#registrystate) | Sync state of the UploadRegistry. |
### `UsageTrendsMeasurement`
Represents a recorded measurement (object count) for the Admins.
......
# frozen_string_literal: true
module Geo
class AttachmentRegistryFinder < FileRegistryFinder
class AttachmentLegacyRegistryFinder < FileRegistryFinder
def registry_class
Geo::UploadRegistry
end
......
# frozen_string_literal: true
module Geo
class UploadRegistryFinder
include FrameworkRegistryFinder
end
end
# frozen_string_literal: true
module Resolvers
module Geo
class UploadRegistriesResolver < BaseResolver
type ::Types::Geo::GeoNodeType.connection_type, null: true
include RegistriesResolver
end
end
end
......@@ -54,6 +54,11 @@ module Types
null: true,
resolver: ::Resolvers::Geo::PagesDeploymentRegistriesResolver,
description: 'Find Pages Deployment registries on this Geo node'
field :upload_registries, ::Types::Geo::UploadRegistryType.connection_type,
null: true,
resolver: ::Resolvers::Geo::UploadRegistriesResolver,
description: 'Find Upload registries on this Geo node',
feature_flag: :geo_upload_replication
end
end
end
# frozen_string_literal: true
module Types
module Geo
# rubocop:disable Graphql/AuthorizeTypes because it is included
class UploadRegistryType < BaseObject
include ::Types::Geo::RegistryType
graphql_name 'UploadRegistry'
description 'Represents the Geo replication and verification state of an upload.'
field :file_id, GraphQL::Types::ID, null: false, description: 'ID of the Upload.'
end
end
end
......@@ -197,15 +197,6 @@ module EE
name: 'wiki',
name_plural: 'wikis'
},
{
data_type: 'blob',
data_type_title: _('File'),
title: _('Upload'),
title_plural: _('Uploads'),
name: 'attachment',
name_plural: 'attachments',
secondary_view: true
},
{
data_type: 'blob',
data_type_title: _('File'),
......@@ -233,6 +224,18 @@ module EE
}
]
if ::Geo::UploadReplicator.disabled?
replicable_types.insert(2, {
data_type: 'blob',
data_type_title: _('File'),
title: _('Upload'),
title_plural: _('Uploads'),
name: 'attachment',
name_plural: 'attachments',
secondary_view: true
})
end
# Adds all the SSF Data Types automatically
enabled_replicator_classes.each do |replicator_class|
replicable_types.push(
......
......@@ -55,7 +55,7 @@ module Geo
raise NotImplementedError
end
# Return the absolute path to locally stored package file
# Return the absolute path to locally stored file
#
# @return [String] File path
def blob_path
......@@ -76,7 +76,7 @@ module Geo
def calculate_checksum
raise 'File is not checksummable' unless checksummable?
model.hexdigest(carrierwave_uploader.path)
model.hexdigest(blob_path)
end
# Returns whether the file exists on disk or in remote storage
......
......@@ -7,7 +7,7 @@ module EE
prepended do
include ::Gitlab::Geo::ReplicableModel
with_replicator Geo::PagesDeploymentReplicator
with_replicator ::Geo::PagesDeploymentReplicator
end
class_methods do
......
......@@ -10,6 +10,9 @@ module EE
prepended do
include ::Gitlab::SQL::Pattern
include ::Gitlab::Geo::ReplicableModel
with_replicator ::Geo::UploadReplicator
after_destroy :log_geo_deleted_event
......
......@@ -2,6 +2,9 @@
class Geo::UploadRegistry < Geo::BaseRegistry
include Geo::Syncable
include ::Geo::ReplicableRegistry
extend ::Gitlab::Utils::Override
MODEL_CLASS = ::Upload
MODEL_FOREIGN_KEY = :file_id
......@@ -51,7 +54,7 @@ class Geo::UploadRegistry < Geo::BaseRegistry
# If false, RegistryConsistencyService will frequently check the end of the
# table to quickly handle new replicables.
def self.has_create_events?
false
::Geo::UploadReplicator.enabled?
end
def self.insert_for_model_ids(attrs)
......@@ -109,4 +112,43 @@ class Geo::UploadRegistry < Geo::BaseRegistry
:failed
end
# TODO Remove this when enabling geo_upload_registry by default
# https://gitlab.com/gitlab-org/gitlab/-/issues/340617
override :registry_consistency_worker_enabled?
def self.registry_consistency_worker_enabled?
true
end
def self.failed
if ::Geo::UploadReplicator.enabled?
with_state(:failed)
else
where(success: false).where.not(retry_count: nil)
end
end
def self.never_attempted_sync
if ::Geo::UploadReplicator.enabled?
pending.where(last_synced_at: nil)
else
where(success: false, retry_count: nil)
end
end
def self.retry_due
if ::Geo::UploadReplicator.enabled?
where(arel_table[:retry_at].eq(nil).or(arel_table[:retry_at].lt(Time.current)))
else
where('retry_at is NULL OR retry_at < ?', Time.current)
end
end
def self.synced
if ::Geo::UploadReplicator.enabled?
with_state(:synced).or(where(success: true))
else
where(success: true)
end
end
end
......@@ -595,7 +595,7 @@ class GeoNodeStatus < ApplicationRecord
end
def attachments_finder
@attachments_finder ||= Geo::AttachmentRegistryFinder.new
@attachments_finder ||= Geo::AttachmentLegacyRegistryFinder.new
end
def job_artifacts_finder
......
# frozen_string_literal: true
module Geo
class UploadReplicator < Gitlab::Geo::Replicator
include ::Geo::BlobReplicatorStrategy
extend ::Gitlab::Utils::Override
def self.model
::Upload
end
def self.replication_enabled_by_default?
false
end
def carrierwave_uploader
model_record.retrieve_uploader
end
# TODO: This method can be removed as part of
# https://gitlab.com/gitlab-org/gitlab/-/issues/340617
override :registry
def registry
super.tap do |record|
# We don't really need this value for SSF, it's only needed to make
# new registry records valid for legacy code in case of disabling the feature.
record.file_type ||= model_record.uploader.delete_suffix("Uploader").underscore
end
end
end
end
......@@ -10,10 +10,11 @@
= link_to admin_geo_projects_path, title: _('Projects') do
%span
= _('Projects')
= nav_link(path: 'admin/geo/uploads#index', html_options: { class: 'gl-pr-2' }) do
= link_to admin_geo_uploads_path, title: _('Uploads') do
%span
= _('Uploads')
- if ::Geo::UploadReplicator.disabled?
= nav_link(path: 'admin/geo/uploads#index', html_options: { class: 'gl-pr-2' }) do
= link_to admin_geo_uploads_path, title: _('Uploads') do
%span
= _('Uploads')
= nav_link(path: 'admin/geo/designs#index', html_options: { class: 'gl-pr-2' }) do
= link_to admin_geo_designs_path, title: _('Designs') do
%span
......
......@@ -71,10 +71,13 @@ module Geo
end
def job_finders
[
Geo::FileDownloadDispatchWorker::AttachmentJobFinder.new(scheduled_file_ids(Gitlab::Geo::Replication::USER_UPLOADS_OBJECT_TYPES)),
Geo::FileDownloadDispatchWorker::JobArtifactJobFinder.new(scheduled_file_ids(:job_artifact))
]
job_finders = [Geo::FileDownloadDispatchWorker::JobArtifactJobFinder.new(scheduled_file_ids(:job_artifact))]
if ::Geo::UploadReplicator.disabled?
job_finders << Geo::FileDownloadDispatchWorker::AttachmentJobFinder.new(scheduled_file_ids(Gitlab::Geo::Replication::USER_UPLOADS_OBJECT_TYPES))
end
job_finders
end
def scheduled_file_ids(file_types)
......
......@@ -6,7 +6,7 @@ module Geo
EXCEPT_RESOURCE_IDS_KEY = :except_ids
def registry_finder
@registry_finder ||= Geo::AttachmentRegistryFinder.new
@registry_finder ||= Geo::AttachmentLegacyRegistryFinder.new
end
private
......
---
name: geo_upload_replication
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68278
rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/340617
milestone: '14.4'
type: development
group: group::geo
default_enabled: false
......@@ -47,7 +47,6 @@ namespace :admin do
# Old Routes Replaced in 13.0
get '/projects', to: redirect(path: 'admin/geo/replication/projects')
get '/uploads', to: redirect(path: 'admin/geo/replication/uploads')
get '/designs', to: redirect(path: 'admin/geo/replication/designs')
resources :nodes, only: [:index, :create, :new, :edit, :update]
......@@ -70,7 +69,7 @@ namespace :admin do
resources :designs, only: [:index]
resources :uploads, only: [:index, :destroy]
resources :uploads, only: [:index, :destroy], path: 'legacy-uploads'
get '/:replicable_name_plural', to: 'replicables#index', as: 'replicables'
end
......
# frozen_string_literal: true
class PrepareFileRegistryForSsf < ActiveRecord::Migration[6.1]
def change
change_column_default :file_registry, :retry_count, from: nil, to: 0
add_column :file_registry, :state, :integer, null: false, limit: 2, default: 0
add_column :file_registry, :last_synced_at, :datetime_with_timezone
add_column :file_registry, :last_sync_failure, :string, limit: 255 # rubocop:disable Migration/PreventStrings see https://gitlab.com/gitlab-org/gitlab/-/issues/323806
end
end
# frozen_string_literal: true
class FixStateColumnInFileRegistry < ActiveRecord::Migration[6.1]
include Gitlab::Database::MigrationHelpers
disable_ddl_transaction!
# The following cop is disabled because of https://gitlab.com/gitlab-org/gitlab/issues/33470
# rubocop:disable Migration/UpdateColumnInBatches
def up
update_column_in_batches(:file_registry, :state, 2) do |table, query|
query.where(table[:success].eq(true)) # rubocop:disable CodeReuse/ActiveRecord
end
end
# rubocop:enable Migration/UpdateColumnInBatches
def down
# no-op
end
end
......@@ -54,9 +54,12 @@ ActiveRecord::Schema.define(version: 2021_08_20_152707) do
t.string "sha256"
t.datetime "created_at", null: false
t.boolean "success", default: false, null: false
t.integer "retry_count"
t.integer "retry_count", default: 0
t.datetime "retry_at"
t.boolean "missing_on_primary", default: false, null: false
t.integer "state", limit: 2, default: 0, null: false
t.datetime_with_timezone "last_synced_at"
t.string "last_sync_failure", limit: 255
t.index ["file_type", "file_id"], name: "index_file_registry_on_file_type_and_file_id", unique: true
t.index ["file_type"], name: "index_file_registry_on_file_type"
t.index ["retry_at"], name: "index_file_registry_on_retry_at"
......
......@@ -35,7 +35,7 @@ module EE
expose :db_replication_lag_seconds
expose :attachments_replication_enabled
expose :attachments_replication_enabled, if: -> (*) { ::Geo::UploadReplicator.disabled? }
expose :job_artifacts_replication_enabled
expose :container_repositories_replication_enabled
expose :design_repositories_replication_enabled
......
......@@ -27,7 +27,8 @@ module Gitlab
::Geo::SnippetRepositoryReplicator,
::Geo::GroupWikiRepositoryReplicator,
::Geo::PipelineArtifactReplicator,
::Geo::PagesDeploymentReplicator
::Geo::PagesDeploymentReplicator,
::Geo::UploadReplicator
].freeze
def self.current_node
......
......@@ -253,6 +253,8 @@ module Gitlab
end
def print_attachments_status
return if ::Geo::UploadReplicator.enabled?
print 'Attachments: '.rjust(GEO_STATUS_COLUMN_WIDTH)
show_failed_value(current_node_status.attachments_failed_count)
print "#{current_node_status.attachments_synced_count}/#{current_node_status.attachments_count} "
......
......@@ -159,6 +159,10 @@ module Gitlab
default_enabled: replication_enabled_by_default?)
end
def self.disabled?
!enabled?
end
# Replication is set behind a feature flag, which is enabled by default.
# If you want it disabled by default, override this method.
def self.replication_enabled_by_default?
......
......@@ -7,15 +7,17 @@ RSpec.describe Admin::Geo::UploadsController, :geo do
let_it_be(:admin) { create(:admin) }
let_it_be(:secondary) { create(:geo_node) }
let_it_be(:synced_registry) { create(:geo_upload_registry, :with_file, :attachment, success: true) }
let_it_be(:failed_registry) { create(:geo_upload_registry, :failed) }
let_it_be(:never_registry) { create(:geo_upload_registry, :failed, retry_count: nil) }
let_it_be(:synced_registry) { create(:geo_upload_legacy_registry, :with_file, :attachment, success: true) }
let_it_be(:failed_registry) { create(:geo_upload_legacy_registry, :failed) }
let_it_be(:never_registry) { create(:geo_upload_legacy_registry, :failed, retry_count: nil) }
def css_id(registry)
"#upload-#{registry.id}-header"
end
before do
stub_feature_flags(geo_upload_replication: false)
sign_in(admin)
end
......@@ -94,7 +96,7 @@ RSpec.describe Admin::Geo::UploadsController, :geo do
subject { delete :destroy, params: { id: registry } }
it_behaves_like 'license required' do
let(:registry) { create(:geo_upload_registry) }
let(:registry) { create(:geo_upload_legacy_registry) }
end
context 'with a valid license' do
......@@ -103,7 +105,7 @@ RSpec.describe Admin::Geo::UploadsController, :geo do
end
context 'with an orphaned registry' do
let(:registry) { create(:geo_upload_registry, success: true) }
let(:registry) { create(:geo_upload_legacy_registry, success: true) }
it 'removes the registry' do
registry.update_column(:file_id, -1)
......@@ -115,7 +117,7 @@ RSpec.describe Admin::Geo::UploadsController, :geo do
end
context 'with a regular registry' do
let(:registry) { create(:geo_upload_registry, :avatar, :with_file, success: true) }
let(:registry) { create(:geo_upload_legacy_registry, :avatar, :with_file, success: true) }
it 'does not delete the registry and gives an error' do
expect(subject).to redirect_to(admin_geo_uploads_path)
......
# frozen_string_literal: true
FactoryBot.define do
factory :geo_upload_registry, class: 'Geo::UploadRegistry' do
factory :geo_upload_legacy_registry, class: 'Geo::UploadRegistry' do
sequence(:file_id)
file_type { :file }
success { true }
......@@ -40,3 +40,30 @@ FactoryBot.define do
end
end
end
FactoryBot.define do
factory :geo_upload_registry, class: 'Geo::UploadRegistry' do
association(:upload, :with_file)
sequence(:file_id)
file_type { :file }
state { Geo::UploadRegistry.state_value(:pending) }
trait :synced do
state { Geo::UploadRegistry.state_value(:synced) }
last_synced_at { 5.days.ago }
end
trait :failed do
state { Geo::UploadRegistry.state_value(:failed) }
last_synced_at { 1.day.ago }
retry_count { 2 }
last_sync_failure { 'Random error' }
end
trait :started do
state { Geo::UploadRegistry.state_value(:started) }
last_synced_at { 1.day.ago }
retry_count { 0 }
end
end
end
......@@ -33,7 +33,11 @@ RSpec.describe 'admin Geo Replication Nav', :js, :geo do
end
end
describe 'visit admin/geo/replication/uploads' do
describe 'visit admin/geo/replication/legacy-uploads' do
before do
stub_feature_flags(geo_upload_replication: false)
end
it_behaves_like 'active sidebar link', 'Uploads' do
let(:path) { admin_geo_uploads_path }
end
......
......@@ -4,10 +4,12 @@ require 'spec_helper'
RSpec.describe 'admin Geo Uploads', :js, :geo do
let!(:geo_node) { create(:geo_node) }
let!(:synced_registry) { create(:geo_upload_registry, :with_file, :attachment, success: true) }
let!(:synced_registry) { create(:geo_upload_legacy_registry, :with_file, :attachment, success: true) }
before do
allow(Gitlab::Geo).to receive(:license_allows?).and_return(true)
stub_feature_flags(geo_upload_replication: false)
admin = create(:admin)
sign_in(admin)
gitlab_enable_admin_mode_sign_in(admin)
......
......@@ -2,7 +2,11 @@
require 'spec_helper'
RSpec.describe Geo::AttachmentRegistryFinder, :geo do
RSpec.describe Geo::AttachmentLegacyRegistryFinder, :geo do
before do
stub_feature_flags(geo_upload_replication: false )
end
it_behaves_like 'a file registry finder' do
let_it_be(:project) { create(:project) }
......@@ -16,13 +20,13 @@ RSpec.describe Geo::AttachmentRegistryFinder, :geo do
let_it_be(:replicable_8) { create(:upload, :object_storage, model: project) }
let_it_be(:replicable_9) { create(:upload, :object_storage, model: project) }
let_it_be(:registry_1) { create(:geo_upload_registry, :attachment, :failed, file_id: replicable_1.id) }
let_it_be(:registry_2) { create(:geo_upload_registry, :attachment, file_id: replicable_2.id, missing_on_primary: true) }
let_it_be(:registry_3) { create(:geo_upload_registry, :attachment, :never_synced, file_id: replicable_3.id) }
let_it_be(:registry_4) { create(:geo_upload_registry, :attachment, :failed, file_id: replicable_4.id) }
let_it_be(:registry_5) { create(:geo_upload_registry, :attachment, file_id: replicable_5.id, missing_on_primary: true, retry_at: 1.day.ago) }
let_it_be(:registry_6) { create(:geo_upload_registry, :attachment, :failed, file_id: replicable_6.id) }
let_it_be(:registry_7) { create(:geo_upload_registry, :attachment, :failed, file_id: replicable_7.id, missing_on_primary: true) }
let_it_be(:registry_8) { create(:geo_upload_registry, :attachment, :never_synced, file_id: replicable_8.id) }
let_it_be(:registry_1) { create(:geo_upload_legacy_registry, :attachment, :failed, file_id: replicable_1.id) }
let_it_be(:registry_2) { create(:geo_upload_legacy_registry, :attachment, file_id: replicable_2.id, missing_on_primary: true) }
let_it_be(:registry_3) { create(:geo_upload_legacy_registry, :attachment, :never_synced, file_id: replicable_3.id) }
let_it_be(:registry_4) { create(:geo_upload_legacy_registry, :attachment, :failed, file_id: replicable_4.id) }
let_it_be(:registry_5) { create(:geo_upload_legacy_registry, :attachment, file_id: replicable_5.id, missing_on_primary: true, retry_at: 1.day.ago) }
let_it_be(:registry_6) { create(:geo_upload_legacy_registry, :attachment, :failed, file_id: replicable_6.id) }
let_it_be(:registry_7) { create(:geo_upload_legacy_registry, :attachment, :failed, file_id: replicable_7.id, missing_on_primary: true) }
let_it_be(:registry_8) { create(:geo_upload_legacy_registry, :attachment, :never_synced, file_id: replicable_8.id) }
end
end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::UploadRegistryFinder do
it_behaves_like 'a framework registry finder', :geo_upload_registry
end
......@@ -157,6 +157,18 @@
"replication_slots_used_count",
"replication_slots_used_in_percentage",
"replication_slots_max_retained_wal_bytes",
"uploads_count",
"uploads_synced_count",
"uploads_failed_count",
"uploads_registry_count",
"uploads_synced_in_percentage",
"uploads_checksum_total_count",
"uploads_checksummed_count",
"uploads_checksum_failed_count",
"uploads_verification_failed_count",
"uploads_verification_total_count",
"uploads_verified_count",
"uploads_verified_in_percentage",
"git_fetch_event_count_weekly",
"git_push_event_count_weekly",
"last_event_id",
......@@ -321,6 +333,18 @@
"repositories_verification_total_count": { "type": ["integer", "null"] },
"repositories_verified_in_percentage": { "type": "string" },
"repositories_checksum_mismatch_count": { "type": ["integer", "null"] },
"uploads_count": { "type": ["integer", "null"] },
"uploads_synced_count": { "type": ["integer", "null"] },
"uploads_failed_count": { "type": ["integer", "null"] },
"uploads_registry_count": { "type": ["integer", "null"] },
"uploads_synced_in_percentage": { "type": "string" },
"uploads_checksummed_count": { "type": ["integer", "null"] },
"uploads_checksum_failed_count": { "type": ["integer", "null"] },
"uploads_checksum_total_count": { "type": ["integer", "null"] },
"uploads_verification_failed_count": { "type": ["integer", "null"] },
"uploads_verification_total_count": { "type": ["integer", "null"] },
"uploads_verified_count": { "type": ["integer", "null"] },
"uploads_verified_in_percentage": { "type": "string" },
"wikis_verified_count": { "type": ["integer", "null"] },
"wikis_verification_failed_count": { "type": ["integer", "null"] },
"wikis_verification_total_count": { "type": ["integer", "null"] },
......
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Resolvers::Geo::UploadRegistriesResolver do
it_behaves_like 'a Geo registries resolver', :geo_upload_registry
end
......@@ -14,9 +14,8 @@ RSpec.describe GitlabSchema.types['GeoNode'] do
minimum_reverification_interval merge_request_diff_registries
package_file_registries snippet_repository_registries
terraform_state_version_registries group_wiki_repository_registries
lfs_object_registries
pipeline_artifact_registries
pages_deployment_registries
pages_deployment_registries lfs_object_registries pipeline_artifact_registries
upload_registries
]
expect(described_class).to have_graphql_fields(*expected_fields)
......
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe GitlabSchema.types['UploadRegistry'] do
it_behaves_like 'a Geo registry type'
it 'has the expected fields (other than those included in RegistryType)' do
expected_fields = %i[file_id]
expect(described_class).to have_graphql_fields(*expected_fields).at_least
end
end
......@@ -29,7 +29,7 @@ RSpec.describe EE::GeoHelper do
repositories
wikis
lfs_objects
attachments
uploads
job_artifacts
container_repositories
design_repositories
......
......@@ -23,7 +23,7 @@ RSpec.describe Gitlab::Geo::GeoNodeStatusCheck do
/Verified Repositories: /,
/Wikis: /,
/Verified Wikis: /,
/Attachments: /,
/Uploads: /,
/CI job artifacts: /,
/Container repositories: /,
/Design repositories: /
......
......@@ -27,7 +27,7 @@ RSpec.describe Gitlab::Geo::LogCursor::Events::UploadDeletedEvent, :clean_gitlab
end
it 'removes the tracking database entry if exist' do
create(:geo_upload_registry, :avatar, file_id: upload.id)
create(:geo_upload_legacy_registry, :avatar, file_id: upload.id)
expect { subject.process }.to change(Geo::UploadRegistry, :count).by(-1)
end
......
# frozen_string_literal: true
require 'spec_helper'
require_migration!
RSpec.describe FixStateColumnInFileRegistry do
let(:registry) { table(:file_registry) }
it 'correctly sets registry state value' do
reg0 = registry.create!(file_id: 1, state: 0, success: false, file_type: 'placeholder')
reg1 = registry.create!(file_id: 2, state: 0, success: true, file_type: 'placeholder')
reg2 = registry.create!(file_id: 3, state: 1, success: false, file_type: 'placeholder')
reg3 = registry.create!(file_id: 4, state: 2, success: true, file_type: 'placeholder')
reg4 = registry.create!(file_id: 5, state: 3, success: false, file_type: 'placeholder')
migrate!
expect(registry.where(state: 0)).to contain_exactly(reg0)
expect(registry.where(state: 2)).to contain_exactly(reg1, reg3)
expect(registry.where(state: 1)).to contain_exactly(reg2)
expect(registry.where(state: 3)).to contain_exactly(reg4)
end
end
......@@ -5,13 +5,17 @@ require 'spec_helper'
RSpec.describe Geo::UploadRegistry, :geo do
include EE::GeoHelpers
before do
stub_feature_flags(geo_upload_replication: false)
end
it_behaves_like 'a BulkInsertSafe model', Geo::UploadRegistry do
let(:valid_items_for_bulk_insertion) { build_list(:geo_upload_registry, 10, created_at: Time.zone.now) }
let(:valid_items_for_bulk_insertion) { build_list(:geo_upload_legacy_registry, 10, created_at: Time.zone.now) }
let(:invalid_items_for_bulk_insertion) { [] } # class does not have any validations defined
end
it 'finds associated Upload record' do
registry = create(:geo_upload_registry, :attachment, :with_file)
registry = create(:geo_upload_legacy_registry, :attachment, :with_file)
expect(described_class.find(registry.id).upload).to be_an_instance_of(Upload)
end
......@@ -35,13 +39,13 @@ RSpec.describe Geo::UploadRegistry, :geo do
it 'returns untracked IDs as well as tracked IDs that are unused', :aggregate_failures do
max_id = Upload.maximum(:id)
create(:geo_upload_registry, :avatar, file_id: upload_1.id)
create(:geo_upload_registry, :file, file_id: upload_3.id)
create(:geo_upload_registry, :avatar, file_id: upload_5.id)
create(:geo_upload_registry, :personal_file, file_id: upload_6.id)
create(:geo_upload_registry, :avatar, file_id: upload_7.id)
unused_registry_1 = create(:geo_upload_registry, :attachment, file_id: max_id + 1)
unused_registry_2 = create(:geo_upload_registry, :personal_file, file_id: max_id + 2)
create(:geo_upload_legacy_registry, :avatar, file_id: upload_1.id)
create(:geo_upload_legacy_registry, :file, file_id: upload_3.id)
create(:geo_upload_legacy_registry, :avatar, file_id: upload_5.id)
create(:geo_upload_legacy_registry, :personal_file, file_id: upload_6.id)
create(:geo_upload_legacy_registry, :avatar, file_id: upload_7.id)
unused_registry_1 = create(:geo_upload_legacy_registry, :attachment, file_id: max_id + 1)
unused_registry_2 = create(:geo_upload_legacy_registry, :personal_file, file_id: max_id + 2)
range = 1..(max_id + 2)
untracked, unused = described_class.find_registry_differences(range)
......@@ -65,8 +69,8 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.failed' do
it 'returns registries in the failed state' do
failed = create(:geo_upload_registry, :failed)
create(:geo_upload_registry)
failed = create(:geo_upload_legacy_registry, :failed)
create(:geo_upload_legacy_registry)
expect(described_class.failed).to match_ids(failed)
end
......@@ -74,8 +78,8 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.synced' do
it 'returns registries in the synced state' do
create(:geo_upload_registry, :failed)
synced = create(:geo_upload_registry)
create(:geo_upload_legacy_registry, :failed)
synced = create(:geo_upload_legacy_registry)
expect(described_class.synced).to match_ids(synced)
end
......@@ -83,10 +87,10 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.retry_due' do
it 'returns registries in the synced state' do
failed = create(:geo_upload_registry, :failed)
synced = create(:geo_upload_registry)
retry_yesterday = create(:geo_upload_registry, retry_at: Date.yesterday)
create(:geo_upload_registry, retry_at: Date.tomorrow)
failed = create(:geo_upload_legacy_registry, :failed)
synced = create(:geo_upload_legacy_registry)
retry_yesterday = create(:geo_upload_legacy_registry, retry_at: Date.yesterday)
create(:geo_upload_legacy_registry, retry_at: Date.tomorrow)
expect(described_class.retry_due).to match_ids([failed, synced, retry_yesterday])
end
......@@ -94,9 +98,9 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.never_attempted_sync' do
it 'returns registries that are never synced' do
create(:geo_upload_registry, :failed)
create(:geo_upload_registry)
pending = create(:geo_upload_registry, retry_count: nil, success: false)
create(:geo_upload_legacy_registry, :failed)
create(:geo_upload_legacy_registry)
pending = create(:geo_upload_legacy_registry, retry_count: nil, success: false)
expect(described_class.never_attempted_sync).to match_ids([pending])
end
......@@ -125,7 +129,7 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.with_search' do
it 'searches registries on path' do
upload = create(:upload, path: 'uploads/-/system/project/avatar/my-awesome-avatar.png')
upload_registry = create(:geo_upload_registry, file_id: upload.id, file_type: :avatar)
upload_registry = create(:geo_upload_legacy_registry, file_id: upload.id, file_type: :avatar)
expect(described_class.with_search('awesome-avatar')).to match_ids(upload_registry)
end
......@@ -134,28 +138,28 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '#file' do
it 'returns the path of the upload of a registry' do
upload = create(:upload, :with_file)
registry = create(:geo_upload_registry, :file, file_id: upload.id)
registry = create(:geo_upload_legacy_registry, :file, file_id: upload.id)
expect(registry.file).to eq(upload.path)
end
it 'return "removed" message when the upload no longer exists' do
registry = create(:geo_upload_registry, :avatar)
registry = create(:geo_upload_legacy_registry, :avatar)
expect(registry.file).to match(/^Removed avatar with id/)
end
end
describe '#synchronization_state' do
let_it_be(:failed) { create(:geo_upload_registry, :failed) }
let_it_be(:synced) { create(:geo_upload_registry) }
let_it_be(:failed) { create(:geo_upload_legacy_registry, :failed) }
let_it_be(:synced) { create(:geo_upload_legacy_registry) }
it 'returns :synced for a successful synced registry' do
expect(synced.synchronization_state).to eq(:synced)
end
it 'returns :never for a successful registry never synced' do
never = build(:geo_upload_registry, success: false, retry_count: nil)
never = build(:geo_upload_legacy_registry, success: false, retry_count: nil)
expect(never.synchronization_state).to eq(:never)
end
......@@ -165,3 +169,13 @@ RSpec.describe Geo::UploadRegistry, :geo do
end
end
end
RSpec.describe Geo::UploadRegistry, :geo, type: :model do
let_it_be(:registry) { create(:geo_upload_registry) }
specify 'factory is valid' do
expect(registry).to be_valid
end
include_examples 'a Geo framework registry'
end
......@@ -149,39 +149,24 @@ RSpec.describe GeoNodeStatus, :geo do
create_list(:user, 3, avatar: fixture_file_upload('spec/fixtures/dk.png', 'image/png'))
uploads = Upload.pluck(:id)
create(:geo_upload_registry, :avatar, file_id: uploads[0])
create(:geo_upload_registry, :avatar, file_id: uploads[1])
create(:geo_upload_registry, :avatar, :failed, file_id: uploads[2])
create(:geo_upload_registry, :synced, file_id: uploads[0])
create(:geo_upload_registry, :synced, file_id: uploads[1])
create(:geo_upload_registry, :failed, file_id: uploads[2])
expect(subject.attachments_synced_count).to eq(2)
end
end
describe '#attachments_synced_missing_on_primary_count' do
it 'only counts successful syncs' do
create_list(:user, 3, avatar: fixture_file_upload('spec/fixtures/dk.png', 'image/png'))
uploads = Upload.pluck(:id)
create(:geo_upload_registry, :avatar, file_id: uploads[0], missing_on_primary: true)
create(:geo_upload_registry, :avatar, file_id: uploads[1])
create(:geo_upload_registry, :avatar, :failed, file_id: uploads[2])
expect(subject.attachments_synced_missing_on_primary_count).to eq(1)
end
end
describe '#attachments_failed_count' do
it 'counts failed avatars, attachment, personal snippets and files' do
# These two should be ignored
create(:geo_lfs_object_registry, :failed)
create(:geo_upload_registry, :with_file)
create(:geo_upload_registry)
create(:geo_upload_registry, :with_file, :failed, file_type: :personal_file)
create(:geo_upload_registry, :with_file, :failed, file_type: :attachment)
create(:geo_upload_registry, :avatar, :with_file, :failed)
create(:geo_upload_registry, :with_file, :failed)
create(:geo_upload_registry, :failed)
create(:geo_upload_registry, :failed)
expect(subject.attachments_failed_count).to eq(4)
expect(subject.attachments_failed_count).to eq(2)
end
end
......@@ -194,10 +179,10 @@ RSpec.describe GeoNodeStatus, :geo do
create_list(:user, 4, avatar: fixture_file_upload('spec/fixtures/dk.png', 'image/png'))
uploads = Upload.pluck(:id)
create(:geo_upload_registry, :avatar, file_id: uploads[0])
create(:geo_upload_registry, :avatar, file_id: uploads[1])
create(:geo_upload_registry, :avatar, :failed, file_id: uploads[2])
create(:geo_upload_registry, :avatar, :never_synced, file_id: uploads[3])
create(:geo_upload_registry, :synced, file_id: uploads[0])
create(:geo_upload_registry, :synced, file_id: uploads[1])
create(:geo_upload_registry, :failed, file_id: uploads[2])
create(:geo_upload_registry, :started, file_id: uploads[3])
expect(subject.attachments_synced_in_percentage).to be_within(0.0001).of(50)
end
......@@ -247,10 +232,7 @@ RSpec.describe GeoNodeStatus, :geo do
it 'counts failed job artifacts' do
# These should be ignored
create(:geo_upload_registry, :failed)
create(:geo_upload_registry, :avatar, :failed)
create(:geo_upload_registry, :attachment, :failed)
create(:geo_job_artifact_registry, :with_artifact, success: true)
create(:geo_job_artifact_registry, :with_artifact, :failed)
expect(subject.job_artifacts_failed_count).to eq(1)
......@@ -1130,6 +1112,7 @@ RSpec.describe GeoNodeStatus, :geo do
Geo::SnippetRepositoryReplicator | :snippet_repository | :geo_snippet_repository_registry
Geo::GroupWikiRepositoryReplicator | :group_wiki_repository | :geo_group_wiki_repository_registry
Geo::PagesDeploymentReplicator | :pages_deployment | :geo_pages_deployment_registry
Geo::UploadReplicator | :upload | :geo_upload_registry
end
with_them do
......@@ -1344,8 +1327,8 @@ RSpec.describe GeoNodeStatus, :geo do
stub_current_geo_node(primary)
end
it 'does not call AttachmentRegistryFinder#registry_count' do
expect_any_instance_of(Geo::AttachmentRegistryFinder).not_to receive(:registry_count)
it 'does not call AttachmentLegacyRegistryFinder#registry_count' do
expect_any_instance_of(Geo::AttachmentLegacyRegistryFinder).not_to receive(:registry_count)
subject
end
......@@ -1358,8 +1341,8 @@ RSpec.describe GeoNodeStatus, :geo do
end
context 'on the secondary' do
it 'calls AttachmentRegistryFinder#registry_count' do
expect_any_instance_of(Geo::AttachmentRegistryFinder).to receive(:registry_count).twice
it 'calls AttachmentLegacyRegistryFinder#registry_count' do
expect_any_instance_of(Geo::AttachmentLegacyRegistryFinder).to receive(:registry_count).twice
subject
end
......
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::UploadReplicator do
let(:model_record) { create(:upload, :with_file) }
include_examples 'a blob replicator'
end
......@@ -51,4 +51,11 @@ RSpec.describe 'Gets registries' do
registry_factory: :geo_pages_deployment_registry,
registry_foreign_key_field_name: 'pagesDeploymentId'
}
it_behaves_like 'gets registries for', {
field_name: 'uploadRegistries',
registry_class_name: 'UploadRegistry',
registry_factory: :geo_upload_registry,
registry_foreign_key_field_name: 'fileId'
}
end
......@@ -27,14 +27,14 @@ RSpec.describe 'EE-specific admin routing' do
end
describe Admin::Geo::UploadsController, 'routing' do
let!(:upload_registry) { create(:geo_upload_registry, :with_file, :attachment, success: true) }
let!(:upload_registry) { create(:geo_upload_legacy_registry, :with_file, :attachment, success: true) }
it 'routes / to #index' do
expect(get('/admin/geo/replication/uploads')).to route_to('admin/geo/uploads#index')
expect(get('/admin/geo/replication/legacy-uploads')).to route_to('admin/geo/uploads#index')
end
it 'routes delete /:id to #destroy' do
expect(delete("/admin/geo/replication/uploads/#{upload_registry.id}")).to route_to('admin/geo/uploads#destroy', id: upload_registry.to_param)
expect(delete("/admin/geo/replication/legacy-uploads/#{upload_registry.id}")).to route_to('admin/geo/uploads#destroy', id: upload_registry.to_param)
end
end
......
......@@ -11,6 +11,8 @@ RSpec.describe Geo::FileDownloadService do
before do
stub_current_geo_node(secondary)
stub_feature_flags(geo_upload_replication: false)
end
describe '#downloader' do
......@@ -58,7 +60,7 @@ RSpec.describe Geo::FileDownloadService do
context 'with uploads' do
let!(:registry_entry) do
create(:geo_upload_registry, :avatar, success: false, file_id: file.id, retry_count: 31)
create(:geo_upload_legacy_registry, :avatar, success: false, file_id: file.id, retry_count: 31)
end
let(:file) { create(:upload) }
......@@ -233,7 +235,7 @@ RSpec.describe Geo::FileDownloadService do
when 'job_artifact'
create(:geo_job_artifact_registry, success: false, artifact_id: file.id, retry_count: 3, retry_at: 1.hour.ago)
else
create(:geo_upload_registry, file_type.to_sym, success: false, file_id: file.id, retry_count: 3, retry_at: 1.hour.ago)
create(:geo_upload_legacy_registry, file_type.to_sym, success: false, file_id: file.id, retry_count: 3, retry_at: 1.hour.ago)
end
end
......
......@@ -223,7 +223,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
context 'with avatar' do
let!(:upload) { create(:user, :with_avatar).avatar.upload }
let!(:registry) { create(:geo_upload_registry, :avatar, file_id: upload.id) }
let!(:registry) { create(:geo_upload_legacy_registry, :avatar, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes'
......@@ -250,7 +250,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
context 'with attachment' do
let!(:upload) { create(:note, :with_attachment).attachment.upload }
let!(:registry) { create(:geo_upload_registry, :attachment, file_id: upload.id) }
let!(:registry) { create(:geo_upload_legacy_registry, :attachment, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes'
......@@ -284,7 +284,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
Upload.find_by(model: group, uploader: NamespaceFileUploader.name)
end
let!(:registry) { create(:geo_upload_registry, :namespace_file, file_id: upload.id) }
let!(:registry) { create(:geo_upload_legacy_registry, :namespace_file, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes'
......@@ -317,7 +317,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
Upload.find_by(model: snippet, uploader: PersonalFileUploader.name)
end
let!(:registry) { create(:geo_upload_registry, :personal_file, file_id: upload.id) }
let!(:registry) { create(:geo_upload_legacy_registry, :personal_file, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path }
context 'migrated to object storage' do
......@@ -348,7 +348,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
Upload.find_by(model: appearance, uploader: FaviconUploader.name)
end
let!(:registry) { create(:geo_upload_registry, :favicon, file_id: upload.id) }
let!(:registry) { create(:geo_upload_legacy_registry, :favicon, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes'
......@@ -406,5 +406,32 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
end
end
end
context 'with Uploads(after migrating to SSF)' do
let!(:upload) { create(:user, :with_avatar).avatar.upload }
let!(:registry) { create(:geo_upload_registry, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes'
context 'migrated to object storage' do
before do
stub_uploads_object_storage(AvatarUploader)
upload.update_column(:store, AvatarUploader::Store::REMOTE)
end
context 'with object storage enabled' do
it_behaves_like 'removes'
end
context 'with object storage disabled' do
before do
stub_uploads_object_storage(AvatarUploader, enabled: false)
end
it_behaves_like 'removes registry entry'
end
end
end
end
end
......@@ -11,7 +11,7 @@ RSpec.describe Geo::FilesExpireService, :geo do
describe '#execute' do
let(:file_uploader) { build(:file_uploader, project: project) }
let!(:upload) { Upload.find_by(path: file_uploader.upload_path) }
let!(:upload_registry) { create(:geo_upload_registry, file_id: upload.id) }
let!(:upload_registry) { create(:geo_upload_registry, :synced, file_id: upload.id) }
before do
project.update(path: "#{project.path}_renamed")
......@@ -32,7 +32,7 @@ RSpec.describe Geo::FilesExpireService, :geo do
end
it 'removes upload_registry associates with upload' do
expect(upload_registry.success).to be_truthy
expect(upload_registry.synced?).to be_truthy
subject.execute
......
......@@ -19,7 +19,8 @@ RSpec.describe Geo::RegistryConsistencyService, :geo, :use_clean_rails_memory_st
{
Geo::DesignRegistry => :project_with_design,
Geo::MergeRequestDiffRegistry => :external_merge_request_diff,
Geo::PackageFileRegistry => :package_file
Geo::PackageFileRegistry => :package_file,
Geo::UploadRegistry => :upload
}.fetch(registry_class, default_factory_name)
end
......
......@@ -7,13 +7,13 @@ module EE
override :reset_column_information
def reset_column_information(klass)
super
rescue Geo::TrackingBase::SecondaryNotConfigured
rescue ::Geo::TrackingBase::SecondaryNotConfigured
end
override :active_record_base
def active_record_base
if geo_migration?
Geo::TrackingBase
::Geo::TrackingBase
else
super
end
......
......@@ -25,6 +25,6 @@ RSpec.shared_examples 'allowlisted /admin/geo requests' do
it_behaves_like 'allowlisted request', :post, '/admin/geo/replication/projects/1/force_redownload'
it_behaves_like 'allowlisted request', :delete, '/admin/geo/replication/uploads/1'
it_behaves_like 'allowlisted request', :delete, '/admin/geo/replication/legacy-uploads/1'
end
end
......@@ -67,6 +67,8 @@ RSpec.shared_examples 'a blob replicator' do
describe '#handle_after_destroy' do
it 'creates a Geo::Event' do
model_record
expect do
replicator.handle_after_destroy
end.to change { ::Geo::Event.count }.by(1)
......@@ -181,7 +183,7 @@ RSpec.shared_examples 'a blob replicator' do
context 'when the file is locally stored' do
context 'when the file exists' do
it 'returns hexdigest of the file' do
expected = described_class.model.hexdigest(subject.carrierwave_uploader.path)
expected = described_class.model.hexdigest(subject.blob_path)
expect(subject.calculate_checksum).to eq(expected)
end
......@@ -198,7 +200,8 @@ RSpec.shared_examples 'a blob replicator' do
context 'when the file is remotely stored' do
it 'raises an error' do
allow(subject.carrierwave_uploader).to receive(:file_storage?).and_return(false)
carrierwave_uploader = double(file_storage?: false)
allow(subject).to receive(:carrierwave_uploader).and_return(carrierwave_uploader)
expect { subject.calculate_checksum }.to raise_error('File is not checksummable')
end
......
......@@ -330,7 +330,7 @@ RSpec.describe 'geo rake tasks', :geo, :silence_stdout do
/Verified Repositories: /,
/Wikis: /,
/Verified Wikis: /,
/Attachments: /,
/Uploads: /,
/CI job artifacts: /,
/Container repositories: /,
/Design repositories: /,
......
......@@ -17,6 +17,8 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
WebMock.stub_request(:get, /primary-geo-node/).to_return(status: 200, body: "", headers: {})
stub_feature_flags(geo_upload_replication: false)
end
it 'does not schedule anything when tracking database is not configured' do
......@@ -48,7 +50,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
let(:upload) { create(:upload) }
it 'performs Geo::FileDownloadWorker for unsynced attachments' do
create(:geo_upload_registry, :avatar, :never_synced, file_id: upload.id)
create(:geo_upload_legacy_registry, :avatar, :never_synced, file_id: upload.id)
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('avatar', upload.id)
......@@ -56,7 +58,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
it 'performs Geo::FileDownloadWorker for failed-sync attachments' do
create(:geo_upload_registry, :avatar, :failed, file_id: upload.id, bytes: 0)
create(:geo_upload_legacy_registry, :avatar, :failed, file_id: upload.id, bytes: 0)
expect(Geo::FileDownloadWorker).to receive(:perform_async)
.with('avatar', upload.id).once.and_return(spy)
......@@ -65,7 +67,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
it 'does not perform Geo::FileDownloadWorker for synced attachments' do
create(:geo_upload_registry, :avatar, file_id: upload.id, bytes: 1234)
create(:geo_upload_legacy_registry, :avatar, file_id: upload.id, bytes: 1234)
expect(Geo::FileDownloadWorker).not_to receive(:perform_async)
......@@ -73,7 +75,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
it 'does not perform Geo::FileDownloadWorker for synced attachments even with 0 bytes downloaded' do
create(:geo_upload_registry, :avatar, file_id: upload.id, bytes: 0)
create(:geo_upload_legacy_registry, :avatar, file_id: upload.id, bytes: 0)
expect(Geo::FileDownloadWorker).not_to receive(:perform_async)
......@@ -81,10 +83,10 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
context 'with a failed file' do
let(:failed_registry) { create(:geo_upload_registry, :avatar, :failed, file_id: non_existing_record_id) }
let(:failed_registry) { create(:geo_upload_legacy_registry, :avatar, :failed, file_id: non_existing_record_id) }
it 'does not stall backfill' do
unsynced_registry = create(:geo_upload_registry, :avatar, :with_file, :never_synced)
unsynced_registry = create(:geo_upload_legacy_registry, :avatar, :with_file, :never_synced)
stub_const('Geo::Scheduler::SchedulerWorker::DB_RETRIEVE_BATCH_SIZE', 1)
......@@ -101,7 +103,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
it 'does not retry failed files when retry_at is tomorrow' do
failed_registry = create(:geo_upload_registry, :avatar, :failed, file_id: non_existing_record_id, retry_at: Date.tomorrow)
failed_registry = create(:geo_upload_legacy_registry, :avatar, :failed, file_id: non_existing_record_id, retry_at: Date.tomorrow)
expect(Geo::FileDownloadWorker).not_to receive(:perform_async).with('avatar', failed_registry.file_id)
......@@ -109,7 +111,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
it 'retries failed files when retry_at is in the past' do
failed_registry = create(:geo_upload_registry, :avatar, :failed, file_id: non_existing_record_id, retry_at: Date.yesterday)
failed_registry = create(:geo_upload_legacy_registry, :avatar, :failed, file_id: non_existing_record_id, retry_at: Date.yesterday)
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('avatar', failed_registry.file_id)
......@@ -131,7 +133,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
it 'does not retry those files if there is no spare capacity' do
unsynced_registry = create(:geo_upload_registry, :avatar, :with_file, :never_synced)
unsynced_registry = create(:geo_upload_legacy_registry, :avatar, :with_file, :never_synced)
expect(subject).to receive(:db_retrieve_batch_size).and_return(1).twice
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('avatar', unsynced_registry.file_id)
......@@ -140,7 +142,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
it 'does not retry those files if they are already scheduled' do
unsynced_registry = create(:geo_upload_registry, :avatar, :with_file, :never_synced)
unsynced_registry = create(:geo_upload_legacy_registry, :avatar, :with_file, :never_synced)
scheduled_jobs = [{ type: 'avatar', id: synced_upload_with_file_missing_on_primary.id, job_id: 'foo' }]
expect(subject).to receive(:scheduled_jobs).and_return(scheduled_jobs).at_least(1)
......@@ -276,11 +278,11 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
result_object = double(:result, success: true, bytes_downloaded: 100, primary_missing_file: false)
allow_any_instance_of(::Gitlab::Geo::Replication::BaseTransfer).to receive(:download_from_primary).and_return(result_object)
create_list(:geo_upload_registry, 2, :avatar, :with_file, :never_synced)
create_list(:geo_upload_registry, 2, :attachment, :with_file, :never_synced)
create(:geo_upload_registry, :favicon, :with_file, :never_synced)
create(:geo_upload_registry, :import_export, :with_file, :never_synced)
create(:geo_upload_registry, :personal_file, :with_file, :never_synced)
create_list(:geo_upload_legacy_registry, 2, :avatar, :with_file, :never_synced)
create_list(:geo_upload_legacy_registry, 2, :attachment, :with_file, :never_synced)
create(:geo_upload_legacy_registry, :favicon, :with_file, :never_synced)
create(:geo_upload_legacy_registry, :import_export, :with_file, :never_synced)
create(:geo_upload_legacy_registry, :personal_file, :with_file, :never_synced)
create(:geo_job_artifact_registry, :with_artifact, :never_synced)
expect(Geo::FileDownloadWorker).to receive(:perform_async).exactly(8).times.and_call_original
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment