Skip to content

G
gitlab-ce
Commit 21a4999b authored by Michael Kozono's avatar Michael Kozono
Browse files
Options

Merge branch '330981-feature-flag-remove-geo_lfs_object_replication' into 'master' 

Resolve "[Feature flag] Remove geo_lfs_object_replication" [RUN ALL RSPEC] [RUN AS-IF-FOSS]

See merge request gitlab-org/gitlab!62039
parents 8d5fb3c7 3fbea827
Hide whitespace changes
Inline Side-by-side
Showing with 7 additions and 1520 deletions
doc/administration/geo/replication/datatypes.md
View file @ 21a4999b
......@@ -180,7 +180,7 @@ successfully, you must replicate their data using some other means.
|[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | |
|[Group wiki repository](../../../user/project/wiki/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. |
|[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. |
|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). |
|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Behind feature flag `geo_lfs_object_replication`, enabled by default. |
|[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | |
|[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | |
|[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. |
......@@ -205,33 +205,3 @@ successfully, you must replicate their data using some other means.
|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | |
|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. |
|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. |
#### LFS object replication using the self service framework
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276696) in GitLab 13.12.
> - [Deployed behind a feature flag](../../../user/feature_flags.md), enabled by default.
> - Not enabled on GitLab.com as Geo is not enabled.
> - Recommended for production use.
> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-lfs-object-replication-using-the-self-service-framework).
There can be [risks when disabling released features](../../../user/feature_flags.md#risks-when-disabling-released-features).
Refer to this feature's version history for more details.
##### Enable or disable LFS object replication using the self service framework
LFS object replication using the self service framework is under development but ready for production use. It is
deployed behind a feature flag that is **enabled by default**.
[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md)
can opt to disable it.
To enable it:
```ruby
Feature.enable(:geo_lfs_object_replication)
```
To disable it:
```ruby
Feature.disable(:geo_lfs_object_replication)
```
ee/app/finders/geo/lfs_object_legacy_registry_finder.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
module Geo
class LfsObjectLegacyRegistryFinder < FileRegistryFinder
def registry_class
Geo::LfsObjectRegistry
end
end
end
ee/app/helpers/ee/geo_helper.rb
View file @ 21a4999b
......@@ -208,17 +208,6 @@ module EE
}
]
if ::Feature.disabled?(:geo_lfs_object_replication, default_enabled: :yaml)
replicable_types.insert(2, {
data_type: 'blob',
data_type_title: _('File'),
title: _('LFS object'),
title_plural: _('LFS objects'),
name: 'lfs_object',
name_plural: 'lfs_objects'
})
end
# Adds all the SSF Data Types automatically
enabled_replicator_classes.each do |replicator_class|
replicable_types.push(
......
ee/app/models/ee/lfs_object.rb
View file @ 21a4999b
......@@ -15,8 +15,6 @@ module EE
with_replicator Geo::LfsObjectReplicator
after_destroy :log_geo_deleted_event
scope :project_id_in, ->(ids) { joins(:projects).merge(::Project.id_in(ids)) }
end
......@@ -39,7 +37,8 @@ module EE
end
def log_geo_deleted_event
::Geo::LfsObjectDeletedEventStore.new(self).create! if ::Feature.disabled?(:geo_lfs_object_replication, default_enabled: :yaml)
# Keep empty for now. Should be addressed in future
# by https://gitlab.com/gitlab-org/gitlab/-/issues/232917
end
end
end
ee/app/models/geo/event_log.rb
View file @ 21a4999b
......@@ -14,7 +14,6 @@ module Geo
Geo::ResetChecksumEvent
Geo::HashedStorageMigratedEvent
Geo::HashedStorageAttachmentsEvent
Geo::LfsObjectDeletedEvent
Geo::JobArtifactDeletedEvent
Geo::UploadDeletedEvent
Geo::ContainerRepositoryUpdatedEvent
......@@ -52,10 +51,6 @@ module Geo
class_name: 'Geo::HashedStorageAttachmentsEvent',
foreign_key: :hashed_storage_attachments_event_id
belongs_to :lfs_object_deleted_event,
class_name: 'Geo::LfsObjectDeletedEvent',
foreign_key: :lfs_object_deleted_event_id
belongs_to :job_artifact_deleted_event,
class_name: 'Geo::JobArtifactDeletedEvent',
foreign_key: :job_artifact_deleted_event_id
......@@ -104,7 +99,6 @@ module Geo
repositories_changed_event ||
hashed_storage_migrated_event ||
hashed_storage_attachments_event ||
lfs_object_deleted_event ||
job_artifact_deleted_event ||
upload_deleted_event ||
reset_checksum_event ||
......
ee/app/models/geo/lfs_object_deleted_event.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
module Geo
class LfsObjectDeletedEvent < ApplicationRecord
include Geo::Model
include Geo::Eventable
belongs_to :lfs_object
validates :lfs_object, :oid, :file_path, presence: true
end
end
ee/app/models/geo/lfs_object_registry.rb
View file @ 21a4999b
# frozen_string_literal: true
class Geo::LfsObjectRegistry < Geo::BaseRegistry
include ::Geo::Syncable
include ::Geo::ReplicableRegistry
include ::ShaAttribute
sha_attribute :sha256
MODEL_CLASS = ::LfsObject
MODEL_FOREIGN_KEY = :lfs_object_id
belongs_to :lfs_object, class_name: 'LfsObject'
def self.registry_consistency_worker_enabled?
if ::Feature.enabled?(:geo_lfs_object_replication, default_enabled: :yaml)
replicator_class.enabled?
else
true
end
end
def self.failed
if Feature.enabled?(:geo_lfs_object_replication, default_enabled: :yaml)
with_state(:failed)
else
where(success: false).where.not(retry_count: nil)
end
end
def self.never_attempted_sync
if Feature.enabled?(:geo_lfs_object_replication, default_enabled: :yaml)
pending.where(last_synced_at: nil)
else
where(success: false, retry_count: nil)
end
end
def self.retry_due
if Feature.enabled?(:geo_lfs_object_replication, default_enabled: :yaml)
where(arel_table[:retry_at].eq(nil).or(arel_table[:retry_at].lt(Time.current)))
else
where('retry_at is NULL OR retry_at < ?', Time.current)
end
end
def self.synced
if Feature.enabled?(:geo_lfs_object_replication, default_enabled: :yaml)
with_state(:synced).or(where(success: true))
else
where(success: true)
end
end
# If false, RegistryConsistencyService will frequently check the end of the
# table to quickly handle new replicables.
def self.has_create_events?
false
end
def self.delete_for_model_ids(lfs_object_ids)
lfs_object_ids.map do |lfs_object_id|
delete_worker_class.perform_async(:lfs, lfs_object_id)
end
end
def self.delete_worker_class
::Geo::FileRegistryRemovalWorker
end
end
ee/app/models/geo_node_status.rb
View file @ 21a4999b
......@@ -64,7 +64,6 @@ class GeoNodeStatus < ApplicationRecord
repositories_replication_enabled
repositories_synced_count
repositories_failed_count
lfs_objects_replication_enabled
attachments_replication_enabled
attachments_count
attachments_synced_count
......@@ -132,12 +131,6 @@ class GeoNodeStatus < ApplicationRecord
wikis_verified_count: 'Number of wikis verified on secondary',
wikis_verification_failed_count: 'Number of wikis failed to verify on secondary',
wikis_checksum_mismatch_count: 'Number of wikis that checksum mismatch on secondary',
lfs_objects_replication_enabled: 'Boolean denoting if replication is enabled for LFS Objects',
lfs_objects_count: 'Total number of syncable LFS objects available on primary',
lfs_objects_synced_count: 'Number of syncable LFS objects synced on secondary',
lfs_objects_failed_count: 'Number of syncable LFS objects failed to sync on secondary',
lfs_objects_registry_count: 'Number of LFS objects in the registry',
lfs_objects_synced_missing_on_primary_count: 'Number of LFS objects marked as synced due to the file missing on the primary',
job_artifacts_replication_enabled: 'Boolean denoting if replication is enabled for Job Artifacts',
job_artifacts_count: 'Total number of syncable job artifacts available on primary',
job_artifacts_synced_count: 'Number of syncable job artifacts synced on secondary',
......@@ -298,7 +291,6 @@ class GeoNodeStatus < ApplicationRecord
self.container_repositories_replication_enabled = Geo::ContainerRepositoryRegistry.replication_enabled?
self.design_repositories_replication_enabled = Geo::DesignRegistry.replication_enabled?
self.job_artifacts_replication_enabled = Geo::JobArtifactRegistry.replication_enabled?
self.lfs_objects_replication_enabled = Geo::LfsObjectRegistry.replication_enabled?
self.repositories_replication_enabled = Geo::ProjectRegistry.replication_enabled?
end
end
......@@ -446,7 +438,6 @@ class GeoNodeStatus < ApplicationRecord
self.repository_deleted_max_id = Geo::RepositoryDeletedEvent.maximum(:id)
self.repository_renamed_max_id = Geo::RepositoryRenamedEvent.maximum(:id)
self.repositories_changed_max_id = Geo::RepositoriesChangedEvent.maximum(:id)
self.lfs_object_deleted_max_id = Geo::LfsObjectDeletedEvent.maximum(:id)
self.job_artifact_deleted_max_id = Geo::JobArtifactDeletedEvent.maximum(:id)
self.hashed_storage_migrated_max_id = Geo::HashedStorageMigratedEvent.maximum(:id)
self.hashed_storage_attachments_max_id = Geo::HashedStorageAttachmentsEvent.maximum(:id)
......@@ -473,7 +464,6 @@ class GeoNodeStatus < ApplicationRecord
self.cursor_last_event_date = Geo::EventLog.find_by(id: self.cursor_last_event_id)&.created_at
load_repositories_data
load_lfs_objects_data
load_job_artifacts_data
load_attachments_data
load_container_registry_data
......@@ -490,17 +480,6 @@ class GeoNodeStatus < ApplicationRecord
self.wikis_failed_count = Geo::ProjectRegistry.sync_failed(:wiki).count
end
def load_lfs_objects_data
return unless lfs_objects_replication_enabled
return if Feature.enabled?(:geo_lfs_object_replication, default_enabled: :yaml)
self.lfs_objects_count = lfs_objects_finder.registry_count
self.lfs_objects_synced_count = lfs_objects_finder.synced_count
self.lfs_objects_failed_count = lfs_objects_finder.failed_count
self.lfs_objects_registry_count = lfs_objects_finder.registry_count
self.lfs_objects_synced_missing_on_primary_count = lfs_objects_finder.synced_missing_on_primary_count
end
def load_job_artifacts_data
return unless job_artifacts_replication_enabled
......@@ -619,10 +598,6 @@ class GeoNodeStatus < ApplicationRecord
@attachments_finder ||= Geo::AttachmentRegistryFinder.new
end
def lfs_objects_finder
@lfs_objects_finder ||= Geo::LfsObjectLegacyRegistryFinder.new
end
def job_artifacts_finder
@job_artifacts_finder ||= Geo::JobArtifactRegistryFinder.new
end
......
ee/app/services/geo/base_file_service.rb
View file @ 21a4999b
......@@ -40,10 +40,6 @@ module Geo
object_type == :job_artifact
end
def lfs?
object_type == :lfs
end
# This is called by LogHelpers to build json log with context info
#
# @see ::Gitlab::Geo::LogHelpers
......
ee/app/services/geo/file_download_service.rb
View file @ 21a4999b
......@@ -38,7 +38,6 @@ module Geo
def downloader_klass
return Gitlab::Geo::Replication::FileDownloader if user_upload?
return Gitlab::Geo::Replication::JobArtifactDownloader if job_artifact?
return Gitlab::Geo::Replication::LfsDownloader if lfs?
fail_unimplemented_klass!(type: 'Downloader')
end
......@@ -62,8 +61,6 @@ module Geo
strong_memoize(:registry) do
if job_artifact?
Geo::JobArtifactRegistry.find_or_initialize_by(artifact_id: object_db_id)
elsif lfs?
Geo::LfsObjectRegistry.find_or_initialize_by(lfs_object_id: object_db_id)
else
Geo::UploadRegistry.find_or_initialize_by(file_type: object_type, file_id: object_db_id)
end
......
ee/app/services/geo/file_registry_removal_service.rb
View file @ 21a4999b
......@@ -43,8 +43,6 @@ module Geo
strong_memoize(:file_registry) do
if job_artifact?
::Geo::JobArtifactRegistry.find_by(artifact_id: object_db_id)
elsif lfs?
::Geo::LfsObjectRegistry.find_by(lfs_object_id: object_db_id)
elsif user_upload?
::Geo::UploadRegistry.find_by(file_type: object_type, file_id: object_db_id)
elsif replicator
......@@ -104,8 +102,6 @@ module Geo
def file_uploader
strong_memoize(:file_uploader) do
case object_type
when :lfs
LfsObject.find(object_db_id).file
when :job_artifact
Ci::JobArtifact.find(object_db_id).file
when *Gitlab::Geo::Replication::USER_UPLOADS_OBJECT_TYPES
......
ee/app/services/geo/file_upload_service.rb
View file @ 21a4999b
......@@ -27,7 +27,6 @@ module Geo
def retriever_klass
return Gitlab::Geo::Replication::FileRetriever if user_upload?
return Gitlab::Geo::Replication::JobArtifactRetriever if job_artifact?
return Gitlab::Geo::Replication::LfsRetriever if lfs?
fail_unimplemented_klass!(type: 'Retriever')
end
......
ee/app/services/geo/lfs_object_deleted_event_store.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
module Geo
class LfsObjectDeletedEventStore < EventStore
extend ::Gitlab::Utils::Override
self.event_type = :lfs_object_deleted_event
attr_reader :lfs_object
def initialize(lfs_object)
@lfs_object = lfs_object
end
private
def build_event
Geo::LfsObjectDeletedEvent.new(
lfs_object: lfs_object,
oid: lfs_object.oid,
file_path: relative_file_path
)
end
def relative_file_path
lfs_object.file.relative_path if lfs_object.file.present?
end
# This is called by LogHelpers to build json log with context info
#
# @see ::Gitlab::Geo::LogHelpers
def extra_log_data
{
lfs_object_id: lfs_object.id,
file_path: lfs_object.file.path
}.compact
end
end
end
ee/app/workers/geo/file_download_dispatch_worker.rb
View file @ 21a4999b
......@@ -75,7 +75,6 @@ module Geo
def job_finders
[
Geo::FileDownloadDispatchWorker::AttachmentJobFinder.new(scheduled_file_ids(Gitlab::Geo::Replication::USER_UPLOADS_OBJECT_TYPES)),
Geo::FileDownloadDispatchWorker::LfsObjectJobFinder.new(scheduled_file_ids(:lfs)),
Geo::FileDownloadDispatchWorker::JobArtifactJobFinder.new(scheduled_file_ids(:job_artifact))
]
end
......
ee/app/workers/geo/file_download_dispatch_worker/lfs_object_job_finder.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
module Geo
class FileDownloadDispatchWorker # rubocop:disable Scalability/IdempotentWorker
class LfsObjectJobFinder < JobFinder # rubocop:disable Scalability/IdempotentWorker
RESOURCE_ID_KEY = :lfs_object_id
EXCEPT_RESOURCE_IDS_KEY = :except_ids
FILE_SERVICE_OBJECT_TYPE = :lfs
def registry_finder
@registry_finder ||= Geo::LfsObjectLegacyRegistryFinder.new
end
end
end
end
ee/lib/ee/api/entities/geo_node_status.rb
View file @ 21a4999b
......@@ -36,7 +36,6 @@ module EE
expose :db_replication_lag_seconds
expose :attachments_replication_enabled
expose :lfs_objects_replication_enabled, if: -> (*) { ::Feature.enabled?(:geo_lfs_object_replication, default_enabled: :yaml) }
expose :job_artifacts_replication_enabled
expose :container_repositories_replication_enabled
expose :design_repositories_replication_enabled
......
ee/lib/gitlab/geo/geo_node_status_check.rb
View file @ 21a4999b
......@@ -28,7 +28,6 @@ module Gitlab
print_verified_repositories
print_wikis_status
print_verified_wikis
print_lfs_objects_status
print_attachments_status
print_ci_job_artifacts_status
print_container_repositories_status
......@@ -50,7 +49,6 @@ module Gitlab
print_verified_repositories
print_wikis_status
print_verified_wikis
print_lfs_objects_status
print_attachments_status
print_ci_job_artifacts_status
print_container_repositories_status
......@@ -75,7 +73,6 @@ module Gitlab
replicables = [
["repositories", Gitlab::Geo.repository_verification_enabled?],
["wikis", Gitlab::Geo.repository_verification_enabled?],
["lfs_objects", false],
["job_artifacts", false],
["attachments", false],
["design_repositories", false]
......@@ -255,15 +252,6 @@ module Gitlab
end
end
def print_lfs_objects_status
return if Feature.enabled?(:geo_lfs_object_replication, default_enabled: :yaml)
print 'LFS Objects: '.rjust(GEO_STATUS_COLUMN_WIDTH)
show_failed_value(current_node_status.lfs_objects_failed_count)
print "#{current_node_status.lfs_objects_synced_count}/#{current_node_status.lfs_objects_count} "
puts using_percentage(current_node_status.lfs_objects_synced_in_percentage)
end
def print_attachments_status
print 'Attachments: '.rjust(GEO_STATUS_COLUMN_WIDTH)
show_failed_value(current_node_status.attachments_failed_count)
......
ee/lib/gitlab/geo/log_cursor/events/lfs_object_deleted_event.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
module Gitlab
module Geo
module LogCursor
module Events
class LfsObjectDeletedEvent
include BaseEvent
def process
# Must always schedule, regardless of shard health
job_id = ::Geo::FileRegistryRemovalWorker.perform_async(:lfs, event.lfs_object_id, file_path)
log_event(job_id)
end
private
def file_path
@file_path ||= File.join(LfsObjectUploader.root, event.file_path)
end
def log_event(job_id)
super(
'Delete LFS object scheduled',
oid: event.oid,
file_id: event.lfs_object_id,
file_path: event.file_path,
job_id: job_id)
end
end
end
end
end
end
ee/lib/gitlab/geo/replication/lfs_downloader.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
module Gitlab
module Geo
module Replication
# This class is responsible for:
# * Finding a LfsObject record
# * Requesting and downloading the LfsObject's file from the primary
# * Returning a detailed Result
#
class LfsDownloader < BaseDownloader
private
def resource
strong_memoize(:resource) { ::LfsObject.find_by_id(object_db_id) }
end
def transfer
strong_memoize(:transfer) { ::Gitlab::Geo::Replication::LfsTransfer.new(resource) }
end
def object_store_enabled?
::LfsObjectUploader.object_store_enabled?
end
end
end
end
end
ee/lib/gitlab/geo/replication/lfs_retriever.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
module Gitlab
module Geo
module Replication
# This class is responsible for:
# * Finding an LfsObject record
# * Returning the necessary response data to send the file back
#
class LfsRetriever < BaseRetriever
def execute
return error('Invalid request') unless valid?
return error('LFS object not found') unless lfs_object
return error('LFS object not found') unless matches_checksum?
unless lfs_object.file.present? && lfs_object.file.exists?
log_error("Could not upload LFS object because it does not have a file", id: lfs_object.id)
return file_not_found(lfs_object)
end
success(lfs_object.file)
end
private
def lfs_object
strong_memoize(:lfs_object) do
LfsObject.find_by_id(object_db_id)
end
end
def valid?
!extra_params.nil?
end
def matches_checksum?
extra_params[:checksum] == lfs_object.oid
end
end
end
end
end
ee/lib/gitlab/geo/replication/lfs_transfer.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
module Gitlab
module Geo
module Replication
# This class is responsible for:
# * Requesting an LfsObject file from the primary
# * Saving it in the right place on successful download
# * Returning a detailed Result object
class LfsTransfer < BaseTransfer
# Initialize a transfer service for a specified LfsObject
#
# @param [LfsObject] lfs_object
def initialize(lfs_object)
if lfs_object.local_store?
super(**local_lfs_object_attributes(lfs_object))
else
super(**remote_lfs_object_attributes(lfs_object))
end
end
private
def uploader
resource.file
end
def local_lfs_object_attributes(lfs_object)
{
resource: lfs_object,
file_type: :lfs,
file_id: lfs_object.id,
filename: lfs_object.file.path,
expected_checksum: lfs_object.oid,
request_data: lfs_request_data(lfs_object)
}
end
def remote_lfs_object_attributes(lfs_object)
{
resource: lfs_object,
file_type: :lfs,
file_id: lfs_object.id,
expected_checksum: lfs_object.oid,
request_data: lfs_request_data(lfs_object)
}
end
def lfs_request_data(lfs_object)
{
checksum: lfs_object.oid,
file_type: :lfs,
file_id: lfs_object.id
}
end
end
end
end
end
ee/spec/factories/geo/event_log.rb
View file @ 21a4999b
......@@ -26,10 +26,6 @@ FactoryBot.define do
hashed_storage_attachments_event factory: :geo_hashed_storage_attachments_event
end
trait :lfs_object_deleted_event do
lfs_object_deleted_event factory: :geo_lfs_object_deleted_event
end
trait :job_artifact_deleted_event do
job_artifact_deleted_event factory: :geo_job_artifact_deleted_event
end
......@@ -126,18 +122,6 @@ FactoryBot.define do
new_attachments_path { Storage::Hashed.new(project).disk_path }
end
factory :geo_lfs_object_deleted_event, class: 'Geo::LfsObjectDeletedEvent' do
lfs_object { association(:lfs_object, :with_file) }
after(:build, :stub) do |event, _|
local_store_path = Pathname.new(LfsObjectUploader.root)
relative_path = Pathname.new(event.lfs_object.file.path).relative_path_from(local_store_path)
event.oid = event.lfs_object.oid
event.file_path = relative_path
end
end
factory :geo_job_artifact_deleted_event, class: 'Geo::JobArtifactDeletedEvent' do
job_artifact { association(:ci_job_artifact, :archive) }
......
ee/spec/factories/geo/lfs_object_registry.rb
View file @ 21a4999b
# frozen_string_literal: true
FactoryBot.define do
factory :geo_lfs_object_legacy_registry, class: 'Geo::LfsObjectRegistry' do
sequence(:lfs_object_id)
success { true }
trait :failed do
success { false }
retry_count { 1 }
end
trait :never_synced do
success { false }
retry_count { nil }
end
trait :with_lfs_object do
after(:build, :stub) do |registry, _|
lfs_object = create(:lfs_object)
registry.lfs_object_id = lfs_object.id
end
end
end
end
FactoryBot.define do
factory :geo_lfs_object_registry, class: 'Geo::LfsObjectRegistry' do
lfs_object
......
ee/spec/factories/geo/upload_registry.rb
View file @ 21a4999b
......@@ -29,9 +29,7 @@ FactoryBot.define do
trait :with_file do
after(:build, :stub) do |registry, _|
file =
if registry.file_type.to_sym == :lfs
raise NotImplementedError, 'Use create(:geo_lfs_object_registry, :with_lfs_object) instead'
elsif registry.file_type.to_sym == :job_artifact
if registry.file_type.to_sym == :job_artifact
raise NotImplementedError, 'Use create(:geo_job_artifact_registry, :with_artifact) instead'
else
create(:upload)
......
ee/spec/factories/geo_node_statuses.rb
View file @ 21a4999b
......@@ -57,7 +57,6 @@ FactoryBot.define do
container_repositories_replication_enabled { false }
design_repositories_replication_enabled { true }
job_artifacts_replication_enabled { false }
lfs_objects_replication_enabled { true }
repositories_replication_enabled { true }
repository_verification_enabled { true }
......
ee/spec/finders/geo/lfs_object_legacy_registry_finder_spec.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::LfsObjectLegacyRegistryFinder, :geo do
before do
stub_feature_flags(geo_lfs_object_replication: false )
end
it_behaves_like 'a file registry finder' do
before do
stub_lfs_object_storage
end
let_it_be(:replicable_1) { create(:lfs_object) }
let_it_be(:replicable_2) { create(:lfs_object) }
let_it_be(:replicable_3) { create(:lfs_object) }
let_it_be(:replicable_4) { create(:lfs_object) }
let_it_be(:replicable_5) { create(:lfs_object) }
let!(:replicable_6) { create(:lfs_object, :object_storage) }
let!(:replicable_7) { create(:lfs_object, :object_storage) }
let!(:replicable_8) { create(:lfs_object, :object_storage) }
let_it_be(:registry_1) { create(:geo_lfs_object_legacy_registry, :failed, lfs_object_id: replicable_1.id) }
let_it_be(:registry_2) { create(:geo_lfs_object_legacy_registry, lfs_object_id: replicable_2.id, missing_on_primary: true) }
let_it_be(:registry_3) { create(:geo_lfs_object_legacy_registry, :never_synced, lfs_object_id: replicable_3.id) }
let_it_be(:registry_4) { create(:geo_lfs_object_legacy_registry, :failed, lfs_object_id: replicable_4.id) }
let_it_be(:registry_5) { create(:geo_lfs_object_legacy_registry, lfs_object_id: replicable_5.id, missing_on_primary: true, retry_at: 1.day.ago) }
let!(:registry_6) { create(:geo_lfs_object_legacy_registry, :failed, lfs_object_id: replicable_6.id) }
let!(:registry_7) { create(:geo_lfs_object_legacy_registry, :failed, lfs_object_id: replicable_7.id, missing_on_primary: true) }
let!(:registry_8) { create(:geo_lfs_object_legacy_registry, :never_synced, lfs_object_id: replicable_8.id) }
end
end
ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json
View file @ 21a4999b
......@@ -11,7 +11,6 @@
"attachments_failed_count",
"attachments_synced_count",
"attachments_synced_missing_on_primary_count",
"lfs_objects_replication_enabled",
"lfs_objects_count",
"lfs_objects_failed_count",
"lfs_objects_synced_count",
......@@ -171,7 +170,6 @@
"attachments_synced_missing_on_primary_count": { "type": ["integer", "null"] },
"attachments_synced_in_percentage": { "type": "string" },
"db_replication_lag_seconds": { "type": ["integer", "null"] },
"lfs_objects_replication_enabled": { "type": ["boolean", "null"] },
"lfs_objects_count": { "type": "integer" },
"lfs_objects_failed_count": { "type": ["integer", "null"] },
"lfs_objects_synced_count": { "type": ["integer", "null"] },
......
ee/spec/lib/ee/api/entities/geo_node_status_spec.rb
View file @ 21a4999b
......@@ -65,22 +65,6 @@ RSpec.describe EE::API::Entities::GeoNodeStatus do
end
end
describe '#lfs_objects_synced_in_percentage' do
context 'LFS with SSF is disabled' do
before do
stub_feature_flags(geo_lfs_object_replication: false)
end
it 'formats as percentage' do
geo_node_status.assign_attributes(lfs_objects_registry_count: 256,
lfs_objects_failed_count: 12,
lfs_objects_synced_count: 123)
expect(subject[:lfs_objects_synced_in_percentage]).to eq '48.05%'
end
end
end
describe '#job_artifacts_synced_in_percentage' do
it 'formats as percentage' do
geo_node_status.assign_attributes(job_artifacts_count: 256,
......
ee/spec/lib/gitlab/geo/geo_node_status_check_spec.rb
View file @ 21a4999b
......@@ -17,17 +17,12 @@ RSpec.describe Gitlab::Geo::GeoNodeStatusCheck do
end
context 'with legacy replication' do
before do
stub_feature_flags(geo_lfs_object_replication: false)
end
it 'prints messages for all legacy replication and verification checks' do
checks = [
/Repositories: /,
/Verified Repositories: /,
/Wikis: /,
/Verified Wikis: /,
/LFS Objects: /,
/Attachments: /,
/CI job artifacts: /,
/Container repositories: /,
......
ee/spec/lib/gitlab/geo/log_cursor/events/lfs_object_deleted_event_spec.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Gitlab::Geo::LogCursor::Events::LfsObjectDeletedEvent, :clean_gitlab_redis_shared_state do
let(:logger) { Gitlab::Geo::LogCursor::Logger.new(described_class, Logger::INFO) }
let(:event_log) { create(:geo_event_log, :lfs_object_deleted_event) }
let!(:event_log_state) { create(:geo_event_log_state, event_id: event_log.id - 1) }
let(:lfs_object_deleted_event) { event_log.lfs_object_deleted_event }
let(:lfs_object) { lfs_object_deleted_event.lfs_object }
subject { described_class.new(lfs_object_deleted_event, Time.now, logger) }
around do |example|
Sidekiq::Testing.inline! { example.run }
end
describe '#process' do
it 'does not create a tracking database entry' do
expect { subject.process }.not_to change(Geo::LfsObjectRegistry, :count)
end
it 'removes the tracking database entry if exist' do
create(:geo_lfs_object_registry, lfs_object_id: lfs_object.id)
expect { subject.process }.to change(Geo::LfsObjectRegistry, :count).by(-1)
end
it 'schedules a Geo::FileRegistryRemovalWorker job' do
expect(::Geo::FileRegistryRemovalWorker).to receive(:perform_async).with(:lfs, lfs_object_deleted_event.lfs_object_id, lfs_object.file.path)
subject.process
end
it_behaves_like 'logs event source info'
end
end
ee/spec/lib/gitlab/geo/replication/base_transfer_spec.rb
View file @ 21a4999b
......@@ -94,39 +94,4 @@ RSpec.describe Gitlab::Geo::Replication::BaseTransfer do
expect(subject.can_transfer?).to be_truthy
end
end
describe '#stream_from_primary_to_object_storage' do
let_it_be(:lfs_object) { create(:lfs_object, :with_file, :correct_oid) }
let(:auth_headers) { { 'Authorization' => 'Bearer 12345' } }
let(:download_link) { 'http://download.link' }
subject do
Gitlab::Geo::Replication::LfsTransfer.new(lfs_object)
end
before do
stub_current_geo_node(secondary_node)
end
it 'downloads file successfully' do
allow_next_instance_of(Gitlab::Geo::TransferRequest) do |request|
allow(request).to receive(:headers).and_return(auth_headers)
end
stub_request(:get, primary_node.geo_transfers_url(:lfs, lfs_object.id.to_s))
.to_return(status: 302, headers: { 'Location' => download_link })
# This stub acts as assertion that auth headers are not present,
# otherwise we would get 500 error
stub_request(:get, download_link)
.with(headers: auth_headers)
.to_return(status: 500)
stub_request(:get, download_link)
.to_return(status: 200)
expect(subject.stream_from_primary_to_object_storage.success).to be_truthy
end
end
end
ee/spec/lib/gitlab/geo/replication/lfs_downloader_spec.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Gitlab::Geo::Replication::LfsDownloader, :geo do
include ::EE::GeoHelpers
describe '#execute' do
let_it_be(:secondary, reload: true) { create(:geo_node) }
before do
stub_current_geo_node(secondary)
end
context 'with LFS object' do
context 'on local storage' do
let(:lfs_object) { create(:lfs_object) }
subject(:downloader) { described_class.new(:lfs, lfs_object.id) }
it 'downloads the LFS file from the primary' do
result = Gitlab::Geo::Replication::BaseTransfer::Result.new(success: true, bytes_downloaded: 1)
expect_next_instance_of(Gitlab::Geo::Replication::LfsTransfer) do |instance|
expect(instance).to receive(:download_from_primary).and_return(result)
end
expect(downloader.execute).to have_attributes(success: true, bytes_downloaded: 1)
end
end
context 'on object storage' do
before do
stub_lfs_object_storage
end
let!(:lfs_object) { create(:lfs_object, :object_storage) }
subject(:downloader) { described_class.new(:lfs, lfs_object.id) }
it 'streams the LFS file from the primary to object storage' do
result = Gitlab::Geo::Replication::BaseTransfer::Result.new(success: true, bytes_downloaded: 1)
expect_next_instance_of(Gitlab::Geo::Replication::LfsTransfer) do |instance|
expect(instance).to receive(:stream_from_primary_to_object_storage).and_return(result)
end
expect(downloader.execute).to have_attributes(success: true, bytes_downloaded: 1)
end
context 'with object storage sync disabled' do
before do
secondary.update_column(:sync_object_storage, false)
end
it 'returns a result indicating a failure before a transfer was attempted' do
result = downloader.execute
expect(result).to have_attributes(
success: false,
failed_before_transfer: true,
reason: 'Skipping transfer as this secondary node is not allowed to replicate content on Object Storage'
)
end
end
context 'with object storage disabled' do
before do
stub_lfs_object_storage(enabled: false)
end
it 'returns a result indicating a failure before a transfer was attempted' do
result = downloader.execute
expect(result).to have_attributes(
success: false,
failed_before_transfer: true,
reason: 'Skipping transfer as this secondary node is not configured to store lfs on Object Storage'
)
end
end
end
end
context 'with unknown object ID' do
let(:unknown_id) { LfsObject.maximum(:id).to_i + 1 }
subject(:downloader) { described_class.new(:lfs, unknown_id) }
it 'returns a result indicating a failure before a transfer was attempted' do
result = downloader.execute
expect(result).to have_attributes(
success: false,
failed_before_transfer: true,
reason: "Skipping transfer as the lfs (ID = #{unknown_id}) could not be found"
)
end
end
end
end
ee/spec/lib/gitlab/geo/replication/lfs_retriever_spec.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Gitlab::Geo::Replication::LfsRetriever, :geo do
describe '#execute' do
subject { retriever.execute }
context 'when the LFS object exists' do
let(:retriever) { described_class.new(lfs_object.id, extra_params) }
before do
expect(LfsObject).to receive(:find_by).with(id: lfs_object.id).and_return(lfs_object)
end
context 'when the LFS object has a file' do
let(:lfs_object) { create(:lfs_object, :with_file) }
let(:extra_params) { { checksum: lfs_object.oid } }
context 'when the extra_params checksum matches the LFS object oid' do
it 'returns the file in a success hash' do
expect(subject).to eq(code: :ok, message: 'Success', file: lfs_object.file)
end
end
context 'when the extra_params checksum does not match the LFS object oid' do
let(:extra_params) { { checksum: 'foo' } }
it 'returns an error hash' do
expect(subject).to include(code: :not_found, message: 'LFS object not found')
end
end
end
context 'when the LFS object does not have a file' do
let(:lfs_object) { create(:lfs_object) }
let(:extra_params) { { checksum: lfs_object.oid } }
it 'returns an error hash' do
expect(subject).to include(code: :not_found, geo_code: 'FILE_NOT_FOUND', message: match(/LfsObject #\d+ file not found/))
end
it 'logs the missing file' do
expect(retriever).to receive(:log_error).with('Could not upload LFS object because it does not have a file', id: lfs_object.id)
subject
end
end
end
context 'when the LFS object does not exist' do
let(:retriever) { described_class.new(10000, {}) }
it 'returns an error hash' do
expect(subject).to eq(code: :not_found, message: 'LFS object not found')
end
end
end
end
ee/spec/lib/gitlab/geo/replication/lfs_transfer_spec.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Gitlab::Geo::Replication::LfsTransfer do
include ::EE::GeoHelpers
let_it_be(:primary_node) { create(:geo_node, :primary) }
let_it_be(:secondary_node) { create(:geo_node) }
let_it_be(:lfs_object) { create(:lfs_object, :with_file, :correct_oid) }
subject do
described_class.new(lfs_object)
end
describe '#download_from_primary' do
before do
stub_current_geo_node(secondary_node)
end
context 'when the destination filename is a directory' do
it 'returns a failed result' do
allow(lfs_object).to receive(:file).and_return(double(path: '/tmp'))
result = subject.download_from_primary
expect_result(result, success: false, bytes_downloaded: 0, primary_missing_file: false)
end
end
context 'when the HTTP response is successful' do
it 'returns a successful result' do
content = lfs_object.file.read
size = content.bytesize
stub_request(:get, subject.resource_url).to_return(status: 200, body: content)
result = subject.download_from_primary
expect_result(result, success: true, bytes_downloaded: size, primary_missing_file: false)
stat = File.stat(lfs_object.file.path)
expect(stat.size).to eq(size)
expect(stat.mode & 0777).to eq(0666 - File.umask)
expect(File.binread(lfs_object.file.path)).to eq(content)
end
end
context 'when the HTTP response is unsuccessful' do
context 'when the HTTP response indicates a missing file on the primary' do
it 'returns a failed result indicating primary_missing_file' do
stub_request(:get, subject.resource_url)
.to_return(status: 404,
headers: { content_type: 'application/json' },
body: { geo_code: Gitlab::Geo::Replication::FILE_NOT_FOUND_GEO_CODE }.to_json)
result = subject.download_from_primary
expect_result(result, success: false, bytes_downloaded: 0, primary_missing_file: true)
end
end
context 'when the HTTP response does not indicate a missing file on the primary' do
it 'returns a failed result' do
stub_request(:get, subject.resource_url).to_return(status: 404, body: 'Not found')
result = subject.download_from_primary
expect_result(result, success: false, bytes_downloaded: 0, primary_missing_file: false)
end
end
end
context 'when Tempfile fails' do
it 'returns a failed result' do
expect(Tempfile).to receive(:new).and_raise(Errno::ENAMETOOLONG)
result = subject.download_from_primary
expect(result.success).to eq(false)
expect(result.bytes_downloaded).to eq(0)
end
end
context "invalid path" do
it 'logs an error if the destination directory could not be created' do
allow(lfs_object).to receive(:file).and_return(double(path: '/foo/bar'))
allow(FileUtils).to receive(:mkdir_p) { raise Errno::EEXIST }
expect(subject).to receive(:log_error).with("Unable to create directory /foo: File exists").once
expect(subject).to receive(:log_error).with("Skipping transfer as we cannot create the destination directory").once
result = subject.download_from_primary
expect(result.success).to eq(false)
expect(result.bytes_downloaded).to eq(0)
end
end
context 'when the checksum of the downloaded file does not match' do
it 'returns a failed result' do
bad_content = 'corrupted!!!'
stub_request(:get, subject.resource_url)
.to_return(status: 200, body: bad_content)
result = subject.download_from_primary
expect_result(result, success: false, bytes_downloaded: bad_content.bytesize, primary_missing_file: false)
end
end
end
def expect_result(result, success:, bytes_downloaded:, primary_missing_file:)
expect(result.success).to eq(success)
expect(result.bytes_downloaded).to eq(bytes_downloaded)
expect(result.primary_missing_file).to eq(primary_missing_file)
# Sanity check to help ensure a valid test
expect(success).not_to be_nil
expect(primary_missing_file).not_to be_nil
end
end
ee/spec/models/geo/event_log_spec.rb
View file @ 21a4999b
......@@ -13,7 +13,6 @@ RSpec.describe Geo::EventLog, type: :model do
it { is_expected.to belong_to(:reset_checksum_event).class_name('Geo::ResetChecksumEvent').with_foreign_key('reset_checksum_event_id') }
it { is_expected.to belong_to(:hashed_storage_migrated_event).class_name('Geo::HashedStorageMigratedEvent').with_foreign_key('hashed_storage_migrated_event_id') }
it { is_expected.to belong_to(:hashed_storage_attachments_event).class_name('Geo::HashedStorageAttachmentsEvent').with_foreign_key('hashed_storage_attachments_event_id') }
it { is_expected.to belong_to(:lfs_object_deleted_event).class_name('Geo::LfsObjectDeletedEvent').with_foreign_key('lfs_object_deleted_event_id') }
it { is_expected.to belong_to(:job_artifact_deleted_event).class_name('Geo::JobArtifactDeletedEvent').with_foreign_key('job_artifact_deleted_event_id') }
it { is_expected.to belong_to(:container_repository_updated_event).class_name('Geo::ContainerRepositoryUpdatedEvent').with_foreign_key('container_repository_updated_event_id') }
end
......@@ -101,13 +100,6 @@ RSpec.describe Geo::EventLog, type: :model do
expect(subject.event).to eq hashed_storage_attachments_event
end
it 'returns lfs_object_deleted_event when set' do
lfs_object_deleted_event = build(:geo_lfs_object_deleted_event)
subject.lfs_object_deleted_event = lfs_object_deleted_event
expect(subject.event).to eq lfs_object_deleted_event
end
it 'returns job_artifact_deleted_event when set' do
job_artifact_deleted_event = build(:geo_job_artifact_deleted_event)
subject.job_artifact_deleted_event = job_artifact_deleted_event
......
ee/spec/models/geo/lfs_object_deleted_event_spec.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::LfsObjectDeletedEvent, type: :model do
describe 'relationships' do
it { is_expected.to belong_to(:lfs_object) }
end
describe 'validations' do
it { is_expected.to validate_presence_of(:lfs_object) }
it { is_expected.to validate_presence_of(:oid) }
it { is_expected.to validate_presence_of(:file_path) }
end
end
ee/spec/models/geo/lfs_object_registry_spec.rb
View file @ 21a4999b
......@@ -2,209 +2,6 @@
require 'spec_helper'
RSpec.describe Geo::LfsObjectRegistry, :geo do
include EE::GeoHelpers
it_behaves_like 'a BulkInsertSafe model', Geo::LfsObjectRegistry do
let(:valid_items_for_bulk_insertion) { build_list(:geo_lfs_object_legacy_registry, 10) }
let(:invalid_items_for_bulk_insertion) { [] } # class does not have any validations defined
end
describe 'relationships' do
it { is_expected.to belong_to(:lfs_object).class_name('LfsObject') }
end
describe '.find_registry_differences' do
let_it_be(:secondary) { create(:geo_node) }
let_it_be(:synced_group) { create(:group) }
let_it_be(:nested_group_1) { create(:group, parent: synced_group) }
let_it_be(:synced_project) { create(:project, group: synced_group) }
let_it_be(:synced_project_in_nested_group) { create(:project, group: nested_group_1) }
let_it_be(:unsynced_project) { create(:project) }
let_it_be(:project_broken_storage) { create(:project, :broken_storage) }
before do
stub_current_geo_node(secondary)
stub_lfs_object_storage
end
let_it_be(:lfs_object_1) { create(:lfs_object) }
let_it_be(:lfs_object_2) { create(:lfs_object) }
let_it_be(:lfs_object_3) { create(:lfs_object) }
let_it_be(:lfs_object_4) { create(:lfs_object) }
let_it_be(:lfs_object_5) { create(:lfs_object) }
let!(:lfs_object_remote_1) { create(:lfs_object, :object_storage) }
let!(:lfs_object_remote_2) { create(:lfs_object, :object_storage) }
let!(:lfs_object_remote_3) { create(:lfs_object, :object_storage) }
context 'untracked IDs' do
before do
create(:geo_lfs_object_legacy_registry, lfs_object_id: lfs_object_1.id)
create(:geo_lfs_object_legacy_registry, :failed, lfs_object_id: lfs_object_3.id)
create(:geo_lfs_object_legacy_registry, lfs_object_id: lfs_object_4.id)
create(:lfs_objects_project, project: synced_project, lfs_object: lfs_object_1)
create(:lfs_objects_project, project: synced_project_in_nested_group, lfs_object: lfs_object_2)
create(:lfs_objects_project, project: synced_project_in_nested_group, lfs_object: lfs_object_3)
create(:lfs_objects_project, project: unsynced_project, lfs_object: lfs_object_4)
create(:lfs_objects_project, project: project_broken_storage, lfs_object: lfs_object_5)
end
it 'includes LFS object IDs without an entry on the tracking database' do
untracked_ids, _ = described_class.find_registry_differences(LfsObject.first.id..LfsObject.last.id)
expect(untracked_ids).to match_array(
[lfs_object_2.id, lfs_object_5.id, lfs_object_remote_1.id,
lfs_object_remote_2.id, lfs_object_remote_3.id])
end
it 'excludes LFS objects outside the ID range' do
untracked_ids, _ = described_class.find_registry_differences(lfs_object_3.id..lfs_object_remote_2.id)
expect(untracked_ids).to match_array(
[lfs_object_5.id, lfs_object_remote_1.id,
lfs_object_remote_2.id])
end
context 'with selective sync by namespace' do
let(:secondary) { create(:geo_node, selective_sync_type: 'namespaces', namespaces: [synced_group]) }
it 'excludes LFS object IDs that are not in selectively synced projects' do
untracked_ids, _ = described_class.find_registry_differences(LfsObject.first.id..LfsObject.last.id)
expect(untracked_ids).to match_array([lfs_object_2.id])
end
end
context 'with selective sync by shard' do
let(:secondary) { create(:geo_node, selective_sync_type: 'shards', selective_sync_shards: ['broken']) }
it 'excludes LFS object IDs that are not in selectively synced projects' do
untracked_ids, _ = described_class.find_registry_differences(LfsObject.first.id..LfsObject.last.id)
expect(untracked_ids).to match_array([lfs_object_5.id])
end
end
context 'with object storage sync disabled' do
let(:secondary) { create(:geo_node, :local_storage_only) }
it 'excludes LFS objects in object storage' do
untracked_ids, _ = described_class.find_registry_differences(LfsObject.first.id..LfsObject.last.id)
expect(untracked_ids).to match_array([lfs_object_2.id, lfs_object_5.id])
end
end
end
context 'unused tracked IDs' do
context 'with an orphaned registry' do
let!(:orphaned) { create(:geo_lfs_object_legacy_registry, lfs_object_id: non_existing_record_id) }
it 'includes tracked IDs that do not exist in the model table' do
range = non_existing_record_id..non_existing_record_id
_, unused_tracked_ids = described_class.find_registry_differences(range)
expect(unused_tracked_ids).to match_array([non_existing_record_id])
end
it 'excludes IDs outside the ID range' do
range = 1..1000
_, unused_tracked_ids = described_class.find_registry_differences(range)
expect(unused_tracked_ids).to be_empty
end
end
context 'with selective sync by namespace' do
let(:secondary) { create(:geo_node, selective_sync_type: 'namespaces', namespaces: [synced_group]) }
context 'with a tracked LFS object' do
let!(:registry_entry) { create(:geo_lfs_object_legacy_registry, lfs_object_id: lfs_object_1.id) }
let(:range) { lfs_object_1.id..lfs_object_1.id }
context 'excluded from selective sync' do
it 'includes tracked LFS object IDs that exist but are not in a selectively synced project' do
_, unused_tracked_ids = described_class.find_registry_differences(range)
expect(unused_tracked_ids).to match_array([lfs_object_1.id])
end
end
context 'included in selective sync' do
let!(:join_record) { create(:lfs_objects_project, project: synced_project, lfs_object: lfs_object_1) }
it 'excludes tracked LFS object IDs that are in selectively synced projects' do
_, unused_tracked_ids = described_class.find_registry_differences(range)
expect(unused_tracked_ids).to be_empty
end
end
end
end
context 'with selective sync by shard' do
let(:secondary) { create(:geo_node, selective_sync_type: 'shards', selective_sync_shards: ['broken']) }
context 'with a tracked LFS object' do
let!(:registry_entry) { create(:geo_lfs_object_legacy_registry, lfs_object_id: lfs_object_1.id) }
let(:range) { lfs_object_1.id..lfs_object_1.id }
context 'excluded from selective sync' do
it 'includes tracked LFS object IDs that exist but are not in a selectively synced project' do
_, unused_tracked_ids = described_class.find_registry_differences(range)
expect(unused_tracked_ids).to match_array([lfs_object_1.id])
end
end
context 'included in selective sync' do
let!(:join_record) { create(:lfs_objects_project, project: project_broken_storage, lfs_object: lfs_object_1) }
it 'excludes tracked LFS object IDs that are in selectively synced projects' do
_, unused_tracked_ids = described_class.find_registry_differences(range)
expect(unused_tracked_ids).to be_empty
end
end
end
end
context 'with object storage sync disabled' do
let(:secondary) { create(:geo_node, :local_storage_only) }
context 'with a tracked LFS object' do
context 'in object storage' do
it 'includes tracked LFS object IDs that are in object storage' do
create(:geo_lfs_object_legacy_registry, lfs_object_id: lfs_object_remote_1.id)
range = lfs_object_remote_1.id..lfs_object_remote_1.id
_, unused_tracked_ids = described_class.find_registry_differences(range)
expect(unused_tracked_ids).to match_array([lfs_object_remote_1.id])
end
end
context 'not in object storage' do
it 'excludes tracked LFS object IDs that are not in object storage' do
create(:geo_lfs_object_legacy_registry, lfs_object_id: lfs_object_1.id)
range = lfs_object_1.id..lfs_object_1.id
_, unused_tracked_ids = described_class.find_registry_differences(range)
expect(unused_tracked_ids).to be_empty
end
end
end
end
end
end
end
RSpec.describe Geo::LfsObjectRegistry, :geo, type: :model do
let_it_be(:registry) { create(:geo_lfs_object_registry) }
......
ee/spec/models/geo_node_status_spec.rb
View file @ 21a4999b
......@@ -173,7 +173,7 @@ RSpec.describe GeoNodeStatus, :geo do
describe '#attachments_failed_count' do
it 'counts failed avatars, attachment, personal snippets and files' do
# These two should be ignored
create(:geo_lfs_object_legacy_registry, :with_lfs_object, :failed)
create(:geo_lfs_object_registry, :failed)
create(:geo_upload_registry, :with_file)
create(:geo_upload_registry, :with_file, :failed, file_type: :personal_file)
......@@ -219,57 +219,6 @@ RSpec.describe GeoNodeStatus, :geo do
end
end
context 'LFS replication with SSF is disabled' do
before do
stub_feature_flags(geo_lfs_object_replication: false)
end
describe '#lfs_objects_synced_count' do
it 'counts synced LFS objects' do
# These four should be ignored
create(:geo_upload_registry, :failed)
create(:geo_upload_registry, :avatar)
create(:geo_upload_registry, file_type: :attachment)
create(:geo_lfs_object_legacy_registry, :failed)
create(:geo_lfs_object_legacy_registry)
create(:geo_lfs_object_legacy_registry, missing_on_primary: true)
expect(subject.lfs_objects_synced_missing_on_primary_count).to eq(1)
end
end
describe '#lfs_objects_failed_count' do
it 'counts failed LFS objects' do
# These four should be ignored
create(:geo_upload_registry, :failed)
create(:geo_upload_registry, :avatar, :failed)
create(:geo_upload_registry, :failed, file_type: :attachment)
create(:geo_lfs_object_legacy_registry)
create(:geo_lfs_object_legacy_registry, :failed)
expect(subject.lfs_objects_failed_count).to eq(1)
end
end
describe '#lfs_objects_synced_in_percentage' do
it 'returns 0 when there are no registries' do
expect(subject.lfs_objects_synced_in_percentage).to eq(0)
end
it 'returns the right percentage' do
create(:geo_lfs_object_legacy_registry)
create(:geo_lfs_object_legacy_registry, :failed)
create(:geo_lfs_object_legacy_registry, :never_synced)
create(:geo_lfs_object_legacy_registry, :never_synced)
expect(subject.lfs_objects_synced_in_percentage).to be_within(0.0001).of(25)
end
end
end
describe '#job_artifacts_synced_count' do
it 'counts synced job artifacts' do
# These should be ignored
......@@ -1394,12 +1343,6 @@ RSpec.describe GeoNodeStatus, :geo do
stub_current_geo_node(primary)
end
it 'does not call LfsObjectRegistryFinder#registry_count' do
expect_any_instance_of(Geo::LfsObjectLegacyRegistryFinder).not_to receive(:registry_count)
subject
end
it 'does not call AttachmentRegistryFinder#registry_count' do
expect_any_instance_of(Geo::AttachmentRegistryFinder).not_to receive(:registry_count)
......@@ -1414,13 +1357,6 @@ RSpec.describe GeoNodeStatus, :geo do
end
context 'on the secondary' do
it 'calls LfsObjectRegistryFinder#registry_count' do
stub_feature_flags(geo_lfs_object_replication: false)
expect_any_instance_of(Geo::LfsObjectLegacyRegistryFinder).to receive(:registry_count).twice
subject
end
it 'calls AttachmentRegistryFinder#registry_count' do
expect_any_instance_of(Geo::AttachmentRegistryFinder).to receive(:registry_count).twice
......
ee/spec/models/lfs_object_spec.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe LfsObject do
include EE::GeoHelpers
before do
stub_feature_flags(geo_lfs_object_replication: false)
end
describe '#destroy' do
subject { create(:lfs_object, :with_file) }
context 'when running in a Geo primary node' do
let_it_be(:primary) { create(:geo_node, :primary) }
let_it_be(:secondary) { create(:geo_node) }
it 'logs an event to the Geo event log' do
stub_current_geo_node(primary)
expect { subject.destroy }.to change(Geo::LfsObjectDeletedEvent, :count).by(1)
end
end
end
end
ee/spec/requests/api/geo_spec.rb
View file @ 21a4999b
......@@ -274,59 +274,6 @@ RSpec.describe API::Geo do
end
end
end
describe 'GET /geo/transfers/lfs/1' do
let(:resource) { create(:lfs_object, :with_file) }
let(:transfer) { Gitlab::Geo::Replication::LfsTransfer.new(resource) }
let(:req_header) { Gitlab::Geo::TransferRequest.new(transfer.request_data).headers }
context 'invalid requests' do
before do
allow_next_instance_of(Gitlab::Geo::TransferRequest) do |instance|
allow(instance).to receive(:requesting_node).and_return(secondary_node)
end
end
it 'responds with 401 when an invalid auth header is provided' do
get api("/geo/transfers/lfs/#{resource.id}"), headers: invalid_geo_auth_header
expect(response).to have_gitlab_http_status(:unauthorized)
end
it 'responds with 404 when resource does not exist' do
get api("/geo/transfers/lfs/100000"), headers: not_found_req_header
expect(response).to have_gitlab_http_status(:not_found)
end
end
context 'LFS object exists' do
context 'file exists' do
subject(:request) { get api("/geo/transfers/lfs/#{resource.id}"), headers: req_header }
it 'responds with 200 with X-Sendfile' do
request
expect(response).to have_gitlab_http_status(:ok)
expect(response.headers['Content-Type']).to eq('application/octet-stream')
expect(response.headers['X-Sendfile']).to eq(resource.file.path)
end
it_behaves_like 'with terms enforced'
end
context 'file does not exist' do
it 'responds with 404 and a specific geo code' do
File.unlink(resource.file.path)
get api("/geo/transfers/lfs/#{resource.id}"), headers: req_header
expect(response).to have_gitlab_http_status(:not_found)
expect(json_response['geo_code']).to eq(Gitlab::Geo::Replication::FILE_NOT_FOUND_GEO_CODE)
end
end
end
end
end
describe 'POST /geo/status' do
......@@ -375,7 +322,6 @@ RSpec.describe API::Geo do
container_repositories_replication_enabled: true,
design_repositories_replication_enabled: false,
job_artifacts_replication_enabled: true,
lfs_objects_replication_enabled: true,
repositories_replication_enabled: true,
repository_verification_enabled: true
}
......
ee/spec/requests/api/graphql/geo/registries_spec.rb
View file @ 21a4999b
......@@ -38,13 +38,6 @@ RSpec.describe 'Gets registries' do
registry_foreign_key_field_name: 'groupWikiRepositoryId'
}
it_behaves_like 'gets registries for', {
field_name: 'lfsObjectRegistries',
registry_class_name: 'LfsObjectRegistry',
registry_factory: :geo_lfs_object_registry,
registry_foreign_key_field_name: 'lfsObjectId'
}
it_behaves_like 'gets registries for', {
field_name: 'pipelineArtifactRegistries',
registry_class_name: 'PipelineArtifactRegistry',
......
ee/spec/services/geo/file_download_service_spec.rb
View file @ 21a4999b
......@@ -22,12 +22,6 @@ RSpec.describe Geo::FileDownloadService do
end
end
it "returns a LfsDownloader given object_type is lfs" do
subject = described_class.new('lfs', 1)
expect(subject.downloader).to be_a(Gitlab::Geo::Replication::LfsDownloader)
end
it "returns a JobArtifactDownloader given object_type is job_artifact" do
subject = described_class.new('job_artifact', 1)
......@@ -103,8 +97,6 @@ RSpec.describe Geo::FileDownloadService do
case file_type
when 'job_artifact'
Geo::JobArtifactRegistry
when 'lfs'
Geo::LfsObjectRegistry
else
Geo::UploadRegistry
end
......@@ -240,8 +232,6 @@ RSpec.describe Geo::FileDownloadService do
case file_type
when 'job_artifact'
create(:geo_job_artifact_registry, success: false, artifact_id: file.id, retry_count: 3, retry_at: 1.hour.ago)
when 'lfs'
create(:geo_lfs_object_registry, success: false, lfs_object_id: file.id, retry_count: 3, retry_at: 1.hour.ago)
else
create(:geo_upload_registry, file_type.to_sym, success: false, file_id: file.id, retry_count: 3, retry_at: 1.hour.ago)
end
......@@ -440,16 +430,6 @@ RSpec.describe Geo::FileDownloadService do
it_behaves_like 'a service that handles orphaned uploads', 'issuable_metric_image'
end
context 'LFS object' do
before do
stub_feature_flags(geo_lfs_object_replication: false)
end
it_behaves_like "a service that downloads the file and registers the sync result", 'lfs' do
let(:file) { create(:lfs_object) }
end
end
context 'job artifacts' do
it_behaves_like "a service that downloads the file and registers the sync result", 'job_artifact' do
let(:file) { create(:ci_job_artifact) }
......
ee/spec/services/geo/file_registry_removal_service_spec.rb
View file @ 21a4999b
......@@ -165,53 +165,6 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
end
end
context 'with LFS object' do
let!(:lfs_object) { create(:lfs_object, :with_file) }
let!(:registry) { create(:geo_lfs_object_registry, lfs_object_id: lfs_object.id) }
let!(:file_path) { lfs_object.file.path }
it_behaves_like 'removes LFS object'
context 'migrated to object storage' do
before do
stub_lfs_object_storage
lfs_object.update_column(:file_store, LfsObjectUploader::Store::REMOTE)
end
context 'with object storage enabled' do
it_behaves_like 'removes LFS object'
end
context 'with object storage disabled' do
before do
stub_lfs_object_storage(enabled: false)
end
it_behaves_like 'removes LFS object registry'
end
end
context 'no lfs_object record' do
before do
lfs_object.delete
end
it_behaves_like 'removes LFS object' do
subject(:service) { described_class.new('lfs', registry.lfs_object_id, file_path) }
end
end
context 'with orphaned registry' do
before do
lfs_object.delete
end
it_behaves_like 'removes LFS object registry' do
subject(:service) { described_class.new('lfs', registry.lfs_object_id) }
end
end
end
context 'with job artifact' do
let!(:job_artifact) { create(:ci_job_artifact, :archive) }
let!(:registry) { create(:geo_job_artifact_registry, artifact_id: job_artifact.id) }
......
ee/spec/services/geo/file_upload_service_spec.rb
View file @ 21a4999b
......@@ -20,12 +20,6 @@ RSpec.describe Geo::FileUploadService do
end
end
it "returns a LfsRetriever given object_type is lfs" do
subject = described_class.new({ type: 'lfs', id: 1 }, 'request-data')
expect(subject.retriever).to be_a(Gitlab::Geo::Replication::LfsRetriever)
end
it "returns a JobArtifactRetriever given object_type is job_artifact" do
subject = described_class.new({ type: 'job_artifact', id: 1 }, 'request-data')
......@@ -180,23 +174,6 @@ RSpec.describe Geo::FileUploadService do
include_examples 'no decoded params'
end
context 'LFS Object' do
let(:lfs_object) { create(:lfs_object, :with_file) }
let(:params) { { id: lfs_object.id, type: 'lfs' } }
let(:request_data) { Gitlab::Geo::Replication::LfsTransfer.new(lfs_object).request_data }
it 'sends LFS file' do
service = described_class.new(params, request_data)
response = service.execute
expect(response[:code]).to eq(:ok)
expect(response[:file].path).to eq(lfs_object.file.path)
end
include_examples 'no decoded params'
end
context 'job artifact' do
let(:job_artifact) { create(:ci_job_artifact, :with_file) }
let(:params) { { id: job_artifact.id, type: 'job_artifact' } }
......
ee/spec/services/geo/lfs_object_deleted_event_store_spec.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::LfsObjectDeletedEventStore do
include EE::GeoHelpers
let_it_be(:secondary_node) { create(:geo_node) }
let(:lfs_object) { create(:lfs_object, :with_file, oid: 'b68143e6463773b1b6c6fd009a76c32aeec041faff32ba2ed42fd7f708a00004') }
subject { described_class.new(lfs_object) }
describe '#create!' do
it_behaves_like 'a Geo event store', Geo::LfsObjectDeletedEvent do
let(:file_subject) { lfs_object }
end
context 'when running on a primary node' do
before do
stub_primary_node
end
it 'tracks LFS object attributes' do
subject.create!
expect(Geo::LfsObjectDeletedEvent.last).to have_attributes(
lfs_object_id: lfs_object.id,
oid: lfs_object.oid,
file_path: 'b6/81/43e6463773b1b6c6fd009a76c32aeec041faff32ba2ed42fd7f708a00004'
)
end
it 'logs an error message when event creation fail' do
invalid_lfs_object = create(:lfs_object)
subject = described_class.new(invalid_lfs_object)
expected_message = {
class: "Geo::LfsObjectDeletedEventStore",
host: "localhost",
lfs_object_id: invalid_lfs_object.id,
message: "Lfs object deleted event could not be created",
error: "Validation failed: File path can't be blank"
}
expect(Gitlab::Geo::Logger).to receive(:error)
.with(expected_message).and_call_original
subject.create!
end
end
end
end
ee/spec/services/geo/metrics_update_service_spec.rb
View file @ 21a4999b
......@@ -167,7 +167,6 @@ RSpec.describe Geo::MetricsUpdateService, :geo, :prometheus do
expect(metric_value(:geo_lfs_objects)).to eq(100)
expect(metric_value(:geo_lfs_objects_synced)).to eq(50)
expect(metric_value(:geo_lfs_objects_failed)).to eq(12)
expect(metric_value(:geo_lfs_objects_synced_missing_on_primary)).to eq(4)
expect(metric_value(:geo_job_artifacts)).to eq(100)
expect(metric_value(:geo_job_artifacts_synced)).to eq(50)
expect(metric_value(:geo_job_artifacts_failed)).to eq(12)
......
ee/spec/tasks/geo_rake_spec.rb
View file @ 21a4999b
......@@ -366,18 +366,6 @@ RSpec.describe 'geo rake tasks', :geo do
expect { run_rake_task('geo:status') }.not_to output(/Health Status Summary/).to_stdout
end
context 'with legacy LFS replication enabled' do
before do
stub_feature_flags(geo_lfs_object_replication: false)
end
it 'prints messages for all the checks' do
(checks << /LFS Objects: /).each do |text|
expect { run_rake_task('geo:status') }.to output(text).to_stdout
end
end
end
context 'with SSF LFS replication eneabled' do
it 'prints messages for all the checks' do
checks.each do |text|
......
ee/spec/workers/geo/file_download_dispatch_worker_spec.rb
View file @ 21a4999b
......@@ -12,7 +12,6 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
before do
stub_current_geo_node(secondary)
stub_exclusive_lease(renew: true)
stub_feature_flags(geo_lfs_object_replication: false)
allow_next_instance_of(described_class) do |instance|
allow(instance).to receive(:over_time?).and_return(false)
end
......@@ -45,47 +44,6 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
subject.perform
end
it 'does not schedule duplicated jobs' do
lfs_object_1 = create(:lfs_object, :with_file)
lfs_object_2 = create(:lfs_object, :with_file)
create(:geo_lfs_object_legacy_registry, :never_synced, lfs_object: lfs_object_1)
create(:geo_lfs_object_legacy_registry, :failed, lfs_object: lfs_object_2)
stub_const('Geo::Scheduler::SchedulerWorker::DB_RETRIEVE_BATCH_SIZE', 5)
secondary.update!(files_max_capacity: 4)
allow(Gitlab::SidekiqStatus).to receive(:job_status).with([]).and_return([]).twice
allow(Gitlab::SidekiqStatus).to receive(:job_status).with(%w[123 456]).and_return([true, true], [true, true], [false, false])
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_1.id).once.and_return('123')
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_2.id).once.and_return('456')
subject.perform
end
it 'does not schedule duplicated jobs because of query cache' do
lfs_object_1 = create(:lfs_object, :with_file)
lfs_object_2 = create(:lfs_object, :with_file)
lfs_object_3 = create(:lfs_object, :with_file)
create(:geo_lfs_object_legacy_registry, :never_synced, lfs_object: lfs_object_1)
create(:geo_lfs_object_legacy_registry, :never_synced, lfs_object: lfs_object_2)
create(:geo_lfs_object_legacy_registry, :never_synced, lfs_object: lfs_object_3)
stub_const('Geo::Scheduler::SchedulerWorker::DB_RETRIEVE_BATCH_SIZE', 3)
secondary.update!(files_max_capacity: 6)
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_1.id).once do
Thread.new do
# Rails will invalidate the query cache if the update happens in the same thread
Geo::LfsObjectRegistry.update(success: true) # rubocop:disable Rails/SaveBang
end
end
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_2.id).once
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_3.id).once
subject.perform
end
context 'with attachments (Upload records)' do
let(:upload) { create(:upload) }
......@@ -193,61 +151,6 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end
end
context 'with LFS objects' do
let!(:lfs_object_local_store) { create(:lfs_object, :with_file) }
let!(:lfs_object_remote_store) { create(:lfs_object, :with_file, :object_storage) }
before do
stub_lfs_object_storage
end
context 'with files missing on the primary' do
let!(:lfs_object_file_missing_on_primary) { create(:lfs_object, :with_file) }
context 'with lfs_object_registry entries' do
before do
create(:geo_lfs_object_legacy_registry, :never_synced, lfs_object: lfs_object_local_store)
create(:geo_lfs_object_legacy_registry, :failed, lfs_object: lfs_object_remote_store)
Geo::LfsObjectRegistry.create!(lfs_object_id: lfs_object_file_missing_on_primary.id, bytes: 1234, success: true, missing_on_primary: true)
end
it 'enqueues file downloads if there is spare capacity' do
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_local_store.id)
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_file_missing_on_primary.id)
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_remote_store.id)
subject.perform
end
it 'does not retry those files if there is no spare capacity' do
expect(subject).to receive(:db_retrieve_batch_size).and_return(1).twice
expect(Geo::FileDownloadWorker).to receive(:perform_async).once
subject.perform
end
it 'does not retry those files if they are already scheduled' do
scheduled_jobs = [{ type: 'lfs', id: lfs_object_file_missing_on_primary.id, job_id: 'foo' }]
expect(subject).to receive(:scheduled_jobs).and_return(scheduled_jobs).at_least(1)
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_local_store.id)
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('lfs', lfs_object_remote_store.id)
subject.perform
end
end
context 'with no lfs_object_registry entries' do
it 'does not enqueue file downloads' do
expect(Geo::FileDownloadWorker).not_to receive(:perform_async)
subject.perform
end
end
end
end
context 'with job artifacts' do
it 'performs Geo::FileDownloadWorker for unsynced job artifacts' do
registry = create(:geo_job_artifact_registry, :with_artifact, :never_synced)
......@@ -373,7 +276,6 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
result_object = double(:result, success: true, bytes_downloaded: 100, primary_missing_file: false)
allow_any_instance_of(::Gitlab::Geo::Replication::BaseTransfer).to receive(:download_from_primary).and_return(result_object)
create_list(:geo_lfs_object_legacy_registry, 2, :with_lfs_object, :never_synced)
create_list(:geo_upload_registry, 2, :avatar, :with_file, :never_synced)
create_list(:geo_upload_registry, 2, :attachment, :with_file, :never_synced)
create(:geo_upload_registry, :favicon, :with_file, :never_synced)
......@@ -381,14 +283,14 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
create(:geo_upload_registry, :personal_file, :with_file, :never_synced)
create(:geo_job_artifact_registry, :with_artifact, :never_synced)
expect(Geo::FileDownloadWorker).to receive(:perform_async).exactly(10).times.and_call_original
expect(Geo::FileDownloadWorker).to receive(:perform_async).exactly(8).times.and_call_original
# For 10 downloads, we expect four database reloads:
# 1. Load the first batch of 5.
# 2. 4 get sent out, 1 remains. This triggers another reload, which loads in the next 5.
# 3. Those 4 get sent out, and 1 remains.
# 3. Since the second reload filled the pipe with 4, we need to do a final reload to ensure
# zero are left.
expect(subject).to receive(:load_pending_resources).exactly(4).times.and_call_original
expect(subject).to receive(:load_pending_resources).exactly(3).times.and_call_original
Sidekiq::Testing.inline! do
subject.perform
......
ee/spec/workers/remove_unreferenced_lfs_objects_worker_spec.rb deleted 100644 → 0
View file @ 8d5fb3c7
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe RemoveUnreferencedLfsObjectsWorker do
include EE::GeoHelpers
describe '#perform' do
context 'when running in a Geo primary node' do
let_it_be(:primary) { create(:geo_node, :primary) }
let_it_be(:secondary) { create(:geo_node) }
it 'logs an event to the Geo event log for every unreferenced LFS objects' do
stub_feature_flags(geo_lfs_object_replication: false)
stub_current_geo_node(primary)
unreferenced_lfs_object_1 = create(:lfs_object, :with_file)
unreferenced_lfs_object_2 = create(:lfs_object, :with_file)
referenced_lfs_object = create(:lfs_object)
create(:lfs_objects_project, lfs_object: referenced_lfs_object)
expect { subject.perform }.to change(Geo::LfsObjectDeletedEvent, :count).by(2)
expect(Geo::LfsObjectDeletedEvent.where(lfs_object: unreferenced_lfs_object_1.id)).to exist
expect(Geo::LfsObjectDeletedEvent.where(lfs_object: unreferenced_lfs_object_2.id)).to exist
expect(Geo::LfsObjectDeletedEvent.where(lfs_object: referenced_lfs_object.id)).not_to exist
end
end
end
end
locale/gitlab.pot
View file @ 21a4999b
......@@ -19162,9 +19162,6 @@ msgstr ""
msgid "LFS"
msgstr ""
msgid "LFS object"
msgstr ""
msgid "LFS objects"
msgstr ""
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7