Commit a2578194 authored by Stan Hu's avatar Stan Hu

Reduce excessive GC on pull mirrors

In https://gitlab.com/gitlab-org/gitaly/issues/1686 and
https://gitlab.com/gitlab-com/gl-infra/scalability/issues/20, we saw a
lot of I/O consumed by Git garbage collection running on pull mirrors.

It looks like the problem happened because
`Projects::AfterImportService` always ran after completion of a project
import sync, which caused `HousekeepingService` to run. Since the
`pushes_since_gc` counter never changed, `HousekeepingService` would
always run GC again.

This commit increments the `pushes_since_gc` counter so that GCs are
only run every 200 syncs. Ideally we'd only run `HousekeepingService`
only if the repository actually changed, but right now we don't have
an easy way of detecting that.

Note that this problem does NOT happen if the mirror is a imported
directly via a URL. It only happens if the `import_type` is
`Gitlab::ImportSources.importer_names`.

Closes https://gitlab.com/gitlab-org/gitaly/issues/1686
parent 6e1f17da
......@@ -9,9 +9,16 @@ module Projects
end
def execute
Projects::HousekeepingService.new(@project).execute do
service = Projects::HousekeepingService.new(@project)
service.execute do
repository.delete_all_refs_except(RESERVED_REF_PREFIXES)
end
# Right now we don't actually have a way to know if a project
# import actually changed, so we increment the counter to avoid
# causing GC to run every time.
service.increment!
rescue Projects::HousekeepingService::LeaseTaken => e
Rails.logger.info( # rubocop:disable Gitlab/RailsLogger
"Could not perform housekeeping for project #{@project.full_path} (#{@project.id}): #{e}")
......
---
title: Reduce excessive GC on pull mirrors
merge_request: 17931
author:
type: performance
......@@ -19,6 +19,8 @@ describe Projects::AfterImportService do
allow(housekeeping_service)
.to receive(:execute).and_yield
expect(housekeeping_service).to receive(:increment!)
end
it 'performs housekeeping' do
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment