Commit f2545996 authored by Dylan Griffith's avatar Dylan Griffith

Add exclusive lock on project in ElasticCommitIndexerWorker

Indexing the same project twice at the same time causes conflict errors
in Elasticsearch and this usually leads to failures which can lead to a
project never being successfully indexed correctly. The simplest
solution is to use a lock in the worker and just fail immediately if
it's already running and let Sideikiq retries requeue the project if it
does have more updates that need indexing. When it next runs it will
check if there are new commits that need indexing.

Closes https://gitlab.com/gitlab-org/gitlab/-/issues/32648
parent 0f5b14d7
......@@ -3,6 +3,7 @@
class ElasticCommitIndexerWorker
include ApplicationWorker
prepend Elastic::IndexingControl
include Gitlab::ExclusiveLeaseHelpers
feature_category :global_search
sidekiq_options retry: 2
......@@ -24,6 +25,8 @@ class ElasticCommitIndexerWorker
project = Project.find(project_id)
return true unless project.use_elasticsearch?
in_lock("#{self.class.name}/#{project_id}/#{wiki}", ttl: 1.hour, retries: 0) do
Gitlab::Elastic::Indexer.new(project, wiki: wiki).run
end
end
end
---
title: Don't index the same project in Elasticsearch in parallel
merge_request: 35842
author:
type: performance
......@@ -34,5 +34,14 @@ RSpec.describe ElasticCommitIndexerWorker do
subject.perform(project.id, nil, nil, true)
end
it 'does not run index when it is locked' do
expect(subject).to receive(:in_lock) # Mock and don't yield
.with("ElasticCommitIndexerWorker/#{project.id}/false", ttl: 1.hour, retries: 0)
expect(Gitlab::Elastic::Indexer).not_to receive(:new)
subject.perform(project.id)
end
end
end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment