Commit 4b48ce4e authored by Sean McGivern's avatar Sean McGivern

Speed up UpdateAllMirrorsWorker query on GitLab.com

On GitLab.com, we check for the license on a project-by-project basis,
as different namespaces can have different licenses. Pull mirroring is a
Bronze feature, meaning it should only be available to:

1. Paid projects.
2. Public projects.

Previously, we offered pull mirroring for free even on private projects.
After we stopped offering that, we started checking the plans in the
query to fetch mirrors, to avoid fetching lots of projects we were going
to do nothing with.

However, this query got slower over time, because we order by
`next_execution_timestamp`, and `next_execution_timestamp` is stuck at
some point before 2020-03-28 for these free private mirrors until the
project is made public, or paid for. That means that the database has to
do a large amount of work discarding those projects that we know we will
never process.

For GitLab.com ONLY, we work around this by explicitly filtering on
`next_execution_timestamp > '2020-03-28'`. This makes the query go from
taking several seconds, to around 150 milliseconds.

In the long term, we should look to remove this workaround and simplify
the query.
parent d1618331
......@@ -42,6 +42,17 @@ class UpdateAllMirrorsWorker # rubocop:disable Scalability/IdempotentWorker
last = nil
scheduled = 0
# On GitLab.com, we stopped processing free mirrors for private
# projects on 2020-03-27. Including mirrors with
# next_execution_timestamp of that date or earlier in the query will
# lead to higher query times:
# <https://gitlab.com/gitlab-org/gitlab/-/issues/216252>
#
# We should remove this workaround in favour of a simpler solution:
# <https://gitlab.com/gitlab-org/gitlab/-/issues/216783>
#
last = Time.utc(2020, 3, 28) if Gitlab.com?
while capacity > 0
batch_size = [capacity * 2, 500].min
projects = pull_mirrors_batch(freeze_at: now, batch_size: batch_size, offset_at: last).to_a
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment