Speed up UpdateAllMirrorsWorker query on GitLab.com
On GitLab.com, we check for the license on a project-by-project basis, as different namespaces can have different licenses. Pull mirroring is a Bronze feature, meaning it should only be available to: 1. Paid projects. 2. Public projects. Previously, we offered pull mirroring for free even on private projects. After we stopped offering that, we started checking the plans in the query to fetch mirrors, to avoid fetching lots of projects we were going to do nothing with. However, this query got slower over time, because we order by `next_execution_timestamp`, and `next_execution_timestamp` is stuck at some point before 2020-03-28 for these free private mirrors until the project is made public, or paid for. That means that the database has to do a large amount of work discarding those projects that we know we will never process. For GitLab.com ONLY, we work around this by explicitly filtering on `next_execution_timestamp > '2020-03-28'`. This makes the query go from taking several seconds, to around 150 milliseconds. In the long term, we should look to remove this workaround and simplify the query.
Showing
Please register or sign in to comment