Skip to content

G
gitlab-ce
Switch branch/tag
  1. 17 May, 2021 1 commit
    • Sean McGivern's avatar
      Allow migrating scheduled and retried Sidekiq jobs to new queues · 03b5e73a
      Sean McGivern authored 
      We now allow Sidekiq worker routing to be configured by administrators.
For example, they can say 'all jobs go to the default queue', or
'project export and import workers share a queue'. Right now, the only
really useful case is to re-route jobs to the default queue, but we will
support other options in future.

Migrating this sounds simple: listen to the old and new queues, update
the worker routing, wait for the old queue to be empty, and stop
listening to the old queue. But there's a catch: Sidekiq maintains two
sorted sets with jobs that are to be run in the future. There is the
scheduled set (for jobs that use `perform_in` or `perform_at` or
similar, where we choose to run a job in the future) and the retry
set (after failing, a job will get retried with some back-off).

Both of those sets are 'global' - there isn't one for each possible
destination queue. That means that the set entries themselves contain
information about their destination queue. And in the migration case
above, the destination queue might be the old queue and no longer
listened to.

This adds two Rake tasks (one for the retry set and one for the
scheduled set) to allow administrators to rewrite the job data in those
sorted sets

It uses these Redis commands:

1. ZCARD to get the initial size of the set, for showing to the
   operator. This is O(1) and is called once.
2. ZSCAN to iterate over the sets. This is O(1) per call, and provides
   useful guarantees about iterating over a set that may be changing as
   it's operated on.
3. ZREM to remove the old job hash. This is O(log(N)) per call, where N
   is the number of elements in the set.
4. ZADD to add the new job hash with the new queue name. This is also
   O(log(N)) per call.

ZREM and ZADD will each be called once per item to be migrated, so there
may be many invocations of these commands during this task's run.

Changelog: added
      03b5e73a
  3. 12 May, 2021 39 commits
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7