• Sean McGivern's avatar
    Allow migrating scheduled and retried Sidekiq jobs to new queues · 03b5e73a
    Sean McGivern authored
    We now allow Sidekiq worker routing to be configured by administrators.
    For example, they can say 'all jobs go to the default queue', or
    'project export and import workers share a queue'. Right now, the only
    really useful case is to re-route jobs to the default queue, but we will
    support other options in future.
    
    Migrating this sounds simple: listen to the old and new queues, update
    the worker routing, wait for the old queue to be empty, and stop
    listening to the old queue. But there's a catch: Sidekiq maintains two
    sorted sets with jobs that are to be run in the future. There is the
    scheduled set (for jobs that use `perform_in` or `perform_at` or
    similar, where we choose to run a job in the future) and the retry
    set (after failing, a job will get retried with some back-off).
    
    Both of those sets are 'global' - there isn't one for each possible
    destination queue. That means that the set entries themselves contain
    information about their destination queue. And in the migration case
    above, the destination queue might be the old queue and no longer
    listened to.
    
    This adds two Rake tasks (one for the retry set and one for the
    scheduled set) to allow administrators to rewrite the job data in those
    sorted sets
    
    It uses these Redis commands:
    
    1. ZCARD to get the initial size of the set, for showing to the
       operator. This is O(1) and is called once.
    2. ZSCAN to iterate over the sets. This is O(1) per call, and provides
       useful guarantees about iterating over a set that may be changing as
       it's operated on.
    3. ZREM to remove the old job hash. This is O(log(N)) per call, where N
       is the number of elements in the set.
    4. ZADD to add the new job hash with the new queue name. This is also
       O(log(N)) per call.
    
    ZREM and ZADD will each be called once per item to be migrated, so there
    may be many invocations of these commands during this task's run.
    
    Changelog: added
    03b5e73a
allow-migrating-sidekiq-scheduled-and-retry-jobs.yml 117 Bytes