• Vincent Pelletier's avatar
    Products.CMFActivity: Distribute all activities pending distribution · 4cbc4eea
    Vincent Pelletier authored
    Activity distribution (actually: validation) is fundamentally a bottleneck
    of the current CMFActivity design: there can be only one distribution/
    validation node, and it must check every single activity which express a
    dependency over any activity, until these dependencies are satisfied.
    As a result, distribution/validation is in the critical path between an
    activity being spawned and it being executed, and this work cannot be
    parallelised. So care should be taken to waste as little time as possible,
    in order to reduce the activity execution latency.
    
    Before this change, CMFActivity would distribute at most 1000 activities
    (MAX_VALIDATED_LIMIT) per queue per timerserver wake-up.
    In a typical ERP5 setup, timerserver ticks once per second, which means
    CMFActivity was unable to validate more than 1000 activities per second
    per queue. Maybe there are more activities which are possible to validate
    but still the code was forcing the node to sleep until the next wake-up,
    which is a tremendous waste of time.
    
    This change fixes that issue by having ActivityTool.distribute keep looping
    until there has been an iteration over all activity queues which
    simultaneously did not find any activity they could validate. This lets
    Zope yield CPU control when it would be better spent processing activities
    (those which are preventing the validation of any activities remaining to
    validate) without imposing a maximum effective validation-per-second
    hard limit.
    4cbc4eea
SQLBase.py 48.2 KB