• Daniel Jordan's avatar
    padata: unbind parallel jobs from specific CPUs · bfde23ce
    Daniel Jordan authored
    Padata binds the parallel part of a job to a single CPU and round-robins
    over all CPUs in the system for each successive job.  Though the serial
    parts rely on per-CPU queues for correct ordering, they're not necessary
    for parallel work, and it improves performance to run the job locally on
    NUMA machines and let the scheduler pick the CPU within a node on a busy
    system.
    
    So, make the parallel workqueue unbound.
    
    Update the parallel workqueue's cpumask when the instance's parallel
    cpumask changes.
    
    Now that parallel jobs no longer run on max_active=1 workqueues, two or
    more parallel works that hash to the same CPU may run simultaneously,
    finish out of order, and so be serialized out of order.  Prevent this by
    keeping the works sorted on the reorder list by sequence number and
    checking that in the reordering logic.
    
    padata_get_next becomes padata_find_next so it can be reused for the end
    of padata_reorder, where it's used to avoid uselessly queueing work when
    the next job by sequence number isn't finished yet but a later job that
    hashed to the same CPU has.
    
    The ENODATA case in padata_find_next no longer makes sense because
    parallel jobs aren't bound to specific CPUs.  The EINPROGRESS case takes
    care of the scenario where a parallel job is potentially running on the
    same CPU as padata_find_next, and with only one error code left, just
    use NULL instead.
    Signed-off-by: default avatarDaniel Jordan <daniel.m.jordan@oracle.com>
    Cc: Herbert Xu <herbert@gondor.apana.org.au>
    Cc: Lai Jiangshan <jiangshanlai@gmail.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Steffen Klassert <steffen.klassert@secunet.com>
    Cc: Tejun Heo <tj@kernel.org>
    Cc: linux-crypto@vger.kernel.org
    Cc: linux-kernel@vger.kernel.org
    Signed-off-by: default avatarHerbert Xu <herbert@gondor.apana.org.au>
    bfde23ce
padata.c 26.8 KB