1. 03 Jul, 2019 6 commits
    • Ard Biesheuvel's avatar
      crypto: arm64/aes-ce - implement 5 way interleave for ECB, CBC and CTR · 7367bfeb
      Ard Biesheuvel authored
      This implements 5-way interleaving for ECB, CBC decryption and CTR,
      resulting in a speedup of ~11% on Marvell ThunderX2, which has a
      very deep pipeline and therefore a high issue latency for NEON
      instructions operating on the same registers.
      
      Note that XTS is left alone: implementing 5-way interleave there
      would either involve spilling of the calculated tweaks to the
      stack, or recalculating them after the encryption operation, and
      doing either of those would most likely penalize low end cores.
      
      For ECB, this is not a concern at all, given that we have plenty
      of spare registers. For CTR and CBC decryption, we take advantage
      of the fact that v16 is not used by the CE version of the code
      (which is the only one targeted by the optimization), and so we
      can reshuffle the code a bit and avoid having to spill to memory
      (with the exception of one extra reload in the CBC routine)
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarHerbert Xu <herbert@gondor.apana.org.au>
      7367bfeb
    • Ard Biesheuvel's avatar
      crypto: arm64/aes-ce - add 5 way interleave routines · e2174139
      Ard Biesheuvel authored
      In preparation of tweaking the accelerated AES chaining mode routines
      to be able to use a 5-way stride, implement the core routines to
      support processing 5 blocks of input at a time. While at it, drop
      the 2 way versions, which have been unused for a while now.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarHerbert Xu <herbert@gondor.apana.org.au>
      e2174139
    • Christophe Leroy's avatar
      crypto: talitos - drop icv_ool · d45b1714
      Christophe Leroy authored
      icv_ool is not used anymore, drop it.
      
      Fixes: e345177d ("crypto: talitos - fix AEAD processing.")
      Signed-off-by: default avatarChristophe Leroy <christophe.leroy@c-s.fr>
      Signed-off-by: default avatarHerbert Xu <herbert@gondor.apana.org.au>
      d45b1714
    • Christophe Leroy's avatar
      crypto: talitos - fix hash on SEC1. · 58cdbc6d
      Christophe Leroy authored
      On SEC1, hash provides wrong result when performing hashing in several
      steps with input data SG list has more than one element. This was
      detected with CONFIG_CRYPTO_MANAGER_EXTRA_TESTS:
      
      [   44.185947] alg: hash: md5-talitos test failed (wrong result) on test vector 6, cfg="random: may_sleep use_finup src_divs=[<reimport>25.88%@+8063, <flush>24.19%@+9588, 28.63%@+16333, <reimport>4.60%@+6756, 16.70%@+16281] dst_divs=[71.61%@alignmask+16361, 14.36%@+7756, 14.3%@+"
      [   44.325122] alg: hash: sha1-talitos test failed (wrong result) on test vector 3, cfg="random: inplace use_final src_divs=[<flush,nosimd>16.56%@+16378, <reimport>52.0%@+16329, 21.42%@alignmask+16380, 10.2%@alignmask+16380] iv_offset=39"
      [   44.493500] alg: hash: sha224-talitos test failed (wrong result) on test vector 4, cfg="random: use_final nosimd src_divs=[<reimport>52.27%@+7401, <reimport>17.34%@+16285, <flush>17.71%@+26, 12.68%@+10644] iv_offset=43"
      [   44.673262] alg: hash: sha256-talitos test failed (wrong result) on test vector 4, cfg="random: may_sleep use_finup src_divs=[<reimport>60.6%@+12790, 17.86%@+1329, <reimport>12.64%@alignmask+16300, 8.29%@+15, 0.40%@+13506, <reimport>0.51%@+16322, <reimport>0.24%@+16339] dst_divs"
      
      This is due to two issues:
      - We have an overlap between the buffer used for copying the input
      data (SEC1 doesn't do scatter/gather) and the chained descriptor.
      - Data copy is wrong when the previous hash left less than one
      blocksize of data to hash, implying a complement of the previous
      block with a few bytes from the new request.
      
      Fix it by:
      - Moving the second descriptor after the buffer, as moving the buffer
      after the descriptor would make it more complex for other cipher
      operations (AEAD, ABLKCIPHER)
      - Skip the bytes taken from the new request to complete the previous
      one by moving the SG list forward.
      
      Fixes: 37b5e889 ("crypto: talitos - chain in buffered data for ahash on SEC1")
      Cc: stable@vger.kernel.org
      Signed-off-by: default avatarChristophe Leroy <christophe.leroy@c-s.fr>
      Signed-off-by: default avatarHerbert Xu <herbert@gondor.apana.org.au>
      58cdbc6d
    • Christophe Leroy's avatar
      crypto: talitos - move struct talitos_edesc into talitos.h · d44769e4
      Christophe Leroy authored
      Moves struct talitos_edesc into talitos.h so that it can be used
      from any place in talitos.c
      
      It will be required for next patch ("crypto: talitos - fix hash
      on SEC1")
      Signed-off-by: default avatarChristophe Leroy <christophe.leroy@c-s.fr>
      Cc: stable@vger.kernel.org
      Signed-off-by: default avatarHerbert Xu <herbert@gondor.apana.org.au>
      d44769e4
    • Christophe Leroy's avatar
      lib/scatterlist: Fix mapping iterator when sg->offset is greater than PAGE_SIZE · aeb87246
      Christophe Leroy authored
      All mapping iterator logic is based on the assumption that sg->offset
      is always lower than PAGE_SIZE.
      
      But there are situations where sg->offset is such that the SG item
      is on the second page. In that case sg_copy_to_buffer() fails
      properly copying the data into the buffer. One of the reason is
      that the data will be outside the kmapped area used to access that
      data.
      
      This patch fixes the issue by adjusting the mapping iterator
      offset and pgoffset fields such that offset is always lower than
      PAGE_SIZE.
      Signed-off-by: default avatarChristophe Leroy <christophe.leroy@c-s.fr>
      Fixes: 4225fc85 ("lib/scatterlist: use page iterator in the mapping iterator")
      Cc: stable@vger.kernel.org
      Signed-off-by: default avatarHerbert Xu <herbert@gondor.apana.org.au>
      aeb87246
  2. 28 Jun, 2019 1 commit
  3. 27 Jun, 2019 10 commits
  4. 20 Jun, 2019 15 commits
  5. 13 Jun, 2019 8 commits