• Vincent Mailhol's avatar
    x86/asm/bitops: Use __builtin_ctzl() to evaluate constant expressions · fdb6649a
    Vincent Mailhol authored
    If x is not 0, __ffs(x) is equivalent to:
      (unsigned long)__builtin_ctzl(x)
    And if x is not ~0UL, ffz(x) is equivalent to:
      (unsigned long)__builtin_ctzl(~x)
    Because __builting_ctzl() returns an int, a cast to (unsigned long) is
    necessary to avoid potential warnings on implicit casts.
    
    Concerning the edge cases, __builtin_ctzl(0) is always undefined,
    whereas __ffs(0) and ffz(~0UL) may or may not be defined, depending on
    the processor. Regardless, for both functions, developers are asked to
    check against 0 or ~0UL so replacing __ffs() or ffz() by
    __builting_ctzl() is safe.
    
    For x86_64, the current __ffs() and ffz() implementations do not
    produce optimized code when called with a constant expression. On the
    contrary, the __builtin_ctzl() folds into a single instruction.
    
    However, for non constant expressions, the __ffs() and ffz() asm
    versions of the kernel remains slightly better than the code produced
    by GCC (it produces a useless instruction to clear eax).
    
    Use __builtin_constant_p() to select between the kernel's
    __ffs()/ffz() and the __builtin_ctzl() depending on whether the
    argument is constant or not.
    
    ** Statistics **
    
    On a allyesconfig, before...:
    
      $ objdump -d vmlinux.o | grep tzcnt | wc -l
      3607
    
    ...and after:
    
      $ objdump -d vmlinux.o | grep tzcnt | wc -l
      2600
    
    So, roughly 27.9% of the calls to either __ffs() or ffz() were using
    constant expressions and could be optimized out.
    
    (tests done on linux v5.18-rc5 x86_64 using GCC 11.2.1)
    
    Note: on x86_64, the BSF instruction produces TZCNT when used with the
    REP prefix (which explain the use of `grep tzcnt' instead of `grep bsf'
    in above benchmark). c.f. [1]
    
    [1] e26a44a2 ("x86: Use REP BSF unconditionally")
    
      [ bp: Massage commit message. ]
    Signed-off-by: default avatarVincent Mailhol <mailhol.vincent@wanadoo.fr>
    Signed-off-by: default avatarBorislav Petkov <bp@suse.de>
    Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
    Reviewed-by: default avatarYury Norov <yury.norov@gmail.com>
    Link: https://lore.kernel.org/r/20220511160319.1045812-1-mailhol.vincent@wanadoo.fr
    fdb6649a
bitops.h 10.8 KB