• Ilya Leoshkevich's avatar
    bpf: Support 64-bit pointers to kfuncs · 1cf3bfc6
    Ilya Leoshkevich authored
    test_ksyms_module fails to emit a kfunc call targeting a module on
    s390x, because the verifier stores the difference between kfunc
    address and __bpf_call_base in bpf_insn.imm, which is s32, and modules
    are roughly (1 << 42) bytes away from the kernel on s390x.
    
    Fix by keeping BTF id in bpf_insn.imm for BPF_PSEUDO_KFUNC_CALLs,
    and storing the absolute address in bpf_kfunc_desc.
    
    Introduce bpf_jit_supports_far_kfunc_call() in order to limit this new
    behavior to the s390x JIT. Otherwise other JITs need to be modified,
    which is not desired.
    
    Introduce bpf_get_kfunc_addr() instead of exposing both
    find_kfunc_desc() and struct bpf_kfunc_desc.
    
    In addition to sorting kfuncs by imm, also sort them by offset, in
    order to handle conflicting imms from different modules. Do this on
    all architectures in order to simplify code.
    
    Factor out resolving specialized kfuncs (XPD and dynptr) from
    fixup_kfunc_call(). This was required in the first place, because
    fixup_kfunc_call() uses find_kfunc_desc(), which returns a const
    pointer, so it's not possible to modify kfunc addr without stripping
    const, which is not nice. It also removes repetition of code like:
    
    	if (bpf_jit_supports_far_kfunc_call())
    		desc->addr = func;
    	else
    		insn->imm = BPF_CALL_IMM(func);
    
    and separates kfunc_desc_tab fixups from kfunc_call fixups.
    Suggested-by: default avatarJiri Olsa <olsajiri@gmail.com>
    Signed-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
    Acked-by: default avatarJiri Olsa <jolsa@kernel.org>
    Link: https://lore.kernel.org/r/20230412230632.885985-1-iii@linux.ibm.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
    1cf3bfc6
verifier.c 553 KB