• Florent Revest's avatar
    bpf: Implement formatted output helpers with bstr_printf · 48cac3f4
    Florent Revest authored
    BPF has three formatted output helpers: bpf_trace_printk, bpf_seq_printf
    and bpf_snprintf. Their signatures specify that all arguments are
    provided from the BPF world as u64s (in an array or as registers). All
    of these helpers are currently implemented by calling functions such as
    snprintf() whose signatures take a variable number of arguments, then
    placed in a va_list by the compiler to call vsnprintf().
    
    "d9c9e4db bpf: Factorize bpf_trace_printk and bpf_seq_printf" introduced
    a bpf_printf_prepare function that fills an array of u64 sanitized
    arguments with an array of "modifiers" which indicate what the "real"
    size of each argument should be (given by the format specifier). The
    BPF_CAST_FMT_ARG macro consumes these arrays and casts each argument to
    its real size. However, the C promotion rules implicitely cast them all
    back to u64s. Therefore, the arguments given to snprintf are u64s and
    the va_list constructed by the compiler will use 64 bits for each
    argument. On 64 bit machines, this happens to work well because 32 bit
    arguments in va_lists need to occupy 64 bits anyway, but on 32 bit
    architectures this breaks the layout of the va_list expected by the
    called function and mangles values.
    
    In "88a5c690 bpf: fix bpf_trace_printk on 32 bit archs", this problem
    had been solved for bpf_trace_printk only with a "horrid workaround"
    that emitted multiple calls to trace_printk where each call had
    different argument types and generated different va_list layouts. One of
    the call would be dynamically chosen at runtime. This was ok with the 3
    arguments that bpf_trace_printk takes but bpf_seq_printf and
    bpf_snprintf accept up to 12 arguments. Because this approach scales
    code exponentially, it is not a viable option anymore.
    
    Because the promotion rules are part of the language and because the
    construction of a va_list is an arch-specific ABI, it's best to just
    avoid variadic arguments and va_lists altogether. Thankfully the
    kernel's snprintf() has an alternative in the form of bstr_printf() that
    accepts arguments in a "binary buffer representation". These binary
    buffers are currently created by vbin_printf and used in the tracing
    subsystem to split the cost of printing into two parts: a fast one that
    only dereferences and remembers values, and a slower one, called later,
    that does the pretty-printing.
    
    This patch refactors bpf_printf_prepare to construct binary buffers of
    arguments consumable by bstr_printf() instead of arrays of arguments and
    modifiers. This gets rid of BPF_CAST_FMT_ARG and greatly simplifies the
    bpf_printf_prepare usage but there are a few gotchas that change how
    bpf_printf_prepare needs to do things.
    
    Currently, bpf_printf_prepare uses a per cpu temporary buffer as a
    generic storage for strings and IP addresses. With this refactoring, the
    temporary buffers now holds all the arguments in a structured binary
    format.
    
    To comply with the format expected by bstr_printf, certain format
    specifiers also need to be pre-formatted: %pB and %pi6/%pi4/%pI4/%pI6.
    Because vsnprintf subroutines for these specifiers are hard to expose,
    we pre-format these arguments with calls to snprintf().
    Reported-by: default avatarRasmus Villemoes <linux@rasmusvillemoes.dk>
    Signed-off-by: default avatarFlorent Revest <revest@chromium.org>
    Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
    Link: https://lore.kernel.org/bpf/20210427174313.860948-3-revest@chromium.org
    48cac3f4
bpf_trace.c 52.2 KB