• Toke Høiland-Jørgensen's avatar
    bpf: Add "live packet" mode for XDP in BPF_PROG_RUN · b530e9e1
    Toke Høiland-Jørgensen authored
    This adds support for running XDP programs through BPF_PROG_RUN in a mode
    that enables live packet processing of the resulting frames. Previous uses
    of BPF_PROG_RUN for XDP returned the XDP program return code and the
    modified packet data to userspace, which is useful for unit testing of XDP
    programs.
    
    The existing BPF_PROG_RUN for XDP allows userspace to set the ingress
    ifindex and RXQ number as part of the context object being passed to the
    kernel. This patch reuses that code, but adds a new mode with different
    semantics, which can be selected with the new BPF_F_TEST_XDP_LIVE_FRAMES
    flag.
    
    When running BPF_PROG_RUN in this mode, the XDP program return codes will
    be honoured: returning XDP_PASS will result in the frame being injected
    into the networking stack as if it came from the selected networking
    interface, while returning XDP_TX and XDP_REDIRECT will result in the frame
    being transmitted out that interface. XDP_TX is translated into an
    XDP_REDIRECT operation to the same interface, since the real XDP_TX action
    is only possible from within the network drivers themselves, not from the
    process context where BPF_PROG_RUN is executed.
    
    Internally, this new mode of operation creates a page pool instance while
    setting up the test run, and feeds pages from that into the XDP program.
    The setup cost of this is amortised over the number of repetitions
    specified by userspace.
    
    To support the performance testing use case, we further optimise the setup
    step so that all pages in the pool are pre-initialised with the packet
    data, and pre-computed context and xdp_frame objects stored at the start of
    each page. This makes it possible to entirely avoid touching the page
    content on each XDP program invocation, and enables sending up to 9
    Mpps/core on my test box.
    
    Because the data pages are recycled by the page pool, and the test runner
    doesn't re-initialise them for each run, subsequent invocations of the XDP
    program will see the packet data in the state it was after the last time it
    ran on that particular page. This means that an XDP program that modifies
    the packet before redirecting it has to be careful about which assumptions
    it makes about the packet content, but that is only an issue for the most
    naively written programs.
    
    Enabling the new flag is only allowed when not setting ctx_out and data_out
    in the test specification, since using it means frames will be redirected
    somewhere else, so they can't be returned.
    Signed-off-by: default avatarToke Høiland-Jørgensen <toke@redhat.com>
    Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
    Acked-by: default avatarMartin KaFai Lau <kafai@fb.com>
    Link: https://lore.kernel.org/bpf/20220309105346.100053-2-toke@redhat.com
    b530e9e1
Kconfig 2.98 KB