Commit b530e9e1 authored by Toke Høiland-Jørgensen's avatar Toke Høiland-Jørgensen Committed by Alexei Starovoitov

bpf: Add "live packet" mode for XDP in BPF_PROG_RUN

This adds support for running XDP programs through BPF_PROG_RUN in a mode
that enables live packet processing of the resulting frames. Previous uses
of BPF_PROG_RUN for XDP returned the XDP program return code and the
modified packet data to userspace, which is useful for unit testing of XDP
programs.

The existing BPF_PROG_RUN for XDP allows userspace to set the ingress
ifindex and RXQ number as part of the context object being passed to the
kernel. This patch reuses that code, but adds a new mode with different
semantics, which can be selected with the new BPF_F_TEST_XDP_LIVE_FRAMES
flag.

When running BPF_PROG_RUN in this mode, the XDP program return codes will
be honoured: returning XDP_PASS will result in the frame being injected
into the networking stack as if it came from the selected networking
interface, while returning XDP_TX and XDP_REDIRECT will result in the frame
being transmitted out that interface. XDP_TX is translated into an
XDP_REDIRECT operation to the same interface, since the real XDP_TX action
is only possible from within the network drivers themselves, not from the
process context where BPF_PROG_RUN is executed.

Internally, this new mode of operation creates a page pool instance while
setting up the test run, and feeds pages from that into the XDP program.
The setup cost of this is amortised over the number of repetitions
specified by userspace.

To support the performance testing use case, we further optimise the setup
step so that all pages in the pool are pre-initialised with the packet
data, and pre-computed context and xdp_frame objects stored at the start of
each page. This makes it possible to entirely avoid touching the page
content on each XDP program invocation, and enables sending up to 9
Mpps/core on my test box.

Because the data pages are recycled by the page pool, and the test runner
doesn't re-initialise them for each run, subsequent invocations of the XDP
program will see the packet data in the state it was after the last time it
ran on that particular page. This means that an XDP program that modifies
the packet before redirecting it has to be careful about which assumptions
it makes about the packet content, but that is only an issue for the most
naively written programs.

Enabling the new flag is only allowed when not setting ctx_out and data_out
in the test specification, since using it means frames will be redirected
somewhere else, so they can't be returned.
Signed-off-by: default avatarToke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
Acked-by: default avatarMartin KaFai Lau <kafai@fb.com>
Link: https://lore.kernel.org/bpf/20220309105346.100053-2-toke@redhat.com
parent 3399dd9f
...@@ -1232,6 +1232,8 @@ enum { ...@@ -1232,6 +1232,8 @@ enum {
/* If set, run the test on the cpu specified by bpf_attr.test.cpu */ /* If set, run the test on the cpu specified by bpf_attr.test.cpu */
#define BPF_F_TEST_RUN_ON_CPU (1U << 0) #define BPF_F_TEST_RUN_ON_CPU (1U << 0)
/* If set, XDP frames will be transmitted after processing */
#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)
/* type for BPF_ENABLE_STATS */ /* type for BPF_ENABLE_STATS */
enum bpf_stats_type { enum bpf_stats_type {
...@@ -1393,6 +1395,7 @@ union bpf_attr { ...@@ -1393,6 +1395,7 @@ union bpf_attr {
__aligned_u64 ctx_out; __aligned_u64 ctx_out;
__u32 flags; __u32 flags;
__u32 cpu; __u32 cpu;
__u32 batch_size;
} test; } test;
struct { /* anonymous struct used by BPF_*_GET_*_ID */ struct { /* anonymous struct used by BPF_*_GET_*_ID */
......
...@@ -30,6 +30,7 @@ config BPF_SYSCALL ...@@ -30,6 +30,7 @@ config BPF_SYSCALL
select TASKS_TRACE_RCU select TASKS_TRACE_RCU
select BINARY_PRINTF select BINARY_PRINTF
select NET_SOCK_MSG if NET select NET_SOCK_MSG if NET
select PAGE_POOL if NET
default n default n
help help
Enable the bpf() system call that allows to manipulate BPF programs Enable the bpf() system call that allows to manipulate BPF programs
......
...@@ -3336,7 +3336,7 @@ static int bpf_prog_query(const union bpf_attr *attr, ...@@ -3336,7 +3336,7 @@ static int bpf_prog_query(const union bpf_attr *attr,
} }
} }
#define BPF_PROG_TEST_RUN_LAST_FIELD test.cpu #define BPF_PROG_TEST_RUN_LAST_FIELD test.batch_size
static int bpf_prog_test_run(const union bpf_attr *attr, static int bpf_prog_test_run(const union bpf_attr *attr,
union bpf_attr __user *uattr) union bpf_attr __user *uattr)
......
This diff is collapsed.
...@@ -1232,6 +1232,8 @@ enum { ...@@ -1232,6 +1232,8 @@ enum {
/* If set, run the test on the cpu specified by bpf_attr.test.cpu */ /* If set, run the test on the cpu specified by bpf_attr.test.cpu */
#define BPF_F_TEST_RUN_ON_CPU (1U << 0) #define BPF_F_TEST_RUN_ON_CPU (1U << 0)
/* If set, XDP frames will be transmitted after processing */
#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)
/* type for BPF_ENABLE_STATS */ /* type for BPF_ENABLE_STATS */
enum bpf_stats_type { enum bpf_stats_type {
...@@ -1393,6 +1395,7 @@ union bpf_attr { ...@@ -1393,6 +1395,7 @@ union bpf_attr {
__aligned_u64 ctx_out; __aligned_u64 ctx_out;
__u32 flags; __u32 flags;
__u32 cpu; __u32 cpu;
__u32 batch_size;
} test; } test;
struct { /* anonymous struct used by BPF_*_GET_*_ID */ struct { /* anonymous struct used by BPF_*_GET_*_ID */
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment