- 25 Aug, 2021 22 commits
-
-
Magnus Karlsson authored
Preface all options with opt_ and make them booleans. Signed-off-by:
Magnus Karlsson <magnus.karlsson@intel.com> Signed-off-by:
Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/20210825093722.10219-17-magnus.karlsson@gmail.com
-
Magnus Karlsson authored
Make enums lower case as that is the standard. Also drop the unnecessary TEST_MODE_UNCONFIGURED mode. Signed-off-by:
-
Magnus Karlsson authored
Generate packets from a specification instead of something hard coded. The idea is that a test generates one or more packet specifications and provides it/them to both Tx and Rx. The Tx thread will generate from this specification and Rx will validate that it receives what is in the specification. The specification can be the same on both ends, meaning that everything that was sent should be received, or different which means that Rx will only receive part of the sent packets. Currently, the packet specification is the same for both Rx and Tx and the same for each test. This will change in later work as features and tests are added. The data path functions are also renamed to better reflect what actions they are performing after introducing this feature. Signed-off-by:
-
Magnus Karlsson authored
Generate the packet directly in the umem instead of in a temporary buffer that is copied out. Simplifies the code and improves performance. Signed-off-by:
-
Magnus Karlsson authored
Simpify the cleanup of ifobjects right before the program exits by introducing functions for creating and destroying these objects. Signed-off-by:
-
Magnus Karlsson authored
Decrease sending speed to avoid potentially overflowing some buffers in the skb case that leads to dropped packets we cannot control (and thus the tests may generate false negatives). Decrease batch size and introduce a usleep in the transmit thread to not overflow the receiver. Signed-off-by:
-
Magnus Karlsson authored
Validate the tx stats on the Tx thread instead of the Rx thread. Depending on your settings, you might not be allowed to query the statistics of a socket you do not own, so better to do this on the correct thread to start with. Signed-off-by:
-
Magnus Karlsson authored
Simplify packet validation in the xsk selftests by performing it at once for every packet. The current code performed this per batch and did this on copied packet data. Make it simpler and faster by validating it at once and on the umem packet data thus skipping the copy and the memory allocation for the temprary buffer. The optional packet dump feature is also simplified in the same manner. Memory allocation and copying is removed and the dump is performed directly on the umem data. Signed-off-by:
-
Magnus Karlsson authored
Rename worker_* functions that are not thread entry points to something else. This was confusing. Now only thread entry points are worker_something. Signed-off-by:
-
Magnus Karlsson authored
Disassociate the number of packets sent with the number of buffers in the umem. This so we can loop over the umem to test more things. Set the size of the umem to be a multiple of 2M. A requirement for huge pages that are needed in unaligned mode. Signed-off-by:
-
Magnus Karlsson authored
Get rid of the end-of-test packet and just count the number of packets received and quit when the expected number as been received. Simplifies the code. Signed-off-by:
-
Magnus Karlsson authored
Simplify the retry code and make it more efficient by waiting first, instead of trying immediately which always fails due to the asynchronous nature of xsk socket close. Also decrease the wait time to significantly lower the run-time of the test suite. Signed-off-by:
-
Magnus Karlsson authored
Return the correct error codes so they can be printed correctly. Signed-off-by:
-
Magnus Karlsson authored
Remove unused variables and typedefs. The *_npkts variables are incremented but never used. Signed-off-by:
-
Magnus Karlsson authored
Remove the number of tx packet option as this should be decided by the test itself. Also change the number of packets to be sent to 4096 speeding up the execution. Signed-off-by:
-
Magnus Karlsson authored
Remove color mode since it does not add any value and having less code means less maintenance which is a good thing. Signed-off-by:
-
Alexei Starovoitov authored
Daniel Xu says: ==================== The motivation behind this helper is to access userspace pt_regs in a kprobe handler. uprobe's ctx is the userspace pt_regs. kprobe's ctx is the kernelspace pt_regs. bpf_task_pt_regs() allows accessing userspace pt_regs in a kprobe handler. The final case (kernelspace pt_regs in uprobe) is pretty rare (usermode helper) so I think that can be solved later if necessary. More concretely, this helper is useful in doing BPF-based DWARF stack unwinding. Currently the kernel can only do framepointer based stack unwinds for userspace code. This is because the DWARF state machines are too fragile to be computed in kernelspace [0]. The idea behind DWARF-based stack unwinds w/ BPF is to copy a chunk of the userspace stack (while in prog context) and send it up to userspace for unwinding (probably with libunwind) [1]. This would effectively enable profiling applications with -fomit-frame-pointer using kprobes and uprobes. [0]: https://lkml.org/lkml/2012/2/10/356 [1]: https://github.com/danobi/bpf-dwarf-walk Changes from v1: - Conwolidate BTF_ID decls for task_struct - Enable bpf_get_current_task_btf() for all prog types - Enable bpf_task_pt_regs() for all prog types - Use ASSERT_* macros instead of CHECK ==================== Signed-off-by:
-
Daniel Xu authored
This test retrieves the uprobe's pt_regs in two different ways and compares the contents in an arch-agnostic way. Signed-off-by:
Daniel Xu <dxu@dxuuu.xyz> Signed-off-by:
-
Daniel Xu authored
The motivation behind this helper is to access userspace pt_regs in a kprobe handler. uprobe's ctx is the userspace pt_regs. kprobe's ctx is the kernelspace pt_regs. bpf_task_pt_regs() allows accessing userspace pt_regs in a kprobe handler. The final case (kernelspace pt_regs in uprobe) is pretty rare (usermode helper) so I think that can be solved later if necessary. More concretely, this helper is useful in doing BPF-based DWARF stack unwinding. Currently the kernel can only do framepointer based stack unwinds for userspace code. This is because the DWARF state machines are too fragile to be computed in kernelspace [0]. The idea behind DWARF-based stack unwinds w/ BPF is to copy a chunk of the userspace stack (while in prog context) and send it up to userspace for unwinding (probably with libunwind) [1]. This would effectively enable profiling applications with -fomit-frame-pointer using kprobes and uprobes. [0]: https://lkml.org/lkml/2012/2/10/356 [1]: https://github.com/danobi/bpf-dwarf-walkSigned-off-by:
-
Daniel Xu authored
bpf_get_current_task() is already supported so it's natural to also include the _btf() variant for btf-powered helpers. This is required for non-tracing progs to use bpf_task_pt_regs() in the next commit. Signed-off-by:
-
Daniel Xu authored
No need to have it defined 5 times. Once is enough. Signed-off-by:
-
Daniel Xu authored
Same as BTF_ID_LIST_SINGLE macro except defines a global ID. Signed-off-by:
-
- 24 Aug, 2021 18 commits
-
-
Alexei Starovoitov authored
Kumar Kartikeya says: ==================== This set revamps XDP samples related to redirection to show better output and implement missing features consolidating all their differences and giving them a consistent look and feel, by implementing common features and command line options. Some of the TODO items like reporting redirect error numbers (ENETDOWN, EINVAL, ENOSPC, etc.) have also been implemented. Some of the features are: * Received packet statistics * xdp_redirect/xdp_redirect_map tracepoint statistics * xdp_redirect_err/xdp_redirect_map_err tracepoint statistics (with support for showing exact errno) * xdp_cpumap_enqueue/xdp_cpumap_kthread tracepoint statistics * xdp_devmap_xmit tracepoint statistics * xdp_exception tracepoint statistics * Per ifindex pair devmap_xmit stats shown dynamically (for xdp_monitor) to decompose the total. * Use of BPF skeleton and BPF static linking to share BPF programs. * Use of vmlinux.h and tp_btf for raw_tracepoint support. * Removal of redundant -N/--native-mode option (enforced by default now) * ... and massive cleanups all over the place. All tracepoints also use raw_tp now, and tracepoints like xdp_redirect are only enabled when requested explicitly to capture successful redirection statistics. The set of programs converted as part of this series are: * xdp_redirect_cpu * xdp_redirect_map_multi * xdp_redirect_map * xdp_redirect * xdp_monitor Explanation of the output: There is now a concise output mode by default that shows primarily four fields: rx/s Number of packets received per second redir/s Number of packets successfully redirected per second err,drop/s Aggregated count of errors per second (including dropped packets) xmit/s Number of packets transmitted on the output device per second Some examples: ; sudo ./xdp_redirect_map veth0 veth1 -s Redirecting from veth0 (ifindex 15; driver veth) to veth1 (ifindex 14; driver veth) veth0->veth1 0 rx/s 0 redir/s 0 err,drop/s 0 xmit/s veth0->veth1 9,998,660 rx/s 9,998,658 redir/s 0 err,drop/s 9,998,654 xmit/s ... There is also a verbose mode, that can also be enabled by default using -v (--verbose). The output mode can be switched dynamically at runtime using Ctrl + \ (SIGQUIT). To make the concise output more useful, the errors that occur are expanded inline (as if verbose mode was enabled) to let the user pin down the source of the problem without having to clutter output (or possibly miss it) or always use verbose mode. For instance, let's consider a case where the output device link state is set to down while redirection is happening: [...] veth0->veth1 24,503,376 rx/s 0 err,drop/s 24,503,372 xmit/s veth0->veth1 25,044,775 rx/s 0 err,drop/s 25,044,783 xmit/s veth0->veth1 25,263,046 rx/s 4 err,drop/s 25,263,028 xmit/s redirect_err 4 error/s ENETDOWN 4 error/s [...] The same holds for xdp_exception actions. An example of how a complete xdp_redirect_map session would look: ; sudo ./xdp_redirect_map veth0 veth1 Redirecting from veth0 (ifindex 5; driver veth) to veth1 (ifindex 4; driver veth) veth0->veth1 7,411,506 rx/s 0 err,drop/s 7,411,470 xmit/s veth0->veth1 8,931,770 rx/s 0 err,drop/s 8,931,771 xmit/s ^\ veth0->veth1 8,787,295 rx/s 0 err,drop/s 8,787,325 xmit/s receive total 8,787,295 pkt/s 0 drop/s 0 error/s cpu:7 8,787,295 pkt/s 0 drop/s 0 error/s redirect_err 0 error/s xdp_exception 0 hit/s xmit veth0->veth1 8,787,325 xmit/s 0 drop/s 0 drv_err/s 2.00 bulk-avg cpu:7 8,787,325 xmit/s 0 drop/s 0 drv_err/s 2.00 bulk-avg veth0->veth1 8,842,610 rx/s 0 err,drop/s 8,842,606 xmit/s receive total 8,842,610 pkt/s 0 drop/s 0 error/s cpu:7 8,842,610 pkt/s 0 drop/s 0 error/s redirect_err 0 error/s xdp_exception 0 hit/s xmit veth0->veth1 8,842,606 xmit/s 0 drop/s 0 drv_err/s 2.00 bulk-avg cpu:7 8,842,606 xmit/s 0 drop/s 0 drv_err/s 2.00 bulk-avg ^C Packets received : 33,973,181 Average packets/s : 4,246,648 Packets transmitted : 33,973,172 Average transmit/s : 4,246,647 The xdp_redirect tracepoint (for success stats) needs to be enabled explicitly using --stats/-s. Documentation for entire output and options is provided when user specifies --help/-h with a sample. Changelog: ---------- v3 -> v4: v3: https://lore.kernel.org/bpf/20210728165552.435050-1-memxor@gmail.com * Address all feedback from Daniel * Use READ_ONCE/WRITE_ONCE from linux/compiler.h (cannot directly include due to conflicts with vmlinux.h) * Fix MAX_CPUS hardcoding by switching to mmapable array maps, that are resized based on the value of libbpf_num_possible_cpus * s/ELEMENTS_OF/ARRAY_SIZE/g * Use tools/include/linux/hashtable.h * Coding style fixes * Remove hyperlinks for tracepoints * Split into smaller reviewable changes * Restore support for specifying custom xdp_redirect_cpu cpumap prog with some enhancements, including built-in programs for common actions (pass, drop, redirect). By default, cpumap prog is now disabled. * Misc bug fixes all over the place The printing stuff is a lot more basic without hyperlink support, hence it has not been exported into a more general facility. v2 -> v3 v2: https://lore.kernel.org/bpf/20210721212833.701342-1-memxor@gmail.com * Address all feedback from Andrii * Replace usage of libbpf hashmap (internal API) with custom one * Rename ATOMIC_* macros to NO_TEAR_* to better reflect their use * Use size_t as a portable word sized data type * Set libbpf_set_strict_mode * Invert conditions in BPF programs to exit early and reduce nesting * Use canonical SEC("xdp") naming for all XDP BPF progams * Add missing help description for cpumap enqueue and kthread tracepoints * Move private struct declarations from xdp_sample_user.h to .c file * Improve help output for cpumap enqueue and cpumap kthread tracepoints * Fix a bug where keys array for BPF_MAP_LOOKUP_BATCH is overallocated * Fix some conditions for printing stats (earlier only checked pps, now pps, drop, err and print if any is greater than zero) * Fix alloc_stats_record to properly return and cleanup allocated memory on allocation failure instead of calling exit(3) * Bump bpf_map_lookup_batch count to 32 to reduce lookup time with multiple devices in map * Fix a bug where devmap_xmit_multi stats are not printed when previous record is missing (i.e. when the first time stats are printed), by simply using a dummy record that is zeroed out * Also print per-CPU counts for devmap_xmit_multi which we collect already * Change mac_map to be BPF_MAP_TYPE_HASH instead of array to prevent resizing to a large size when max_ifindex is high, in xdp_redirect_map_multi * Fix instance of strerror(errno) in sample_install_xdp to use saved errno * Provide a usage function from samples helper * Provide a fix where incorrect stats are shown for parallel sessions of xdp_redirect_* samples by introducing matching support for input device(s), output device(s) and cpumap map id for enqueue and kthread stats. Only xdp_monitor doesn't filter stats, all others do. RFC (v1) -> v2 RFC (v1): https://lore.kernel.org/bpf/20210528235250.2635167-1-memxor@gmail.com * Address all feedback from Andrii * Use BPF static linking * Use vmlinux.h * Use BPF_PROG macro * Use global variables instead of maps * Use of tp_btf for raw_tracepoint progs * Switch to timerfd for polling * Use libbpf hashmap for maintaing device sets for per ifindex pair devmap_xmit stats * Fix Makefile to specify object dependencies properly * Use in-tree bpftool * ... misc fixes and cleanups all over the place ==================== Signed-off-by: -
Kumar Kartikeya Dwivedi authored
Use the libbpf skeleton facility and other utilities provided by XDP samples helper. Also adapt to change of type of mac address map, so that no resizing is required. Add a new flag for sample mask that skips priting the from_device->to_device heading for each line, as xdp_redirect_map_multi may have two devices but the flow of data may be bidirectional, so the output would be confusing. Signed-off-by:
Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
One of the notable changes is using a BPF_MAP_TYPE_HASH instead of array map to store mac addresses of devices, as the resizing behavior was based on max_ifindex, which unecessarily maximized the capacity of map beyond what was needed. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
Use the libbpf skeleton facility and other utilities provided by XDP samples helper. Since get_mac_addr is already provided by XDP samples helper, we drop it. Also convert to XDP samples helper similar to prior samples to minimize duplication of code. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
Also update it to use consistent SEC("xdp") and SEC("xdp_devmap") naming, and use global variable instead of BPF map for copying the mac address. Signed-off-by: -
Kumar Kartikeya Dwivedi authored
Use the libbpf skeleton facility and other utilities provided by XDP samples helper. Similar to xdp_monitor, xdp_redirect_cpu was quite featureful except a few minor omissions (e.g. redirect errno reporting). All of these have been moved to XDP samples helper, hence drop the unneeded code and convert to usage of helpers provided by it. One of the important changes here is dropping of mprog-disable option, as we make that the default. Also, we support built-in programs for some common actions on the packet when it reaches kthread (pass, drop, redirect to device). If the user still needs to install a custom program, they can still supply a BPF object, however the program should be suitably tagged with SEC("xdp_cpumap") annotation so that the expected attach type is correct when updating our cpumap map element. Signed-off-by: -
Kumar Kartikeya Dwivedi authored
Similar to xdp_monitor_kern, a lot of these BPF programs have been reimplemented properly consolidating missing features from other XDP samples. Hence, drop the unneeded code and rename to .bpf.c suffix. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
Use the libbpf skeleton facility and other utilities provided by XDP samples helper. One important note: The XDP samples helper handles ownership of installed XDP programs on devices, including responding to SIGINT and SIGTERM, so drop the code here and use the helpers we provide going forward for all xdp_redirect* conversions. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
We moved swap_src_dst_mac to xdp_sample.bpf.h to be shared with other potential users, so drop it while moving code to the new file. Also, consistently use SEC("xdp") naming instead. Signed-off-by: -
Kumar Kartikeya Dwivedi authored
Use the libbpf skeleton facility and other utilities provided by XDP samples helper. A lot of the code in xdp_monitor and xdp_redirect_cpu has been moved to the xdp_sample_user.o helper, so we remove the duplicate functions here that are no longer needed. Thanks to BPF skeleton, we no longer depend on order of tracepoints to uninstall them on startup. Instead, the sample mask is used to install the needed tracepoints. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
We already moved all the functionality it provided in XDP samples helper userspace and kernel BPF object, so just delete the unneeded code. We also add generation of BPF skeleton and compilation using clang -target bpf for files ending with .bpf.c suffix (to denote that they use vmlinux.h). Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
Also, take this opportunity to depend on in-tree bpftool, so that we can use static linking support in subsequent commits for XDP samples BPF helper object. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
This adds support for retrieval and printing for devmap_xmit total and mutli mode tracepoint. For multi mode, we keep a hash map entry for each redirection stream, such that we can dynamically add and remove entries on output. The from_match and to_match will be set by individual samples when setting up the XDP program on these devices. The multi mode tracepoint is also handy for xdp_redirect_map_multi, where up to 32 devices can be specified. Also add samples_init_pre_load macro to finally set up the resized maps and mmap them in place for low overhead stats retrieval. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
This adds support for the devmap_xmit tracepoint, and its multi device variant that can be used to obtain streams for each individual net_device to net_device redirection. This is useful for decomposing total xmit stats in xdp_monitor. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
This consolidates retrieval and printing into the XDP sample helper. For the kthread stats, it expands xdp_stats separately with its own per-CPU stats. For cpumap enqueue, we display FROM->TO stats also with its per-CPU stats. The help out explains in detail the various aspects of the output. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
These are invoked in two places, when the XDP frame or SKB (for generic XDP) enqueued to the ptr_ring (cpumap_enqueue) and when kthread processes the frame after invoking the CPUMAP program for it (returning stats for the batch). We use cpumap_map_id to filter on the map_id as a way to avoid printing incorrect stats for parallel sessions of xdp_redirect_cpu. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
This implements the retrieval and printing, as well the help output. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
This would allow us to store stats for each XDP action, including their per-CPU counts. Consolidating this here allows all redirect samples to detect xdp_exception events. Signed-off-by:
-
