- 25 Nov, 2019 24 commits
-
-
Daniel Borkmann authored
Add tracking of constant keys into tail call maps. The signature of bpf_tail_call_proto is that arg1 is ctx, arg2 map pointer and arg3 is a index key. The direct call approach for tail calls can be enabled if the verifier asserted that for all branches leading to the tail call helper invocation, the map pointer and index key were both constant and the same. Tracking of map pointers we already do from prior work via c93552c4 ("bpf: properly enforce index mask to prevent out-of-bounds speculation") and 09772d92 ("bpf: avoid retpoline for lookup/update/ delete calls on maps"). Given the tail call map index key is not on stack but directly in the register, we can add similar tracking approach and later in fixup_bpf_calls() add a poke descriptor to the progs poke_tab with the relevant information for the JITing phase. We internally reuse insn->imm for the rewritten BPF_JMP | BPF_TAIL_CALL instruction in order to point into the prog's poke_tab, and keep insn->imm as 0 as indicator that current indirect tail call emission must be used. Note that publishing to the tracker must happen at the end of fixup_bpf_calls() since adding elements to the poke_tab reallocates its memory, so we need to wait until its in final state. Future work can generalize and add similar approach to optimize plain array map lookups. Difference there is that we need to look into the key value that sits on stack. For clarity in bpf_insn_aux_data, map_state has been renamed into map_ptr_state, so we get map_{ptr,key}_state as trackers. Signed-off-by:
Daniel Borkmann <daniel@iogearbox.net> Signed-off-by:
Alexei Starovoitov <ast@kernel.org> Acked-by:
Andrii Nakryiko <andriin@fb.com> Link: https://lore.kernel.org/bpf/e8db37f6b2ae60402fa40216c96738ee9b316c32.1574452833.git.daniel@iogearbox.net
-
Daniel Borkmann authored
This work adds program tracking to prog array maps. This is needed such that upon prog array updates/deletions we can fix up all programs which make use of this tail call map. We add ops->map_poke_{un,}track() helpers to maps to maintain the list of programs and ops->map_poke_run() for triggering the actual update. bpf_array_aux is extended to contain the list head and poke_mutex in order to serialize program patching during updates/deletions. bpf_free_used_maps() will untrack the program shortly before dropping the reference to the map. For clearing out the prog array once all urefs are dropped we need to use schedule_work() to have a sleepable context. The prog_array_map_poke_run() is triggered during updates/deletions and walks the maintained prog list. It checks in their poke_tabs whether the map and key is matching and runs the actual bpf_arch_text_poke() for patching in the nop or new jmp location. Depending on the type of update, we use one of BPF_MOD_{NOP_TO_JUMP,JUMP_TO_NOP,JUMP_TO_JUMP}. Signed-off-by: -
Daniel Borkmann authored
Add initial poke table data structures and management to the BPF prog that can later be used by JITs. Also add an instance of poke specific data for tail call maps; plan for later work is to extend this also for BPF static keys. Signed-off-by:
-
Daniel Borkmann authored
We're going to extend this with further information which is only relevant for prog array at this point. Given this info is not used in critical path, move it into its own structure such that the main array map structure can be kept on diet. Signed-off-by:
-
Daniel Borkmann authored
We later on are going to need a sleepable context as opposed to plain RCU callback in order to untrack programs we need to poke at runtime and tracking as well as image update is performed under mutex. Signed-off-by:
-
Daniel Borkmann authored
Add BPF_MOD_{NOP_TO_JUMP,JUMP_TO_JUMP,JUMP_TO_NOP} patching for x86 JIT in order to be able to patch direct jumps or nop them out. We need this facility in order to patch tail call jumps and in later work also BPF static keys. Signed-off-by: -
Alexei Starovoitov authored
Add a test that benchmarks different ways of attaching BPF program to a kernel function. Here are the results for 2.4Ghz x86 cpu on a kernel without mitigations: $ ./test_progs -n 49 -v|grep events task_rename base 2743K events per sec task_rename kprobe 2419K events per sec task_rename kretprobe 1876K events per sec task_rename raw_tp 2578K events per sec task_rename fentry 2710K events per sec task_rename fexit 2685K events per sec On a kernel with retpoline: $ ./test_progs -n 49 -v|grep events task_rename base 2401K events per sec task_rename kprobe 1930K events per sec task_rename kretprobe 1485K events per sec task_rename raw_tp 2053K events per sec task_rename fentry 2351K events per sec task_rename fexit 2185K events per sec All 5 approaches: - kprobe/kretprobe in __set_task_comm() - raw tracepoint in trace_task_rename() - fentry/fexit in __set_task_comm() are roughly equivalent. __set_task_comm() by itself is quite fast, so any extra instructions add up. Until BPF trampoline was introduced the fastest mechanism was raw tracepoint. kprobe via ftrace was second best. kretprobe is slow due to trap. New fentry/fexit methods via BPF trampoline are clearly the fastest and the difference is more pronounced with retpoline on, since BPF trampoline doesn't use indirect jumps. Signed-off-by:
John Fastabend <john.fastabend@gmail.com> Link: https://lore.kernel.org/bpf/20191122011515.255371-1-ast@kernel.org
-
Alexei Starovoitov authored
Yonghong Song says: ==================== With latest llvm, bpf selftest test_progs, which has +alu32 enabled, failed for strobemeta.o and a few other subtests. The reason is due to that verifier did not provide better var_off.mask after jmp32 instructions. This patch set addressed this issue and after the fix, test_progs passed with alu32. Patch #1 provided detailed explanation of the problem and the fix. Patch #2 added three tests in test_verifier. Changelog: v1 -> v2: - do not directly manipulate tnum.{value,mask} in __reg_bound_offset32(), using tnum_lshift/tnum_rshift functions instead - do __reg_bound_offset32() after regular 64bit __reg_bound_offset() since the latter may give a better upper 32bit var_off, which can be inherited by __reg_bound_offset32(). ==================== Signed-off-by: -
Andrii Nakryiko authored
test_core_reloc_kernel.c selftest is the only CO-RE test that reads and returns for validation calling thread's information (pid, tgid, comm). Thus it has to make sure that only test_prog's invocations are honored. Fixes: df36e621 ("selftests/bpf: add CO-RE relocs testing setup") Reported-by:
-
Yonghong Song authored
Three test cases are added. Test 1: jmp32 'reg op imm'. Test 2: jmp32 'reg op reg' where dst 'reg' has unknown constant and src 'reg' has known constant Test 3: jmp32 'reg op reg' where dst 'reg' has known constant and src 'reg' has unknown constant Signed-off-by:Yonghong Song <yhs@fb.com> Signed-off-by:
-
Andrii Nakryiko authored
If bpf_object__open_file() gets path like "some/dir/obj.o", it should derive BPF object's name as "obj" (unless overriden through opts->object_name). Instead, due to using `path` as a fallback value for opts->obj_name, path is used as is for object name, so for above example BPF object's name will be verbatim "some/dir/obj", which leads to all sorts of troubles, especially when internal maps are concern (they are using up to 8 characters of object name). Fix that by ensuring object_name stays NULL, unless overriden. Fixes: 291ee02b ("libbpf: Refactor bpf_object__open APIs to use common opts") Signed-off-by:
-
Yonghong Song authored
With latest llvm (trunk https://github.com/llvm/llvm-project), test_progs, which has +alu32 enabled, failed for strobemeta.o. The verifier output looks like below with edit to replace large decimal numbers with hex ones. 193: (85) call bpf_probe_read_user_str#114 R0=inv(id=0) 194: (26) if w0 > 0x1 goto pc+4 R0_w=inv(id=0,umax_value=0xffffffff00000001) 195: (6b) *(u16 *)(r7 +80) = r0 196: (bc) w6 = w0 R6_w=inv(id=0,umax_value=0xffffffff,var_off=(0x0; 0xffffffff)) 197: (67) r6 <<= 32 R6_w=inv(id=0,smax_value=0x7fffffff00000000,umax_value=0xffffffff00000000, var_off=(0x0; 0xffffffff00000000)) 198: (77) r6 >>= 32 R6=inv(id=0,umax_value=0xffffffff,var_off=(0x0; 0xffffffff)) ... 201: (79) r8 = *(u64 *)(r10 -416) R8_w=map_value(id=0,off=40,ks=4,vs=13872,imm=0) 202: (0f) r8 += r6 R8_w=map_value(id=0,off=40,ks=4,vs=13872,umax_value=0xffffffff,var_off=(0x0; 0xffffffff)) 203: (07) r8 += 9696 R8_w=map_value(id=0,off=9736,ks=4,vs=13872,umax_value=0xffffffff,var_off=(0x0; 0xffffffff)) ... 255: (bf) r1 = r8 R1_w=map_value(id=0,off=9736,ks=4,vs=13872,umax_value=0xffffffff,var_off=(0x0; 0xffffffff)) ... 257: (85) call bpf_probe_read_user_str#114 R1 unbounded memory access, make sure to bounds check any array access into a map The value range for register r6 at insn 198 should be really just 0/1. The umax_value=0xffffffff caused later verification failure. After jmp instructions, the current verifier already tried to use just obtained information to get better register range. The current mechanism is for 64bit register only. This patch implemented to tighten the range for 32bit sub-registers after jmp32 instructions. With the patch, we have the below range ranges for the above code sequence: 193: (85) call bpf_probe_read_user_str#114 R0=inv(id=0) 194: (26) if w0 > 0x1 goto pc+4 R0_w=inv(id=0,smax_value=0x7fffffff00000001,umax_value=0xffffffff00000001, var_off=(0x0; 0xffffffff00000001)) 195: (6b) *(u16 *)(r7 +80) = r0 196: (bc) w6 = w0 R6_w=inv(id=0,umax_value=0xffffffff,var_off=(0x0; 0x1)) 197: (67) r6 <<= 32 R6_w=inv(id=0,umax_value=0x100000000,var_off=(0x0; 0x100000000)) 198: (77) r6 >>= 32 R6=inv(id=0,umax_value=1,var_off=(0x0; 0x1)) ... 201: (79) r8 = *(u64 *)(r10 -416) R8_w=map_value(id=0,off=40,ks=4,vs=13872,imm=0) 202: (0f) r8 += r6 R8_w=map_value(id=0,off=40,ks=4,vs=13872,umax_value=1,var_off=(0x0; 0x1)) 203: (07) r8 += 9696 R8_w=map_value(id=0,off=9736,ks=4,vs=13872,umax_value=1,var_off=(0x0; 0x1)) ... 255: (bf) r1 = r8 R1_w=map_value(id=0,off=9736,ks=4,vs=13872,umax_value=1,var_off=(0x0; 0x1)) ... 257: (85) call bpf_probe_read_user_str#114 ... At insn 194, the register R0 has better var_off.mask and smax_value. Especially, the var_off.mask ensures later lshift and rshift maintains proper value range. Suggested-by:
-
Toke Høiland-Jørgensen authored
Tetsuo pointed out that it was not only the device unregister hook that was broken for devmap_hash types, it was also cleanup on map free. So better fix this as well. While we're at it, there's no reason to allocate the netdev_map array for DEVMAP_HASH, so skip that and adjust the cost accordingly. Fixes: 6f9d451a ("xdp: Add devmap_hash map type for looking up devices by hashed index") Reported-by:
Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> Signed-off-by:
Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by:
-
Alexei Starovoitov authored
Andrii Nakryiko says: ==================== This patch set salvages all the non-extern-specific changes out of blocked externs patch set ([0]). In addition to small clean ups, it also refactors libbpf's handling of relocations and allows support for global (non-static) variables. [0] https://patchwork.ozlabs.org/project/netdev/list/?series=143358&state=* ==================== Signed-off-by:
-
Andrii Nakryiko authored
Add exra level of verboseness, activated by -vvv argument. When -vv is specified, verbose libbpf and verifier log (level 1) is output, even for successful tests. With -vvv, verifier log goes to level 2. This is extremely useful to debug verifier failures, as well as just see the state and flow of verification. Before this, you'd have to go and modify load_program()'s source code inside libbpf to specify extra log_level flags, which is suboptimal to say the least. Currently -vv and -vvv triggering verifier output is integrated into test_stub's bpf_prog_load as well as bpf_verif_scale.c tests. Signed-off-by:
-
Andrii Nakryiko authored
Initialized global variables are no different in ELF from static variables, and don't require any extra support from libbpf. But they are matching semantics of global data (backed by BPF maps) more closely, preventing LLVM/Clang from aggressively inlining constant values and not requiring volatile incantations to prevent those. This patch enables global variables. It still disables uninitialized variables, which will be put into special COM (common) ELF section, because BPF doesn't allow uninitialized data to be accessed. Signed-off-by:
-
Jakub Kicinski authored
If selftests are copied over to another machine/location for execution the build test of bpftool will obviously not work, since the sources are not copied. Skip it if we can't find bpftool's Makefile. Reported-by:
Naresh Kamboju <naresh.kamboju@linaro.org> Signed-off-by:
Jakub Kicinski <jakub.kicinski@netronome.com> Signed-off-by:
Quentin Monnet <quentin.monnet@netronome.com> Signed-off-by:
-
Andrii Nakryiko authored
Fix a bunch of warnings and errors reported by checkpatch.pl, to make it easier to spot new problems. Signed-off-by:
-
Quentin Monnet authored
The trap on EXIT is used to clean up any temporary directory left by the build attempts. It is not needed when the user simply calls the script with its --help option, and may not be needed either if we add checks (e.g. on the availability of bpftool files) before the build attempts. Let's move this trap and related variables lower down in the code, so that we don't accidentally change the value returned from the script on early exits at pre-checks. Signed-off-by:
-
Andrii Nakryiko authored
Relocation handling code is convoluted and unnecessarily deeply nested. Split out per-relocation logic into separate function. Also refactor the logic to be more a sequence of per-relocation type checks and processing steps, making it simpler to follow control flow. This makes it easier to further extends it to new kinds of relocations (e.g., support for extern variables). This patch also makes relocation's section verification more robust. Previously relocations against not yet supported externs were silently ignored because of obj->efile.text_shndx was zero, when all BPF programs had custom section names and there was no .text section. Also, invalid LDIMM64 relocations against non-map sections were passed through, if they were pointing to a .text section (or 0, which is invalid section). All these bugs are fixed within this refactoring and checks are made more appropriate for each type of relocation. Signed-off-by:
-
Quentin Monnet authored
Building selftests with 'make TARGETS=bpf kselftest' was fixed in commit 55d554f5 ("tools: bpf: Use !building_out_of_srctree to determine srctree"). However, by updating $(srctree) in tools/bpf/Makefile for in-tree builds only, we leave out the case where we pass an output directory to build BPF tools, but $(srctree) is not set. This typically happens for: $ make -s tools/bpf O=/tmp/foo Makefile:40: /tools/build/Makefile.feature: No such file or directory Fix it by updating $(srctree) in the Makefile not only for out-of-tree builds, but also if $(srctree) is empty. Detected with test_bpftool_build.sh. Fixes: 55d554f5 ("tools: bpf: Use !building_out_of_srctree to determine srctree") Signed-off-by:
-
Andrii Nakryiko authored
Add -mattr=dwarfris attribute to llc to avoid having relocations against DWARF data. These relocations make it impossible to inspect DWARF contents: all strings are invalid. Signed-off-by:
-
Quentin Monnet authored
When building bpftool, a warning was introduced by commit a9436460 ("bpftool: Allow to read btf as raw data"), because the return value from a call to 'read()' is ignored. Let's address it. Signed-off-by:
-
Luc Van Oostenryck authored
xsk_poll() is defined as returning 'unsigned int' but the .poll method is declared as returning '__poll_t', a bitwise type. Fix this by using the proper return type and using the EPOLL constants instead of the POLL ones, as required for __poll_t. Signed-off-by:
Luc Van Oostenryck <luc.vanoostenryck@gmail.com> Signed-off-by:
Björn Töpel <bjorn.topel@intel.com> Link: https://lore.kernel.org/bpf/20191120001042.30830-1-luc.vanoostenryck@gmail.com
-
- 24 Nov, 2019 16 commits
-
-
Eric Dumazet authored
A patch in net-next triggered a compile error on powerpc: include/linux/u64_stats_sync.h: In function 'u64_stats_read': include/asm-generic/local64.h:30:37: warning: passing argument 1 of 'local_read' discards 'const' qualifier from pointer target type This seems reasonable to relax powerpc local_read() requirements. Fixes: 316580b6 ("u64_stats: provide u64_stats_t type") Signed-off-by:
Eric Dumazet <edumazet@google.com> Reported-by:
kbuild test robot <lkp@intel.com> Acked-by:
Michael Ellerman <mpe@ellerman.id.au> Tested-by: Stephen Rothwell <sfr@canb.auug.org.au> # build only Signed-off-by:
-
Jakub Kicinski authored
Michael Chan says: ==================== bnxt_en: Updates. This patchset contains these main features: 1. Add the proper logic to support suspend/resume on the new 57500 chips. 2. Allow Phy configurations from user on a Multihost function if supported by fw. 3. devlink NVRAM flashing support. 4. Add a couple of chip IDs, PHY loopback enhancement, and provide more RSS contexts to VFs. v2: Dropped the devlink info patches to address some feedback and resubmit for the 5.6 kernel. ==================== Signed-off-by: -
Vasundhara Volam authored
Use the same bnxt_flash_package_from_file() function to support devlink flash operation. Cc: Jiri Pirko <jiri@mellanox.com> Signed-off-by:
Vasundhara Volam <vasundhara-v.volam@broadcom.com> Signed-off-by:
Michael Chan <michael.chan@broadcom.com> Signed-off-by:
-
Michael Chan authored
Currently, the driver does not allow PHY settings on a multi-function or NPAR NIC whose port is shared by more than one function. Newer firmware now allows PHY settings on some of these NICs. Check for this new firmware setting and allow the user to set the PHY settings accordingly. Signed-off-by:
-
Michael Chan authored
If the link settings have been changed by another function sharing the port, firmware will send us an async. message. In response, we will call the new bnxt_init_ethtool_link_settings() function to update the current settings. Signed-off-by:
-
Michael Chan authored
Refactor this logic in bnxt_probe_phy() into a separate function bnxt_init_ethtool_link_settings(). It used to be that the settable link settings will never be changed without going through ethtool. So we only needed to do this once in bnxt_probe_phy(). Now, another function sharing the port may change it and we may need to re-initialize the ethtool settings again in run-time. Signed-off-by:
-
Michael Chan authored
New firmware allows PHY loopback to be set without disabling autoneg first. Check this capability and skip disabling autoneg when it is supported by firmware. Using this scheme, loopback will always work even if the PHY only supports autoneg. Signed-off-by:
-
Michael Chan authored
The driver currently only assignes 1 RSS context to each VF. This works for the Linux VF driver. But other drivers, such as DPDK, can make use of additional RSS contexts. Modify the code to divide up and assign RSS contexts to VFs just like other resources. Signed-off-by:
-
Michael Chan authored
Some chips that need host context memory as a backing store requires the memory to be initialized to a non-zero value. Query the value from firmware and initialize the context memory accordingly. Signed-off-by:
-
Vasundhara Volam authored
Driver calls HWRM_FUNC_RESET firmware call while resuming the device which clears the context memory backing store. Because of which allocating firmware resources would eventually fail. Fix it by freeing all context memory during suspend and reallocate the memory during resume. Call bnxt_hwrm_queue_qportcfg() in resume path. This firmware call is needed on the 57500 chips so that firmware will set up the proper queue mapping in relation to the context memory. Signed-off-by:
-
Vasundhara Volam authored
After driver unregister, firmware is erasing the information that driver supports new resource management. Send FUNC_RESOURCE_QCAPS command to inform the firmware that driver supports new resource management while resuming from hibernation. Otherwise, we fallback to the older resource allocation scheme. Also, move driver register after sending FUNC_RESOURCE_QCAPS command to be consistent with the normal initialization sequence. Signed-off-by:
-
Vasundhara Volam authored
Everytime driver registers with firmware, driver is required to register for async event notifications as well. These 2 calls are done using the same firmware command and can be combined. We are also missing the 2nd step to register for async events in the suspend/resume path and this will fix it. Prior to this, we were getting only default notifications. ULP can register for additional async events for the RDMA driver, so we add a parameter to the new function to only do step 2 when it is called from ULP. Signed-off-by:
-
Vasundhara Volam authored
In the bnxt_init_one() failure path, if the driver has already called firmware to register the driver, it is not undoing the driver registration. Add this missing step to unregister for correctness, so that the firmware knows that the driver has unloaded. Signed-off-by:
-
Michael Chan authored
Disable Bus master during suspend to prevent DMAs after the device goes into D3hot state. The new 57500 devices may continue to DMA from context memory after the system goes into D3hot state. This may cause some PCIe errors on some system. Re-enable it during resume. Signed-off-by:
-
Michael Chan authored
Fix BNXT_CHIP_NUM_5645X() to include 57452 and 56454 chip IDs, so that these chips will be properly classified as P4 chips to take advantage of the P4 fixes and features. Signed-off-by:
-
Jakub Kicinski authored
The rfs members of struct efx_channel are under CONFIG_RFS_ACCEL. Ethtool stats which access those need to be as well. Reported-by:
-
