1. 01 Jun, 2020 40 commits
    • Lorenzo Bianconi's avatar
      xdp: Introduce xdp_convert_frame_to_buff utility routine · fc379872
      Lorenzo Bianconi authored
      Introduce xdp_convert_frame_to_buff utility routine to initialize xdp_buff
      fields from xdp_frames ones. Rely on xdp_convert_frame_to_buff in veth xdp
      code.
      Suggested-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
      Signed-off-by: default avatarLorenzo Bianconi <lorenzo@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
      Link: https://lore.kernel.org/bpf/87acf133073c4b2d4cbb8097e8c2480c0a0fac32.1590698295.git.lorenzo@kernel.org
      fc379872
    • Alexei Starovoitov's avatar
      Merge branch 'bpf_setsockopt-SO_BINDTODEVICE' · c48a24f0
      Alexei Starovoitov authored
      Ferenc Fejes says:
      
      ====================
      This option makes it possible to programatically bind sockets
      to netdevices. With the help of this option sockets
      of VRF unaware applications could be distributed between
      multiple VRFs with an eBPF program. This lets the applications
      benefit from multiple possible routes.
      
      v2:
      - splitting up the patch to three parts
      - lock_sk parameter for optional locking in sock_bindtoindex - Stanislav Fomichev
      - testing the SO_BINDTODEVICE option - Andrii Nakryiko
      ====================
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      c48a24f0
    • Ferenc Fejes's avatar
      selftests/bpf: Add test for SO_BINDTODEVICE opt of bpf_setsockopt · 9c441fe4
      Ferenc Fejes authored
      This test intended to verify if SO_BINDTODEVICE option works in
      bpf_setsockopt. Because we already in the SOL_SOCKET level in this
      connect bpf prog its safe to verify the sanity in the beginning of
      the connect_v4_prog by calling the bind_to_device test helper.
      
      The testing environment already created by the test_sock_addr.sh
      script so this test assume that two netdevices already existing in
      the system: veth pair with names test_sock_addr1 and test_sock_addr2.
      The test will try to bind the socket to those devices first.
      Then the test assume there are no netdevice with "nonexistent_dev"
      name so the bpf_setsockopt will give use ENODEV error.
      At the end the test remove the device binding from the socket
      by binding it to an empty name.
      Signed-off-by: default avatarFerenc Fejes <fejes@inf.elte.hu>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/3f055b8e45c65639c5c73d0b4b6c589e60b86f15.1590871065.git.fejes@inf.elte.hu
      9c441fe4
    • Ferenc Fejes's avatar
      bpf: Allow SO_BINDTODEVICE opt in bpf_setsockopt · 70c58997
      Ferenc Fejes authored
      Extending the supported sockopts in bpf_setsockopt with
      SO_BINDTODEVICE. We call sock_bindtoindex with parameter
      lock_sk = false in this context because we already owning
      the socket.
      Signed-off-by: default avatarFerenc Fejes <fejes@inf.elte.hu>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/4149e304867b8d5a606a305bc59e29b063e51f49.1590871065.git.fejes@inf.elte.hu
      70c58997
    • Ferenc Fejes's avatar
      net: Make locking in sock_bindtoindex optional · 8ea204c2
      Ferenc Fejes authored
      The sock_bindtoindex intended for kernel wide usage however
      it will lock the socket regardless of the context. This modification
      relax this behavior optionally: locking the socket will be optional
      by calling the sock_bindtoindex with lock_sk = true.
      
      The modification applied to all users of the sock_bindtoindex.
      Signed-off-by: default avatarFerenc Fejes <fejes@inf.elte.hu>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/bee6355da40d9e991b2f2d12b67d55ebb5f5b207.1590871065.git.fejes@inf.elte.hu
      8ea204c2
    • Denis Efremov's avatar
      bpf: Change kvfree to kfree in generic_map_lookup_batch() · bb2359f4
      Denis Efremov authored
      buf_prevkey in generic_map_lookup_batch() is allocated with
      kmalloc(). It's safe to free it with kfree().
      
      Fixes: cb4d03ab ("bpf: Add generic support for lookup batch op")
      Signed-off-by: default avatarDenis Efremov <efremov@linux.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Link: https://lore.kernel.org/bpf/20200601162814.17426-1-efremov@linux.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      bb2359f4
    • Alexei Starovoitov's avatar
      Merge branch 'fix-ktls-with-sk_skb_verdict' · 7b805819
      Alexei Starovoitov authored
      John Fastabend says:
      
      ====================
      If a socket is running a BPF_SK_SKB_SREAM_VERDICT program and KTLS is
      enabled the data stream may be broken if both TLS stream parser and
      BPF stream parser try to handle data. Fix this here by making KTLS
      stream parser run first to ensure TLS messages are received correctly
      and then calling the verdict program. This analogous to how we handle
      a similar conflict on the TX side.
      
      Note, this is a fix but it doesn't make sense to push this late to
      bpf tree so targeting bpf-next and keeping fixes tags.
      ====================
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      7b805819
    • Alexei Starovoitov's avatar
      tools/bpf: sync bpf.h · df8fe57c
      Alexei Starovoitov authored
      Sync bpf.h into tool/include/uapi/
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      df8fe57c
    • John Fastabend's avatar
      bpf, selftests: Add test for ktls with skb bpf ingress policy · 463bac5f
      John Fastabend authored
      This adds a test for bpf ingress policy. To ensure data writes happen
      as expected with extra TLS headers we run these tests with data
      verification enabled by default. This will test receive packets have
      "PASS" stamped into the front of the payload.
      Signed-off-by: default avatarJohn Fastabend <john.fastabend@gmail.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/159079363965.5745.3390806911628980210.stgit@john-Precision-5820-TowerSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      463bac5f
    • Alexei Starovoitov's avatar
      Merge branch 'xdp_devmap' · 906312ca
      Alexei Starovoitov authored
      David Ahern says:
      
      ====================
      Implementation of Daniel's proposal for allowing DEVMAP entries to be
      a device index, program fd pair.
      
      Programs are run after XDP_REDIRECT and have access to both Rx device
      and Tx device.
      
      v4
      - moved struct bpf_devmap_val from uapi to devmap.c, named the union
        and dropped the prefix from the elements - Jesper
      - fixed 2 bugs in selftests
      
      v3
      - renamed struct to bpf_devmap_val
      - used offsetofend to check for expected map size, modification of
        Toke's comment
      - check for explicit value sizes
      - adjusted switch statement in dev_map_run_prog per Andrii's comment
      - changed SEC shortcut to xdp_devmap
      - changed selftests to use skeleton and new map declaration
      
      v2
      - moved dev_map_ext_val definition to uapi to formalize the API for devmap
        extensions; add bpf_ prefix to the prog_fd and prog_id entries
      - changed devmap code to handle struct in a way that it can support future
        extensions
      - fixed subject in libbpf patch
      
      v1
      - fixed prog put on invalid program - Toke
      - changed write value from id to fd per Toke's comments about capabilities
      - add test cases
      ====================
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      906312ca
    • John Fastabend's avatar
      bpf: Fix running sk_skb program types with ktls · e91de6af
      John Fastabend authored
      KTLS uses a stream parser to collect TLS messages and send them to
      the upper layer tls receive handler. This ensures the tls receiver
      has a full TLS header to parse when it is run. However, when a
      socket has BPF_SK_SKB_STREAM_VERDICT program attached before KTLS
      is enabled we end up with two stream parsers running on the same
      socket.
      
      The result is both try to run on the same socket. First the KTLS
      stream parser runs and calls read_sock() which will tcp_read_sock
      which in turn calls tcp_rcv_skb(). This dequeues the skb from the
      sk_receive_queue. When this is done KTLS code then data_ready()
      callback which because we stacked KTLS on top of the bpf stream
      verdict program has been replaced with sk_psock_start_strp(). This
      will in turn kick the stream parser again and eventually do the
      same thing KTLS did above calling into tcp_rcv_skb() and dequeuing
      a skb from the sk_receive_queue.
      
      At this point the data stream is broke. Part of the stream was
      handled by the KTLS side some other bytes may have been handled
      by the BPF side. Generally this results in either missing data
      or more likely a "Bad Message" complaint from the kTLS receive
      handler as the BPF program steals some bytes meant to be in a
      TLS header and/or the TLS header length is no longer correct.
      
      We've already broke the idealized model where we can stack ULPs
      in any order with generic callbacks on the TX side to handle this.
      So in this patch we do the same thing but for RX side. We add
      a sk_psock_strp_enabled() helper so TLS can learn a BPF verdict
      program is running and add a tls_sw_has_ctx_rx() helper so BPF
      side can learn there is a TLS ULP on the socket.
      
      Then on BPF side we omit calling our stream parser to avoid
      breaking the data stream for the KTLS receiver. Then on the
      KTLS side we call BPF_SK_SKB_STREAM_VERDICT once the KTLS
      receiver is done with the packet but before it posts the
      msg to userspace. This gives us symmetry between the TX and
      RX halfs and IMO makes it usable again. On the TX side we
      process packets in this order BPF -> TLS -> TCP and on
      the receive side in the reverse order TCP -> TLS -> BPF.
      
      Discovered while testing OpenSSL 3.0 Alpha2.0 release.
      
      Fixes: d829e9c4 ("tls: convert to generic sk_msg interface")
      Signed-off-by: default avatarJohn Fastabend <john.fastabend@gmail.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/159079361946.5745.605854335665044485.stgit@john-Precision-5820-TowerSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      e91de6af
    • David Ahern's avatar
      selftest: Add tests for XDP programs in devmap entries · d39aec79
      David Ahern authored
      Add tests to verify ability to add an XDP program to a
      entry in a DEVMAP.
      
      Add negative tests to show DEVMAP programs can not be
      attached to devices as a normal XDP program, and accesses
      to egress_ifindex require BPF_XDP_DEVMAP attach type.
      Signed-off-by: default avatarDavid Ahern <dsahern@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarToke Høiland-Jørgensen <toke@redhat.com>
      Link: https://lore.kernel.org/bpf/20200529220716.75383-6-dsahern@kernel.orgSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      d39aec79
    • John Fastabend's avatar
      bpf: Refactor sockmap redirect code so its easy to reuse · ca2f5f21
      John Fastabend authored
      We will need this block of code called from tls context shortly
      lets refactor the redirect logic so its easy to use. This also
      cleans up the switch stmt so we have fewer fallthrough cases.
      
      No logic changes are intended.
      
      Fixes: d829e9c4 ("tls: convert to generic sk_msg interface")
      Signed-off-by: default avatarJohn Fastabend <john.fastabend@gmail.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Reviewed-by: default avatarJakub Sitnicki <jakub@cloudflare.com>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Link: https://lore.kernel.org/bpf/159079360110.5745.7024009076049029819.stgit@john-Precision-5820-TowerSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      ca2f5f21
    • David Ahern's avatar
      libbpf: Add SEC name for xdp programs attached to device map · 27787970
      David Ahern authored
      Support SEC("xdp_devmap*") as a short cut for loading the program with
      type BPF_PROG_TYPE_XDP and expected attach type BPF_XDP_DEVMAP.
      Signed-off-by: default avatarDavid Ahern <dsahern@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarToke Høiland-Jørgensen <toke@redhat.com>
      Link: https://lore.kernel.org/bpf/20200529220716.75383-5-dsahern@kernel.orgSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      27787970
    • David Ahern's avatar
      xdp: Add xdp_txq_info to xdp_buff · 64b59025
      David Ahern authored
      Add xdp_txq_info as the Tx counterpart to xdp_rxq_info. At the
      moment only the device is added. Other fields (queue_index)
      can be added as use cases arise.
      
      >From a UAPI perspective, add egress_ifindex to xdp context for
      bpf programs to see the Tx device.
      
      Update the verifier to only allow accesses to egress_ifindex by
      XDP programs with BPF_XDP_DEVMAP expected attach type.
      Signed-off-by: default avatarDavid Ahern <dsahern@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarToke Høiland-Jørgensen <toke@redhat.com>
      Link: https://lore.kernel.org/bpf/20200529220716.75383-4-dsahern@kernel.orgSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      64b59025
    • David Ahern's avatar
      bpf: Add support to attach bpf program to a devmap entry · fbee97fe
      David Ahern authored
      Add BPF_XDP_DEVMAP attach type for use with programs associated with a
      DEVMAP entry.
      
      Allow DEVMAPs to associate a program with a device entry by adding
      a bpf_prog.fd to 'struct bpf_devmap_val'. Values read show the program
      id, so the fd and id are a union. bpf programs can get access to the
      struct via vmlinux.h.
      
      The program associated with the fd must have type XDP with expected
      attach type BPF_XDP_DEVMAP. When a program is associated with a device
      index, the program is run on an XDP_REDIRECT and before the buffer is
      added to the per-cpu queue. At this point rxq data is still valid; the
      next patch adds tx device information allowing the prorgam to see both
      ingress and egress device indices.
      
      XDP generic is skb based and XDP programs do not work with skb's. Block
      the use case by walking maps used by a program that is to be attached
      via xdpgeneric and fail if any of them are DEVMAP / DEVMAP_HASH with
      
      Block attach of BPF_XDP_DEVMAP programs to devices.
      Signed-off-by: default avatarDavid Ahern <dsahern@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarToke Høiland-Jørgensen <toke@redhat.com>
      Link: https://lore.kernel.org/bpf/20200529220716.75383-3-dsahern@kernel.orgSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      fbee97fe
    • Yonghong Song's avatar
      bpf: Use strncpy_from_unsafe_strict() in bpf_seq_printf() helper · b36e62eb
      Yonghong Song authored
      In bpf_seq_printf() helper, when user specified a "%s" in the
      format string, strncpy_from_unsafe() is used to read the actual string
      to a buffer. The string could be a format string or a string in
      the kernel data structure. It is really unlikely that the string
      will reside in the user memory.
      
      This is different from Commit b2a5212f ("bpf: Restrict bpf_trace_printk()'s %s
      usage and add %pks, %pus specifier") which still used
      strncpy_from_unsafe() for "%s" to preserve the old behavior.
      
      If in the future, bpf_seq_printf() indeed needs to read user
      memory, we can implement "%pus" format string.
      
      Based on discussion in [1], if the intent is to read kernel memory,
      strncpy_from_unsafe_strict() should be used. So this patch
      changed to use strncpy_from_unsafe_strict().
      
      [1]: https://lore.kernel.org/bpf/20200521152301.2587579-1-hch@lst.de/T/
      
      Fixes: 492e639f ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Cc: Christoph Hellwig <hch@lst.de>
      Link: https://lore.kernel.org/bpf/20200529004810.3352219-1-yhs@fb.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      b36e62eb
    • David Ahern's avatar
      devmap: Formalize map value as a named struct · 7f1c0426
      David Ahern authored
      Add 'struct bpf_devmap_val' to formalize the expected values that can
      be passed in for a DEVMAP. Update devmap code to use the struct.
      Signed-off-by: default avatarDavid Ahern <dsahern@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarToke Høiland-Jørgensen <toke@redhat.com>
      Link: https://lore.kernel.org/bpf/20200529220716.75383-2-dsahern@kernel.orgSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      7f1c0426
    • Amritha Nambiar's avatar
      bpf: Add rx_queue_mapping to bpf_sock · c3c16f2e
      Amritha Nambiar authored
      Add "rx_queue_mapping" to bpf_sock. This gives read access for the
      existing field (sk_rx_queue_mapping) of struct sock from bpf_sock.
      Semantics for the bpf_sock rx_queue_mapping access are similar to
      sk_rx_queue_get(), i.e the value NO_QUEUE_MAPPING is not allowed
      and -1 is returned in that case. This is useful for transmit queue
      selection based on the received queue index which is cached in the
      socket in the receive path.
      
      v3: Addressed review comments to add usecase in patch description,
          and fixed default value for rx_queue_mapping.
      v2: fixed build error for CONFIG_XPS wrapping, reported by
          kbuild test robot <lkp@intel.com>
      Signed-off-by: default avatarAmritha Nambiar <amritha.nambiar@intel.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      c3c16f2e
    • Daniel Borkmann's avatar
      Merge branch 'bpf-ring-buffer' · e255d327
      Daniel Borkmann authored
      Andrii Nakryiko says:
      
      ====================
      Implement a new BPF ring buffer, as presented at BPF virtual conference ([0]).
      It presents an alternative to perf buffer, following its semantics closely,
      but allowing sharing same instance of ring buffer across multiple CPUs
      efficiently.
      
      Most patches have extensive commentary explaining various aspects, so I'll
      keep cover letter short. Overall structure of the patch set:
      - patch #1 adds BPF ring buffer implementation to kernel and necessary
        verifier support;
      - patch #2 adds libbpf consumer implementation for BPF ringbuf;
      - patch #3 adds selftest, both for single BPF ring buf use case, as well as
        using it with array/hash of maps;
      - patch #4 adds extensive benchmarks and provide some analysis in commit
        message, it builds upon selftests/bpf's bench runner.
      - patch #5 adds most of patch #1 commit message as a doc under
        Documentation/bpf/ringbuf.rst.
      
      Litmus tests, validating consumer/producer protocols and memory orderings,
      were moved out as discussed in [1] and are going to be posted against -rcu
      tree and put under Documentation/litmus-tests/bpf-rb.
      
        [0] https://docs.google.com/presentation/d/18ITdg77Bj6YDOH2LghxrnFxiPWe0fAqcmJY95t_qr0w
        [1] https://lkml.org/lkml/2020/5/22/1011
      
      v3->v4:
      - fix ringbuf freeing (vunmap, __free_page); verified with a trivial loop
        creating and closing ringbuf map endlessly (Daniel);
      
      v2->v3:
      - dropped unnecessary smp_wmb() (Paul);
      - verifier reference type enhancement patch was dropped (Alexei);
      - better verifier message for various memory access checks (Alexei);
      - clarified a bit roundup_len() bit shifting (Alexei);
      - converted doc to .rst (Alexei);
      - fixed warning on 32-bit arches regarding tautological ring area size check.
      
      v1->v2:
      - commit()/discard()/output() accept flags (NO_WAKEUP/FORCE_WAKEUP) (Stanislav);
      - bpf_ringbuf_query() added, returning available data size, ringbuf size,
        consumer/producer positions, needed to implement smarter notification policy
        (Stanislav);
      - added ringbuf UAPI constants to include/uapi/linux/bpf.h (Jonathan);
      - fixed sample size check, added proper ringbuf size check (Jonathan, Alexei);
      - wake_up_all() is done through irq_work (Alexei);
      - consistent use of smp_load_acquire/smp_store_release, no
        READ_ONCE/WRITE_ONCE (Alexei);
      - added Documentation/bpf/ringbuf.txt (Stanislav);
      - updated litmus test with smp_load_acquire/smp_store_release changes;
      - added ring_buffer__consume() API to libbpf for busy-polling;
      - ring_buffer__poll() on success returns number of records consumed;
      - fixed EPOLL notifications, don't assume available data, done similarly to
        perfbuf's implementation;
      - both ringbuf and perfbuf now have --rb-sampled mode, instead of
        pb-raw/pb-custom mode, updated benchmark results;
      - extended ringbuf selftests to validate epoll logic/manual notification
        logic, as well as bpf_ringbuf_query().
      ====================
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      e255d327
    • Anton Protopopov's avatar
      selftests/bpf: Add tests for write-only stacks/queues · 43dd115b
      Anton Protopopov authored
      For write-only stacks and queues bpf_map_update_elem should be allowed, but
      bpf_map_lookup_elem and bpf_map_lookup_and_delete_elem should fail with EPERM.
      Signed-off-by: default avatarAnton Protopopov <a.s.protopopov@gmail.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200527185700.14658-6-a.s.protopopov@gmail.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      43dd115b
    • Andrii Nakryiko's avatar
      docs/bpf: Add BPF ring buffer design notes · 97abb2b3
      Andrii Nakryiko authored
      Add commit description from patch #1 as a stand-alone documentation under
      Documentation/bpf, as it might be more convenient format, in long term
      perspective.
      Suggested-by: default avatarStanislav Fomichev <sdf@google.com>
      Signed-off-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200529075424.3139988-6-andriin@fb.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      97abb2b3
    • Andrii Nakryiko's avatar
      bpf: Add BPF ringbuf and perf buffer benchmarks · c97099b0
      Andrii Nakryiko authored
      Extend bench framework with ability to have benchmark-provided child argument
      parser for custom benchmark-specific parameters. This makes bench generic code
      modular and independent from any specific benchmark.
      
      Also implement a set of benchmarks for new BPF ring buffer and existing perf
      buffer. 4 benchmarks were implemented: 2 variations for each of BPF ringbuf
      and perfbuf:,
        - rb-libbpf utilizes stock libbpf ring_buffer manager for reading data;
        - rb-custom implements custom ring buffer setup and reading code, to
          eliminate overheads inherent in generic libbpf code due to callback
          functions and the need to update consumer position after each consumed
          record, instead of batching updates (due to pessimistic assumption that
          user callback might take long time and thus could unnecessarily hold ring
          buffer space for too long);
        - pb-libbpf uses stock libbpf perf_buffer code with all the default
          settings, though uses higher-performance raw event callback to minimize
          unnecessary overhead;
        - pb-custom implements its own custom consumer code to minimize any possible
          overhead of generic libbpf implementation and indirect function calls.
      
      All of the test support default, no data notification skipped, mode, as well
      as sampled mode (with --rb-sampled flag), which allows to trigger epoll
      notification less frequently and reduce overhead. As will be shown, this mode
      is especially critical for perf buffer, which suffers from high overhead of
      wakeups in kernel.
      
      Otherwise, all benchamrks implement similar way to generate a batch of records
      by using fentry/sys_getpgid BPF program, which pushes a bunch of records in
      a tight loop and records number of successful and dropped samples. Each record
      is a small 8-byte integer, to minimize the effect of memory copying with
      bpf_perf_event_output() and bpf_ringbuf_output().
      
      Benchmarks that have only one producer implement optional back-to-back mode,
      in which record production and consumption is alternating on the same CPU.
      This is the highest-throughput happy case, showing ultimate performance
      achievable with either BPF ringbuf or perfbuf.
      
      All the below scenarios are implemented in a script in
      benchs/run_bench_ringbufs.sh. Tests were performed on 28-core/56-thread
      Intel Xeon CPU E5-2680 v4 @ 2.40GHz CPU.
      
      Single-producer, parallel producer
      ==================================
      rb-libbpf            12.054 ± 0.320M/s (drops 0.000 ± 0.000M/s)
      rb-custom            8.158 ± 0.118M/s (drops 0.001 ± 0.003M/s)
      pb-libbpf            0.931 ± 0.007M/s (drops 0.000 ± 0.000M/s)
      pb-custom            0.965 ± 0.003M/s (drops 0.000 ± 0.000M/s)
      
      Single-producer, parallel producer, sampled notification
      ========================================================
      rb-libbpf            11.563 ± 0.067M/s (drops 0.000 ± 0.000M/s)
      rb-custom            15.895 ± 0.076M/s (drops 0.000 ± 0.000M/s)
      pb-libbpf            9.889 ± 0.032M/s (drops 0.000 ± 0.000M/s)
      pb-custom            9.866 ± 0.028M/s (drops 0.000 ± 0.000M/s)
      
      Single producer on one CPU, consumer on another one, both running at full
      speed. Curiously, rb-libbpf has higher throughput than objectively faster (due
      to more lightweight consumer code path) rb-custom. It appears that faster
      consumer causes kernel to send notifications more frequently, because consumer
      appears to be caught up more frequently. Performance of perfbuf suffers from
      default "no sampling" policy and huge overhead that causes.
      
      In sampled mode, rb-custom is winning very significantly eliminating too
      frequent in-kernel wakeups, the gain appears to be more than 2x.
      
      Perf buffer achieves even more impressive wins, compared to stock perfbuf
      settings, with 10x improvements in throughput with 1:500 sampling rate. The
      trade-off is that with sampling, application might not get next X events until
      X+1st arrives, which is not always acceptable. With steady influx of events,
      though, this shouldn't be a problem.
      
      Overall, single-producer performance of ring buffers seems to be better no
      matter the sampled/non-sampled modes, but it especially beats ring buffer
      without sampling due to its adaptive notification approach.
      
      Single-producer, back-to-back mode
      ==================================
      rb-libbpf            15.507 ± 0.247M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf-sampled    14.692 ± 0.195M/s (drops 0.000 ± 0.000M/s)
      rb-custom            21.449 ± 0.157M/s (drops 0.000 ± 0.000M/s)
      rb-custom-sampled    20.024 ± 0.386M/s (drops 0.000 ± 0.000M/s)
      pb-libbpf            1.601 ± 0.015M/s (drops 0.000 ± 0.000M/s)
      pb-libbpf-sampled    8.545 ± 0.064M/s (drops 0.000 ± 0.000M/s)
      pb-custom            1.607 ± 0.022M/s (drops 0.000 ± 0.000M/s)
      pb-custom-sampled    8.988 ± 0.144M/s (drops 0.000 ± 0.000M/s)
      
      Here we test a back-to-back mode, which is arguably best-case scenario both
      for BPF ringbuf and perfbuf, because there is no contention and for ringbuf
      also no excessive notification, because consumer appears to be behind after
      the first record. For ringbuf, custom consumer code clearly wins with 21.5 vs
      16 million records per second exchanged between producer and consumer. Sampled
      mode actually hurts a bit due to slightly slower producer logic (it needs to
      fetch amount of data available to decide whether to skip or force notification).
      
      Perfbuf with wakeup sampling gets 5.5x throughput increase, compared to
      no-sampling version. There also doesn't seem to be noticeable overhead from
      generic libbpf handling code.
      
      Perfbuf back-to-back, effect of sample rate
      ===========================================
      pb-sampled-1         1.035 ± 0.012M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-5         3.476 ± 0.087M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-10        5.094 ± 0.136M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-25        7.118 ± 0.153M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-50        8.169 ± 0.156M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-100       8.887 ± 0.136M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-250       9.180 ± 0.209M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-500       9.353 ± 0.281M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-1000      9.411 ± 0.217M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-2000      9.464 ± 0.167M/s (drops 0.000 ± 0.000M/s)
      pb-sampled-3000      9.575 ± 0.273M/s (drops 0.000 ± 0.000M/s)
      
      This benchmark shows the effect of event sampling for perfbuf. Back-to-back
      mode for highest throughput. Just doing every 5th record notification gives
      3.5x speed up. 250-500 appears to be the point of diminishing return, with
      almost 9x speed up. Most benchmarks use 500 as the default sampling for pb-raw
      and pb-custom.
      
      Ringbuf back-to-back, effect of sample rate
      ===========================================
      rb-sampled-1         1.106 ± 0.010M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-5         4.746 ± 0.149M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-10        7.706 ± 0.164M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-25        12.893 ± 0.273M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-50        15.961 ± 0.361M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-100       18.203 ± 0.445M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-250       19.962 ± 0.786M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-500       20.881 ± 0.551M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-1000      21.317 ± 0.532M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-2000      21.331 ± 0.535M/s (drops 0.000 ± 0.000M/s)
      rb-sampled-3000      21.688 ± 0.392M/s (drops 0.000 ± 0.000M/s)
      
      Similar benchmark for ring buffer also shows a great advantage (in terms of
      throughput) of skipping notifications. Skipping every 5th one gives 4x boost.
      Also similar to perfbuf case, 250-500 seems to be the point of diminishing
      returns, giving roughly 20x better results.
      
      Keep in mind, for this test, notifications are controlled manually with
      BPF_RB_NO_WAKEUP and BPF_RB_FORCE_WAKEUP. As can be seen from previous
      benchmarks, adaptive notifications based on consumer's positions provides same
      (or even slightly better due to simpler load generator on BPF side) benefits in
      favorable back-to-back scenario. Over zealous and fast consumer, which is
      almost always caught up, will make thoughput numbers smaller. That's the case
      when manual notification control might prove to be extremely beneficial.
      
      Ringbuf back-to-back, reserve+commit vs output
      ==============================================
      reserve              22.819 ± 0.503M/s (drops 0.000 ± 0.000M/s)
      output               18.906 ± 0.433M/s (drops 0.000 ± 0.000M/s)
      
      Ringbuf sampled, reserve+commit vs output
      =========================================
      reserve-sampled      15.350 ± 0.132M/s (drops 0.000 ± 0.000M/s)
      output-sampled       14.195 ± 0.144M/s (drops 0.000 ± 0.000M/s)
      
      BPF ringbuf supports two sets of APIs with various usability and performance
      tradeoffs: bpf_ringbuf_reserve()+bpf_ringbuf_commit() vs bpf_ringbuf_output().
      This benchmark clearly shows superiority of reserve+commit approach, despite
      using a small 8-byte record size.
      
      Single-producer, consumer/producer competing on the same CPU, low batch count
      =============================================================================
      rb-libbpf            3.045 ± 0.020M/s (drops 3.536 ± 0.148M/s)
      rb-custom            3.055 ± 0.022M/s (drops 3.893 ± 0.066M/s)
      pb-libbpf            1.393 ± 0.024M/s (drops 0.000 ± 0.000M/s)
      pb-custom            1.407 ± 0.016M/s (drops 0.000 ± 0.000M/s)
      
      This benchmark shows one of the worst-case scenarios, in which producer and
      consumer do not coordinate *and* fight for the same CPU. No batch count and
      sampling settings were able to eliminate drops for ringbuffer, producer is
      just too fast for consumer to keep up. But ringbuf and perfbuf still able to
      pass through quite a lot of messages, which is more than enough for a lot of
      applications.
      
      Ringbuf, multi-producer contention
      ==================================
      rb-libbpf nr_prod 1  10.916 ± 0.399M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 2  4.931 ± 0.030M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 3  4.880 ± 0.006M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 4  3.926 ± 0.004M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 8  4.011 ± 0.004M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 12 3.967 ± 0.016M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 16 2.604 ± 0.030M/s (drops 0.001 ± 0.002M/s)
      rb-libbpf nr_prod 20 2.233 ± 0.003M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 24 2.085 ± 0.015M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 28 2.055 ± 0.004M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 32 1.962 ± 0.004M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 36 2.089 ± 0.005M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 40 2.118 ± 0.006M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 44 2.105 ± 0.004M/s (drops 0.000 ± 0.000M/s)
      rb-libbpf nr_prod 48 2.120 ± 0.058M/s (drops 0.000 ± 0.001M/s)
      rb-libbpf nr_prod 52 2.074 ± 0.024M/s (drops 0.007 ± 0.014M/s)
      
      Ringbuf uses a very short-duration spinlock during reservation phase, to check
      few invariants, increment producer count and set record header. This is the
      biggest point of contention for ringbuf implementation. This benchmark
      evaluates the effect of multiple competing writers on overall throughput of
      a single shared ringbuffer.
      
      Overall throughput drops almost 2x when going from single to two
      highly-contended producers, gradually dropping with additional competing
      producers.  Performance drop stabilizes at around 20 producers and hovers
      around 2mln even with 50+ fighting producers, which is a 5x drop compared to
      non-contended case. Good kernel implementation in kernel helps maintain decent
      performance here.
      
      Note, that in the intended real-world scenarios, it's not expected to get even
      close to such a high levels of contention. But if contention will become
      a problem, there is always an option of sharding few ring buffers across a set
      of CPUs.
      Signed-off-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200529075424.3139988-5-andriin@fb.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      c97099b0
    • Andrii Nakryiko's avatar
      selftests/bpf: Add BPF ringbuf selftests · cb1c9ddd
      Andrii Nakryiko authored
      Both singleton BPF ringbuf and BPF ringbuf with map-in-map use cases are tested.
      Also reserve+submit/discards and output variants of API are validated.
      Signed-off-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200529075424.3139988-4-andriin@fb.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      cb1c9ddd
    • Andrii Nakryiko's avatar
      libbpf: Add BPF ring buffer support · bf99c936
      Andrii Nakryiko authored
      Declaring and instantiating BPF ring buffer doesn't require any changes to
      libbpf, as it's just another type of maps. So using existing BTF-defined maps
      syntax with __uint(type, BPF_MAP_TYPE_RINGBUF) and __uint(max_elements,
      <size-of-ring-buf>) is all that's necessary to create and use BPF ring buffer.
      
      This patch adds BPF ring buffer consumer to libbpf. It is very similar to
      perf_buffer implementation in terms of API, but also attempts to fix some
      minor problems and inconveniences with existing perf_buffer API.
      
      ring_buffer support both single ring buffer use case (with just using
      ring_buffer__new()), as well as allows to add more ring buffers, each with its
      own callback and context. This allows to efficiently poll and consume
      multiple, potentially completely independent, ring buffers, using single
      epoll instance.
      
      The latter is actually a problem in practice for applications
      that are using multiple sets of perf buffers. They have to create multiple
      instances for struct perf_buffer and poll them independently or in a loop,
      each approach having its own problems (e.g., inability to use a common poll
      timeout). struct ring_buffer eliminates this problem by aggregating many
      independent ring buffer instances under the single "ring buffer manager".
      
      Second, perf_buffer's callback can't return error, so applications that need
      to stop polling due to error in data or data signalling the end, have to use
      extra mechanisms to signal that polling has to stop. ring_buffer's callback
      can return error, which will be passed through back to user code and can be
      acted upon appropariately.
      
      Two APIs allow to consume ring buffer data:
        - ring_buffer__poll(), which will wait for data availability notification
          and will consume data only from reported ring buffer(s); this API allows
          to efficiently use resources by reading data only when it becomes
          available;
        - ring_buffer__consume(), will attempt to read new records regardless of
          data availablity notification sub-system. This API is useful for cases
          when lowest latency is required, in expense of burning CPU resources.
      Signed-off-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200529075424.3139988-3-andriin@fb.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      bf99c936
    • Andrii Nakryiko's avatar
      bpf: Implement BPF ring buffer and verifier support for it · 457f4436
      Andrii Nakryiko authored
      This commit adds a new MPSC ring buffer implementation into BPF ecosystem,
      which allows multiple CPUs to submit data to a single shared ring buffer. On
      the consumption side, only single consumer is assumed.
      
      Motivation
      ----------
      There are two distinctive motivators for this work, which are not satisfied by
      existing perf buffer, which prompted creation of a new ring buffer
      implementation.
        - more efficient memory utilization by sharing ring buffer across CPUs;
        - preserving ordering of events that happen sequentially in time, even
        across multiple CPUs (e.g., fork/exec/exit events for a task).
      
      These two problems are independent, but perf buffer fails to satisfy both.
      Both are a result of a choice to have per-CPU perf ring buffer.  Both can be
      also solved by having an MPSC implementation of ring buffer. The ordering
      problem could technically be solved for perf buffer with some in-kernel
      counting, but given the first one requires an MPSC buffer, the same solution
      would solve the second problem automatically.
      
      Semantics and APIs
      ------------------
      Single ring buffer is presented to BPF programs as an instance of BPF map of
      type BPF_MAP_TYPE_RINGBUF. Two other alternatives considered, but ultimately
      rejected.
      
      One way would be to, similar to BPF_MAP_TYPE_PERF_EVENT_ARRAY, make
      BPF_MAP_TYPE_RINGBUF could represent an array of ring buffers, but not enforce
      "same CPU only" rule. This would be more familiar interface compatible with
      existing perf buffer use in BPF, but would fail if application needed more
      advanced logic to lookup ring buffer by arbitrary key. HASH_OF_MAPS addresses
      this with current approach. Additionally, given the performance of BPF
      ringbuf, many use cases would just opt into a simple single ring buffer shared
      among all CPUs, for which current approach would be an overkill.
      
      Another approach could introduce a new concept, alongside BPF map, to
      represent generic "container" object, which doesn't necessarily have key/value
      interface with lookup/update/delete operations. This approach would add a lot
      of extra infrastructure that has to be built for observability and verifier
      support. It would also add another concept that BPF developers would have to
      familiarize themselves with, new syntax in libbpf, etc. But then would really
      provide no additional benefits over the approach of using a map.
      BPF_MAP_TYPE_RINGBUF doesn't support lookup/update/delete operations, but so
      doesn't few other map types (e.g., queue and stack; array doesn't support
      delete, etc).
      
      The approach chosen has an advantage of re-using existing BPF map
      infrastructure (introspection APIs in kernel, libbpf support, etc), being
      familiar concept (no need to teach users a new type of object in BPF program),
      and utilizing existing tooling (bpftool). For common scenario of using
      a single ring buffer for all CPUs, it's as simple and straightforward, as
      would be with a dedicated "container" object. On the other hand, by being
      a map, it can be combined with ARRAY_OF_MAPS and HASH_OF_MAPS map-in-maps to
      implement a wide variety of topologies, from one ring buffer for each CPU
      (e.g., as a replacement for perf buffer use cases), to a complicated
      application hashing/sharding of ring buffers (e.g., having a small pool of
      ring buffers with hashed task's tgid being a look up key to preserve order,
      but reduce contention).
      
      Key and value sizes are enforced to be zero. max_entries is used to specify
      the size of ring buffer and has to be a power of 2 value.
      
      There are a bunch of similarities between perf buffer
      (BPF_MAP_TYPE_PERF_EVENT_ARRAY) and new BPF ring buffer semantics:
        - variable-length records;
        - if there is no more space left in ring buffer, reservation fails, no
          blocking;
        - memory-mappable data area for user-space applications for ease of
          consumption and high performance;
        - epoll notifications for new incoming data;
        - but still the ability to do busy polling for new data to achieve the
          lowest latency, if necessary.
      
      BPF ringbuf provides two sets of APIs to BPF programs:
        - bpf_ringbuf_output() allows to *copy* data from one place to a ring
          buffer, similarly to bpf_perf_event_output();
        - bpf_ringbuf_reserve()/bpf_ringbuf_commit()/bpf_ringbuf_discard() APIs
          split the whole process into two steps. First, a fixed amount of space is
          reserved. If successful, a pointer to a data inside ring buffer data area
          is returned, which BPF programs can use similarly to a data inside
          array/hash maps. Once ready, this piece of memory is either committed or
          discarded. Discard is similar to commit, but makes consumer ignore the
          record.
      
      bpf_ringbuf_output() has disadvantage of incurring extra memory copy, because
      record has to be prepared in some other place first. But it allows to submit
      records of the length that's not known to verifier beforehand. It also closely
      matches bpf_perf_event_output(), so will simplify migration significantly.
      
      bpf_ringbuf_reserve() avoids the extra copy of memory by providing a memory
      pointer directly to ring buffer memory. In a lot of cases records are larger
      than BPF stack space allows, so many programs have use extra per-CPU array as
      a temporary heap for preparing sample. bpf_ringbuf_reserve() avoid this needs
      completely. But in exchange, it only allows a known constant size of memory to
      be reserved, such that verifier can verify that BPF program can't access
      memory outside its reserved record space. bpf_ringbuf_output(), while slightly
      slower due to extra memory copy, covers some use cases that are not suitable
      for bpf_ringbuf_reserve().
      
      The difference between commit and discard is very small. Discard just marks
      a record as discarded, and such records are supposed to be ignored by consumer
      code. Discard is useful for some advanced use-cases, such as ensuring
      all-or-nothing multi-record submission, or emulating temporary malloc()/free()
      within single BPF program invocation.
      
      Each reserved record is tracked by verifier through existing
      reference-tracking logic, similar to socket ref-tracking. It is thus
      impossible to reserve a record, but forget to submit (or discard) it.
      
      bpf_ringbuf_query() helper allows to query various properties of ring buffer.
      Currently 4 are supported:
        - BPF_RB_AVAIL_DATA returns amount of unconsumed data in ring buffer;
        - BPF_RB_RING_SIZE returns the size of ring buffer;
        - BPF_RB_CONS_POS/BPF_RB_PROD_POS returns current logical possition of
          consumer/producer, respectively.
      Returned values are momentarily snapshots of ring buffer state and could be
      off by the time helper returns, so this should be used only for
      debugging/reporting reasons or for implementing various heuristics, that take
      into account highly-changeable nature of some of those characteristics.
      
      One such heuristic might involve more fine-grained control over poll/epoll
      notifications about new data availability in ring buffer. Together with
      BPF_RB_NO_WAKEUP/BPF_RB_FORCE_WAKEUP flags for output/commit/discard helpers,
      it allows BPF program a high degree of control and, e.g., more efficient
      batched notifications. Default self-balancing strategy, though, should be
      adequate for most applications and will work reliable and efficiently already.
      
      Design and implementation
      -------------------------
      This reserve/commit schema allows a natural way for multiple producers, either
      on different CPUs or even on the same CPU/in the same BPF program, to reserve
      independent records and work with them without blocking other producers. This
      means that if BPF program was interruped by another BPF program sharing the
      same ring buffer, they will both get a record reserved (provided there is
      enough space left) and can work with it and submit it independently. This
      applies to NMI context as well, except that due to using a spinlock during
      reservation, in NMI context, bpf_ringbuf_reserve() might fail to get a lock,
      in which case reservation will fail even if ring buffer is not full.
      
      The ring buffer itself internally is implemented as a power-of-2 sized
      circular buffer, with two logical and ever-increasing counters (which might
      wrap around on 32-bit architectures, that's not a problem):
        - consumer counter shows up to which logical position consumer consumed the
          data;
        - producer counter denotes amount of data reserved by all producers.
      
      Each time a record is reserved, producer that "owns" the record will
      successfully advance producer counter. At that point, data is still not yet
      ready to be consumed, though. Each record has 8 byte header, which contains
      the length of reserved record, as well as two extra bits: busy bit to denote
      that record is still being worked on, and discard bit, which might be set at
      commit time if record is discarded. In the latter case, consumer is supposed
      to skip the record and move on to the next one. Record header also encodes
      record's relative offset from the beginning of ring buffer data area (in
      pages). This allows bpf_ringbuf_commit()/bpf_ringbuf_discard() to accept only
      the pointer to the record itself, without requiring also the pointer to ring
      buffer itself. Ring buffer memory location will be restored from record
      metadata header. This significantly simplifies verifier, as well as improving
      API usability.
      
      Producer counter increments are serialized under spinlock, so there is
      a strict ordering between reservations. Commits, on the other hand, are
      completely lockless and independent. All records become available to consumer
      in the order of reservations, but only after all previous records where
      already committed. It is thus possible for slow producers to temporarily hold
      off submitted records, that were reserved later.
      
      Reservation/commit/consumer protocol is verified by litmus tests in
      Documentation/litmus-test/bpf-rb.
      
      One interesting implementation bit, that significantly simplifies (and thus
      speeds up as well) implementation of both producers and consumers is how data
      area is mapped twice contiguously back-to-back in the virtual memory. This
      allows to not take any special measures for samples that have to wrap around
      at the end of the circular buffer data area, because the next page after the
      last data page would be first data page again, and thus the sample will still
      appear completely contiguous in virtual memory. See comment and a simple ASCII
      diagram showing this visually in bpf_ringbuf_area_alloc().
      
      Another feature that distinguishes BPF ringbuf from perf ring buffer is
      a self-pacing notifications of new data being availability.
      bpf_ringbuf_commit() implementation will send a notification of new record
      being available after commit only if consumer has already caught up right up
      to the record being committed. If not, consumer still has to catch up and thus
      will see new data anyways without needing an extra poll notification.
      Benchmarks (see tools/testing/selftests/bpf/benchs/bench_ringbuf.c) show that
      this allows to achieve a very high throughput without having to resort to
      tricks like "notify only every Nth sample", which are necessary with perf
      buffer. For extreme cases, when BPF program wants more manual control of
      notifications, commit/discard/output helpers accept BPF_RB_NO_WAKEUP and
      BPF_RB_FORCE_WAKEUP flags, which give full control over notifications of data
      availability, but require extra caution and diligence in using this API.
      
      Comparison to alternatives
      --------------------------
      Before considering implementing BPF ring buffer from scratch existing
      alternatives in kernel were evaluated, but didn't seem to meet the needs. They
      largely fell into few categores:
        - per-CPU buffers (perf, ftrace, etc), which don't satisfy two motivations
          outlined above (ordering and memory consumption);
        - linked list-based implementations; while some were multi-producer designs,
          consuming these from user-space would be very complicated and most
          probably not performant; memory-mapping contiguous piece of memory is
          simpler and more performant for user-space consumers;
        - io_uring is SPSC, but also requires fixed-sized elements. Naively turning
          SPSC queue into MPSC w/ lock would have subpar performance compared to
          locked reserve + lockless commit, as with BPF ring buffer. Fixed sized
          elements would be too limiting for BPF programs, given existing BPF
          programs heavily rely on variable-sized perf buffer already;
        - specialized implementations (like a new printk ring buffer, [0]) with lots
          of printk-specific limitations and implications, that didn't seem to fit
          well for intended use with BPF programs.
      
        [0] https://lwn.net/Articles/779550/Signed-off-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200529075424.3139988-2-andriin@fb.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      457f4436
    • Anton Protopopov's avatar
      bpf: Fix map permissions check · 1ea0f912
      Anton Protopopov authored
      The map_lookup_and_delete_elem() function should check for both FMODE_CAN_WRITE
      and FMODE_CAN_READ permissions because it returns a map element to user space.
      
      Fixes: bd513cd0 ("bpf: add MAP_LOOKUP_AND_DELETE_ELEM syscall")
      Signed-off-by: default avatarAnton Protopopov <a.s.protopopov@gmail.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200527185700.14658-5-a.s.protopopov@gmail.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      1ea0f912
    • Anton Protopopov's avatar
      selftests/bpf: Cleanup comments in test_maps · efbc3b8f
      Anton Protopopov authored
      Make comments inside the test_map_rdonly and test_map_wronly tests
      consistent with logic.
      Signed-off-by: default avatarAnton Protopopov <a.s.protopopov@gmail.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200527185700.14658-4-a.s.protopopov@gmail.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      efbc3b8f
    • Anton Protopopov's avatar
      selftests/bpf: Cleanup some file descriptors in test_maps · 36ef9a2d
      Anton Protopopov authored
      The test_map_rdonly and test_map_wronly tests should close file descriptors
      which they open.
      Signed-off-by: default avatarAnton Protopopov <a.s.protopopov@gmail.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200527185700.14658-3-a.s.protopopov@gmail.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      36ef9a2d
    • Anton Protopopov's avatar
      selftests/bpf: Fix a typo in test_maps · 204fb041
      Anton Protopopov authored
      Trivial fix to a typo in the test_map_wronly test: "read" -> "write"
      Signed-off-by: default avatarAnton Protopopov <a.s.protopopov@gmail.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Link: https://lore.kernel.org/bpf/20200527185700.14658-2-a.s.protopopov@gmail.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      204fb041
    • Eelco Chaudron's avatar
      libbpf: Fix perf_buffer__free() API for sparse allocs · 601b05ca
      Eelco Chaudron authored
      In case the cpu_bufs are sparsely allocated they are not all
      free'ed. These changes will fix this.
      
      Fixes: fb84b822 ("libbpf: add perf buffer API")
      Signed-off-by: default avatarEelco Chaudron <echaudro@redhat.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Link: https://lore.kernel.org/bpf/159056888305.330763.9684536967379110349.stgit@ebuildSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      601b05ca
    • John Fastabend's avatar
      bpf, selftests: Test probe_* helpers from SCHED_CLS · ee103e9f
      John Fastabend authored
      Lets test using probe* in SCHED_CLS network programs as well just
      to be sure these keep working. Its cheap to add the extra test
      and provides a second context to test outside of sk_msg after
      we generalized probe* helpers to all networking types.
      Signed-off-by: default avatarJohn Fastabend <john.fastabend@gmail.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarYonghong Song <yhs@fb.com>
      Acked-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Link: https://lore.kernel.org/bpf/159033911685.12355.15951980509828906214.stgit@john-Precision-5820-TowerSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      ee103e9f
    • John Fastabend's avatar
      bpf, selftests: Add sk_msg helpers load and attach test · 1d9c037a
      John Fastabend authored
      The test itself is not particularly useful but it encodes a common
      pattern we have.
      
      Namely do a sk storage lookup then depending on data here decide if
      we need to do more work or alternatively allow packet to PASS. Then
      if we need to do more work consult task_struct for more information
      about the running task. Finally based on this additional information
      drop or pass the data. In this case the suspicious check is not so
      realisitic but it encodes the general pattern and uses the helpers
      so we test the workflow.
      
      This is a load test to ensure verifier correctly handles this case.
      Signed-off-by: default avatarJohn Fastabend <john.fastabend@gmail.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Link: https://lore.kernel.org/bpf/159033909665.12355.6166415847337547879.stgit@john-Precision-5820-TowerSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      1d9c037a
    • John Fastabend's avatar
    • John Fastabend's avatar
      bpf: Extend bpf_base_func_proto helpers with probe_* and *current_task* · f470378c
      John Fastabend authored
      Often it is useful when applying policy to know something about the
      task. If the administrator has CAP_SYS_ADMIN rights then they can
      use kprobe + networking hook and link the two programs together to
      accomplish this. However, this is a bit clunky and also means we have
      to call both the network program and kprobe program when we could just
      use a single program and avoid passing metadata through sk_msg/skb->cb,
      socket, maps, etc.
      
      To accomplish this add probe_* helpers to bpf_base_func_proto programs
      guarded by a perfmon_capable() check. New supported helpers are the
      following,
      
       BPF_FUNC_get_current_task
       BPF_FUNC_probe_read_user
       BPF_FUNC_probe_read_kernel
       BPF_FUNC_probe_read_user_str
       BPF_FUNC_probe_read_kernel_str
      Signed-off-by: default avatarJohn Fastabend <john.fastabend@gmail.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarYonghong Song <yhs@fb.com>
      Link: https://lore.kernel.org/bpf/159033905529.12355.4368381069655254932.stgit@john-Precision-5820-TowerSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      f470378c
    • John Fastabend's avatar
      bpf, sk_msg: Add some generic helpers that may be useful from sk_msg · abe3cac8
      John Fastabend authored
      Add these generic helpers that may be useful to use from sk_msg programs.
      The helpers do not depend on ctx so we can simply add them here,
      
       BPF_FUNC_perf_event_output
       BPF_FUNC_get_current_uid_gid
       BPF_FUNC_get_current_pid_tgid
       BPF_FUNC_get_current_cgroup_id
       BPF_FUNC_get_current_ancestor_cgroup_id
       BPF_FUNC_get_cgroup_classid
      Signed-off-by: default avatarJohn Fastabend <john.fastabend@gmail.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarYonghong Song <yhs@fb.com>
      Link: https://lore.kernel.org/bpf/159033903373.12355.15489763099696629346.stgit@john-Precision-5820-TowerSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      abe3cac8
    • Yauheni Kaliuta's avatar
      libbpf: Use .so dynamic symbols for abi check · 55983299
      Yauheni Kaliuta authored
      Since dynamic symbols are used for dynamic linking it makes sense to
      use them (readelf --dyn-syms) for abi check.
      
      Found with some configuration on powerpc where linker puts
      local *.plt_call.* symbols into .so.
      Signed-off-by: default avatarYauheni Kaliuta <yauheni.kaliuta@redhat.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Link: https://lore.kernel.org/bpf/20200525061846.16524-1-yauheni.kaliuta@redhat.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      55983299
    • Chris Packham's avatar
    • Nikolay Borisov's avatar
      libbpf: Install headers as part of make install · 93581359
      Nikolay Borisov authored
      Current 'make install' results in only pkg-config and library binaries
      being installed. For consistency also install headers as part of
      "make install"
      Signed-off-by: default avatarNikolay Borisov <nborisov@suse.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Link: https://lore.kernel.org/bpf/20200526174612.5447-1-nborisov@suse.comSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      93581359
    • Eelco Chaudron's avatar
      libbpf: Add API to consume the perf ring buffer content · 272d51af
      Eelco Chaudron authored
      This new API, perf_buffer__consume, can be used as follows:
      
      - When you have a perf ring where wakeup_events is higher than 1,
        and you have remaining data in the rings you would like to pull
        out on exit (or maybe based on a timeout).
      
      - For low latency cases where you burn a CPU that constantly polls
        the queues.
      Signed-off-by: default avatarEelco Chaudron <echaudro@redhat.com>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarAndrii Nakryiko <andriin@fb.com>
      Link: https://lore.kernel.org/bpf/159048487929.89441.7465713173442594608.stgit@ebuildSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      272d51af