1. 30 Oct, 2018 3 commits
    • yonghong-song's avatar
      allow packed structure in bpf program in python API (#2020) · 9410c86d
      yonghong-song authored
      Fix issue #2017.
      
      For python programs, the map data passed from C++ library
      is parsed through the key/value desc types which are
      generated by C++ json map declaration visitor.
      
      The map declaration visitor visits the map key/value
      declaration types and generate a string to represent
      the type, which is later used by python to reconstruct
      the cttype.
      
      The map declaration already tries to add all the padding
      to the type string in order to make sure C++ and python
      see the same layout.
      
      This patch further added packed support such that if
      C++ json map visitor has applied padding, the python
      type reconstructor is free to add _pack_=1 for structure
      type since the structure is already packed.
      
      For example, for a type,
        struct t { char a; int b; }
      the structure after json visitor will look like
        struct t { char a; char __pad[3]; int b; }
      
      If the type is
        struct t { char a; int b; } __packed;
      the structure after json visitor will look like
        struct t { char a; int b; }
      
      In either case, it will be okay to add __packed attribute
      for the type generated by json map visitor in order to
      match the original declaration.
      
      Thanks Chaitanya for filing the issue and providing the test case!
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      9410c86d
    • yonghong-song's avatar
      Merge pull request #2021 from cyphar/32bit-uint128 · b81dcb7c
      yonghong-song authored
      frontends: disable uint128_t on 32-bit builds
      b81dcb7c
    • Aleksa Sarai's avatar
      frontends: disable uint128_t on 32-bit builds · 329faafd
      Aleksa Sarai authored
      On 32-bit, __uint128_t is generally not available. Removing it fixes
      builds on those systems.
      Signed-off-by: default avatarAleksa Sarai <cyphar@cyphar.com>
      329faafd
  2. 29 Oct, 2018 2 commits
    • yonghong-song's avatar
      Merge pull request #2019 from iovisor/yhs_dev · 895aefda
      yonghong-song authored
      correct certain tracepoint types in tp frontend action
      895aefda
    • Yonghong Song's avatar
      use proper sync for function pointers · 1589a4f0
      Yonghong Song authored
      This is a correction for previous patch where
      function pointer is not proper specified
        static int bpf_map_push_elem(struct bpf_map *map, const void *value, u64 flags) =
           (void *) BPF_FUNC_map_push_elem;
      and some old gcc compiler (e.g., ubuntu 16.04) will complain
      with error:
        Illegal initializer (only variables can be initialized)
      The correct way is
        static int (*bpf_map_push_elem)(struct bpf_map *map, const void *value, u64 flags) =
           (void *) BPF_FUNC_map_push_elem;
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      1589a4f0
  3. 28 Oct, 2018 3 commits
    • Yonghong Song's avatar
      correct certain tracepoint types in tp frontend action · 0e02fec8
      Yonghong Song authored
      Fix issue #2010.
      
      Alastair reported some missing cases in tp frontend action
      where certain types are not adjusted.
      For example, for tracepoint syscalls:sys_enter_kill,
      the kernel format:
              ...
              field:int __syscall_nr; offset:8;       size:4; signed:1;
              field:pid_t pid;        offset:16;      size:8; signed:0;
              field:int sig;  offset:24;      size:8; signed:0;
      
      The size for "pid_t pid" is 8 bytes, but the kernel pid_t
      is "int", so it is needed to change its type to "s64"
      to be consistent to its size.
      
      This patch also added conversion for gid_t and uid_t as Alastair
      discovered that they are also used in some tracepoints with size 8.
      For type gid_t and uid_t, the corresponding kernel type is "unsigned int",
      so it should be converted to "u64".
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      0e02fec8
    • 4ast's avatar
      Merge pull request #2018 from iovisor/yhs_dev · f3fc87aa
      4ast authored
      sync with latest net-next
      f3fc87aa
    • Yonghong Song's avatar
      sync with latest net-next · 2a68bd8b
      Yonghong Song authored
      Sync bpf uapi header bpf.h with latest net-next
      and update related documentation.
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      2a68bd8b
  4. 15 Oct, 2018 2 commits
    • 4ast's avatar
      Merge pull request #2011 from iovisor/yhs_dev · afd00154
      4ast authored
      fix bcc build failure with latest llvm/clang trunk
      afd00154
    • Yonghong Song's avatar
      fix bcc build failure with latest llvm/clang trunk · 331ab236
      Yonghong Song authored
      The clang commit
        commit 247be3e41c41201641af01888ff0d84484c4e0f4
        Author: Simon Pilgrim <llvm-dev@redking.me.uk>
        Date:   Fri Oct 12 19:14:43 2018 +0000
      
          Fix a clang::driver::ArgStringList usage I missed in rL344398. NFCI.
      removed
        using ArgStringList = llvm::opt::ArgStringList;
      in clang:include/clang/Driver/Job.h,
      and makes bcc compilation failure.
      
      Using llvm::opt::ArgStringList fixed the issue and should
      still work with previous llvm versions.
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      331ab236
  5. 12 Oct, 2018 1 commit
    • jeromemarchand's avatar
      Tools fixes (#2009) · 09f9d3c8
      jeromemarchand authored
      * tools: the argument of get_kprobe_functions() should be a bytes object
      
      It fixes the following error:
      Traceback (most recent call last):
        File "/usr/share/bcc/tools/ext4dist", line 189, in <module>
          if BPF.get_kprobe_functions('ext4_file_read_iter'):
        File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 519, in get_kprobe_functions
          if (t.lower() in [b't', b'w']) and re.match(event_re, fn) \
        File "/usr/lib64/python3.6/re.py", line 172, in match
          return _compile(pattern, flags).match(string)
      TypeError: cannot use a string pattern on a bytes-like object
      
      * tools: fix freeze in offwaketime
      
      Since commit 47cecb65, the sigint signal is ignored
      unconditionally which prevents offwaketime to be stopped by
      pressing Ctrl-C when run without a specified duration.
      09f9d3c8
  6. 11 Oct, 2018 1 commit
  7. 10 Oct, 2018 2 commits
    • yonghong-song's avatar
      sync with latest net-next bpf headers (#2001) · d4b23876
      yonghong-song authored
      Sync compat bpf.h and virtual_bpf.h with latest net-next versions
      
      Also add a barrier for the test_brb bpf program like below
               meta.prog_id = skb->cb[0];
      +        asm volatile("" ::: "memory");
               meta.rx_port_id = skb->cb[1];
      so two cb read does not become a 8 byte load which is not
      allowed for ubuntu 16.04 kernel (4.4 based). The 8 byte load
      is allowed for skb->cb's for later kernels.
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      d4b23876
    • Marko Myllynen's avatar
      utools Tcl support (#2005) · 9f3662ee
      Marko Myllynen authored
      * Add perl* utools man page links
      
      Matches other languages, related to commit 9162be45.
      
      * uthreads: explicitly list C as supported
      
      uthreads supports C (pthreads) thread tracing, make this explicit.
      
      Earlier use of -l java and -l none unchanged.
      
      Add the corresponding man page and usage examples.
      
      * Add Tcl support for ucalls / uflow / uobjnew / ustat
      
      For some reason we need to use proc__args instead of proc__entry
      in uflow.py to get similar results as with e.g. Python.
      9f3662ee
  8. 09 Oct, 2018 3 commits
  9. 07 Oct, 2018 1 commit
  10. 05 Oct, 2018 3 commits
  11. 04 Oct, 2018 1 commit
  12. 03 Oct, 2018 1 commit
    • Marek Vavruša's avatar
      lua: codegen tests, tracking inverse form of null guards, IPv6 header builtins (#1991) · d1791f0b
      Marek Vavruša authored
      * lua: eliding repetitive NULL checks, fixes for read types, luacheck
      
      This fixes verifier failures in some kernel versions that track whether
      a possible NULL pointer has been check (and shouldn't be checked twice).
      
      It also fixes accessing memory from maps with composite keys.
      
      It fixes a case when temporary variable is used in condition,
      and the body of the condition contains just an assignment from
      immediate value to a variable that existed before the condition, e.g.:
      
      ```
      local x = 1 -- Assign constant to 'x', it will be materialized here
      if skb.len > 0 then -- New BB, not possible to fold as condition isn't const
          x = 2 -- Assign constant, it failed to materialize here
      end -- End of BB
      -- Value of x is always 1
      ```
      
      The `bpf.socket()` support ljsyscall socket as first parameter, and
      fixes compatibility with newer ljsyscall (type name has changed).
      
      * reverse BPF_LD ntohl() semantics for <= 32bit loads as well
      
      The loads using traditional instructions always do ntohl():
      
      https://www.kernel.org/doc/Documentation/networking/filter.txt
      
      They are however needed to support reads not using `skb`, e.g. NET_OFF
      
      * proto: add builtins for traversing IPv6 header and extension headers
      
      The IPv6 header is fixed size (40B), but it may contain either
      extension header, or transport  protocol after it, so the caller
      must check the `next_header` field before traversing to determine
      which dissector to use to traverse it, e.g.:
      
      ```
      if ip6.next_header == 44 then
         ip6 = ip6.ip6_opt -- Skip fragment header
      end
      if ip6.next_header == c.ip.proto_tcp then
         local tcp = ip6.tcp -- Finally, TCP
      end
      ```
      
      * reverse ntohl() for indirect BPF_LD as well
      
      * lua: started codegen tests, direct skb support, bugfixes
      
      This starts adding compiler correctness tests for basic expressions,
      variable source tracking, and control flow.
      
      Added:
      * direct skb->data access (in addition to BPF_LDABS, BPF_LDIND)
      * loads and stores from skb->data and map value backed memory
      * unified variable source tracking (ptr_to_ctx, ptr_to_map[_or_null], ptr_to_skb, ptr_to_pkt, ptr_to_stack)
      * BPF constants introduced between 4.10-4.15
      * bpf.dump_string() to dump generated assembly (text) to string
      
      Fixes:
      * pointer nil check tracking
      * dissectors for map value backed memory
      * ljsyscall extensions when the version is too old
      * KPRI nil variables used in conditions
      * wrongly elided constant materialization on condition-less jumps
      * loads/stores from stack memory using variable offset
      
      * lua: track inverse null guards (if x ~= nil)
      
      The verifier prohibits pointer comparisons except the first NULL check,
      so both forms must be tracked, otherwise it will check for NULL twice
      and verifier will reject it.
      
      * lua: support cdata and constants larger than i32, fix shadowed variables
      
      This adds support for numeric cdata constants (either in program, or
      retrieved with GGET/UGET). Values larger than i32 are coerced into i64.
      
      This also fixes shadowing of variables, and fixes materialization of
      result of variable copy at the end of basic block.
      d1791f0b
  13. 01 Oct, 2018 1 commit
  14. 28 Sep, 2018 1 commit
    • Mauricio Vásquez's avatar
      Add support for shared tables (#1988) · 54e377d2
      Mauricio Vásquez authored
      Currently tables can be created in two modes:
      1. private to the ebpf program
      2. shared among all the ebpf programs created by the same user-space
      process
      
      This commit extends bcc allowing to create a table that is shared among
      a set of specific ebpf programs.
      This is useful when creating network functions that are composed
      by more than 1 ebpf program, a map is shared among all the programs of
      the network function but isolated from different instances of that
      function.
      Signed-off-by: default avatarMauricio Vasquez B <mauricio.vasquez@polito.it>
      54e377d2
  15. 27 Sep, 2018 1 commit
  16. 26 Sep, 2018 1 commit
  17. 25 Sep, 2018 1 commit
  18. 21 Sep, 2018 5 commits
  19. 20 Sep, 2018 1 commit
  20. 19 Sep, 2018 3 commits
    • Mark Drayton's avatar
      biolatency: Fix --disks bpf_probe_read() (#1980) · fc245dfe
      Mark Drayton authored
      bpf_probe_read()'s third argument is no longer rewritten. Instead, use a
      temporary variable (like #1973) to avoid a memory access error.
      
      Before:
      
      ```
      $ sudo biolatency -D 1
      bpf: Failed to load program: Permission denied
      0: (79) r1 = *(u64 *)(r1 +112)
      1: (7b) *(u64 *)(r10 -8) = r1
      [..]
      R1 invalid mem access 'inv'
      
      HINT: The invalid mem access 'inv' error can happen if you try to
      dereference memory without first using bpf_probe_read() to copy it to
      the BPF stack. Sometimes the bpf_probe_read is automatic by the bcc
      rewriter, other times you'll need to be explicit.
      ```
      
      After, works as expected.
      fc245dfe
    • yonghong-song's avatar
      support "long" and "unsigned long" type in trace.py (#1977) · f720257c
      yonghong-song authored
      Currently, trace.py does not support "long" and "unsigned long"
      types and it often caught users with a surprise and they are
      not sure what is the problem. For example, for kernel function:
        void blk_mq_delay_kick_requeue_list(struct request_queue *q, unsigned long msecs)
      The following
        $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %lu", msecs'
        list index out of range
      
      With this patch,
        $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %lu", msecs'
        PID     TID     COMM            FUNC             -
        ^C
        $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %ld", msecs'
        PID     TID     COMM            FUNC             -
        ^C
        $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %lx", msecs'
        PID     TID     COMM            FUNC             -
        ^C
        $
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      f720257c
    • yonghong-song's avatar
      fix compilation error with latest llvm 8.0 trunk (#1976) · 72bb0d56
      yonghong-song authored
      In llvm/clang 8 trunk, the following commits
        https://reviews.llvm.org/rL339384
        https://reviews.llvm.org/rL339385
        https://reviews.llvm.org/rL339386
      renames getLocStart to getBeginLoc, and getLocEnd to getEndLoc.
      This caused bcc compilation error with llvm/clang 8.
      To avoid proliferation of #define's based on LLVM version numbers
      and similar logic for many different types,
      this patch fixed the issues by define macros in common
      frontend_action_common.h where macro definition is multi-versioned
      based on llvm version. The macro itself is used in
      {b,tp}_frontend_action.cc.
      
      clang 8 also introduced a new library dependency
        libclangAnalysis.a depends on libclangASTMatchers.a
      This patch fixed this issue as well.
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      72bb0d56
  21. 18 Sep, 2018 1 commit
    • yonghong-song's avatar
      fix a trace.py problem (#1973) · 61484e17
      yonghong-song authored
      Currently, trace.py failed for the following command:
        $ sudo ./trace.py 'filename_lookup(int dfd, struct filename *name) "%s", name->name'
        ...
        0: (bf) r6 = r1
        1: (79) r7 = *(u64 *)(r6 +104)
        ...
        32: (15) if r1 == 0x0 goto pc+5
        R0=inv(id=0) R1=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0)
        R8=inv0 R10=fp0,call_-1 fp-8=0 fp-16=0 fp-24=0 fp-32=0 fp-40=0 fp-48=0 fp-56=0 fp-64=0 fp-72=0 fp-80=0
        33: (79) r3 = *(u64 *)(r7 +0)
        R7 invalid mem access 'inv'
      
      For string format argument, the trace.py generates the below code:
              if (name->name != 0) {
                      bpf_probe_read(&__data.v0, sizeof(__data.v0), (void *)name->name);
              }
      Right now, bcc skips the rewriter for the third argument of bpf_probe_read to avoid
      unnecessary nested bpf_probe_read and other potential issues.
      This causes name->name memory access not transformed with bpf_probe_read and hence
      the verifier complains.
      
      To fix the issue, this patch did the following transformation using an
      temporary variable to hold the src address:
              if (name->name != 0) {
                      void *__tmp = (void *)name->name;
                      bpf_probe_read(&__data.v0, sizeof(__data.v0), __tmp);
              }
      This way, rewriter can do the work properly.
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      61484e17
  22. 13 Sep, 2018 2 commits
    • Nguyen Phuong An's avatar
      Fix syntax error in xdp_redirect_cpu.py (#1969) · 0cae0dd3
      Nguyen Phuong An authored
      Fix syntax error in xdp_redirect_cpu.py
      0cae0dd3
    • zlim's avatar
      Enable Ubuntu/arm64 deb packaging (#1968) · 17f797d9
      zlim authored
      * Bump ubuntu docker build to bionic
      
      Fixes build dependency for arm64
      Signed-off-by: default avatarZi Shen Lim <zlim.lnx@gmail.com>
      
      * Add Build-Depends: dh-python
      
      Fixes build dependency issue observed on bionic/arm64:
      
      dh: unable to load addon python3: Can't locate Debian/Debhelper/Sequence/python3.pm in @INC (you may need to install the Debian::Debhelper::Sequence::python3 module) (@INC contains: /etc/perl /usr/local/lib/aarch64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/aarch64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/aarch64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/aarch64-linux-gnu/perl-base) at (eval 9) line 1.
      BEGIN failed--compilation aborted at (eval 9) line 1.
      Signed-off-by: default avatarZi Shen Lim <zlim.lnx@gmail.com>
      
      * Enable arm64 deb packaging
      
      LLVM6.0 is minimum version required for arm64 build. [PR#1512]
      
      [PR#1512] https://github.com/iovisor/bcc/pull/1512Signed-off-by: default avatarZi Shen Lim <zlim.lnx@gmail.com>
      17f797d9