Otherwise, if llvm is built with assertion on,
we will hit the following error for a simple trace.py script like
  -bash-4.4$ sudo ./trace.py 'SyS_nanosleep'
  python: /home/yhs/work/llvm/lib/ExecutionEngine/MCJIT/MCJIT.cpp:204:
          virtual void llvm::MCJIT::generateCodeForModule(llvm::Module*):
          Assertion `M->getDataLayout() == getDataLayout() && "DataLayout Mismatch"'
          failed.
  Aborted (core dumped)
  -bash-4.4$
Signed-off-by: Yonghong Song <yhs@fb.com>

Sync compat/linux/{bpf.h, bpf_common.h, virtual_bpf.h}
with latest bpf-next. The change in bpf_common.h are comments only.
Signed-off-by: Yonghong Song <yhs@fb.com>

In folded output format when both kernel and user stacks are missing,
the message "[Missing User Stack]" is formatted incorrectly, for example:

chrome;entry_SYSCALL_64_after_hwframe;do_syscall_64;SyS_poll;do_sys_poll;poll_schedule_timeout;schedule_hrtimeout_range;schedule_hrtimeout_range_clock;schedule;--;[Missed Kernel Stack];[;M;i;s;s;e;d; ;U;s;e;r; ;S;t;a;c;k;];fio 4955989

This is because we're incorrectly appending the missing stack message in
the list of stack entries. Fix by appending the message as a single item.
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

Also, fix an incorrect indentation in the same block.

* Allow unused return value in cc source

With llvm-7.0.0,
some annoying warning messeges are raised:

	/home/lecopzer/workspace/bcc/src/cc/libbpf.c:456:3: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result [-Wunused-result]
   	fgets(fmt, sizeof(fmt), f); // pos
   	^~~~~~~~~~~~~~~~~~~~~~~~~~

/home/lecopzer/workspace/bcc/src/cc/libbpf.c: In function ‘bpf_prog_get_tag’:
/home/lecopzer/workspace/bcc/src/cc/libbpf.c:456:3: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result [-Wunused-result]
   fgets(fmt, sizeof(fmt), f); // pos
   ^~~~~~~~~~~~~~~~~~~~~~~~~~
...

/home/lecopzer/workspace/bcc/tests/cc/utils.cc: In function ‘int cmd_scanf(const char*, const char*, ...)’:
/home/lecopzer/workspace/bcc/tests/cc/utils.cc:30:10: warning: ignoring return value of ‘int vfscanf(FILE*, const char*, __va_list_tag*)’, declared with attribute warn_unused_result [-Wunused-result]
   vfscanf(pipe, fmt, args);
   ~~~~~~~^~~~~~~~~~~~~~~~~

Let get rid of them by adding -Wno-unused-result.

* cc: Fix comparison between signed and unsigned value

With llvm-7.0.0:

/home/lecopzer/workspace/bcc/src/cc/common.cc: In function ‘std::__cxx11::string ebpf::get_pid_exe(pid_t)’:
/home/lecopzer/workspace/bcc/src/cc/common.cc:60:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (res >= sizeof(exe_path))
       ~~~~^~~~~~~~~~

As the declaration of `exe_path` is `char exe_path[4096]`,
the `sizeof(exe_path)` would always return 4096 (unsigned), so it's safe
to static cast to `int` unless it's larger than 2^31 - 1.

* tools: Add shmsnoop to spy on shm* syscalls

Adding shmsnoop tool to trace System V shared memory
syscalls: shmget, shmat, shmdt, shmctl

  # ./shmsnoop.py
  PID    COMM                SYS              RET ARGs
  19813  server           SHMGET            10000 key: 0x78020001, size: 20, shmflg: 0x3b6 (IPC_CREAT|0666)
  19813  server            SHMAT     7f1cf8b1f000 shmid: 0x10000, shmaddr: 0x0, shmflg: 0x0
  19816  client           SHMGET            10000 key: 0x78020001, size: 20, shmflg: 0x1b6 (0666)
  19816  client            SHMAT     7f4fd8ee7000 shmid: 0x10000, shmaddr: 0x0, shmflg: 0x0
  19816  client            SHMDT                0 shmaddr: 0x7f4fd8ee7000
  19813  server            SHMDT                0 shmaddr: 0x7f1cf8b1f000
  19813  server           SHMCTL                0 shmid: 0x10000, cmd: 0, buf: 0x0

Every call the shm* syscall (SHM column) is displayed
on separate line together with process info (PID/COMM
columns) and argument details: return value (RET column)
and syscall arguments (ARGs column).

The ARGs column contains 'arg: value' couples that represent
given syscall arguments as described in their manpage.

It supports standard options to filter on pid/tid,
to specify duration of the trace and command name
filter, like:

    ./shmsnoop           # trace all shm*() syscalls
    ./shmsnoop -T        # include timestamps
    ./shmsnoop -p 181    # only trace PID 181
    ./shmsnoop -t 123    # only trace TID 123
    ./shmsnoop -d 10     # trace for 10 seconds only
    ./shmsnoop -n main   # only print process names containing "main"

* tools: Add sofdsnoop to spy on fds passed through socket

The sofdsnoop traces FDs passed through unix sockets.

  # ./sofdsnoop
  ACTION TID    COMM             SOCKET                    FD    NAME
  SEND   2576   Web Content      24:socket:[39763]         51    /dev/shm/org.mozilla.ipc.2576.23874
  RECV   2576   Web Content      49:socket:[809997]        51
  SEND   2576   Web Content      24:socket:[39763]         58    N/A
  RECV   2464   Gecko_IOThread   75:socket:[39753]         55

Every file descriptor that is passed via unix sockets os displayed
on separate line together with process info (TID/COMM columns),
ACTION details (SEND/RECV), file descriptor number (FD) and its
translation to file if available (NAME).

examples:
    ./sofdsnoop           # trace file descriptors passes
    ./sofdsnoop -T        # include timestamps
    ./sofdsnoop -p 181    # only trace PID 181
    ./sofdsnoop -t 123    # only trace TID 123
    ./sofdsnoop -d 10     # trace for 10 seconds only
    ./sofdsnoop -n main   # only print process names containing "main"

* inject: add option to limit number of overrides

* inject: update man pages with new count option
Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>

* add kernel stack trace option to capable
Signed-off-by: Andrea Righi <andrea@betterlinux.com>

* capable: avoid stack trace overwrite

Use 0 in get_stack() to avoid overwriting the stack trace in the
kernel for the same stack_id.
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

* capable: print both TID and PID
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

* add user-space stack trace option to capable
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

* capable: avoid catching itself
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

* capable: drop unused member pid_tgid in struct
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

* capable: report a proper error message when stack trace is missing

Print [Missed User Stack] or [Missed Kernel Stack] when get_stackid()
returns an error.
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

* capable: add missing errno import
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

* capable: remove dependency from python-enum module
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

This reorders the struct data_t members so that the definition
is consistent across the c code and the corresponding python
ct.Structure.

Upon running the script with the arguments option, the user and
kernel stack ids read from the stack maps are incorrect as they
are read off the wrong structure offsets. When a stack walk is
attempted on an incorrect stack id, we end up with a KeyError.

This has been verified as shown below.

  $ sudo ./funcslower.py c:inet_pton -u1 -a1 -UK
  $ ping -6 ::1

Before:
  Tracing function calls slower than 1 us... Ctrl+C to quit.
  COMM           PID    LAT(us)             RVAL FUNC ARGS
  ping           33541    47.93                1 c:inet_pton 0xfffffff2000001a0
  Traceback (most recent call last):
    File "_ctypes/callbacks.c", line 315, in 'calling callback function'
    File "/usr/lib/python2.7/site-packages/bcc/table.py", line 573, in raw_cb_
      callback(cpu, data, size)
    File "./funcslower.py", line 337, in print_event
      print_stack(event)
    File "./funcslower.py", line 301, in print_stack
      user_stack = stack_traces.walk(event.user_stack_id)
    File "/usr/lib/python2.7/site-packages/bcc/table.py", line 768, in walk
      return StackTrace.StackWalker(self[self.Key(stack_id)], resolve)
    File "/usr/lib/python2.7/site-packages/bcc/table.py", line 212, in __getitem__
      raise KeyError
  KeyError

After:
  Tracing function calls slower than 1 us... Ctrl+C to quit.
  COMM           PID    LAT(us)             RVAL FUNC ARGS
  ping           34672    48.20                1 c:inet_pton 0xa
      gaih_inet.constprop.7
      [unknown]
      getaddrinfo
      [unknown]
      generic_start_main.isra.0
      __libc_start_main

Fixes: 925bac87 ("Adding user and kernel stack frames option to funcslower")
Signed-off-by: Sandipan Das <sandipan@linux.ibm.com>

Fix issue #2035.

For code like
    int i;
    int j = 0;
    const int last_index = payload_offset + 7;
    for (i = payload_offset ; i < last_index ; i++) {
               p[j] = load_byte(skb , i);

Here, the payload_offset is unknown. llvm 7.0 and trunk
compiler seems generating code like
   p[0] = load_byte(skb, payload_offset)
   if (payload_offset + 1 < last_index) {
      p[1] = ...
      ...
      p[6] = ...
   } else {
      /* do nothing */
   }
   /* accessing p[0], p[1], ..., p[6] */

The compiler did the above transformation because the potential
overflow for last_index and/or payload_offset + 1 in which case
compiler preserved both branches.

This caused a problem for verifier as in the else branch
p[1] is not assigned and the verifier will reject the program.

Changing the loop to simply iterate from 0 to 6 fixed the problem.
Signed-off-by: Yonghong Song <yhs@fb.com>

install docs - on ubuntu bionic use llvm 6.0

* Use correct entry point for uprobes on powerpc64

For powerpc64 (big endian), the address of a function is the
address of the corresponding function descriptor. While the
actual functions reside in the ".text" section, the function
descriptors are present in the ".opd" section. According to
the ABI, each descriptor is a tri-doubleword data structure
where the first doubleword is the actual entry point address.

The symbol table entries do not list actual entry points but
instead provide the location of the function descriptor. So,
when attaching a probe, the location should be changed to the
actual entry point by peeking into the function descriptor.

This has been verified as shown below.

  $ readelf -S /usr/lib64/power8/libc-2.26.so | grep -A1 ".opd"
    [30] .opd              PROGBITS         0000000000213648  00203648
         000000000000bcb8  0000000000000000  WA       0     0     8

The first column shows the index of the ".opd" section.

  $ readelf -s /usr/lib64/power8/libc-2.26.so | grep "inet_pton$"
    3405: 000000000021d168    96 FUNC    LOCAL  DEFAULT   30 __inet_pton
    3990: 000000000021d168    96 FUNC    LOCAL  DEFAULT   30 __GI___inet_pton
    5167: 000000000021d168    96 FUNC    LOCAL  DEFAULT   30 __GI_inet_pton
    6514: 000000000021d168    96 FUNC    WEAK   DEFAULT   30 inet_pton

The seventh column shows the index of the section to which the
symbols belong. This implies that all of these symbols are from
the ".opd" section.

  $ objdump -d --section=.opd /usr/lib64/power8/libc-2.26.so | grep -A5 "inet_pton>:"
  000000000021d168 <inet_pton>:
    21d168:       00 00 00 00     .long 0x0
    21d16c:       00 17 2b 40     .long 0x172b40
    21d170:       00 00 00 00     .long 0x0
    21d174:       00 22 73 00     .long 0x227300

  $ objdump -d /usr/lib64/power8/libc-2.26.so | grep -A5 "inet_pton>:"
  0000000000172b40 <.__inet_pton>:
    172b40:       7c 08 02 a6     mflr    r0
    172b44:       fb c1 ff f0     std     r30,-16(r1)
    172b48:       fb e1 ff f8     std     r31,-8(r1)
    172b4c:       7c 7f 1b 78     mr      r31,r3
    172b50:       7c 83 23 78     mr      r3,r4

The first doubleword in the descriptor of "inet_pton" gives the
actual entry point address i.e. 0x172b40. So, the probe must be
attached here and not 0x21d168.

  $ sudo trace "c:inet_pton" -U
  PID     TID     COMM            FUNC
  40769   40769   ping            inet_pton
          __GI___inet_pton+0x0 [libc-2.26.so]
          gaih_inet.constprop.7+0xf4c [libc-2.26.so]
          __GI_getaddrinfo+0x15c [libc-2.26.so]
          [unknown] [ping]
          generic_start_main.isra.0+0x150 [libc-2.26.so]
          __libc_start_main+0xbc [libc-2.26.so]

  $ ping -6 ::1
  PING ::1(::1) 56 data bytes
  64 bytes from ::1: icmp_seq=1 ttl=64 time=0.271 ms
  64 bytes from ::1: icmp_seq=2 ttl=64 time=0.039 ms
  ^C
  --- ::1 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1058ms
  rtt min/avg/max/mdev = 0.039/0.155/0.271/0.116 ms

Previously, the event was not triggered upon running ping.
Signed-off-by: Sandipan Das <sandipan@linux.ibm.com>

* Use correct entry point for uprobes on powerpc64le

For powerpc64le, functions have a Global Entry Point (GEP)
and a Local Entry Point (LEP). When using the GEP, there
are some additional instructions at the beginning of the
function that setup the TOC pointer. However, for all local
calls, the TOC pointer is not required and a function can
be called into directly via the LEP.

While placing a uprobe, we should always prefer the LEP as
the probe location since this will be encountered for any
call through either the GEP or the LEP. Currently, the GEP
is used as the probe location and hence the corresponding
event is never triggered when the function is called via
it's LEP.

Information about the LEP can be obtained from the st_other
field of an Elf symbol. While this field typically provides
visibility information, the three most significant bits can
provide additional information about the offset of the Local
Entry Point (LEP) from the Global Entry Point (GEP) for any
symbol in case of powerpc64le.

This has been verified as shown below.

  $ readelf -s /usr/lib64/libc-2.27.so | grep "inet_pton "
  3522: 0000000000164610   104 FUNC    LOCAL  DEFAULT   11 __inet_pton  [<localentry>: 8]
  4188: 0000000000164610   104 FUNC    LOCAL  DEFAULT   11 __GI___inet_pton     [<localentry>: 8]
  5528: 0000000000164610   104 FUNC    LOCAL  DEFAULT   11 __GI_inet_pton       [<localentry>: 8]
  6925: 0000000000164610   104 FUNC    WEAK   DEFAULT   11 inet_pton    [<localentry>: 8]

  $ sudo trace "c:inet_pton" -U
  PID     TID     COMM            FUNC
  25383   25383   ping            inet_pton
          __GI___inet_pton+0x8 [libc-2.27.so]
          gaih_inet.constprop.7+0x1040 [libc-2.27.so]
          getaddrinfo+0x164 [libc-2.27.so]
          [unknown] [ping]
          generic_start_main.isra.0+0x138 [libc-2.27.so]
          __libc_start_main+0xc4 [libc-2.27.so]

  $ ping -6 ::1
  PING ::1(::1) 56 data bytes
  64 bytes from ::1: icmp_seq=1 ttl=64 time=0.140 ms
  64 bytes from ::1: icmp_seq=2 ttl=64 time=0.029 ms
  ^C
  --- ::1 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1022ms
  rtt min/avg/max/mdev = 0.029/0.084/0.140/0.056 ms

Previously, the event was not triggered upon running ping.
Signed-off-by: Sandipan Das <sandipan@linux.ibm.com>

Fix the lua standalone test

When kprobe/kreprobe/tracepoint attachment fails, print the name of the
failing function and its target.
Signed-off-by: Oriol Arcas <oriol@starflownetworks.com>

This fixes the subtest that checks if bcc-lua depends on
libluajit.
Signed-off-by: Sandipan Das <sandipan@linux.ibm.com>

Fix symbol resolution across mount namespaces (#2029)

This updates the python and lua bindings for libbcc to be
able to support attaching kprobes at custom offsets.
Signed-off-by: Sandipan Das <sandipan@linux.ibm.com>

docs: fix kernel versions for a few newly added helpers.

Fix parsing of /proc/pid/maps

* Coverity #def53: COPY_PASTE_ERROR

* Coverity #def18: DC.STREAM_BUFFER. Double-check max length of dev

* Coverity #def44: MISSING_BREAK. This looks like it should be here

* Coverity #def67: STRING_NULL: potential OOB read if 0 bytes read.

* Coverity #def66: FORWARD_NULL: potential null ptr deref

* Coverity #def17: RESOURCE_LEAK: missing free()

* Dont free the result of dirname

dirname() may return pointers to statically allocated memory. Don't
free the pointer it returns.

Fix USDT semaphore address calculation for position independent executables (resolves #1998) (#2023)

Fix USDT semaphore address calculation for position independent executables

Fix issue #2017.

For python programs, the map data passed from C++ library
is parsed through the key/value desc types which are
generated by C++ json map declaration visitor.

The map declaration visitor visits the map key/value
declaration types and generate a string to represent
the type, which is later used by python to reconstruct
the cttype.

The map declaration already tries to add all the padding
to the type string in order to make sure C++ and python
see the same layout.

This patch further added packed support such that if
C++ json map visitor has applied padding, the python
type reconstructor is free to add _pack_=1 for structure
type since the structure is already packed.

For example, for a type,
  struct t { char a; int b; }
the structure after json visitor will look like
  struct t { char a; char __pad[3]; int b; }

If the type is
  struct t { char a; int b; } __packed;
the structure after json visitor will look like
  struct t { char a; int b; }

In either case, it will be okay to add __packed attribute
for the type generated by json map visitor in order to
match the original declaration.

Thanks Chaitanya for filing the issue and providing the test case!
Signed-off-by: Yonghong Song <yhs@fb.com>

frontends: disable uint128_t on 32-bit builds

On 32-bit, __uint128_t is generally not available. Removing it fixes
builds on those systems.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

correct certain tracepoint types in tp frontend action

This is a correction for previous patch where
function pointer is not proper specified
  static int bpf_map_push_elem(struct bpf_map *map, const void *value, u64 flags) =
     (void *) BPF_FUNC_map_push_elem;
and some old gcc compiler (e.g., ubuntu 16.04) will complain
with error:
  Illegal initializer (only variables can be initialized)
The correct way is
  static int (*bpf_map_push_elem)(struct bpf_map *map, const void *value, u64 flags) =
     (void *) BPF_FUNC_map_push_elem;
Signed-off-by: Yonghong Song <yhs@fb.com>

Fix issue #2010.

Alastair reported some missing cases in tp frontend action
where certain types are not adjusted.
For example, for tracepoint syscalls:sys_enter_kill,
the kernel format:
        ...
        field:int __syscall_nr; offset:8;       size:4; signed:1;
        field:pid_t pid;        offset:16;      size:8; signed:0;
        field:int sig;  offset:24;      size:8; signed:0;

The size for "pid_t pid" is 8 bytes, but the kernel pid_t
is "int", so it is needed to change its type to "s64"
to be consistent to its size.

This patch also added conversion for gid_t and uid_t as Alastair
discovered that they are also used in some tracepoints with size 8.
For type gid_t and uid_t, the corresponding kernel type is "unsigned int",
so it should be converted to "u64".
Signed-off-by: Yonghong Song <yhs@fb.com>

sync with latest net-next

Sync bpf uapi header bpf.h with latest net-next
and update related documentation.
Signed-off-by: Yonghong Song <yhs@fb.com>

fix bcc build failure with latest llvm/clang trunk

The clang commit
  commit 247be3e41c41201641af01888ff0d84484c4e0f4
  Author: Simon Pilgrim <llvm-dev@redking.me.uk>
  Date:   Fri Oct 12 19:14:43 2018 +0000

    Fix a clang::driver::ArgStringList usage I missed in rL344398. NFCI.
removed
  using ArgStringList = llvm::opt::ArgStringList;
in clang:include/clang/Driver/Job.h,
and makes bcc compilation failure.

Using llvm::opt::ArgStringList fixed the issue and should
still work with previous llvm versions.
Signed-off-by: Yonghong Song <yhs@fb.com>

* tools: the argument of get_kprobe_functions() should be a bytes object

It fixes the following error:
Traceback (most recent call last):
  File "/usr/share/bcc/tools/ext4dist", line 189, in <module>
    if BPF.get_kprobe_functions('ext4_file_read_iter'):
  File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 519, in get_kprobe_functions
    if (t.lower() in [b't', b'w']) and re.match(event_re, fn) \
  File "/usr/lib64/python3.6/re.py", line 172, in match
    return _compile(pattern, flags).match(string)
TypeError: cannot use a string pattern on a bytes-like object

* tools: fix freeze in offwaketime

Since commit 47cecb65, the sigint signal is ignored
unconditionally which prevents offwaketime to be stopped by
pressing Ctrl-C when run without a specified duration.

Fix issue #2002
Signed-off-by: Yonghong Song <yhs@fb.com>

Sync compat bpf.h and virtual_bpf.h with latest net-next versions

Also add a barrier for the test_brb bpf program like below
         meta.prog_id = skb->cb[0];
+        asm volatile("" ::: "memory");
         meta.rx_port_id = skb->cb[1];
so two cb read does not become a 8 byte load which is not
allowed for ubuntu 16.04 kernel (4.4 based). The 8 byte load
is allowed for skb->cb's for later kernels.
Signed-off-by: Yonghong Song <yhs@fb.com>

* Add perl* utools man page links

Matches other languages, related to commit 9162be45.

* uthreads: explicitly list C as supported

uthreads supports C (pthreads) thread tracing, make this explicit.

Earlier use of -l java and -l none unchanged.

Add the corresponding man page and usage examples.

* Add Tcl support for ucalls / uflow / uobjnew / ustat

For some reason we need to use proc__args instead of proc__entry
in uflow.py to get similar results as with e.g. Python.

* tools: uses 'replace' error handler by default in decode()

Tools might encouter characters from non utf-8 charset (e.g. a file
name). When this happen, it's better to replace the unexpected
character by a question mark than crash the tool when all we do is
to print the string.

* tools: fix a bytes/string issue in attach_perf_event()

Fixed Context::enable_probe method so that it does not discard the result of Probe::enable.

add bpf function call support link in docs.
Signed-off-by: Alban Crequy <alban@kinvolk.io>

kbuild_helper.h: Add required #include ftw.h