fix a race condition between perf_reader munmap and read

adding filtering example to trace.py help

Fix issue #1533.

Currently, there exist a race condition between perf_reader
buffer munmap and read if they are happening in two different
threads, crash is possible as in issue #1533.

          thread 1                    thread 2
          perf_reader_event_read      ...
                                      detach_probe
                                      munmap
          access ring buffer

detach_probe may happen as part of bpf object exit cleanup process
at which point thread 1 is still alive. In this case, accessing
ring buffer may cause segfault since the original mmap'ed memory
is not available any more.

It is hard to fix such races outside bcc since user
calls kprobe_poll which has valid BPF object when it is called,
but race happens inside the kprobe_poll.

This patch adds a state of the ring buffer and the read will
not happen once the state comes to the munmap.
Signed-off-by: Yonghong Song <yhs@fb.com>

Signed-off-by: Yonghong Song <yhs@fb.com>

Add a few trace.py examples in tutorial.md

Allow for sharing of python tools/ scripts

Use of this argparse constructor keyword is causing regression
test failures, and its use was nice-to-have - this means the -e
shorthand for --ebpf will be available iff its not been used in
another add_argument call.  Neither -e/--ebpf are advertised in
the usage message anyway.

Signed-off-by: Yonghong Song <yhs@fb.com>

Small fixes to some man pages for bcc tools

Fix folding option

Add bpf_override_return() helper definition

Adding minimum latency filter to tcpconnlat

Using the "folding" option produces an error.

/usr/share/bcc/tools/offwaketime -f 3 > out.stacks

Traceback (most recent call last):
  File "/usr/share/bcc/tools/offwaketime", line 305, in <module>
    print("%s %d" % (str.join(";", line), v.value))
TypeError: sequence item 0: expected str instance, bytes found

It seems that k.target and k.waker are always type = bytes.  This edit resolves this error.
Thank you for your time.

Use /proc/kallsyms for regex filtering

flag to print bpf program. Also fixed naming in tcpconnlat man page.

Added more examples in tcpconnlat_example.txt to show min duration
usage.

instead of /sys/kernel/debug/tracing/available_filter_functions.
available_filter_functions does not contain all tracable functions,
only those ftrace can use.

1. Adding `-m` and `-u` parameters for filtering tcp connects with greater the specified minimum latency.
2. Adding -v option to print out the bpf program. This is helpful for debugging and is a bit more consistent with some of the other bpf tools.

The latency check is only added if `-u` or `-m` have been set to greater than 0.

fix a trace_pipe output issue

For trace_pipe output, the /sys/kernel/debug/tracing/trace_pipe
output format looks like below for kernel 4.12 and earlier:
```
 <COMM>-<PID> [CPU] FLAGS TIMESTAMP: [SYMBOL]: USER_MESSAGES
```
where if an internal address is not able to be resolved to
a kernel symbol, `SYMBOL` will not be printed out at all.

kernel 4.13 and later will have format like:
```
 <COMM>-<PID> [CPU] FLAGS TIMESTAMP: [SYMBOL_OR_ADDR]: USER_MESSAGES
```
Basically, if an address is not able to resolved into a
kernel symbol, the address itself will be printed out.
The change was introduced by below commit:
```
commit feaf1283d11794b9d518fcfd54b6bf8bee1f0b4b
Author: Steven Rostedt (VMware) <rostedt@goodmis.org>
Date:   Thu Jun 22 17:04:55 2017 -0400

    tracing: Show address when function names are not found

    Currently, when a function is not found in kallsyms, instead of simply
    showing the function address, it shows nothing at all:

     # echo ':mod:kvm_intel' > /sys/kernel/tracing/set_ftrace_filter
     # echo function > /sys/kernel/tracing/set_ftrace_filter
     # qemu -enable-kvm /home/my-qemu-image
       <Ctrl-C>
     # rmmod kvm_intel
     # cat /sys/kernel/tracing/trace
     qemu-system-x86-2408  [001] d..2   135.013238:  <-kvm_arch_hardware_enable
     qemu-system-x86-2408  [001] ....   135.014574:  <-kvm_arch_vm_ioctl
     qemu-system-x86-2408  [001] ....   135.015420:  <-kvm_vm_ioctl_check_extension
     qemu-system-x86-2408  [001] ....   135.045411:  <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ....   135.045412:  <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ....   135.045412:  <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ....   135.045412:  <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ...1   135.045413:  <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ....   135.045413:  <-__do_cpuid_ent

    When it should show:

     qemu-system-x86-2408  [001] d..2   135.013238: 0xffffffffa02a39f0 <-kvm_arch_hardware_enable
     qemu-system-x86-2408  [001] ....   135.014574: 0xffffffffa02a2ba0 <-kvm_arch_vm_ioctl
     qemu-system-x86-2408  [001] ....   135.015420: 0xffffffffa029e4e0 <-kvm_vm_ioctl_check_extension
     qemu-system-x86-2408  [001] ....   135.045411: 0xffffffffa02a1380 <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ....   135.045412: 0xffffffffa029e160 <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ....   135.045412: 0xffffffffa029e180 <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ....   135.045412: 0xffffffffa029e520 <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ...1   135.045413: 0xffffffffa02a13b0 <-__do_cpuid_ent
     qemu-system-x86-2408  [001] ....   135.045413: 0xffffffffa02a1380 <-__do_cpuid_ent

    instead.
```

This patch fixed the issue by parsing the trace_pipe output considering the
`SYMBOL_OR_ADDRESS` field may or may not be empty.
Signed-off-by: Yonghong Song <yhs@fb.com>

As discussed in https://github.com/iovisor/bcc/pull/1531
review comments.
Signed-off-by: Nathan Scott <nathans@redhat.com>

Several python tools allow their eBPF code to be printed to
stdout for debugging.  There are other projects that would
like to share these program definitions however, instead of
duplicating code.  Formalise an -e/--ebpf option, and start
using it in several tools (more to come).
Signed-off-by: Nathan Scott <nathans@redhat.com>

The exception raised in the BPF class constructor assumed
that the "src_file" argument was used to pass eBPF code -
this is not the case for many of the tools scripts, which
tend to use "text".
Signed-off-by: Nathan Scott <nathans@redhat.com>

usdt: ignore location with incorrect probe virtual address

usdt: better handling for probes existing in multiple binary files

tools: syscount: add --errno=EPERM filter

Signed-off-by: Yonghong Song <yhs@fb.com>

Currently, for usdt, the commands where both a pid and a binary path
are specified are not well supported. For example,
```
funccount -p <pid> 'u:<binary_path>:probe'
```
will count `probe` occurances for all binary paths in `pid`, not just
`<binary_path>`. The command
```
argdist -p <pid> 'u:<binary_path>:probe():s64:arg1'
```
will also count `probe` occurances for all binary paths in `pid`
with my previous patch.

Furthermore, suppose user want to trace linker `setjmp` probe point
with command
```
trace.py -p <pid> 'u:/usr/lib64/ld-2.17.so:setjmp'
```
Without my previous patch, user will have incorrect results as both
`libc:setjmp` and `rtld:setjmp` exists and the bcc just picks the first
one which is `libc:setjmp`. My last patch will cause `enable_probe`
failures if in the same usdt context, two probes have the same
probe_name but different provider name.

To fix all these issues, this patch passes additional binary path
to the pid-based usdt context, so that only probes from that particular
binary will be added to the context. This solved all the above
mentioned issues.
Signed-off-by: Yonghong Song <yhs@fb.com>

Fixing issue #1515.

Currently, each usdt probe (provider, probe_name) can only
have locations from the single binary. It is possible that
USDT probes are defined in a header file which eventually causes
the same usdt probe having locations in several different
binary/shared_objects. In such cases, we are not able to attach
the same bpf program to all these locations.

This patch addresses this issue by defining each location to
be `bin_path + addr_offset` vs. previous `addr_offset` only.
This way, each internal Probe class can represent all locations
for the same (provider, probe_name) pair.

The `tplist.py` output is re-organized with the (provider, probe_name)
in the top level like below:
```
...
rtld:lll_futex_wake [sema 0x0]
  location #1 /usr/lib64/ld-2.17.so 0xaac8
    argument #1 8 unsigned bytes @ di
    argument #2 4 signed   bytes @ 1
    argument #3 4 signed   bytes @ 0
  location #2 /usr/lib64/ld-2.17.so 0xe9b9
    argument #1 8 unsigned bytes @ di
    argument #2 4 signed   bytes @ 1
    argument #3 4 signed   bytes @ 0
  location #3 /usr/lib64/ld-2.17.so 0xef3b
    argument #1 8 unsigned bytes @ di
    argument #2 4 signed   bytes @ 1
    argument #3 4 signed   bytes @ 0
...
```

Tested with the following commands
```
  tplist.py
  trace.py -p <pid> 'u::probe "arg1 = %d", arg1'
  trace.py u:<binary_path>:probe "arg1 = %d", arg1'
  argdist.py -p <pid> 'u::probe():s64:arg1'
  argdist.py -C 'u:<binary_path>:probe():s64:arg1'
  funccount.py -p <pid> 'u:<binary_path>:probe'
  funccount.py 'u:<binary_path>:probe'
```
Signed-off-by: Yonghong Song <yhs@fb.com>

Similar to `--filter` which reports all failing syscalls,
`--errno=ENOENT` reports syscalls failing with the specified errno value.

```
$ sudo bcc/tools/syscount.py -e ENOENT
Tracing syscalls, printing top 10... Ctrl+C to quit.
^C[12:07:13]
SYSCALL                   COUNT
open                        330
stat                        240
access                       63
execve                       22
readlink                      3
```

The patch fixed issue #1528. The linker ld.gold may generate
a probe with invalid virtual address if the section which the probe
relocates against is discarded.

Instead of failing, a simple checking is added such that
if the probe virtual address is less than the address of the first
instruction, print out a warning and ignore this probe.
Signed-off-by: Yonghong Song <yhs@fb.com>