Currently running "trace.py --max-events #" (shortname "trace.py -M #")
will hang like below:

  -bash-4.2$ sudo ./trace.py -M 2 'SyS_futex'
  PID     TID     COMM            FUNC
  137727  138229  IOThreadPool0   SyS_futex
  138250  151288  OBC Dispatcher  SyS_futex
  ^C^Z
  [1]+  Stopped                 sudo ./trace.py -M 2 'SyS_futex'
  -bash-4.2$

The hang happens in perf_reader_free.

Commit cd5d4a6c ("fix a race condition between perf_reader
munmap and read") fixed a race condition by adding some coordination
between perf_reader_event_read and perf_reader_free.

In this case, however, the callback function inside perf_reader_event_read
never returns and actually calling exit(). This is because
the maximum number of perf_events have been received. The exit()
is calling BPF object cleanup() which calls perf_reader_free to
free the ring buffer. perf_reader_free got stuck since it thinks
perf_reader_event_reader did not finish yet.

To fix this, a checking for thread_id is added so that
perf_reader_free will proceed without locking if the perf_reader_read
tid is the same as perf_reader_free since this signals that
the callback function calls/triggers perf_reader_free.

After the fix,

  -bash-4.2$ sudo ./trace.py -M 2 'SyS_futex'
  PID     TID     COMM            FUNC
  137727  138178  load-monitor    SyS_futex
  6359    6440    load-monitor    SyS_futex
  -bash-4.2$
Reported-by: Teng Qin <qinteng@fb.com>
Signed-off-by: Yonghong Song <yhs@fb.com>

fix percpu table support in c++ api

Upgrade snap llvm toolchain version

Do not rely on RTLD_DI_ORIGIN

This will allow the bcc to be built on Alpine without additional
patches to musl.

The percpu table support in the c++ api was wrong.

This commit creates two new table classes BPFPercpuArrayTable
and BPFPercpuHashTable, it also extends the BPFTable that allows
to access map using strings.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>

llvm-3.7 toolchain no longer exists in 17.10.

By switching to llvm-4.0, this snap builds in both 16.04 and 17.10.

add a BPFModule API to disable rw_engine sscanf/snprintf functions

Currently, for every table of a module, corresponding
key/value sscanf/snprintf functions will be generated.
These functions are mostly used for python API to
facilitate passing keys between python and C++.

It is a known issue that these sscanf/snprintf functions
can consume a lot of system resources esp. memory for
large arrays. Commit 22eae8c6 ("Use late-binding to
finalize snprintf/sscanf") avoids unnecessary code
generation for snprintf/sscanf until they are called.
Even with this commit, however, the overhead can still
be significant for large arrays.

For example, the following large array,
  #define TCP6_RXMIT_BUCKET_BITS 18
  struct tcp6_rxmit_tbl {
    __u64 buckets[1 << TCP6_RXMIT_BUCKET_BITS];
  };
  BPF_ARRAY(rxmit_marking_map, struct tcp6_rxmit_tbl, 1);
is used inside Facebook.
If it is added to examples/cpp/HelloWorld.cpp,
the HelloWorld RSS memory will increase from 7MB to
370MB.

This patch add a BPFModule API and a C++ API to
disable rw_engine sscanf/snprintf function through additional
constructor parameters. rw_engine support for Python is not affected.
Signed-off-by: Yonghong Song <yhs@fb.com>

This commit fixes the following error with python3:

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/usr/lib/python3.6/site-packages/bcc/table.py", line 508, in raw_cb_
    callback(cpu, data, size)
  File "./opensnoop.py", line 169, in print_event
    if args.name and args.name not in event.comm:
TypeError: a bytes-like object is required, not 'str'
Signed-off-by: Gary Lin <glin@suse.com>

Explicitly include sys/types.h

This will help build bcc on Alpine Linux.

scripts: avoid check-helpers.sh's writes to disk

Allow to delete elements from prog tables

Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>

sync bpf compat headers with latest net-next

Signed-off-by: Yonghong Song <yhs@fb.com>

Script to check that the lists of helpers are in sync

It is allowed in the kernel from some time ago:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/bpf/arraymap.c?h=v4.4#n267Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>

Checks src/cc/libbpf.c, docs/kernel-versions.md, src/cc/compat/linux/bpf.h,
and src/cc/export/helpers.h.

Cosmetic: Make kernel-versions commit hashes monospace

Discourage use of str() type strings in python API

Fix two issues in examples

Implement get_text for Python USDT class

Destruct USDTContext correctly in Python

<linux/blkdev.h> is not needed.
Signed-off-by: Liu Bo <liub.liubo@gmail.com>

A function definition doesn't need a tail semicolon.
Signed-off-by: Liu Bo <liub.liubo@gmail.com>

Conform to bytes encoding for some portion of the tools/tests, such that
smoke tests pass on python3. More conversions are surely required.
Signed-off-by: Brenden Blanco <bblanco@gmail.com>

Move bcc internals to sanitize all arguments as bytes instead of str
type. For now, disable warnings or actual assertion, to be turned on
incrementally.

Introduce some helpers for managing bytes/unicode objects in a way that
bridges the gap from python2 to 3.

1. Add printb() helper for writing bytes output directly to stdout. This
avoids complaints from print() in python3, which expects a unicode
str(). Since python 3.5, `b"" % bytes()` style format strings should
work and we can write tools with common code, once we convert format
strings to bytes.
http://legacy.python.org/dev/peps/pep-0461/

2. Add a class for wrapping command line arguments that are intended for
comparing to debugged memory, for instance running process COMM or
kernel pathname data. The approach takes some of the discussion from
http://legacy.python.org/dev/peps/pep-0383/ into account, though
unfortunately the python2-future implementation of "surrogateescape" is
buggy, therefore this iteration is partial.

The object instance should only ever be coerced into a bytes object.
This silently invokes encode(sys.getfilesystemencoding()), which if it
fails implies that the tool was passed junk characters on the command
line. Thereafter the tool should implement only bytes-bytes comparisons
(for instance re.search(b"", b"")) and bytes stdout printing (see
printb).

3. Add an _assert_is_bytes helper to check for improper usage of str
objects in python arguments. The behavior of the assertion can be
tweaked by changing the bcc.utils._strict_bytes bool.

Going forward, one should never invoke decode() on a bpf data stream,
e.g. the result of a table lookup or perf ring output. Leave that data
in the native bytes() representation.
Signed-off-by: Brenden Blanco <bblanco@gmail.com>

Signed-off-by: Brenden Blanco <bblanco@gmail.com>

BCC cross compilation support

BCC at the moment builds eBPF only considering the local architecture
instead of the one that the user's target kernel is running on.
For cross-compiler environments, the ARCH environment variable is
used to specify which ARCH to build the kernel for. In this patch we
add support to read ARCH and if that's not set, then fallback to
detecting based on local architecture.

This patch borrows some code from a patch doing a similar thing for
eBPF samples in the kenrel that I submitted recently [1]

[1] https://patchwork.kernel.org/patch/9961801/Signed-off-by: Joel Fernandes <joelaf@google.com>

Many developers have kernel sources at an specific location. Allow
clang to build from there by allowing to specify a single absolute
path to the kernel sources through a new env var BCC_KERNEL_SOURCE.
Signed-off-by: Joel Fernandes <joelaf@google.com>

Updated the FAQ with the error produced if python[2-3]-bcc isn't inst…

Document bpf_tail_call()