Port fixes of biosnoop from Python to Lua

Fix bugs introduced in Lua Perf Buffer handling

docs: fix commit hash and links for bpf_set_hash() and bpf_setsockopt()

tools/cachestat: Rewrite it so it makes more sense

cachstat separation between read and writes doesn't make much sense. During tests
its seen that a workload that is supposed to be completely in the missed category
appears to be 50% hit.

This patch rewrites the BCC tool to be more inline with Brendan Gregg's original
cachestat perf tool which makes more sense:
https://github.com/brendangregg/perf-tools/blob/master/fs/cachestat
http://www.brendangregg.com/blog/2014-12-31/linux-page-cache-hit-ratio.html

Fixes Issue: #1586
Signed-off-by: Joel Fernandes <joelaf@google.com>

Helpers bpf_set_hash() and bpf_setsockopt() do not have the correct link
(bpf_set_hash()) and hash (both) in the list of BPF features per kernel
version. Fix them.

Avoid including common.h in BPFTable.h

Remove PERF_SAMPLE_CALLCHAIN logic from TRACEPOINT events

free llvm engine/context memory when rw_engine is not used

bcc utilizes MCJIT to generate final code and the code
is actually runnable in the process. This, however,
may have negative memory consumption impact.
All data will be actually allocated in memory.

For example, the following large array,
  #define TCP6_RXMIT_BUCKET_BITS 18
  struct tcp6_rxmit_tbl {
    __u64 buckets[1 << TCP6_RXMIT_BUCKET_BITS];
  };
  BPF_ARRAY(rxmit_marking_map, struct tcp6_rxmit_tbl, 1);
is added to examples/cpp/HelloWorld.cpp.
Internally, bcc will define a structure type rxmit_marking_map_table
with members `struct tcp6_rxmit_tbl leaf` and a variable
with this structure type. (see src/cc/export/helpers.h),
so that rw_engine can traverse the type to generate
proper sscanf/snprintf functions.

Even with rw_engine is disabled, the RSS still increased
from 6MB to 12MB. The array size is roughly 2MB.
The additional 4MB overhead is due to some llvm internal overhead
which I did not root cause it.

If rw_engine is disabled, we can actually free llvm
engine and context memory after copying the section data.
We do not need to copy map section data since it is not
used later on. Only map size is needed.

We cannot free llvm context memory if rw_engine is enabled since context
is shared between regular llvm module and sscanf module.

For the above HelloWorld example, if rw_engine is disabled,
this patch is able to reduce the RSS memory from 12MB to 6MB.
Signed-off-by: Yonghong Song <yhs@fb.com>

setup some bpf_module data structures correctly when rw_engine is disabled

Commit db7b8eb0 ("add a BPFModule API to disable rw_engine
sscanf/snprintf functions") permits to disable rw_engine so that
memory can be saved for structures with large arrays. As a result,
the function BPFModule::annotate(), which is used to generate
"sscanf" module, is not called when rw_engine is disabled.

Besides generating "sscanf" module, however, BPFModule::annotate()
also sets up several other data structures which are used for
map/table manipulation. This patch implements BPFModule::annotate_light(),
which will be called when rw_engine is disabled, to
setup these data structures.
Signed-off-by: Yonghong Song <yhs@fb.com>

fix hang with "trace.py --max-events #"

Currently running "trace.py --max-events #" (shortname "trace.py -M #")
will hang like below:

  -bash-4.2$ sudo ./trace.py -M 2 'SyS_futex'
  PID     TID     COMM            FUNC
  137727  138229  IOThreadPool0   SyS_futex
  138250  151288  OBC Dispatcher  SyS_futex
  ^C^Z
  [1]+  Stopped                 sudo ./trace.py -M 2 'SyS_futex'
  -bash-4.2$

The hang happens in perf_reader_free.

Commit cd5d4a6c ("fix a race condition between perf_reader
munmap and read") fixed a race condition by adding some coordination
between perf_reader_event_read and perf_reader_free.

In this case, however, the callback function inside perf_reader_event_read
never returns and actually calling exit(). This is because
the maximum number of perf_events have been received. The exit()
is calling BPF object cleanup() which calls perf_reader_free to
free the ring buffer. perf_reader_free got stuck since it thinks
perf_reader_event_reader did not finish yet.

To fix this, a checking for thread_id is added so that
perf_reader_free will proceed without locking if the perf_reader_read
tid is the same as perf_reader_free since this signals that
the callback function calls/triggers perf_reader_free.

After the fix,

  -bash-4.2$ sudo ./trace.py -M 2 'SyS_futex'
  PID     TID     COMM            FUNC
  137727  138178  load-monitor    SyS_futex
  6359    6440    load-monitor    SyS_futex
  -bash-4.2$
Reported-by: Teng Qin <qinteng@fb.com>
Signed-off-by: Yonghong Song <yhs@fb.com>

fix percpu table support in c++ api

Upgrade snap llvm toolchain version

Do not rely on RTLD_DI_ORIGIN

This will allow the bcc to be built on Alpine without additional
patches to musl.

The percpu table support in the c++ api was wrong.

This commit creates two new table classes BPFPercpuArrayTable
and BPFPercpuHashTable, it also extends the BPFTable that allows
to access map using strings.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>

llvm-3.7 toolchain no longer exists in 17.10.

By switching to llvm-4.0, this snap builds in both 16.04 and 17.10.

add a BPFModule API to disable rw_engine sscanf/snprintf functions

Currently, for every table of a module, corresponding
key/value sscanf/snprintf functions will be generated.
These functions are mostly used for python API to
facilitate passing keys between python and C++.

It is a known issue that these sscanf/snprintf functions
can consume a lot of system resources esp. memory for
large arrays. Commit 22eae8c6 ("Use late-binding to
finalize snprintf/sscanf") avoids unnecessary code
generation for snprintf/sscanf until they are called.
Even with this commit, however, the overhead can still
be significant for large arrays.

For example, the following large array,
  #define TCP6_RXMIT_BUCKET_BITS 18
  struct tcp6_rxmit_tbl {
    __u64 buckets[1 << TCP6_RXMIT_BUCKET_BITS];
  };
  BPF_ARRAY(rxmit_marking_map, struct tcp6_rxmit_tbl, 1);
is used inside Facebook.
If it is added to examples/cpp/HelloWorld.cpp,
the HelloWorld RSS memory will increase from 7MB to
370MB.

This patch add a BPFModule API and a C++ API to
disable rw_engine sscanf/snprintf function through additional
constructor parameters. rw_engine support for Python is not affected.
Signed-off-by: Yonghong Song <yhs@fb.com>

This commit fixes the following error with python3:

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/usr/lib/python3.6/site-packages/bcc/table.py", line 508, in raw_cb_
    callback(cpu, data, size)
  File "./opensnoop.py", line 169, in print_event
    if args.name and args.name not in event.comm:
TypeError: a bytes-like object is required, not 'str'
Signed-off-by: Gary Lin <glin@suse.com>

Explicitly include sys/types.h

This will help build bcc on Alpine Linux.

scripts: avoid check-helpers.sh's writes to disk

Allow to delete elements from prog tables

Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>

sync bpf compat headers with latest net-next

Signed-off-by: Yonghong Song <yhs@fb.com>

Script to check that the lists of helpers are in sync

It is allowed in the kernel from some time ago:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/bpf/arraymap.c?h=v4.4#n267Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>

Checks src/cc/libbpf.c, docs/kernel-versions.md, src/cc/compat/linux/bpf.h,
and src/cc/export/helpers.h.