If a module has more than one executable region, it is reported
multiple times by `bcc_procutils_each_module`. This is fine for
symbol resolution, but we don't need the duplicates for parsing
the ELF header looking for USDT probes: the first appearance of
that module is enough. This also prevents issues with the same
probe appearing multiple times with the same location, which
results in an invalid program when reading USDT arguments.

Fix by storing each visited module in the USDT::Context class,
and ignoring modules that were already visited.

cc: Retry symbol resolution using perfmap

cc: Handle nested functions correctly when resolving symbols

cc: Fix SEGV when there is no build-id section

Fix long running test_debuginfo and python3 fix

Make sure subclass calls super().tearDown to clean up dummy process.
Also, fixup a python3 str.encode().

Fixes: #1013
Signed-off-by: Brenden Blanco <bblanco@gmail.com>

When a symbol lies within a module, and that module doesn't have
debuginfo (or doesn't even have an ELF header), the symbol will
always be resolved as [unknown]. However, the /tmp/perf-PID.map
(perf map) for that process might actually have an entry for that
symbol, if it was dynamically generated by some external tool.
This commit changes the resolution process such that if the desired
address lies in a module but that module doesn't have debuginfo,
we keep trying to resolve it in subsequent modules (including the
perf map). If we resolve it successfully using the perf map, the
reported symbol information will have the original module's name,
so we don't lose fidelity.

The motivation for this change is the way symbols work with .NET
Core on Linux. The runtime binaries are compiled ahead-of-time to
native code, but do not have debuginfo. There is an external tool,
which generates a file similar to a perf map (albeit with relative
addresses) for these binaries. This file can then be merged into
the main perf map for the process and used for symbol resolution,
but only if we keep trying to use the perf map when the symbol is
in a previously-seen module.

Make perf ring buffer size configurable

As discussed in #966, this PR makes the size of the ring buffer used to send
data to userspace configurable. It changes the Python, Lua and C++ APIs to
expose this knob.

It also defaults the buffer size to a larger value (64 pages per CPU, an 8x
increase) for several tools which produce a lot of output, as well as making it
configurable in `trace` via a `-b` flag.

snapcraft: add in some new tools missing from the snapcraft apps list

Support for macros in rewriter

Extends the scope of 98b90974's fix to support macros anywhere in the
rewriter. All SourceRange objects are replaced to use macro expanded locations.

docs: Update eBPF features list

Update of BPF features list, following the release of kernel 4.10.

Add in syscount, dbstat and dbslower to apps list.
Signed-off-by: Colin Ian King <colin.king@canonical.com>

`ProcSyms::Module::find_addr` incorrectly resolves symbols when
functions are nested in each other. Specifically, this was discovered
with libpthread, where there are multiple symbols for `write`,
where `write_nocancel` is strictly nested inside `write`. Fix by
explicitly going backward until we reach a matching symbol -- see
details in `ProcSyms::Module::find_addr` comments.

Improve matching of file-backed memory mappings

Use the same rules as perf to determine if a mapping in /proc/pid/maps is
file-backed. This allows mappings in anonymous huge pages and so on to fall
back to resolving from /tmp/perf-pid.map, if appropriate.

ref: http://lxr.free-electrons.com/source/tools/perf/util/map.c#L28

Fix "for loop initial declarations only in C99" compile error

Fix minor error in test instructions

External debuginfo support and general symbols overhaul

This commit introduces support for tests of the new debuglink and
build-id debuginfo resolution functionality. The tests build a
dummy.c file with a debuglink section, and again with a build-id
section, and make sure that the symbol resolution code can locate
the debug information correctly (in the binary's directory for
debuglink, and in /usr/lib/debug/.build-id for the build-id).

The implementation of `ProcSyms::resolve_name` was only valid for
kernel symbols, when there is no module. When a module was provided,
it would segfault due to the module being null. Fixed by making
`bcc_symcache_resolve_name` take an additional module parameter,
which, for kernel symbols, is simply null (`None` from Python).

Adds support for resolving symbols using external debuginfo files,
which can be retrieved from two locations. First, check the build-id
of the desired binary and look in /usr/lib/debug/.build-id according
to the build-id structure. Second, check the debuglink section of
the desired binary and look in /usr/lib/debug or in the binary's
current directory. These are the rules applied by GDB as well, but
GDB lets the user reconfigure the debug directory path from
/usr/lib/debug to something else; we do not support this.

These changes are based on the following description of how GDB
resolves external debuginfo:

https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

When resolving symbols, ProcSyms would treat position-independent
executables (PIE files) incorrectly, resulting in symbol resolution
failures. Specifically, PIE files are treated like shared objects
for ASLR, which means all symbol addresses in the file need to be
taken relative to the executable load address at runtime, the same
as with dynamic library shared objects.

The fix is in the `is_so()` method on `ProcSyms::Module`, which
now uses the correct `bcc_elf_is_shared` helper for testing if a
file is a shared object rather than just looking at the extension
".so", which is very brittle -- and wrong.

Remove addresses from stack output, as these are not used in the
vast majority of the cases.

The default profile output used to include stack addresses,
which are not used in 99+% of the cases.

Remove usyms.py dependency and replace with new symbols API.

The new API has two main methods: `sym` and `ksym` (the second is
a trivial wrapper of the first one). Both methods accept two
Boolean flags arguments: `show_module` and `show_address`. The first
controls whether the resulting symbol name should contain the name
of the symbol's module, in brackets. For kernel symbols, this is
simply "[kernel]". The second controls whether the resulting symbol
name should contain the instruction offset from the beginning of
the symbol, e.g. "start_thread+0x202".

This class was obsolete and replaced by the SymbolCache class.

Support base + index * scale addressing for USDT arguments

trace, argdist: Treat small USDT arguments correctly

trace and argdist currently only work correctly for USDT arguments
whose size is exactly 8 bytes. Smaller types, such as chars, shorts,
ints (signed or unsigned) are not treated correctly. The reason is
that the produced program would invoke the `bpf_usdt_readarg` helper
with the address of a u64 local variable, and then cast that variable
to the user-specified type derived from the format string. However,
the `bpf_usdt_readarg` rewriting then passes `sizeof(u64)` to the
generated `bpf_..._readarg` macro, which then fails to read anything
because the provided size doesn't match the argument size it knows
about.

The fix is fairly easy: instead of declaring a u64 unconditionally
and reading into that variable with `bpf_usdt_readarg`, declare a
variable that has the correct type according to what we know about
the USDT probe.

Fix file descriptor leak