Binary operator expressions where the left hand-side expression is a
reference to the packet are replaced by a call to the bpf_dins_pkt
helper. When replacing text, the Clang Rewriter tries to maintain a
list of offsets between the original and the new position of tokens.

Replacing the whole binary operator expression with the call to
bpf_dins_pkt confuses the Rewriter and it is unable to track the new
position of the right hand-side expression. Rewriting the binary
operator expression in two times without rewriting the right
hand-side expression itself solves the issue.

python: handle null module in BPF.sym

Symbol resolution with multiple executable regions per module

add XDP return values to python interface

The symbol resolution code used to assume for most purposes that
there is a single executable region per module. When there were
several, there was no crash, but symbols were not resolved correctly.
The reason is that the symbol offsets are relative to the first
executable region's start address, but bcc would resolve them
relative to the region in which they appeared. For example, given
the following regions and spans for a module libfoo.so loaded into
some process:

  1000-2000 r-xp libfoo.so
  2000-3000 rw-p libfoo.so
  3000-4000 r-xp libfoo.so
  4000-5000 r--- libfoo.so

Now, suppose there is a symbol bar() loaded at address 3500. In
the binary on disk, bar() is at offset 2500 from the beginning of
the module (but not the beginning of the 3000-4000 region!). When
we look at the candidate regions, we find 3000-4000, and discover
that 3500 lies within it. Then we subtract 3500-3000 to find the
offset from the beginning of the region, get 500, and now look
for a symbol that contains the relative address 500. As a result,
we might find some random symbol in the region 1000-2000, and
report that address 3500 corresponds to that random symbol rather
than to bar().

This commit fixes the situation by keeping only a single `Module`
instance for each module, even if that module spans multiple
executable regions. We remember all executable region start and
end ranges so we can determine whether an address (like 3500 in
the above example) lies within the module. But for the purpose of
finding the actual symbol, we need only the offset from the start
of the _first_ executable region, and then need to look up a symbol
based on that.

This was discovered and fixed while tracing .NET Core processes on
Linux, where libcoreclr.so (the main CLR binary) has several
executable regions. Resolving symbols from any but the first region
would produce totally bogus results.

This fixes the following error:

$: ./tplist -v -v -l /usr/lib64/dri/i965_dri.so
argument 1: <class 'TypeError'>: wrong type

small typo

cc: Don't parse the same module multiple times for USDT probes

If a module has more than one executable region, it is reported
multiple times by `bcc_procutils_each_module`. This is fine for
symbol resolution, but we don't need the duplicates for parsing
the ELF header looking for USDT probes: the first appearance of
that module is enough. This also prevents issues with the same
probe appearing multiple times with the same location, which
results in an invalid program when reading USDT arguments.

Fix by storing each visited module in the USDT::Context class,
and ignoring modules that were already visited.

Signed-off-by: Gabriel Ganne <gabriel.ganne@enea.com>
Signed-off-by: Romain Ly <romain.ly@enea.com>

Check to see if `module` is None before attempting to call `os.path.basename`
on it. Before:

```
>>> BPF.sym(0x400001, 12345, show_module=True)
Traceback (most recent call last):
..
AttributeError: 'NoneType' object has no attribute 'rfind'
```

After:

```
>>> BPF.sym(0x400001, 12345, show_module=True)
'[unknown]'
```

cc: Retry symbol resolution using perfmap

cc: Handle nested functions correctly when resolving symbols

cc: Fix SEGV when there is no build-id section

Fix long running test_debuginfo and python3 fix

Make sure subclass calls super().tearDown to clean up dummy process.
Also, fixup a python3 str.encode().

Fixes: #1013
Signed-off-by: Brenden Blanco <bblanco@gmail.com>

When a symbol lies within a module, and that module doesn't have
debuginfo (or doesn't even have an ELF header), the symbol will
always be resolved as [unknown]. However, the /tmp/perf-PID.map
(perf map) for that process might actually have an entry for that
symbol, if it was dynamically generated by some external tool.
This commit changes the resolution process such that if the desired
address lies in a module but that module doesn't have debuginfo,
we keep trying to resolve it in subsequent modules (including the
perf map). If we resolve it successfully using the perf map, the
reported symbol information will have the original module's name,
so we don't lose fidelity.

The motivation for this change is the way symbols work with .NET
Core on Linux. The runtime binaries are compiled ahead-of-time to
native code, but do not have debuginfo. There is an external tool,
which generates a file similar to a perf map (albeit with relative
addresses) for these binaries. This file can then be merged into
the main perf map for the process and used for symbol resolution,
but only if we keep trying to use the perf map when the symbol is
in a previously-seen module.

Make perf ring buffer size configurable

As discussed in #966, this PR makes the size of the ring buffer used to send
data to userspace configurable. It changes the Python, Lua and C++ APIs to
expose this knob.

It also defaults the buffer size to a larger value (64 pages per CPU, an 8x
increase) for several tools which produce a lot of output, as well as making it
configurable in `trace` via a `-b` flag.

snapcraft: add in some new tools missing from the snapcraft apps list

Support for macros in rewriter

Extends the scope of 98b90974's fix to support macros anywhere in the
rewriter. All SourceRange objects are replaced to use macro expanded locations.

docs: Update eBPF features list

Update of BPF features list, following the release of kernel 4.10.

Add in syscount, dbstat and dbslower to apps list.
Signed-off-by: Colin Ian King <colin.king@canonical.com>

`ProcSyms::Module::find_addr` incorrectly resolves symbols when
functions are nested in each other. Specifically, this was discovered
with libpthread, where there are multiple symbols for `write`,
where `write_nocancel` is strictly nested inside `write`. Fix by
explicitly going backward until we reach a matching symbol -- see
details in `ProcSyms::Module::find_addr` comments.

Improve matching of file-backed memory mappings

Use the same rules as perf to determine if a mapping in /proc/pid/maps is
file-backed. This allows mappings in anonymous huge pages and so on to fall
back to resolving from /tmp/perf-pid.map, if appropriate.

ref: http://lxr.free-electrons.com/source/tools/perf/util/map.c#L28

Fix "for loop initial declarations only in C99" compile error

Fix minor error in test instructions

External debuginfo support and general symbols overhaul

This commit introduces support for tests of the new debuglink and
build-id debuginfo resolution functionality. The tests build a
dummy.c file with a debuglink section, and again with a build-id
section, and make sure that the symbol resolution code can locate
the debug information correctly (in the binary's directory for
debuglink, and in /usr/lib/debug/.build-id for the build-id).

The implementation of `ProcSyms::resolve_name` was only valid for
kernel symbols, when there is no module. When a module was provided,
it would segfault due to the module being null. Fixed by making
`bcc_symcache_resolve_name` take an additional module parameter,
which, for kernel symbols, is simply null (`None` from Python).

Adds support for resolving symbols using external debuginfo files,
which can be retrieved from two locations. First, check the build-id
of the desired binary and look in /usr/lib/debug/.build-id according
to the build-id structure. Second, check the debuglink section of
the desired binary and look in /usr/lib/debug or in the binary's
current directory. These are the rules applied by GDB as well, but
GDB lets the user reconfigure the debug directory path from
/usr/lib/debug to something else; we do not support this.

These changes are based on the following description of how GDB
resolves external debuginfo:

https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

When resolving symbols, ProcSyms would treat position-independent
executables (PIE files) incorrectly, resulting in symbol resolution
failures. Specifically, PIE files are treated like shared objects
for ASLR, which means all symbol addresses in the file need to be
taken relative to the executable load address at runtime, the same
as with dynamic library shared objects.

The fix is in the `is_so()` method on `ProcSyms::Module`, which
now uses the correct `bcc_elf_is_shared` helper for testing if a
file is a shared object rather than just looking at the extension
".so", which is very brittle -- and wrong.