* Add runqslower tool

* Remove mentions of obsolete enqueue_task_* in tools/runq*

* Use u32 for pid field in runqslower

* Trace all external pointers going through a first map

Currently, MapVisitor only detects maps with external pointers as
values if the value was directly passed from a function's argument.
For example, in the following, the rewriter is currently unable to
detect currsock has an external pointer as value because an
intermediate variable is used instead of passing directly sk as the
map's value.

    int test(struct pt_regs *ctx, struct sock *sk) {
        u32 pid = bpf_get_current_pid_tgid();
        struct sock **skp = &sk;
        currsock.update(&pid, skp);
        return 0;
    };

With this commit, MapVisitor is able to trace any external pointer
derived from the function's argument and used as a map value. This
commit breaks the ProbeVisitor traversal in two distinct traversals.
The first rewrites dereferences of external pointers originating
from function's arguments and helpers, while the second rewrites only
dereferences of external pointers passed through maps.
Maps with external pointers as values are identified between the two
ProbeVisitor traversals.

* New tests for external pointers passed through maps

test_ext_ptr_maps_reverse ensures dereferences are correctly replaced
even if the update happens after the lookup (in the order of
MapVisitor traversal).
test_ext_ptr_maps_indirect ensures the rewriter is able to trace
external pointers used as map values even if using an intermediate
variable.

* Fix USDT probes arguments' encoding in Python3

Running `trace` on a binary's USDT while fetching some arguments (
`sudo python3 trace.py -p $(pidof ruby) 'u:ruby:array__create "%d",
arg1'`) fails with `argument 2: <class 'TypeError'>: wrong type`.

This PR fixes the encoding of the USDT probe name in
udst.py `get_probe_arg_ctype` function. I've tested this works on Python 2 too.

Link to article on how Circonus uses bcc

old/tools: Diverse fixes

Fix dereference replacements for structure members

Dereferences of structure members pointing to external addresses are
now correctly recognized and replaced by bpf_probe_read calls by the
rewriter.

Currently, if a structure member is assigned an external pointer
(pointer to kernel address), the pointer to the structure is marked
as external instead of the structure member. This issue affects all
uses of structures with pointers to external addresses. This commit
fixes it by marking the structure member as external.

Fix "'ArgString' object has no attribute 'rfind'" error

os.path.dirname expects a string.

Fix bpf_trace_printk() for big-endian targets

The format string for bpf_trace_printk() gets garbled during the
conversion of the LLVM IR code to the corresponding BPF object.
This happens because bcc explicitly sets the data layout of the
module to 'e-m:e-p:64:64-i64:64-n32:64-S128' which is valid for
little-endian targets only. If the arch is set to 'bpf', LLVM
selects the correct data layout automatically and there is no
need to set this explicitly.
Signed-off-by: Sandipan Das <sandipan@linux.vnet.ibm.com>
Reported-by: Chandan Rajendra <chandan@linux.vnet.ibm.com>
Fixes: #1723

Remove duplicate ebpf program load in tools/runqlen.py

fix lua compilation issue with latest clang/llvm 7.0

fix compilation error with latest llvm

The clang commit https://reviews.llvm.org/rL331155
changed the clang::SourceManager function prototype
   SourceRange getExpansionRange(SourceRange Range)
to
   CharSourceRange getExpansionRange(SourceRange Range)
and caused the following compilation failure:

  /home/yhs/work/bcc/src/cc/frontends/clang/b_frontend_action.cc:
  In member function ‘clang::SourceRange ebpf::ProbeVisitor::expansionRange(clang::SourceRange)’:
  /home/yhs/work/bcc/src/cc/frontends/clang/b_frontend_action.cc:267:58:
  error: could not convert ‘(&(&((ebpf::ProbeVisitor *)this)->ebpf::ProbeVisitor::rewriter_)
         ->clang::Rewriter::getSourceMgr())->clang::SourceManager::getExpansionRange(range)’
  from ‘clang::CharSourceRange’ to ‘clang::SourceRange’
     return rewriter_.getSourceMgr().getExpansionRange(range);
                                                            ^
  ...

It is hard to find a compatible change which works
for both old llvm and the latest change. So this patch
just fixed the problem for clang 7.0.0 and the old code
is used for clang 6.x and lower.
Signed-off-by: Yonghong Song <yhs@fb.com>

Detect external pointers from context argument

The bcc rewriter is currently unable to detect external pointers
(i.e., to a memory address that requires calls to bpf_probe_read) if
they are not declared as arguments, e.g., if they are retrieved
through the context argument.
For example, although the two following examples translate to the
same C code in the end (the bcc rewriter translates the first into
the second), the sk pointer is recognized as an external pointer only
in the first example.

int test1(struct pt_regs *ctx, struct sock *sk) {
    // sk is correctly recognized as an external pointer.
}
int test2(struct pt_regs *ctx) {
    struct sock *sk = (struct sock *)ctx->di;
    // sk is not recognized as an external pointer.
}

This commit fixes that by detecting member dereferences of the
context argument (i.e., the first argument of externally visible
functions). It also works for the TRACEPOINT_PROBE macro.

Commit 46ebd9ef ("fix compilation with latest clang/llvm 7.0")
tried to fix a compilation introduced with latest clang change.
It did not fix the lua compilation issue since unfortunately
my bcc environment did not have lua setup properly for that commit.

This commit intends to fix for lua compilation issue as well.
The intermediate static library bcc-lua-static is removed and
now bcc-lua-static is just a list of static libraries used
for final linking to produce bcc-lua. This way, we do not
need the special linker flags like -Wl,--whole-archive and
-Wl,--no-whole-archive. The bcc-static and bcc-shared libraries
did not change since they will be installed and may be used
by the application.

Tested on FC27 based system, with the workaround in issue #1685,
all tests passed.

Fixes: 46ebd9ef ("fix compilation with latest clang/llvm 7.0")
Signed-off-by: Yonghong Song <yhs@fb.com>

fix compilation with latest clang/llvm 7.0

profile: update --stack-storage-size default value in help message

The bcc build with latest clang/llvm 7.0 has the following
compilation error:

 /home/yhs/work/llvm/build/install/lib/libclangCodeGen.a(CodeGenAction.cpp.o):
  In function `clang::BackendConsumer::Initialize(clang::ASTContext&)':
  CodeGenAction.cpp:(.text._ZN5clang15BackendConsumer10InitializeERNS_10ASTContextE[_ZN5clang15BackendConsumer10InitializeERNS_10ASTContextE]+0xc): undefined reference to `clang::FrontendTimesIsEnabled'
 ...

The latest clang defined clang::FrontendTimesIsEnabled in
libclangFrontend.a and used in libclangCodeGen.a.

Due to the library order specified in clang_libs.cmake:
  # order is important
  set(clang_libs
    ${libclangFrontend}
    ...
    ${libclangCodeGen}
and the fact clang_libs here is used to build bcc-static,
the definition of clang::FrontendTimesIsEnabled is dropped
since nobody used it before the definition is checked by the compiler.

The fix here is to keep all the definitions/uses for clang_libs/llvm_libs
when buildint static library.

Tested on clang 4.0, 5.0, 6.0 and latest 7.0, all works fine.
Signed-off-by: Yonghong Song <yhs@fb.com>

docs: update BPF features list (tracepoints to debug eBPF were removed)

Tracepoints for debugging eBPF were removed from the kernel (some other
mechanisms were being used instead, and they were not used often enough
to get proper maintenance). Let's specify this on the related line in
BPF features doc.

introduce {attach|detach}_raw_tracepoint API

Update docs/kernel_versions

Update list of main features and supported map types.

Also update list of architectures supporting JIT, and main features
list, with important upcoming changes (x86_32 for JIT, bpfilter and
AF_XDP for main features).

Also add notes on top of the lists about how to obtain the items of each
list.

Remove speculative map types with no link or schedule.

- Update links in doc (make them point from net-next to linux, when
  relevant).
- Add helpers bpf_xdp_adjust_tail() and bpf_skb_get_xfrm_state() to
  documentation and headers.
- Synchronise helpers with latest net-next.

The motivation comes from pull request #1689.
It attached a kprobe bpf program to kernel function
ttwu_do_wakeup for more accurate tracing.
Unfortunately, it broke runqlat.py in my
4.17 environment since ttwu_do_wakeup function
is inlined in my kernel with gcc 7.3.1.

4.17 introduced raw_tracepoint and this patch
added the relevant API to bcc. With this,
we can use tracepoints
sched:{sched_wakeup, sched_wakeup_new, sched_switch}
to measure runq latency more reliably.
Signed-off-by: Yonghong Song <yhs@fb.com>

Misc improvements to C++ API and example