But verify via nint the scenario is sane (e.g. nint=2 means TXintr
RXintr icmp_echo icmp_reply TXintr RXintr ...)

----------------------------------------
     nint          : count     distribution
        0          : 0        |                                        |
        1          : 3        |                                        |
        2          : 24557    |****************************************|
        3          : 11       |                                        |
        4          : 1        |                                        |
     int - icmp_echo (μs) : count     distribution
         0 -> 1          : 23238    |****************************************|
         2 -> 3          : 1031     |*                                       |
         4 -> 7          : 169      |                                        |
         8 -> 15         : 17       |                                        |
        16 -> 31         : 118      |                                        |
        32 -> 63         : 1        |                                        |
     icmp_echo - tx (μs) : count     distribution
         0 -> 1          : 24335    |****************************************|
         2 -> 3          : 186      |                                        |
         4 -> 7          : 39       |                                        |
         8 -> 15         : 1        |                                        |
        16 -> 31         : 16       |                                        |
     int - tx (μs)      : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 23836    |****************************************|
         4 -> 7          : 583      |                                        |
         8 -> 15         : 26       |                                        |
        16 -> 31         : 123      |                                        |
        32 -> 63         : 10       |                                        |
----------------------------------------

i.e. on my deco software processing time for icmp echo request is <= 3μs.

Although last point - it now gets counted from prev TX time which is
wrong to do (nint = 2 and so latencies are high):

----------------------------------------
     nint          : count     distribution
        0          : 0        |                                        |
        1          : 9        |                                        |
        2          : 23421    |****************************************|
        3          : 34       |                                        |
        4          : 7        |                                        |
     int - icmp_echo (μs) : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 12       |                                        |
        32 -> 63         : 26       |                                        |
        64 -> 127        : 23099    |****************************************|
       128 -> 255        : 317      |                                        |
       256 -> 511        : 5        |                                        |
       512 -> 1023       : 6        |                                        |
      1024 -> 2047       : 1        |                                        |
      2048 -> 4095       : 1        |                                        |
      4096 -> 8191       : 2        |                                        |
     icmp_echo - tx (μs) : count     distribution
         0 -> 1          : 23125    |****************************************|
         2 -> 3          : 266      |                                        |
         4 -> 7          : 66       |                                        |
         8 -> 15         : 3        |                                        |
        16 -> 31         : 9        |                                        |
     int - tx (μs)      : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 8        |                                        |
        32 -> 63         : 30       |                                        |
        64 -> 127        : 23084    |****************************************|
       128 -> 255        : 330      |                                        |
       256 -> 511        : 5        |                                        |
       512 -> 1023       : 6        |                                        |
      1024 -> 2047       : 1        |                                        |
      2048 -> 4095       : 1        |                                        |
      4096 -> 8191       : 2        |                                        |
----------------------------------------

Switch to estimate end of net tx by icmp_echo kretprobe (icmp_echo calls
icmp_reply which in turn cals ... -> net_dev_start_xmit).

Draft, does not currently work.

For the following program:

    #include <linux/interrupt.h>

    // remember t(last-interrupt) on interface
    int kprobe__handle_irq_event_percpu(struct pt_regs *ctx, struct irq_desc *desc) {
        const char *irqname = desc->action->name;

        char c;

        bpf_probe_read(&c, 1, &irqname[0]);
        if (c != 'e') return 0;

        bpf_probe_read(&c, 1, &irqname[1]);
        if (c != 't') return 0;

        ...

LLVM gives warnings because irqaction->name is `const char *`:

    /virtual/main.c:10:27: warning: passing 'const char *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        bpf_probe_read(&c, 1, &irqname[0]);
                              ^~~~~~~~~~~
    /virtual/main.c:13:27: warning: passing 'const char *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        bpf_probe_read(&c, 1, &irqname[1]);
                              ^~~~~~~~~~~
    ...

Instead of adding casts in source everywhere fix bpf_probe_read* signature to
indicate the memory referenced by src won't be modified, as it should be.

P.S.

bpf_probe_read_str was in fact already marked so in several places in comments
but not in actual signature.

Even if units was μs and thus factor=1000 `hardirqs -d` and `softirqs
-d` were printing actual data in nanoseconds but with distribution unit
labeled as "usecs", e.g.:

    softirq = sched
         usecs               : count     distribution
             0 -> 1          : 0        |                                        |
             2 -> 3          : 0        |                                        |
             4 -> 7          : 0        |                                        |
             8 -> 15         : 0        |                                        |
            16 -> 31         : 0        |                                        |
            32 -> 63         : 0        |                                        |
            64 -> 127        : 0        |                                        |
           128 -> 255        : 0        |                                        |
           256 -> 511        : 0        |                                        |
           512 -> 1023       : 0        |                                        |
          1024 -> 2047       : 6        |*****                                   |
          2048 -> 4095       : 3        |**                                      |
          4096 -> 8191       : 8        |*******                                 |
          8192 -> 16383      : 31       |*****************************           |
         16384 -> 32767      : 42       |****************************************|
         32768 -> 65535      : 18       |*****************                       |

Fix it.

NOTE: not putting "/ FACTOR" into common code because for counting mode
we do not want to round intermediate results as e.g. there could be lots
of say 0.5μs interrupts that should be all accounted as N*0.5, not N*0.0.

Execsnoop's documentation says -x/--fails means "also include failed
exec()s". However it was programmed to instead skip successful execs on
-x and without -x show all - successful and unsuccessful ones.

The logic was broken in 5b47e0f8 ("execsnoop: use BPF_PERF_OUTPUT
instead of trace pipe").

Fix it.

P.S. current test_tools_smoke.py only provides basic infrastructure for
testing whether tool's BPF program won't break, without anything related
to options handling, so unfortunately the patch comes without
corresponding test.

LPM trie maps require the BPF_F_NO_PREALLOC flag on creation. The
need for this flag is not obvious at first; this new macro should
help avoid the mistake.

Fix edge case when doing symbol name -> address resolution

In order to run, some test programs depend on the availability
of binaries in locations that are part of PATH. So, we add a
generic utility to simplify this.
Signed-off-by: Sandipan Das <sandipan@linux.vnet.ibm.com>

If netperf is not installed or installed at a location that is not
in PATH as recognized by Python, then 'test_brb' and 'test_brb2'
freeze after an OSError is raised. To avoid this, we proactively
check if the 'iperf', 'netserver' and 'netperf' binaries are
available before making the corresponding NSPopen() calls.
Signed-off-by: Sandipan Das <sandipan@linux.vnet.ibm.com>

Fix 'test_debuginfo' from failing if a kernel symbol has multiple aliases

Update LINKS.md

The first symbol from /proc/kallsyms is read by 'test_debuginfo'
and the name obtained here is compared against the name obtained
from the BPF.ksym() library call. In some architectures such as
powerpc64le, a kernel symbol may have multiple aliases that refer
to the same address. So, to avoid the test from failing, we need
to compare the name returned by BPF.ksym() against all possible
aliases for the given address in /proc/kallsyms and look for a
match.
Signed-off-by: Sandipan Das <sandipan@linux.vnet.ibm.com>

Signed-off-by: Gary Lin <glin@suse.com>

Signed-off-by: Colin Ian King <colin.king@canonical.com>

nfsdist: trace NFS operation latency distribution, similar to the other *dist tools.

Fix for incorrect tool arguments in the man pages

Improve string buffer checking on uprobe attach and detach

annotate program tag

Do not create instance for kprobe

Fix for bug in lesson 4 of the Python developer tutorial

during debug of production systems it's difficult to trace back
the kernel reported 'bpf_prog_4985bb0bd6c69631' symbols to the source code
of the program, hence teach bcc to store the main function source
in the /var/tmp/bcc/bpf_prog_4985bb0bd6c69631/ directory.

This program tag is stable. Every time the script is called the tag
will be the same unless source code of the program changes.
During active development of bcc scripts the /var/tmp/bcc/ dir can
get a bunch of stale tags. The users have to trim that dir manually.

Python scripts can be modified to use this feature too, but probably
need to be gated by the flag. For c++ api I think it makes sense
to store the source code always, since the cost is minimal and
c++ api is used by long running services.

Example:
$ ./examples/cpp/LLCStat
$ ls -l /var/tmp/bcc/bpf_prog_4985bb0bd6c69631/
total 16
-rw-r--r--. 1 root root 226 Sep  1 17:30 on_cache_miss.c
-rw-r--r--. 1 root root 487 Sep  1 17:30 on_cache_miss.rewritten.c
-rw-r--r--. 1 root root 224 Sep  1 17:30 on_cache_ref.c
-rw-r--r--. 1 root root 484 Sep  1 17:30 on_cache_ref.rewritten.c

Note that there are two .c files there, since two different
bpf programs have exactly the same bytecode hence same prog_tag.

$ cat /var/tmp/bcc/bpf_prog_4985bb0bd6c69631/on_cache_miss.c
int on_cache_miss(struct bpf_perf_event_data *ctx) {
    struct event_t key = {};
    get_key(&key);

    u64 zero = 0, *val;
    val = miss_count.lookup_or_init(&key, &zero);
...
Signed-off-by: Alexei Starovoitov <ast@fb.com>

bpf_obj_get_info() to retreive prog_tag from the kernel based on prog_fd (kernel 4.13+)
bpf_prog_compute_tag() to compute prog_tag from a set of bpf_insns (kernel independent)
bpf_prog_get_tag() to retrieve prog_tag from /proc/pid/fdinfo/fd (kernel 4.10+)
Signed-off-by: Alexei Starovoitov <ast@fb.com>

docs: update features list

examples: fixed http_filter example

Loop unrolling wasn't happening because of variable upper
bound. Making it const fixes the problem.

Tested with 4.13.0-rc6 on fedora 26

Mention the smoke tests for new tool contributions

examples: fix vlan_learning example

* Initial commit of nfsslower, tracking NFS4_READ, NFS4_WRITE and NFS4_OPEN

* Added in documentation, examples, support for csv and tracing for GETATTR

* Added in man pages, READM mods and example file, to comply with
https://github.com/samuelnair/bcc/blob/master/CONTRIBUTING-SCRIPTS.md

* Changes to address comments from @brendangregg and a small bug regarding
the output header not being printed when tracing all NFS operations

* Added nfsslower to the correct alphabetical postion

* Addressing Sasha's comments. I appreciate the thoroughness of the review

* Added test case for nfsslower and an extra function to check if the NFS kernel
module is loaded