Note this allows us to eliminate the `bpf_ktime_get_ns()`
call from `trace_entry()`.

prevent array subscript expression if base/index is not rewritable

The following command failed:
  trace.py -U 'r::_do_fork (retval == -11) "%llu", ((struct task_struct *)bpf_get_current_task())->signal->rlim[RLIMIT_NPROC].rlim_cur'

as rewriter generates code like
  __data.v0 = (unsigned long long)((struct task_struct *)bpf_get_current_task())->signal->rlim[RLIMIT_NPROC))); _val; }).rlim_cur;

Let us prevent rewriting if either base or index is not rewritable and this fixed the issue.
Signed-off-by: Yonghong Song <yhs@fb.com>

get_offline_table is missing in the BPFTable class.

Add it and some test cases.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>

Adds the support for the DEVMAP in the C++ APIs, which was missing in #1810

bpf_probe_read()'s third argument is no longer rewritten. Instead, use a
temporary variable (like #1973) to avoid a memory access error.

Before:

```
$ sudo biolatency -D 1
bpf: Failed to load program: Permission denied
0: (79) r1 = *(u64 *)(r1 +112)
1: (7b) *(u64 *)(r10 -8) = r1
[..]
R1 invalid mem access 'inv'

HINT: The invalid mem access 'inv' error can happen if you try to
dereference memory without first using bpf_probe_read() to copy it to
the BPF stack. Sometimes the bpf_probe_read is automatic by the bcc
rewriter, other times you'll need to be explicit.
```

After, works as expected.

Currently, trace.py does not support "long" and "unsigned long"
types and it often caught users with a surprise and they are
not sure what is the problem. For example, for kernel function:
  void blk_mq_delay_kick_requeue_list(struct request_queue *q, unsigned long msecs)
The following
  $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %lu", msecs'
  list index out of range

With this patch,
  $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %lu", msecs'
  PID     TID     COMM            FUNC             -
  ^C
  $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %ld", msecs'
  PID     TID     COMM            FUNC             -
  ^C
  $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %lx", msecs'
  PID     TID     COMM            FUNC             -
  ^C
  $
Signed-off-by: Yonghong Song <yhs@fb.com>

In llvm/clang 8 trunk, the following commits
  https://reviews.llvm.org/rL339384
  https://reviews.llvm.org/rL339385
  https://reviews.llvm.org/rL339386
renames getLocStart to getBeginLoc, and getLocEnd to getEndLoc.
This caused bcc compilation error with llvm/clang 8.
To avoid proliferation of #define's based on LLVM version numbers
and similar logic for many different types,
this patch fixed the issues by define macros in common
frontend_action_common.h where macro definition is multi-versioned
based on llvm version. The macro itself is used in
{b,tp}_frontend_action.cc.

clang 8 also introduced a new library dependency
  libclangAnalysis.a depends on libclangASTMatchers.a
This patch fixed this issue as well.
Signed-off-by: Yonghong Song <yhs@fb.com>

Currently, trace.py failed for the following command:
  $ sudo ./trace.py 'filename_lookup(int dfd, struct filename *name) "%s", name->name'
  ...
  0: (bf) r6 = r1
  1: (79) r7 = *(u64 *)(r6 +104)
  ...
  32: (15) if r1 == 0x0 goto pc+5
  R0=inv(id=0) R1=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0)
  R8=inv0 R10=fp0,call_-1 fp-8=0 fp-16=0 fp-24=0 fp-32=0 fp-40=0 fp-48=0 fp-56=0 fp-64=0 fp-72=0 fp-80=0
  33: (79) r3 = *(u64 *)(r7 +0)
  R7 invalid mem access 'inv'

For string format argument, the trace.py generates the below code:
        if (name->name != 0) {
                bpf_probe_read(&__data.v0, sizeof(__data.v0), (void *)name->name);
        }
Right now, bcc skips the rewriter for the third argument of bpf_probe_read to avoid
unnecessary nested bpf_probe_read and other potential issues.
This causes name->name memory access not transformed with bpf_probe_read and hence
the verifier complains.

To fix the issue, this patch did the following transformation using an
temporary variable to hold the src address:
        if (name->name != 0) {
                void *__tmp = (void *)name->name;
                bpf_probe_read(&__data.v0, sizeof(__data.v0), __tmp);
        }
This way, rewriter can do the work properly.
Signed-off-by: Yonghong Song <yhs@fb.com>

Fix syntax error in xdp_redirect_cpu.py

* Bump ubuntu docker build to bionic

Fixes build dependency for arm64
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>

* Add Build-Depends: dh-python

Fixes build dependency issue observed on bionic/arm64:

dh: unable to load addon python3: Can't locate Debian/Debhelper/Sequence/python3.pm in @INC (you may need to install the Debian::Debhelper::Sequence::python3 module) (@INC contains: /etc/perl /usr/local/lib/aarch64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/aarch64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/aarch64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/aarch64-linux-gnu/perl-base) at (eval 9) line 1.
BEGIN failed--compilation aborted at (eval 9) line 1.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>

* Enable arm64 deb packaging

LLVM6.0 is minimum version required for arm64 build. [PR#1512]

[PR#1512] https://github.com/iovisor/bcc/pull/1512Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>

Finish the conversion started with commit 36ce1124.

Minor tweaks to make the tools consistent within themselves.

tcpretrans: use user-space PID displayed to the user

PID collected in tcpretrans' trace_event() is displayed under the
label of "PID" to the user so it would seem more appropriate to use
TGID as PID not kernel's PID.

* Small fix on C++ USDT implementation

Fix an logging error, and small optimization on initialization

* Add move constructor to C++ USDT instance

debian changelog for v0.7.0 tag

Sort language entries while at it.

* tools: Skip backward time entries in xfsslower

While using xfsslower on RHEL7 we occasionally get
following screwed up latencies:

  # xfsslower
  Tracing XFS operations slower than 1 ms
  TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
  13:25:03 git            3385   R 62      4704     18446744073708.55 tmp_pack_bDUbwZ
  13:25:03 git            3385   S 0       0           3.05 tmp_idx_Kjb2bW
  ...

The reason for this is that on RHEL7 it's possible to get backward
timetamp with bpf_ktime_get_ns. This needs to be fixed, but meanwhile
this fix makes sure the latencies with backward times are skipped.

For the rest of the kernels this is just sanity fix with
possibly just single compare instruction overhead.

It's temporary workaround for #728.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>

* tools: Skip backward time entries in ext4dist

While using ext4dist on RHEL7 we occasionally get
following screwed up latencies:

  # ext4dist
  Tracing ext4 operation latency... Hit Ctrl-C to end.
  ^C

  operation = write
               usecs                         : count     distribution
                   0 -> 1                    : 1134529   |********            |
                   2 -> 3                    : 2777582   |********************|
                   4 -> 7                    : 688014    |****                |
                   8 -> 15                   : 36160     |                    |
                  16 -> 31                   : 698       |                    |
                  32 -> 63                   : 6         |                    |
                  64 -> 127                  : 15        |                    |
                 128 -> 255                  : 7         |                    |
                 256 -> 511                  : 1         |                    |
                 512 -> 1023                 : 0         |                    |
                1024 -> 2047                 : 0         |                    |
                2048 -> 4095                 : 2         |                    |
                4096 -> 8191                 : 1         |                    |
                8192 -> 16383                : 5         |                    |
               16384 -> 32767                : 0         |                    |
               32768 -> 65535                : 0         |                    |

    9007199254740992 -> 18014398509481983    : 0         |                    |
   18014398509481984 -> 36028797018963967    : 1         |                    |

The reason for this is that on RHEL7 it's possible to get backward
timestamp with bpf_ktime_get_ns. This needs to be fixed, but meanwhile
this fix makes sure the latencies with backward times are skipped.

For the rest of the kernels this is just sanity fix with
possibly just single compare instruction overhead.

It's temporary workaround for #728.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>

fix a typo

Generating sdist for both python2 and python3 at the same time can fail
as they both use the same temporary files.

Make sure that we generate sdist for each `PYTHON_CMD` sequentially.

Added in RHEL 7.6 Beta information

The poll iteration count 10 sometimes not big enough.
Now let us increase to 100 iterations, but will bail out
if the expected data have received. Hopefully this
will fix flakiness of this test.
Signed-off-by: Yonghong Song <yhs@fb.com>

* copy oomkill.py to old/oomkill.py

* update oomkill

* Update test_tools_smoke.py

argdist -H 'r::__vfs_read(void *file, void *buf, size_t count):size_t
$entry(count):$latency > 1000000'
is a wrong example which cann't excute success, because lack of ":" and cann't split field correctly.

So, the right command is:

argdist -H 'r::__vfs_read(void *file, void *buf, size_t count):size_t:
$entry(count):$latency > 1000000'
Signed-off-by: Ahao Mu <muahao@linux.alibaba.com>

Fix unsiggned typo introduced in tp_frontend_action.cc

all program types, map types and helpers are added based on latest net-next.
Signed-off-by: Yonghong Song <yhs@fb.com>

make types appropriate for ipaddr/protocol etc.

It's useful to know each syscall's total latency
in that period, but not the latency of the last time
in that period.
Signed-off-by: Ahao Mu <muahao@linux.alibaba.com>

provide a parameter to suppress printing a new line at the end of the bytes.
existing behavior is not changed.

fix a rewriter bug for array subscript

additional fix for issue #1850

for the below case in test_clang.py;
  int test(struct pt_regs *ctx, struct mm_struct *mm) {
      return mm->rss_stat.count[MM_ANONPAGES].counter;
  }

the current rewriter generates:
  int test(struct pt_regs *ctx) {
   struct mm_struct *mm = ctx->di;
      return ({ typeof(atomic_long_t) _val;
                __builtin_memset(&_val, 0, sizeof(_val));
                bpf_probe_read(&_val,
                               sizeof(_val),
                               (u64)(&mm->rss_stat.count) + (MM_ANONPAGES));
                _val; }).counter;
  }
The third argument of bpf_probe_read() is incorrect.
The correct third argument should be
   (u64)((&mm->rss_stat.count) + (MM_ANONPAGES))

This patch fixed the issue by adding extra parenthesis for the
outer u64 type casting.

  int test(struct pt_regs *ctx) {
   struct mm_struct *mm = ctx->di;
      return ({ typeof(atomic_long_t) _val;
                __builtin_memset(&_val, 0, sizeof(_val));
                bpf_probe_read(&_val,
                               sizeof(_val),
                               (u64)((&mm->rss_stat.count) + (MM_ANONPAGES)));
                _val; }).counter;
  }
Signed-off-by: Yonghong Song <yhs@fb.com>

LLVM commit https://reviews.llvm.org/D49741 removed
function DEARFContext::getCompileUnitAtIndex() and caused
the bcc compilation failure.

Change usage of getCompileUnitAtIndex() to the one
recommended in the above llvm commit.
Signed-off-by: Yonghong Song <yhs@fb.com>

fix #1921

For newer kernels, bcc tries to fd based kuprobe
attachment. In fd based kprobe attachment,
  config1: for symbol
  config2: for symbol offset

In python API attach_kretprobe, the offset value
is not set in lib.bpf_attach_kprobe and hence it
will be a random value and eventually causing
kretprobe attachment failure.

This is not an issue for old debugfs based attachment
as the offset will not be used if it is a retprobe.
Signed-off-by: Yonghong Song <yhs@fb.com>

Add name to program too large error message

llcstat: print a nicer error message when hardware events are missing

* Adding Fedora 28 to the list of fedora versions

Looking at the repo location, Fedora 28 appears to be supported as well.  Fixing the documentation to include this.

* Update INSTALL.md

* python3: check ksymname calls with _assert_is_bytes

Fixes a bytes/string concatenation error when get/fix_syscall_fnname is
called from a python3 system.

* python3: use env python invocation in tools

In order to facilitate testing, but not necessarily as an example of
good practice, I am changing the invocation of the test tools to use
`/usr/bin/env python`, so that we can control which python (2 vs 3)
gets invoked for the test. On the buildbots, I plan to add an optional
`ln -s /usr/bin/python3 /usr/local/bin/python` on systems that have
python3-bcc package built. This way, we get more test coverage. Having a
cmake mechanism to enable both python2 and python3 testing could be a
further enhancement.

* tools/memleak: add an explicit stdout.flush to print loop

The stdout flush behavior seems to have changed in python3, breaking one
of the tests. I think it makes sense to flush stdout at the end of each
timed interval loop anyway, so adding that to the tool itself.

* tests: add b'' strings and fix dangling handles

Add b'' strings in a few places in the test tools, and fix one dangling
process handle in the memleak test tool runner.

* Fix multiple memory access errors

Fixes a buffer overflow in get_pid_exe(), a use-after-free error in
bcc_usdt_get_probe_argctype() and a possible NULL pointer dereference
in find_debug_via_debuglink().

* Fix multiple ressource leaks

Leaked file descriptors in bpf_attach_uprobe() and verify_checksum().
Memory leaks in  Parser::func_add() and bcc_procutils_language().

* fixup! Fix multiple ressource leaks

* Documentation: Added table for program type and its helper functions

* updated the Program type table to follow the same Markdown format as other tables

* Fixed typos

* fixed typos and added new helper functions

Hardware events such as CACHE_MISSES and CACHE_REFERENCES are usually
not available on virtual machine. Print a more useful message when
this happen.