In llvm/clang 8 trunk, the following commits
  https://reviews.llvm.org/rL339384
  https://reviews.llvm.org/rL339385
  https://reviews.llvm.org/rL339386
renames getLocStart to getBeginLoc, and getLocEnd to getEndLoc.
This caused bcc compilation error with llvm/clang 8.
To avoid proliferation of #define's based on LLVM version numbers
and similar logic for many different types,
this patch fixed the issues by define macros in common
frontend_action_common.h where macro definition is multi-versioned
based on llvm version. The macro itself is used in
{b,tp}_frontend_action.cc.

clang 8 also introduced a new library dependency
  libclangAnalysis.a depends on libclangASTMatchers.a
This patch fixed this issue as well.
Signed-off-by: Yonghong Song <yhs@fb.com>

Currently, trace.py failed for the following command:
  $ sudo ./trace.py 'filename_lookup(int dfd, struct filename *name) "%s", name->name'
  ...
  0: (bf) r6 = r1
  1: (79) r7 = *(u64 *)(r6 +104)
  ...
  32: (15) if r1 == 0x0 goto pc+5
  R0=inv(id=0) R1=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0)
  R8=inv0 R10=fp0,call_-1 fp-8=0 fp-16=0 fp-24=0 fp-32=0 fp-40=0 fp-48=0 fp-56=0 fp-64=0 fp-72=0 fp-80=0
  33: (79) r3 = *(u64 *)(r7 +0)
  R7 invalid mem access 'inv'

For string format argument, the trace.py generates the below code:
        if (name->name != 0) {
                bpf_probe_read(&__data.v0, sizeof(__data.v0), (void *)name->name);
        }
Right now, bcc skips the rewriter for the third argument of bpf_probe_read to avoid
unnecessary nested bpf_probe_read and other potential issues.
This causes name->name memory access not transformed with bpf_probe_read and hence
the verifier complains.

To fix the issue, this patch did the following transformation using an
temporary variable to hold the src address:
        if (name->name != 0) {
                void *__tmp = (void *)name->name;
                bpf_probe_read(&__data.v0, sizeof(__data.v0), __tmp);
        }
This way, rewriter can do the work properly.
Signed-off-by: Yonghong Song <yhs@fb.com>

Fix syntax error in xdp_redirect_cpu.py

* Bump ubuntu docker build to bionic

Fixes build dependency for arm64
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>

* Add Build-Depends: dh-python

Fixes build dependency issue observed on bionic/arm64:

dh: unable to load addon python3: Can't locate Debian/Debhelper/Sequence/python3.pm in @INC (you may need to install the Debian::Debhelper::Sequence::python3 module) (@INC contains: /etc/perl /usr/local/lib/aarch64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/aarch64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/aarch64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/aarch64-linux-gnu/perl-base) at (eval 9) line 1.
BEGIN failed--compilation aborted at (eval 9) line 1.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>

* Enable arm64 deb packaging

LLVM6.0 is minimum version required for arm64 build. [PR#1512]

[PR#1512] https://github.com/iovisor/bcc/pull/1512Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>

Finish the conversion started with commit 36ce1124.

Minor tweaks to make the tools consistent within themselves.

tcpretrans: use user-space PID displayed to the user

PID collected in tcpretrans' trace_event() is displayed under the
label of "PID" to the user so it would seem more appropriate to use
TGID as PID not kernel's PID.

* Small fix on C++ USDT implementation

Fix an logging error, and small optimization on initialization

* Add move constructor to C++ USDT instance

debian changelog for v0.7.0 tag

Sort language entries while at it.

* tools: Skip backward time entries in xfsslower

While using xfsslower on RHEL7 we occasionally get
following screwed up latencies:

  # xfsslower
  Tracing XFS operations slower than 1 ms
  TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
  13:25:03 git            3385   R 62      4704     18446744073708.55 tmp_pack_bDUbwZ
  13:25:03 git            3385   S 0       0           3.05 tmp_idx_Kjb2bW
  ...

The reason for this is that on RHEL7 it's possible to get backward
timetamp with bpf_ktime_get_ns. This needs to be fixed, but meanwhile
this fix makes sure the latencies with backward times are skipped.

For the rest of the kernels this is just sanity fix with
possibly just single compare instruction overhead.

It's temporary workaround for #728.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>

* tools: Skip backward time entries in ext4dist

While using ext4dist on RHEL7 we occasionally get
following screwed up latencies:

  # ext4dist
  Tracing ext4 operation latency... Hit Ctrl-C to end.
  ^C

  operation = write
               usecs                         : count     distribution
                   0 -> 1                    : 1134529   |********            |
                   2 -> 3                    : 2777582   |********************|
                   4 -> 7                    : 688014    |****                |
                   8 -> 15                   : 36160     |                    |
                  16 -> 31                   : 698       |                    |
                  32 -> 63                   : 6         |                    |
                  64 -> 127                  : 15        |                    |
                 128 -> 255                  : 7         |                    |
                 256 -> 511                  : 1         |                    |
                 512 -> 1023                 : 0         |                    |
                1024 -> 2047                 : 0         |                    |
                2048 -> 4095                 : 2         |                    |
                4096 -> 8191                 : 1         |                    |
                8192 -> 16383                : 5         |                    |
               16384 -> 32767                : 0         |                    |
               32768 -> 65535                : 0         |                    |

    9007199254740992 -> 18014398509481983    : 0         |                    |
   18014398509481984 -> 36028797018963967    : 1         |                    |

The reason for this is that on RHEL7 it's possible to get backward
timestamp with bpf_ktime_get_ns. This needs to be fixed, but meanwhile
this fix makes sure the latencies with backward times are skipped.

For the rest of the kernels this is just sanity fix with
possibly just single compare instruction overhead.

It's temporary workaround for #728.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>

fix a typo

Generating sdist for both python2 and python3 at the same time can fail
as they both use the same temporary files.

Make sure that we generate sdist for each `PYTHON_CMD` sequentially.

Added in RHEL 7.6 Beta information

The poll iteration count 10 sometimes not big enough.
Now let us increase to 100 iterations, but will bail out
if the expected data have received. Hopefully this
will fix flakiness of this test.
Signed-off-by: Yonghong Song <yhs@fb.com>

* copy oomkill.py to old/oomkill.py

* update oomkill

* Update test_tools_smoke.py

argdist -H 'r::__vfs_read(void *file, void *buf, size_t count):size_t
$entry(count):$latency > 1000000'
is a wrong example which cann't excute success, because lack of ":" and cann't split field correctly.

So, the right command is:

argdist -H 'r::__vfs_read(void *file, void *buf, size_t count):size_t:
$entry(count):$latency > 1000000'
Signed-off-by: Ahao Mu <muahao@linux.alibaba.com>

Fix unsiggned typo introduced in tp_frontend_action.cc

all program types, map types and helpers are added based on latest net-next.
Signed-off-by: Yonghong Song <yhs@fb.com>

make types appropriate for ipaddr/protocol etc.

It's useful to know each syscall's total latency
in that period, but not the latency of the last time
in that period.
Signed-off-by: Ahao Mu <muahao@linux.alibaba.com>

provide a parameter to suppress printing a new line at the end of the bytes.
existing behavior is not changed.

fix a rewriter bug for array subscript

additional fix for issue #1850

for the below case in test_clang.py;
  int test(struct pt_regs *ctx, struct mm_struct *mm) {
      return mm->rss_stat.count[MM_ANONPAGES].counter;
  }

the current rewriter generates:
  int test(struct pt_regs *ctx) {
   struct mm_struct *mm = ctx->di;
      return ({ typeof(atomic_long_t) _val;
                __builtin_memset(&_val, 0, sizeof(_val));
                bpf_probe_read(&_val,
                               sizeof(_val),
                               (u64)(&mm->rss_stat.count) + (MM_ANONPAGES));
                _val; }).counter;
  }
The third argument of bpf_probe_read() is incorrect.
The correct third argument should be
   (u64)((&mm->rss_stat.count) + (MM_ANONPAGES))

This patch fixed the issue by adding extra parenthesis for the
outer u64 type casting.

  int test(struct pt_regs *ctx) {
   struct mm_struct *mm = ctx->di;
      return ({ typeof(atomic_long_t) _val;
                __builtin_memset(&_val, 0, sizeof(_val));
                bpf_probe_read(&_val,
                               sizeof(_val),
                               (u64)((&mm->rss_stat.count) + (MM_ANONPAGES)));
                _val; }).counter;
  }
Signed-off-by: Yonghong Song <yhs@fb.com>

LLVM commit https://reviews.llvm.org/D49741 removed
function DEARFContext::getCompileUnitAtIndex() and caused
the bcc compilation failure.

Change usage of getCompileUnitAtIndex() to the one
recommended in the above llvm commit.
Signed-off-by: Yonghong Song <yhs@fb.com>

fix #1921

For newer kernels, bcc tries to fd based kuprobe
attachment. In fd based kprobe attachment,
  config1: for symbol
  config2: for symbol offset

In python API attach_kretprobe, the offset value
is not set in lib.bpf_attach_kprobe and hence it
will be a random value and eventually causing
kretprobe attachment failure.

This is not an issue for old debugfs based attachment
as the offset will not be used if it is a retprobe.
Signed-off-by: Yonghong Song <yhs@fb.com>

Add name to program too large error message

llcstat: print a nicer error message when hardware events are missing

* Adding Fedora 28 to the list of fedora versions

Looking at the repo location, Fedora 28 appears to be supported as well.  Fixing the documentation to include this.

* Update INSTALL.md

* python3: check ksymname calls with _assert_is_bytes

Fixes a bytes/string concatenation error when get/fix_syscall_fnname is
called from a python3 system.

* python3: use env python invocation in tools

In order to facilitate testing, but not necessarily as an example of
good practice, I am changing the invocation of the test tools to use
`/usr/bin/env python`, so that we can control which python (2 vs 3)
gets invoked for the test. On the buildbots, I plan to add an optional
`ln -s /usr/bin/python3 /usr/local/bin/python` on systems that have
python3-bcc package built. This way, we get more test coverage. Having a
cmake mechanism to enable both python2 and python3 testing could be a
further enhancement.

* tools/memleak: add an explicit stdout.flush to print loop

The stdout flush behavior seems to have changed in python3, breaking one
of the tests. I think it makes sense to flush stdout at the end of each
timed interval loop anyway, so adding that to the tool itself.

* tests: add b'' strings and fix dangling handles

Add b'' strings in a few places in the test tools, and fix one dangling
process handle in the memleak test tool runner.

* Fix multiple memory access errors

Fixes a buffer overflow in get_pid_exe(), a use-after-free error in
bcc_usdt_get_probe_argctype() and a possible NULL pointer dereference
in find_debug_via_debuglink().

* Fix multiple ressource leaks

Leaked file descriptors in bpf_attach_uprobe() and verify_checksum().
Memory leaks in  Parser::func_add() and bcc_procutils_language().

* fixup! Fix multiple ressource leaks

* Documentation: Added table for program type and its helper functions

* updated the Program type table to follow the same Markdown format as other tables

* Fixed typos

* fixed typos and added new helper functions

Hardware events such as CACHE_MISSES and CACHE_REFERENCES are usually
not available on virtual machine. Print a more useful message when
this happen.

* sslsniff: add NSS support

* sslsniff: update documentation

Fix issue #1910

Otherwise, we will have a type mismatch like below:
  [root@aborniakFC tools]# ./ttysnoop 1
  Traceback (most recent call last):
  File "./ttysnoop", line 102, in <module>
  b = BPF(text=bpf_text)
  File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 337, in __init__
  self._trace_autoload()
  File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 1030, in _trace_autoload
  event=self.fix_syscall_fnname(func_name[8:]),
  File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 569, in fix_syscall_fnname
  if name.startswith(prefix):
  TypeError: startswith first arg must be bytes or a tuple of bytes, not str
  [root@aborniakFC tools]#
Signed-off-by: Yonghong Song <yhs@fb.com>

cmake3 was throwing an error that the package did not exist. cmake exists and the build was successful using it.

* Replace boilerplate for increment with a call to `increment`

from new tooling. Found cases to replace using `ripgrep`[1]:
```
$ rg '\(\*\w+\)\s*\+\+' -l | grep tools | grep -v old
```

[1]: https://github.com/BurntSushi/ripgrep

* Replace boilerplate for bigger than 1 increments with the new
`increment` call

from new tooling. Found cases to replace using `ripgrep`[1]:
```
$ rg '\(\*\w+\)\s*\+=' -l | grep tools | grep -v old
```

[1]: https://github.com/BurntSushi/ripgrep

* Update examples indicating the alternative increment call for hash tables

Add instructions for installing on Amazon Linux from source

* [trace.py]: allow to use STRCMP helper with binary values

Summary:
sometimes in probe you want to compare char* w/ some predefined value
which is not a string. e.g. setsockopt syscall has signature like this:
sys_setsockopt(int fd, int level, int optname, char* optval, int optlen)
and if you want to catch where/who is setting up specific value you are
forced to compare optval against some predefined array. it's not
possible today w/ trace.py and in this diff i'm adding such ability

Test Plan:
as example: we want to catch setsockopt when someone is setting up
IP_TOS equal to 108
trace.py 'sys_setsockopt(int fd, int level, int optname, char* optval,
int optlen)(level==0 && optname == 1 && STRCMP("{0x6C,0x00, 0x00,
0x00}", optval))' -U -M 1 --bin_cmp -v

without this new modifier:
static inline bool streq_0(char const *ignored, uintptr_t str) {
        char needle[] = "{0x6C,0x00, 0x00, 0x00}";
        char haystack[sizeof(needle)];
        bpf_probe_read(&haystack, sizeof(haystack), (void *)str);
        for (int i = 0; i < sizeof(needle) - 1; ++i) {
                if (needle[i] != haystack[i]) {
                        return false;
                }
        }
        return true;
}

// see needle is qouted above

with:

tatic inline bool streq_0(char const *ignored, uintptr_t str) {
        char needle[] = {0x6C,0x00, 0x00, 0x00};
        char haystack[sizeof(needle)];
        bpf_probe_read(&haystack, sizeof(haystack), (void *)str);
        for (int i = 0; i < sizeof(needle) - 1; ++i) {
                if (needle[i] != haystack[i]) {
                        return false;
                }
        }
        return true;
}

...
PID     TID     COMM            FUNC             -
1855611 1863183 worker          sys_setsockopt   found

* adding example of --bin_cmp flag usage

* Allow arbitrary hashtable increments. Fixes #1742

Right now incrementing some datastructure's values like maps or histograms can
be done with some boilerplate[1] or with `increment` which increments a value
by 1.

This patch allows a second optional parameter to use as the increment.

- [1]:
```
u64 zero = 0, *val;
val = map.lookup_or_init(&key, &zero);
(*val) += inc;
```

Notes:
- Some lines in the documentation where changed because of trailing spaces
deletion
- The test is quite simple right now
- Will update the tools to use `increment` in another PR

* CR changes