* tools: the argument of get_kprobe_functions() should be a bytes object

It fixes the following error:
Traceback (most recent call last):
  File "/usr/share/bcc/tools/ext4dist", line 189, in <module>
    if BPF.get_kprobe_functions('ext4_file_read_iter'):
  File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 519, in get_kprobe_functions
    if (t.lower() in [b't', b'w']) and re.match(event_re, fn) \
  File "/usr/lib64/python3.6/re.py", line 172, in match
    return _compile(pattern, flags).match(string)
TypeError: cannot use a string pattern on a bytes-like object

* tools: fix freeze in offwaketime

Since commit 47cecb65, the sigint signal is ignored
unconditionally which prevents offwaketime to be stopped by
pressing Ctrl-C when run without a specified duration.

Fix issue #2002
Signed-off-by: Yonghong Song <yhs@fb.com>

Sync compat bpf.h and virtual_bpf.h with latest net-next versions

Also add a barrier for the test_brb bpf program like below
         meta.prog_id = skb->cb[0];
+        asm volatile("" ::: "memory");
         meta.rx_port_id = skb->cb[1];
so two cb read does not become a 8 byte load which is not
allowed for ubuntu 16.04 kernel (4.4 based). The 8 byte load
is allowed for skb->cb's for later kernels.
Signed-off-by: Yonghong Song <yhs@fb.com>

* Add perl* utools man page links

Matches other languages, related to commit 9162be45.

* uthreads: explicitly list C as supported

uthreads supports C (pthreads) thread tracing, make this explicit.

Earlier use of -l java and -l none unchanged.

Add the corresponding man page and usage examples.

* Add Tcl support for ucalls / uflow / uobjnew / ustat

For some reason we need to use proc__args instead of proc__entry
in uflow.py to get similar results as with e.g. Python.

* tools: uses 'replace' error handler by default in decode()

Tools might encouter characters from non utf-8 charset (e.g. a file
name). When this happen, it's better to replace the unexpected
character by a question mark than crash the tool when all we do is
to print the string.

* tools: fix a bytes/string issue in attach_perf_event()

Fixed Context::enable_probe method so that it does not discard the result of Probe::enable.

add bpf function call support link in docs.
Signed-off-by: Alban Crequy <alban@kinvolk.io>

kbuild_helper.h: Add required #include ftw.h

uflow: convert bytes to str

Python 3 fix, similar to commit 99d14683 and commit 4e4c9e01 for ucalls.

Closes #1996.

* lua: eliding repetitive NULL checks, fixes for read types, luacheck

This fixes verifier failures in some kernel versions that track whether
a possible NULL pointer has been check (and shouldn't be checked twice).

It also fixes accessing memory from maps with composite keys.

It fixes a case when temporary variable is used in condition,
and the body of the condition contains just an assignment from
immediate value to a variable that existed before the condition, e.g.:

```
local x = 1 -- Assign constant to 'x', it will be materialized here
if skb.len > 0 then -- New BB, not possible to fold as condition isn't const
    x = 2 -- Assign constant, it failed to materialize here
end -- End of BB
-- Value of x is always 1
```

The `bpf.socket()` support ljsyscall socket as first parameter, and
fixes compatibility with newer ljsyscall (type name has changed).

* reverse BPF_LD ntohl() semantics for <= 32bit loads as well

The loads using traditional instructions always do ntohl():

https://www.kernel.org/doc/Documentation/networking/filter.txt

They are however needed to support reads not using `skb`, e.g. NET_OFF

* proto: add builtins for traversing IPv6 header and extension headers

The IPv6 header is fixed size (40B), but it may contain either
extension header, or transport  protocol after it, so the caller
must check the `next_header` field before traversing to determine
which dissector to use to traverse it, e.g.:

```
if ip6.next_header == 44 then
   ip6 = ip6.ip6_opt -- Skip fragment header
end
if ip6.next_header == c.ip.proto_tcp then
   local tcp = ip6.tcp -- Finally, TCP
end
```

* reverse ntohl() for indirect BPF_LD as well

* lua: started codegen tests, direct skb support, bugfixes

This starts adding compiler correctness tests for basic expressions,
variable source tracking, and control flow.

Added:
* direct skb->data access (in addition to BPF_LDABS, BPF_LDIND)
* loads and stores from skb->data and map value backed memory
* unified variable source tracking (ptr_to_ctx, ptr_to_map[_or_null], ptr_to_skb, ptr_to_pkt, ptr_to_stack)
* BPF constants introduced between 4.10-4.15
* bpf.dump_string() to dump generated assembly (text) to string

Fixes:
* pointer nil check tracking
* dissectors for map value backed memory
* ljsyscall extensions when the version is too old
* KPRI nil variables used in conditions
* wrongly elided constant materialization on condition-less jumps
* loads/stores from stack memory using variable offset

* lua: track inverse null guards (if x ~= nil)

The verifier prohibits pointer comparisons except the first NULL check,
so both forms must be tracked, otherwise it will check for NULL twice
and verifier will reject it.

* lua: support cdata and constants larger than i32, fix shadowed variables

This adds support for numeric cdata constants (either in program, or
retrieved with GGET/UGET). Values larger than i32 are coerced into i64.

This also fixes shadowing of variables, and fixes materialization of
result of variable copy at the end of basic block.

Update INSTALL.md to include Ubuntu Bionic (18.04) 

Currently tables can be created in two modes:
1. private to the ebpf program
2. shared among all the ebpf programs created by the same user-space
process

This commit extends bcc allowing to create a table that is shared among
a set of specific ebpf programs.
This is useful when creating network functions that are composed
by more than 1 ebpf program, a map is shared among all the programs of
the network function but isolated from different instances of that
function.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>

After https://github.com/iovisor/bcc/pull/1826 version.py is generated
by cmake in order to have the __version__ define in the module. The
previous approach of installing the python module created a link between the
src and the "build" folder, hence the version.py was actually created in the
scr folder, this caused some git noise.

This comit solves that problem by copying files instead of creating a link,
then the version.py file is only created in the "build" dir.
Signed-off-by: Mauricio Vasquez B <mauriciovasquezbernal@gmail.com>

Use argparse in cachestat, add --ebpf support.

Add --ebpf support for u* tools, finalize language sorting.

Remove sole --ebpf string on usage line in tcpsubnet.

Add #include to libbpf.h for base types.

Note this allows us to eliminate the `bpf_ktime_get_ns()`
call from `trace_entry()`.

prevent array subscript expression if base/index is not rewritable

The following command failed:
  trace.py -U 'r::_do_fork (retval == -11) "%llu", ((struct task_struct *)bpf_get_current_task())->signal->rlim[RLIMIT_NPROC].rlim_cur'

as rewriter generates code like
  __data.v0 = (unsigned long long)((struct task_struct *)bpf_get_current_task())->signal->rlim[RLIMIT_NPROC))); _val; }).rlim_cur;

Let us prevent rewriting if either base or index is not rewritable and this fixed the issue.
Signed-off-by: Yonghong Song <yhs@fb.com>

get_offline_table is missing in the BPFTable class.

Add it and some test cases.
Signed-off-by: Mauricio Vasquez B <mauricio.vasquez@polito.it>

Adds the support for the DEVMAP in the C++ APIs, which was missing in #1810

bpf_probe_read()'s third argument is no longer rewritten. Instead, use a
temporary variable (like #1973) to avoid a memory access error.

Before:

```
$ sudo biolatency -D 1
bpf: Failed to load program: Permission denied
0: (79) r1 = *(u64 *)(r1 +112)
1: (7b) *(u64 *)(r10 -8) = r1
[..]
R1 invalid mem access 'inv'

HINT: The invalid mem access 'inv' error can happen if you try to
dereference memory without first using bpf_probe_read() to copy it to
the BPF stack. Sometimes the bpf_probe_read is automatic by the bcc
rewriter, other times you'll need to be explicit.
```

After, works as expected.

Currently, trace.py does not support "long" and "unsigned long"
types and it often caught users with a surprise and they are
not sure what is the problem. For example, for kernel function:
  void blk_mq_delay_kick_requeue_list(struct request_queue *q, unsigned long msecs)
The following
  $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %lu", msecs'
  list index out of range

With this patch,
  $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %lu", msecs'
  PID     TID     COMM            FUNC             -
  ^C
  $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %ld", msecs'
  PID     TID     COMM            FUNC             -
  ^C
  $ sudo ./trace.py 'blk_mq_delay_kick_requeue_list(void *notused, unsigned long msecs) "msecs = %lx", msecs'
  PID     TID     COMM            FUNC             -
  ^C
  $
Signed-off-by: Yonghong Song <yhs@fb.com>

In llvm/clang 8 trunk, the following commits
  https://reviews.llvm.org/rL339384
  https://reviews.llvm.org/rL339385
  https://reviews.llvm.org/rL339386
renames getLocStart to getBeginLoc, and getLocEnd to getEndLoc.
This caused bcc compilation error with llvm/clang 8.
To avoid proliferation of #define's based on LLVM version numbers
and similar logic for many different types,
this patch fixed the issues by define macros in common
frontend_action_common.h where macro definition is multi-versioned
based on llvm version. The macro itself is used in
{b,tp}_frontend_action.cc.

clang 8 also introduced a new library dependency
  libclangAnalysis.a depends on libclangASTMatchers.a
This patch fixed this issue as well.
Signed-off-by: Yonghong Song <yhs@fb.com>

Currently, trace.py failed for the following command:
  $ sudo ./trace.py 'filename_lookup(int dfd, struct filename *name) "%s", name->name'
  ...
  0: (bf) r6 = r1
  1: (79) r7 = *(u64 *)(r6 +104)
  ...
  32: (15) if r1 == 0x0 goto pc+5
  R0=inv(id=0) R1=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0)
  R8=inv0 R10=fp0,call_-1 fp-8=0 fp-16=0 fp-24=0 fp-32=0 fp-40=0 fp-48=0 fp-56=0 fp-64=0 fp-72=0 fp-80=0
  33: (79) r3 = *(u64 *)(r7 +0)
  R7 invalid mem access 'inv'

For string format argument, the trace.py generates the below code:
        if (name->name != 0) {
                bpf_probe_read(&__data.v0, sizeof(__data.v0), (void *)name->name);
        }
Right now, bcc skips the rewriter for the third argument of bpf_probe_read to avoid
unnecessary nested bpf_probe_read and other potential issues.
This causes name->name memory access not transformed with bpf_probe_read and hence
the verifier complains.

To fix the issue, this patch did the following transformation using an
temporary variable to hold the src address:
        if (name->name != 0) {
                void *__tmp = (void *)name->name;
                bpf_probe_read(&__data.v0, sizeof(__data.v0), __tmp);
        }
This way, rewriter can do the work properly.
Signed-off-by: Yonghong Song <yhs@fb.com>

Fix syntax error in xdp_redirect_cpu.py

* Bump ubuntu docker build to bionic

Fixes build dependency for arm64
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>

* Add Build-Depends: dh-python

Fixes build dependency issue observed on bionic/arm64:

dh: unable to load addon python3: Can't locate Debian/Debhelper/Sequence/python3.pm in @INC (you may need to install the Debian::Debhelper::Sequence::python3 module) (@INC contains: /etc/perl /usr/local/lib/aarch64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/aarch64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/aarch64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/aarch64-linux-gnu/perl-base) at (eval 9) line 1.
BEGIN failed--compilation aborted at (eval 9) line 1.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>

* Enable arm64 deb packaging

LLVM6.0 is minimum version required for arm64 build. [PR#1512]

[PR#1512] https://github.com/iovisor/bcc/pull/1512Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>

Finish the conversion started with commit 36ce1124.

Minor tweaks to make the tools consistent within themselves.

tcpretrans: use user-space PID displayed to the user

PID collected in tcpretrans' trace_event() is displayed under the
label of "PID" to the user so it would seem more appropriate to use
TGID as PID not kernel's PID.

* Small fix on C++ USDT implementation

Fix an logging error, and small optimization on initialization

* Add move constructor to C++ USDT instance

debian changelog for v0.7.0 tag

Sort language entries while at it.

* tools: Skip backward time entries in xfsslower

While using xfsslower on RHEL7 we occasionally get
following screwed up latencies:

  # xfsslower
  Tracing XFS operations slower than 1 ms
  TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
  13:25:03 git            3385   R 62      4704     18446744073708.55 tmp_pack_bDUbwZ
  13:25:03 git            3385   S 0       0           3.05 tmp_idx_Kjb2bW
  ...

The reason for this is that on RHEL7 it's possible to get backward
timetamp with bpf_ktime_get_ns. This needs to be fixed, but meanwhile
this fix makes sure the latencies with backward times are skipped.

For the rest of the kernels this is just sanity fix with
possibly just single compare instruction overhead.

It's temporary workaround for #728.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>

* tools: Skip backward time entries in ext4dist

While using ext4dist on RHEL7 we occasionally get
following screwed up latencies:

  # ext4dist
  Tracing ext4 operation latency... Hit Ctrl-C to end.
  ^C

  operation = write
               usecs                         : count     distribution
                   0 -> 1                    : 1134529   |********            |
                   2 -> 3                    : 2777582   |********************|
                   4 -> 7                    : 688014    |****                |
                   8 -> 15                   : 36160     |                    |
                  16 -> 31                   : 698       |                    |
                  32 -> 63                   : 6         |                    |
                  64 -> 127                  : 15        |                    |
                 128 -> 255                  : 7         |                    |
                 256 -> 511                  : 1         |                    |
                 512 -> 1023                 : 0         |                    |
                1024 -> 2047                 : 0         |                    |
                2048 -> 4095                 : 2         |                    |
                4096 -> 8191                 : 1         |                    |
                8192 -> 16383                : 5         |                    |
               16384 -> 32767                : 0         |                    |
               32768 -> 65535                : 0         |                    |

    9007199254740992 -> 18014398509481983    : 0         |                    |
   18014398509481984 -> 36028797018963967    : 1         |                    |

The reason for this is that on RHEL7 it's possible to get backward
timestamp with bpf_ktime_get_ns. This needs to be fixed, but meanwhile
this fix makes sure the latencies with backward times are skipped.

For the rest of the kernels this is just sanity fix with
possibly just single compare instruction overhead.

It's temporary workaround for #728.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>

fix a typo

Generating sdist for both python2 and python3 at the same time can fail
as they both use the same temporary files.

Make sure that we generate sdist for each `PYTHON_CMD` sequentially.

Added in RHEL 7.6 Beta information