Misc fixes on Python tools

syscall.py: Fixes python3 related error

With out the patch:
Traceback (most recent call last):$
  File "/root/bcc/src/python/bcc/syscall.py", line 381, in <module>$
    out = out.split('\n',1)[1]$
TypeError: a bytes-like object is required, not 'str'$
$
During handling of the above exception, another exception occurred:$
$
Traceback (most recent call last):$
  File "/root/bcc/tests/python/test_stat1.py", line 10, in <module>$
    from bcc import BPF$
  File "/root/bcc/src/python/bcc/__init__.py", line 30, in <module>$
    from .syscall import syscall_name$
  File "/root/bcc/src/python/bcc/syscall.py", line 387, in <module>$
    raise Exception("ausyscall: command not found")$
Exception: ausyscall: command not found$

This is because variable 'out' is a byte object type and while split,
code is passing 'str' type.

Tested this on python3 and python2
Signed-off-by: Nageswara R Sastry <rnsastry@linux.vnet.ibm.com>

Handle keyboard interrupt to avoid backtrace printed out when user
presses Ctrl-C. Other tools have been fixed recently. Let's fix these
remaining tools.

Fix issue #2119.

Kernel 5.0 added more usages of asm goto and llvm
does not support asm goto yet. This resulted in
compilation error for virtually any bcc scripts.

The workaround here is to redefine asm_volatile_goto to
  asm volatile("invalid use of asm_volatile_goto")
which can pass clang.
If bpf program does not use asm_volatile_goto,
nothing bad will happen. The functions using
asm_volatile_goto will be thrown away.
If bpf program accidentally uses asm_volatile_goto,
a compilation error like below will be printed out:
  <inline asm>:1:2: error: invalid register/token name
          invalid use of asm_volatile_goto
          ^
  LLVM ERROR: Error parsing inline asm
Signed-off-by: Yonghong Song <yhs@fb.com>

add new Ubuntu bpfcc-tools package name in INSTALL.md

add missing PT_REGS_RET definition for x86

Fix BPFModule error when RW Engine is disabled 

fix several printb usage with python3

The following three tools are recently changed to use
printb in order to flush out the result.
  opensnoop.py, tcpaccept.py, tcpconnect.py

With python3, however, these tools have errors like below:
  TypeError: %b requires a bytes-like object, or an object that implements __bytes__, not 'str'
  Traceback (most recent call last):
    File "_ctypes/callbacks.c", line 234, in 'calling callback function'
    File "/usr/lib/python3.6/site-packages/bcc/table.py", line 572, in raw_cb_
      callback(cpu, data, size)
    File "../../tools/opensnoop.py", line 248, in print_event
      printb(b'%s' % event.fname.decode('utf-8', 'replace'))

This patch fixed printb related issues for these three tools
for python3. The python2 still works with the fix.
Signed-off-by: Yonghong Song <yhs@fb.com>

The bpf program generated with memset() call in this tool contains
BPF_JMP|BPF_CALL instruction with imm=-1. The verifier rejects the
program. Let's use llvm built-in function __builtin_memset

On fc29, compiling bcc from source, the following
compiler warnings are seen:
  /home/yhs/work/bcc/src/cc/bcc_elf.c: In function ‘bcc_free_memory_with_file’:
  /home/yhs/work/bcc/src/cc/bcc_elf.c:802:36: warning: unused variable ‘_fini’ [-Wunused-variable]
       extern unsigned long _start, _fini;
                                    ^~~~~
/home/yhs/work/bcc/src/cc/bcc_elf.c:802:28: warning: unused variable ‘_start’ [-Wunused-variable]
       extern unsigned long _start, _fini;
                            ^~~~~~

These unused externs are accidentally introduced by
Commit 51480d05 ("implement free_bcc_memory() API").
This patch removed them.
Signed-off-by: Yonghong Song <yhs@fb.com>

* python: remove MANIFEST

* cmake/python: fix build race condition

When ARCH is not defined, bcc get the architecture from uname. It then
modifies it to get the name of arch directory in linux source.

When ARCH is defined however, it just copy it as is, without the
translation to the arch directory. If for instance ARCH is set to
x86_64, it tries to look into the include directory
build/arch/x86_64/, which doesn't exist.

It fixes the following issue:
$ echo $ARCH
x86_64
$ /usr/share/bcc/tools/bashreadline
In file included from <built-in>:2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from /lib/modules/4.18.0-49.el8.x86_64/build/include/linux/types.h:6:
/lib/modules/4.18.0-49.el8.x86_64/build/include/uapi/linux/types.h:5:10: fatal error: 'asm/types.h' file
      not found
         ^~~~~~~~~~~~~
1 error generated.
Traceback (most recent call last):
  File "/usr/share/bcc/tools/bashreadline", line 51, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 318, in __init__
    raise Exception("Failed to compile BPF text")
Exception: Failed to compile BPF text

support uid option in tcpconnect.

bpf_trace_printk() does not have `fmt_size` as the second parameter in BCC.

add missing FILTER in the bpf program.

opensnoop: supported UID options

* inject: Add support for alloc_page family of functions

* Add vim backup files to gitignore

* add stdout flushing to some tools

* change printb import source in opensnoop tool

The biolatency tool attaches kprobe at blk_account_io_completion() function
to check when a request is finished. But sometimes request can be finished
partially and blk_account_io_completion() is called again.
The blk_account_io_done() is called when a request is finally
finished and biolatency tool should attach blk_account_io_done(), not
blk_account_io_completion().
Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>

* fixed shebangs in tools (and lib)

* fixed shebangs in examples

* do not mangle shebangs in rpm spec

* renamed style-check.sh to c-style-check.sh

* factored out python linter to a separate file

* added shebang validation to the py-style-check

* added shebangs to all python executables

Comment on duplicate code for parsing double vlans.
I also changed it into a short loop, which I don't think improves (or degrades) the code, but at least makes it clear that the double parsing is intentional.
Signed-off-by: Paul Chaignon <paul.chaignon@orange.com>

Fix typos in hint message 
Signed-off-by: Paul Chaignon <paul.chaignon@orange.com>

align the header in opensnoop in the tutorial.

added basic race condition protection for ustat.py

fix issue #226

The unknown opcode typically happens if the bpf
program has an external reference which does not
get resolved. Note bcc does not even preform
relocations for maps as map_id is directly
used in bpf problem through bpf_pseudo_fd()
intrinsic.

Instead of the error:
  bpf: Failed to load program: Invalid argument
  unknown opcode 00

A little explanation is added like the below:
  HINT: The 'unknown opcode' can happen if you referencea global
  or static variable, or data in read only section.
  For example,'char *p = "hello"' will result in p referencing a
  read only section,and 'char p[] = "hello"' will have "hello"
  stored on the stack.
Signed-off-by: Yonghong Song <yhs@fb.com>

The header file will be in sync with the 4.21 rc1.
Signed-off-by: Yonghong Song <yhs@fb.com>

* opensnoop: print flags, enable filtering

* Add docs, extended_fields option; filter flags in-kernel

* Homogenize documentation

* Add FLAGS to the FIELDS man page section

The main purpose of this API is to proactively release llvm/clang
.text memory which is brought in during compilation.
bcc .text memory for some other functions, e.g., attach_tracepoint,
bpf_prog_load, etc. can also be freed after all these tasks are done.

Note that such memory is reclaimable in kernel since it has
file backup. But certain applicaiton may want to reduce this
memory immediately to satisfy constraints imposed by sysadmin, etc.

The implementation uses madvise with MADV_DONTNEED.
For the case where bcc is static linked into the binary,
we do not really know the start and the end of memory regions
used by bcc, so the implementation here bluntly returned
all .text memory back to kernel. This will incur some performance
overhead as later on executed instructions will need to bring
back to memory again.

For static linked library, instrumented RandomRead example,
without this patch, the RSS memory before load is:
  VmRSS:     63644 kB
  RssAnon:           23876 kB
  RssFile:           39768 kB
  RssShmem:              0 kB

After this patch,
  VmRSS:     34264 kB
  RssAnon:           23880 kB
  RssFile:           10384 kB
  RssShmem:              0 kB

For shared library, a python unit test, test_free_llvm_memory.py, is
added, which shows for a do-nothing bpf program, we have
  Before freeing llvm memory: RssFile:  43000 kB
  After  freeing llvm memory: RssFile:  11992 kB

The RssFile reduction on Facebook internal applications
also ranges in 30-40MB.
Signed-off-by: Yonghong Song <yhs@fb.com>

Compiler shows warning "incompatible integer to pointer conversion
initializing" while compiling bpf program.
This patch adds necessary typecast when assigning PT_REGS_PARAM vaules
to struct pt_regs pointer

fix typos, link reference and package versions for rhel 

* Make dependency on LLVM native target optional

Adds an option ENABLE_LLVM_NATIVECODEGEN with default value ON.
If set to off the "nativecodegen" llvm will not be enabled, thus
reducing dependencies on needed libraries (reduced text size when
building with statically linked libraries).

Code that uses native target will not be compiled reducing text size.
Currently this affects the rw_engine which needs the native target.

BPF api "rw_engine_enabled" will have default value "true" if
ENABLE_LLVM_NATIVECODEGEN="ON" and "false" if
ENABLE_LLVM_NATIVECODEGEN="OFF"

Not needed for BCC to work. It somehow brought in the interpreter and
executionengine which is needed. Those features are added instead.

* Remove garbage in code making it compile again

* Remove interpreter and executionengine LLVM dependencies

These doesn't seem to be needed on a Ubuntu 18.04 system (although
executionengine is heavily used).

Interpreter was added due to runtime dependency on ARM64. It brings in
a dependency on ffi library.

(.text._ZL10ffiTypeForPN4llvm4TypeE+0x3a): undefined reference to `ffi_type_float'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x43): undefined reference to `ffi_type_void'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x53): undefined reference to `ffi_type_pointer'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x63): undefined reference to `ffi_type_double'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x78): undefined reference to `ffi_type_sint8'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x83): undefined reference to `ffi_type_sint16'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x93): undefined reference to `ffi_type_sint64'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0xb3): undefined reference to `ffi_type_sint32'
/usr/lib/llvm-6.0/lib/libLLVMInterpreter.a

fix exception message format in detach_kprobe_event() 

Many tools rely on the user to type Ctrl-C to end, but don't actually
catch the keyboard interrupt and thus show an ugly backtrace when it
happens. Let's catch the interrupt.

dbslower: fix a python3 bytes/string issue int the -x option

In python3, the find method requires a bytes-like object. It fixes the
following error:

$ dbslower mysql -x $(which mysqld)
Traceback (most recent call last):
  File "/usr/share/bcc/tools/dbslower", line 72, in <module>
    if mysql_func_name.find("COM_DATA") >= 0:
TypeError: a bytes-like object is required, not 'str'

Also the -x option is currently undocumented in the man page and the
example file. So let's ix that too.