* add stdout flushing to some tools

* change printb import source in opensnoop tool

The biolatency tool attaches kprobe at blk_account_io_completion() function
to check when a request is finished. But sometimes request can be finished
partially and blk_account_io_completion() is called again.
The blk_account_io_done() is called when a request is finally
finished and biolatency tool should attach blk_account_io_done(), not
blk_account_io_completion().
Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>

* fixed shebangs in tools (and lib)

* fixed shebangs in examples

* do not mangle shebangs in rpm spec

* renamed style-check.sh to c-style-check.sh

* factored out python linter to a separate file

* added shebang validation to the py-style-check

* added shebangs to all python executables

Comment on duplicate code for parsing double vlans.
I also changed it into a short loop, which I don't think improves (or degrades) the code, but at least makes it clear that the double parsing is intentional.
Signed-off-by: Paul Chaignon <paul.chaignon@orange.com>

Fix typos in hint message 
Signed-off-by: Paul Chaignon <paul.chaignon@orange.com>

align the header in opensnoop in the tutorial.

added basic race condition protection for ustat.py

fix issue #226

The unknown opcode typically happens if the bpf
program has an external reference which does not
get resolved. Note bcc does not even preform
relocations for maps as map_id is directly
used in bpf problem through bpf_pseudo_fd()
intrinsic.

Instead of the error:
  bpf: Failed to load program: Invalid argument
  unknown opcode 00

A little explanation is added like the below:
  HINT: The 'unknown opcode' can happen if you referencea global
  or static variable, or data in read only section.
  For example,'char *p = "hello"' will result in p referencing a
  read only section,and 'char p[] = "hello"' will have "hello"
  stored on the stack.
Signed-off-by: Yonghong Song <yhs@fb.com>

The header file will be in sync with the 4.21 rc1.
Signed-off-by: Yonghong Song <yhs@fb.com>

* opensnoop: print flags, enable filtering

* Add docs, extended_fields option; filter flags in-kernel

* Homogenize documentation

* Add FLAGS to the FIELDS man page section

The main purpose of this API is to proactively release llvm/clang
.text memory which is brought in during compilation.
bcc .text memory for some other functions, e.g., attach_tracepoint,
bpf_prog_load, etc. can also be freed after all these tasks are done.

Note that such memory is reclaimable in kernel since it has
file backup. But certain applicaiton may want to reduce this
memory immediately to satisfy constraints imposed by sysadmin, etc.

The implementation uses madvise with MADV_DONTNEED.
For the case where bcc is static linked into the binary,
we do not really know the start and the end of memory regions
used by bcc, so the implementation here bluntly returned
all .text memory back to kernel. This will incur some performance
overhead as later on executed instructions will need to bring
back to memory again.

For static linked library, instrumented RandomRead example,
without this patch, the RSS memory before load is:
  VmRSS:     63644 kB
  RssAnon:           23876 kB
  RssFile:           39768 kB
  RssShmem:              0 kB

After this patch,
  VmRSS:     34264 kB
  RssAnon:           23880 kB
  RssFile:           10384 kB
  RssShmem:              0 kB

For shared library, a python unit test, test_free_llvm_memory.py, is
added, which shows for a do-nothing bpf program, we have
  Before freeing llvm memory: RssFile:  43000 kB
  After  freeing llvm memory: RssFile:  11992 kB

The RssFile reduction on Facebook internal applications
also ranges in 30-40MB.
Signed-off-by: Yonghong Song <yhs@fb.com>

Compiler shows warning "incompatible integer to pointer conversion
initializing" while compiling bpf program.
This patch adds necessary typecast when assigning PT_REGS_PARAM vaules
to struct pt_regs pointer

fix typos, link reference and package versions for rhel 

* Make dependency on LLVM native target optional

Adds an option ENABLE_LLVM_NATIVECODEGEN with default value ON.
If set to off the "nativecodegen" llvm will not be enabled, thus
reducing dependencies on needed libraries (reduced text size when
building with statically linked libraries).

Code that uses native target will not be compiled reducing text size.
Currently this affects the rw_engine which needs the native target.

BPF api "rw_engine_enabled" will have default value "true" if
ENABLE_LLVM_NATIVECODEGEN="ON" and "false" if
ENABLE_LLVM_NATIVECODEGEN="OFF"

Not needed for BCC to work. It somehow brought in the interpreter and
executionengine which is needed. Those features are added instead.

* Remove garbage in code making it compile again

* Remove interpreter and executionengine LLVM dependencies

These doesn't seem to be needed on a Ubuntu 18.04 system (although
executionengine is heavily used).

Interpreter was added due to runtime dependency on ARM64. It brings in
a dependency on ffi library.

(.text._ZL10ffiTypeForPN4llvm4TypeE+0x3a): undefined reference to `ffi_type_float'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x43): undefined reference to `ffi_type_void'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x53): undefined reference to `ffi_type_pointer'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x63): undefined reference to `ffi_type_double'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x78): undefined reference to `ffi_type_sint8'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x83): undefined reference to `ffi_type_sint16'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0x93): undefined reference to `ffi_type_sint64'
(.text._ZL10ffiTypeForPN4llvm4TypeE+0xb3): undefined reference to `ffi_type_sint32'
/usr/lib/llvm-6.0/lib/libLLVMInterpreter.a

fix exception message format in detach_kprobe_event() 

Many tools rely on the user to type Ctrl-C to end, but don't actually
catch the keyboard interrupt and thus show an ugly backtrace when it
happens. Let's catch the interrupt.

dbslower: fix a python3 bytes/string issue int the -x option

In python3, the find method requires a bytes-like object. It fixes the
following error:

$ dbslower mysql -x $(which mysqld)
Traceback (most recent call last):
  File "/usr/share/bcc/tools/dbslower", line 72, in <module>
    if mysql_func_name.find("COM_DATA") >= 0:
TypeError: a bytes-like object is required, not 'str'

Also the -x option is currently undocumented in the man page and the
example file. So let's ix that too.

modified:   __init__.py

* get_kprobe_functions(): detect more non-attachable functions

Improve get_kprobe_functions() to exclude more non-attachable functions
like IRQ routines and functions in kernel modules blacklisted via
NOKPROBE_SYMBOLS().
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

* get_kprobe_functions: fix bytes/str type error
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

manpage: remove non-existent -p option from the biotop manpage

The biotop manpage references a -p option that the tools doesn't have,
and AFAICT, never had. It's only referenced in the manpage option, not
in the synopsis, in "biotop -h" output not biotop_example.txt.

Enables linking bcc-static in binaries compiled with rtti

When ausyscall is used as a fallback plan to generate the
syscall map, the subprocess.check_output() call fails as
the program name and arguments must be passed separately.

Fixes: 218f7482 ("Wcohen/efficiency (#2063)")
Signed-off-by: Sandipan Das <sandipan@linux.ibm.com>

Recent -next kernels don't have blk_start_request() function
anymore. It has been removed in a recent cleanup. bio* tools should be
able to handle the lack of this probe.

* Reduce instrumentation overhead with the sys_enter and sys_exit tracepoints

The ucalls script initially used kprobes and kretprobes on each of the
hundreds of syscalls functions in the system.  This approach causes a
large number of probes to be set up at the start and removed at the
conclusion of the script's execution resulting in slow start up.

Like the syscount.py script the ucall syscall instrumentation has been
modified to use the sys_enter and sys_exit tracepoints.  This only
requires the installation and removal of one or two tracepoints to
implement and results in much shorter times to start and stop the
ucalls script.

Another benefit of this change is syscalls on newer kernels will be
monitored with the "-S" option.  The regular expression used to find
the locations for the kprobes and kretprobes for all the possible
syscall functions would not would match the syscall function naming
convention in newer kernels.

* Update ucalls_examples.txt to match current "-S" option output

* Add required "import subprocess" and remove unneeded "global syscalls"

* Factor out the syscall_name code into a separate python module syscall.py

Multiple scripts are going to find the syscall_name() function useful
when using the syscall tracepoints.  Factoring out this code into a
separate python module avoids having to replicate this code in
multiple scripts.

* Use the syscall_name() function in syscount.py to make it more compact.

* Update the default syscall mappings and the way that they were generated

The default table was missing some newer syscall mapping. Regenerated
the table using the syscallent.h file from Fedora 30
strace-4.25-1.fc30.src.rpm.  Also updated the comment with the command
actually used to generate the mappings.

* Add license information and upsdate the syscalls

The default x86_64 syscall dictionary mapping syscalls numbers to
names has been updated. The following syscall x86_64 names have been
updated:

    18: b"pwrite64",
    60: b"exit",
    166: b"umount2",

The following syscall x86_64 have been added:

    313: b"finit_module",
    314: b"sched_setattr",
    315: b"sched_getattr",
    316: b"renameat2",
    317: b"seccomp",
    318: b"getrandom",
    319: b"memfd_create",
    320: b"kexec_file_load",
    321: b"bpf",
    322: b"execveat",
    323: b"userfaultfd",
    324: b"membarrier",
    325: b"mlock2",
    326: b"copy_file_range",
    327: b"preadv2",
    328: b"pwritev2",
    329: b"pkey_mprotect",
    330: b"pkey_alloc",
    331: b"pkey_free",
    332: b"statx",
    333: b"io_pgetevents",
    334: b"rseq",

* Eliminate stderr output and use of shell features

Redirect all stderr output so it isn't seen.  Also avoid use of the
shell pipe and tail command.  Just strip off the first line in the
python code instead.

* Update lib/ucalls.py smoke test to required linux-4.7

The use of tracepoints in the ucalls.py requires linux-4.7. Changed
the test to only run with a suitable kernel.  The libs/ucalls.py
script is no longer inserting hundreds of kprobes and is much faster
as a result, so removed the timeout adjustment and the comment about
it being slow.

This reverts commit 5b76047f.

The code is actually not duplicated. It is used to process
double vlan's. See comments in:
  https://github.com/iovisor/bcc/pull/1493

With latest llvm/clang, we have the following compilation errors:

    /home/yhs/work/bcc2/src/cc/frontends/clang/b_frontend_action.cc:
      In member function ‘bool ebpf::BTypeVisitor::VisitVarDecl(clang::VarDecl*)’:
    /home/yhs/work/bcc2/src/cc/frontends/clang/b_frontend_action.cc:1130:52:
      error: no matching function for call to ‘clang::Expr::EvaluateAsInt(llvm::APSInt&, clang::ASTContext&)’
               if (I->getInit(idx)->EvaluateAsInt(res, C)) {
                                                    ^
    /home/yhs/work/bcc2/src/cc/frontends/clang/b_frontend_action.cc:1130:52: note: candidate is:
    In file included from /home/yhs/work/llvm/build/install/include/clang/AST/Attr.h:19:0,
                 from /home/yhs/work/llvm/build/install/include/clang/AST/TypeLoc.h:18,
                 from /home/yhs/work/llvm/build/install/include/clang/AST/ASTTypeTraits.h:24,
                 from /home/yhs/work/llvm/build/install/include/clang/AST/ASTContext.h:18,
                 from /home/yhs/work/bcc2/src/cc/frontends/clang/b_frontend_action.cc:23:
    /home/yhs/work/llvm/build/install/include/clang/AST/Expr.h:604:8:
      note: bool clang::Expr::EvaluateAsInt(clang::Expr::EvalResult&, const clang::ASTContext&,
              clang::Expr::SideEffectsKind) const
       bool EvaluateAsInt(EvalResult &Result, const ASTContext &Ctx,
        ^

The error is introduced by the following commit:
https://reviews.llvm.org/rL348053

Basically, the clang FieldDecl method EvaluateAsInt signature got
changed and there is no compatible way in the llvm/clang
to also work in the old versions. So this patch just provided
a new implementation to get field value for llvm version 8 and later.
Signed-off-by: Yonghong Song <yhs@fb.com>

Mostly a cut and paste of the xdp_drop_count.py with needed changes for
doing a source and destination MAC swap.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>

There have been some updates the code in the examples.  The tutorial
text should match up with the current example code. In particular we
want to make ensure that the probe for the syscall functions match up
with what is currently needed for newer kernels that do not have
syscall functions uniformly start with "sys_".

* tcpstates: Add systemd journal logging.

Add a -Y/--journal flag to tcpstates.py, which logs events to the
systemd journal.

* tcpstates: Document systemd journal logging.

Update tcpstates_example.txt and tcpstates.8 to include the "-Y" flag.

auto select current release for ubuntu

Snapcraft has an improved mechanism for wrapping python executables.
Remove all the older snapcraft legacy cruft and update for the new
snapcraft tooling.
Signed-off-by: Colin Ian King <colin.king@canonical.com>

Otherwise, if llvm is built with assertion on,
we will hit the following error for a simple trace.py script like
  -bash-4.4$ sudo ./trace.py 'SyS_nanosleep'
  python: /home/yhs/work/llvm/lib/ExecutionEngine/MCJIT/MCJIT.cpp:204:
          virtual void llvm::MCJIT::generateCodeForModule(llvm::Module*):
          Assertion `M->getDataLayout() == getDataLayout() && "DataLayout Mismatch"'
          failed.
  Aborted (core dumped)
  -bash-4.4$
Signed-off-by: Yonghong Song <yhs@fb.com>

Sync compat/linux/{bpf.h, bpf_common.h, virtual_bpf.h}
with latest bpf-next. The change in bpf_common.h are comments only.
Signed-off-by: Yonghong Song <yhs@fb.com>

In folded output format when both kernel and user stacks are missing,
the message "[Missing User Stack]" is formatted incorrectly, for example:

chrome;entry_SYSCALL_64_after_hwframe;do_syscall_64;SyS_poll;do_sys_poll;poll_schedule_timeout;schedule_hrtimeout_range;schedule_hrtimeout_range_clock;schedule;--;[Missed Kernel Stack];[;M;i;s;s;e;d; ;U;s;e;r; ;S;t;a;c;k;];fio 4955989

This is because we're incorrectly appending the missing stack message in
the list of stack entries. Fix by appending the message as a single item.
Signed-off-by: Andrea Righi <righi.andrea@gmail.com>

Also, fix an incorrect indentation in the same block.

* Allow unused return value in cc source

With llvm-7.0.0,
some annoying warning messeges are raised:

	/home/lecopzer/workspace/bcc/src/cc/libbpf.c:456:3: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result [-Wunused-result]
   	fgets(fmt, sizeof(fmt), f); // pos
   	^~~~~~~~~~~~~~~~~~~~~~~~~~

/home/lecopzer/workspace/bcc/src/cc/libbpf.c: In function ‘bpf_prog_get_tag’:
/home/lecopzer/workspace/bcc/src/cc/libbpf.c:456:3: warning: ignoring return value of ‘fgets’, declared with attribute warn_unused_result [-Wunused-result]
   fgets(fmt, sizeof(fmt), f); // pos
   ^~~~~~~~~~~~~~~~~~~~~~~~~~
...

/home/lecopzer/workspace/bcc/tests/cc/utils.cc: In function ‘int cmd_scanf(const char*, const char*, ...)’:
/home/lecopzer/workspace/bcc/tests/cc/utils.cc:30:10: warning: ignoring return value of ‘int vfscanf(FILE*, const char*, __va_list_tag*)’, declared with attribute warn_unused_result [-Wunused-result]
   vfscanf(pipe, fmt, args);
   ~~~~~~~^~~~~~~~~~~~~~~~~

Let get rid of them by adding -Wno-unused-result.

* cc: Fix comparison between signed and unsigned value

With llvm-7.0.0:

/home/lecopzer/workspace/bcc/src/cc/common.cc: In function ‘std::__cxx11::string ebpf::get_pid_exe(pid_t)’:
/home/lecopzer/workspace/bcc/src/cc/common.cc:60:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (res >= sizeof(exe_path))
       ~~~~^~~~~~~~~~

As the declaration of `exe_path` is `char exe_path[4096]`,
the `sizeof(exe_path)` would always return 4096 (unsigned), so it's safe
to static cast to `int` unless it's larger than 2^31 - 1.

* tools: Add shmsnoop to spy on shm* syscalls

Adding shmsnoop tool to trace System V shared memory
syscalls: shmget, shmat, shmdt, shmctl

  # ./shmsnoop.py
  PID    COMM                SYS              RET ARGs
  19813  server           SHMGET            10000 key: 0x78020001, size: 20, shmflg: 0x3b6 (IPC_CREAT|0666)
  19813  server            SHMAT     7f1cf8b1f000 shmid: 0x10000, shmaddr: 0x0, shmflg: 0x0
  19816  client           SHMGET            10000 key: 0x78020001, size: 20, shmflg: 0x1b6 (0666)
  19816  client            SHMAT     7f4fd8ee7000 shmid: 0x10000, shmaddr: 0x0, shmflg: 0x0
  19816  client            SHMDT                0 shmaddr: 0x7f4fd8ee7000
  19813  server            SHMDT                0 shmaddr: 0x7f1cf8b1f000
  19813  server           SHMCTL                0 shmid: 0x10000, cmd: 0, buf: 0x0

Every call the shm* syscall (SHM column) is displayed
on separate line together with process info (PID/COMM
columns) and argument details: return value (RET column)
and syscall arguments (ARGs column).

The ARGs column contains 'arg: value' couples that represent
given syscall arguments as described in their manpage.

It supports standard options to filter on pid/tid,
to specify duration of the trace and command name
filter, like:

    ./shmsnoop           # trace all shm*() syscalls
    ./shmsnoop -T        # include timestamps
    ./shmsnoop -p 181    # only trace PID 181
    ./shmsnoop -t 123    # only trace TID 123
    ./shmsnoop -d 10     # trace for 10 seconds only
    ./shmsnoop -n main   # only print process names containing "main"

* tools: Add sofdsnoop to spy on fds passed through socket

The sofdsnoop traces FDs passed through unix sockets.

  # ./sofdsnoop
  ACTION TID    COMM             SOCKET                    FD    NAME
  SEND   2576   Web Content      24:socket:[39763]         51    /dev/shm/org.mozilla.ipc.2576.23874
  RECV   2576   Web Content      49:socket:[809997]        51
  SEND   2576   Web Content      24:socket:[39763]         58    N/A
  RECV   2464   Gecko_IOThread   75:socket:[39753]         55

Every file descriptor that is passed via unix sockets os displayed
on separate line together with process info (TID/COMM columns),
ACTION details (SEND/RECV), file descriptor number (FD) and its
translation to file if available (NAME).

examples:
    ./sofdsnoop           # trace file descriptors passes
    ./sofdsnoop -T        # include timestamps
    ./sofdsnoop -p 181    # only trace PID 181
    ./sofdsnoop -t 123    # only trace TID 123
    ./sofdsnoop -d 10     # trace for 10 seconds only
    ./sofdsnoop -n main   # only print process names containing "main"

* inject: add option to limit number of overrides

* inject: update man pages with new count option
Signed-off-by: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>