add statsnoop tool

README.md

@@ -157,6 +157,7 @@ bpftrace contains various tools, which also serve as examples of programming in
 - tools/[loads.bt](tools/loads.bt): Print load averages. [Examples](tools/loads_example.txt).
 - tools/[opensnoop.bt](tools/loads.bt): Trace open() syscalls showing filenames. [Examples](tools/opensnoop_example.txt).
 - tools/[pidpersec.bt](tools/pidpersec.bt): Count new procesess (via fork). [Examples](tools/pidpersec_example.txt).
+- tools/[statsnoop.bt](tools/statsnoop.bt): Trace stat() syscalls for general debugging. [Examples](tools/statsnoop_example.txt).
 - tools/[syncsnoop.bt](tools/syncsnoop.bt): Trace sync() variety of syscalls. [Examples](tools/syncsnoop_example.txt).
 - tools/[vfscount.bt](tools/vfscount.bt): Count VFS calls. [Examples](tools/vfscount_example.txt).
 - tools/[vfsstat.bt](tools/vfsstat.bt): Count some VFS calls, with per-second summaries. [Examples](tools/vfsstat_example.txt).

man/man8/statsnoop.8

+.TH statsnoop 8  "2018-09-08" "USER COMMANDS"
+.SH NAME
+statsnoop.bt \- Trace stat() syscalls. Uses bpftrace/eBPF.
+.SH SYNOPSIS
+.B statsnoop.bt
+.SH DESCRIPTION
+statsnoop traces the stat() syscall, showing which processes are attempting
+to stat which files. This can be useful for determining the location of config
+and log files, or for troubleshooting applications that are failing, specially
+on startup.
+
+This traces the traecepoints for statfs(), statx(), newstat(), and
+newlstat(). These aren't the only the stat syscalls: if you are missing
+activity, you may need to add more variants.
+
+Since this uses BPF, only the root user can use this tool.
+.SH REQUIREMENTS
+CONFIG_BPF and bcc.
+.SH EXAMPLES
+.TP
+Trace all stat() syscalls:
+#
+.B statsnoop.bt
+.SH FIELDS
+PID
+Process ID
+.TP
+TID
+Thread ID
+.TP
+COMM
+Process name
+.TP
+FD
+File descriptor (if success), or -1 (if failed)
+.TP
+ERR
+Error number (see the system's errno.h)
+.TP
+PATH
+Stat path
+.SH OVERHEAD
+This traces the stat tracepoints and prints output for each event. As the
+rate of this is generally expected to be low (< 1000/s), the overhead is also
+expected to be negligible. If you have an application that is calling a high
+rate of stat()s, then test and understand overhead before use.
+.SH SOURCE
+This is from bpftrace.
+.IP
+https://github.com/iovisor/bpftrace
+.PP
+Also look in the bpftrace distribution for a companion _examples.txt file containing
+example usage, output, and commentary for this tool.
+
+This is a bpftrace version of the bcc tool of the same name. The bcc tool
+may provide more options and customizations.
+.IP
+https://github.com/iovisor/bcc
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+opensnoop(8), execsnoop(8)

tools/statsnoop.bt

+/*
+ * statsnoop	Trace stat() syscalls.
+ *		For Linux, uses bpftrace and eBPF.
+ *
+ * This traces the traecepoints for statfs(), statx(), newstat(), and
+ * newlstat(). These aren't the only the stat syscalls: if you are missing
+ * activity, you may need to add more variants.
+ *
+ * Also a basic example of bpftrace.
+ *
+ * USAGE: statsnoop.bt
+ *
+ * This is a bpftrace version of the bcc tool of the same name.
+ *
+ * Copyright 2018 Netflix, Inc.
+ * Licensed under the Apache License, Version 2.0 (the "License")
+ *
+ * 08-Sep-2018	Brendan Gregg	Created this.
+ */
+
+BEGIN
+{
+	printf("Tracing stat syscalls... Hit Ctrl-C to end.\n");
+	printf("%-6s %-16s %3s %s\n", "PID", "COMM", "ERR", "PATH");
+}
+
+tracepoint:syscalls:sys_enter_statfs
+{
+	$path = *(ctx + 16);
+	@filename[tid] = $path;
+}
+
+tracepoint:syscalls:sys_enter_statx
+{
+	$file = *(ctx + 24);
+	@filename[tid] = $file;
+}
+
+tracepoint:syscalls:sys_enter_newstat
+{
+	$file = *(ctx + 16);
+	@filename[tid] = $file;
+}
+
+tracepoint:syscalls:sys_enter_newlstat
+{
+	$file = *(ctx + 16);
+	@filename[tid] = $file;
+}
+
+tracepoint:syscalls:sys_exit_statfs,
+tracepoint:syscalls:sys_exit_statx,
+tracepoint:syscalls:sys_exit_newstat,
+tracepoint:syscalls:sys_exit_newlstat
+/@filename[tid]/
+{
+	$ret = *(ctx + 16);
+	$errno = $ret >= 0 ? 0 : 0 - $ret;
+
+	printf("%-6d %-16s %3d %s\n", pid, comm, $errno,
+	    str(@filename[tid]));
+	delete(@filename[tid]);
+}

tools/statsnoop_example.txt

+Demonstrations of statsnoop, the Linux bpftrace/eBPF version.
+
+
+statsnoop traces different stat() syscalls system-wide, and prints details.
+Example output:
+
+# bpftrace statsnoop.bt
+Attaching 9 probes...
+Tracing stat syscalls... Hit Ctrl-C to end.
+PID    COMM             ERR PATH
+27835  bash               0 .
+27835  bash               2 /usr/local/sbin/iconfig
+27835  bash               2 /usr/local/bin/iconfig
+27835  bash               2 /usr/sbin/iconfig
+27835  bash               2 /usr/bin/iconfig
+27835  bash               2 /sbin/iconfig
+27835  bash               2 /bin/iconfig
+27835  bash               2 /usr/games/iconfig
+27835  bash               2 /usr/local/games/iconfig
+27835  bash               2 /snap/bin/iconfig
+27835  bash               2 /apps/python/bin/iconfig
+30573  command-not-fou    2 /usr/bin/Modules/Setup
+30573  command-not-fou    2 /usr/bin/lib/python3.5/os.py
+30573  command-not-fou    2 /usr/bin/lib/python3.5/os.pyc
+30573  command-not-fou    0 /usr/lib/python3.5/os.py
+30573  command-not-fou    2 /usr/bin/pybuilddir.txt
+30573  command-not-fou    2 /usr/bin/lib/python3.5/lib-dynload
+30573  command-not-fou    0 /usr/lib/python3.5/lib-dynload
+30573  command-not-fou    2 /usr/lib/python35.zip
+30573  command-not-fou    0 /usr/lib
+30573  command-not-fou    2 /usr/lib/python35.zip
+30573  command-not-fou    0 /usr/lib/python3.5/
+30573  command-not-fou    0 /usr/lib/python3.5/
+30573  command-not-fou    0 /usr/lib/python3.5/
+30573  command-not-fou    2 /usr/lib/python3.5/encodings/__init__.cpython-35m-x86_64-linux-
+30573  command-not-fou    2 /usr/lib/python3.5/encodings/__init__.abi3.so
+30573  command-not-fou    2 /usr/lib/python3.5/encodings/__init__.so
+30573  command-not-fou    0 /usr/lib/python3.5/encodings/__init__.py
+30573  command-not-fou    0 /usr/lib/python3.5/encodings/__init__.py
+
+This output has caught me mistyping a command in another shell, "iconfig"
+instead of "ifconfig". The first several lines show the bash shell searching
+the $PATH (why is games in my $PATH??), and failing to find it (ERR == 2 is
+file not found). Then, a "command-not-found" program executes (the name is
+truncated to 16 characters in the COMM field, including the NULL), which
+begins the process of searching for and suggesting a package. ie, this:
+
+# iconfig
+The program 'iconfig' is currently not installed. You can install it by typing:
+apt install ipmiutil
+
+statsnoop can be used for general debugging, to see what file information has
+been requested, and whether those files exist. It can be used as a companion
+to opensnoop, which shows what files were actually opened.
+
+
+There is another version of this tool in bcc: https://github.com/iovisor/bcc
+The bcc version provides options to customize the output.