add oomkill tool

README.md

@@ -150,6 +150,7 @@ bpftrace contains various tools, which also serve as examples of programming in
 - tools/[killsnoop.bt](tools/killsnoop.bt): Trace signals issued by the kill() syscall. [Examples](tools/killsnoop_example.txt).
 - tools/[loads.bt](tools/loads.bt): Print load averages. [Examples](tools/loads_example.txt).
 - tools/[opensnoop.bt](tools/loads.bt): Trace open() syscalls showing filenames. [Examples](tools/opensnoop_example.txt).
+- tools/[oomkill.bt](tools/oomkill.bt): Trace OOM killer. [Examples](tools/oomkill_example.txt).
 - tools/[pidpersec.bt](tools/pidpersec.bt): Count new procesess (via fork). [Examples](tools/pidpersec_example.txt).
 - tools/[statsnoop.bt](tools/statsnoop.bt): Trace stat() syscalls for general debugging. [Examples](tools/statsnoop_example.txt).
 - tools/[syncsnoop.bt](tools/syncsnoop.bt): Trace sync() variety of syscalls. [Examples](tools/syncsnoop_example.txt).

man/man8/oomkill.8

+.TH oomkill 8  "2018-09-07" "USER COMMANDS"
+.SH NAME
+oomkill.bt \- Trace OOM killer. Uses bpftrace/eBPF.
+.SH SYNOPSIS
+.B oomkill.bt
+.SH DESCRIPTION
+This traces the kernel out-of-memory killer, and prints basic details,
+including the system load averages at the time of the OOM kill. This can
+provide more context on the system state at the time: was it getting busier
+or steady, based on the load averages? This tool may also be useful to
+customize for investigations; for example, by adding other task_struct
+details at the time of OOM, or by adding other commands to run at the shell.
+
+Since this uses BPF, only the root user can use this tool.
+.SH REQUIREMENTS
+CONFIG_BPF and bpftrace.
+.SH EXAMPLES
+.TP
+Trace OOM kill events:
+#
+.B oomkill.bt
+.SH FIELDS
+.TP
+Triggered by ...
+The process ID and process name of the task that was running when another task was OOM
+killed.
+.TP
+OOM kill of ...
+The process ID and name of the target process that was OOM killed.
+.TP
+loadavg
+Contents of /proc/loadavg. The first three numbers are 1, 5, and 15 minute
+load averages (where the average is an exponentially damped moving sum, and
+those numbers are constants in the equation); then there is the number of
+running tasks, a slash, and the total number of tasks; and then the last number
+is the last PID to be created.
+.SH OVERHEAD
+Negligible.
+.SH SOURCE
+This is from bpftrace.
+.IP
+https://github.com/iovisor/bpftrace
+.PP
+Also look in the bpftrace distribution for a companion _examples.txt file containing
+example usage, output, and commentary for this tool.
+
+This is a bpftrace version of the bcc tool of the same name. The bcc tool
+may provide more options and customizations.
+.IP
+https://github.com/iovisor/bcc
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+dmesg(1)

tools/oomkill.bt

+/*
+ * oomkill	Trace OOM killer.
+ *		For Linux, uses bpftrace and eBPF.
+ *
+ * This traces the kernel out-of-memory killer, and prints basic details,
+ * including the system load averages. This can provide more context on the
+ * system state at the time of OOM: was it getting busier or steady, based
+ * on the load averages? This tool may also be useful to customize for
+ * investigations; for example, by adding other task_struct details at the
+ * time of the OOM, or other commands in the system() call.
+ *
+ * This currently works by using kernel dynamic tracing of oom_kill_process().
+ *
+ * USAGE: oomkill.bt
+ *
+ * Copyright 2018 Netflix, Inc.
+ * Licensed under the Apache License, Version 2.0 (the "License")
+ *
+ * 07-Sep-2018	Brendan Gregg	Created this.
+ */
+
+#include <linux/oom.h>
+
+BEGIN
+{
+	printf("Tracing oom_kill_process()... Hit Ctrl-C to end.\n");
+}
+
+kprobe:oom_kill_process
+{
+	$oc = (oom_control *)arg1;
+	time("%H:%M:%S ");
+	printf("Triggered by PID %d (\"%s\"), ", pid, comm);
+	printf("OOM kill of PID %d (\"%s\"), %d pages, loadavg: ",
+	    $oc->chosen->pid, $oc->chosen->comm, $oc->totalpages);
+	system("cat /proc/loadavg");
+}

tools/oomkill_example.txt

+Demonstrations of oomkill, the Linux bpftrace/eBPF version.
+
+
+oomkill is a simple program that traces the Linux out-of-memory (OOM) killer,
+and shows basic details on one line per OOM kill:
+
+# ./oomkill
+Tracing oom_kill_process()... Ctrl-C to end.
+21:03:39 Triggered by PID 3297 ("ntpd"), OOM kill of PID 22516 ("perl"), 3850642 pages, loadavg: 0.99 0.39 0.30 3/282 22724
+21:03:48 Triggered by PID 22517 ("perl"), OOM kill of PID 22517 ("perl"), 3850642 pages, loadavg: 0.99 0.41 0.30 2/282 22932
+
+The first line shows that PID 22516, with process name "perl", was OOM killed
+when it reached 3850642 pages (usually 4 Kbytes per page). This OOM kill
+happened to be triggered by PID 3297, process name "ntpd", doing some memory
+allocation.
+
+The system log (dmesg) shows pages of details and system context about an OOM
+kill. What it currently lacks, however, is context on how the system had been
+changing over time. I've seen OOM kills where I wanted to know if the system
+was at steady state at the time, or if there had been a recent increase in
+workload that triggered the OOM event. oomkill provides some context: at the
+end of the line is the load average information from /proc/loadavg. For both
+of the oomkills here, we can see that the system was getting busier at the
+time (a higher 1 minute "average" of 0.99, compared to the 15 minute "average"
+of 0.30).
+
+oomkill can also be the basis of other tools and customizations. For example,
+you can edit it to include other task_struct details from the target PID at
+the time of the OOM kill, or to run other commands from the shell.
+
+There is another version of this tool in bcc: https://github.com/iovisor/bcc