add mdflush tool

README.md

@@ -149,6 +149,7 @@ bpftrace contains various tools, which also serve as examples of programming in
 - tools/[gethostlatency.bt](tools/gethostlatency.bt): Show latency for getaddrinfo/gethostbyname[2] calls. [Examples](tools/gethostlatency_example.txt).
 - tools/[killsnoop.bt](tools/killsnoop.bt): Trace signals issued by the kill() syscall. [Examples](tools/killsnoop_example.txt).
 - tools/[loads.bt](tools/loads.bt): Print load averages. [Examples](tools/loads_example.txt).
+- tools/[mdflush.bt](tools/mdflush.bt): Trace md flush events. [Examples](tools/mdflush_example.txt).
 - tools/[opensnoop.bt](tools/loads.bt): Trace open() syscalls showing filenames. [Examples](tools/opensnoop_example.txt).
 - tools/[oomkill.bt](tools/oomkill.bt): Trace OOM killer. [Examples](tools/oomkill_example.txt).
 - tools/[pidpersec.bt](tools/pidpersec.bt): Count new procesess (via fork). [Examples](tools/pidpersec_example.txt).

man/man8/mdflush.8

+.TH mdflush 8  "2018-09-07" "USER COMMANDS"
+.SH NAME
+mdflush.bt \- Trace md flush events. Uses bpftrace/eBPF.
+.SH SYNOPSIS
+.B mdflush.bt
+.SH DESCRIPTION
+This tool traces flush events by md, the Linux multiple device driver
+(software RAID). The timestamp and md device for the flush are printed.
+Knowing when these flushes happen can be useful for correlation with
+unexplained spikes in disk latency.
+
+This works by tracing the kernel md_flush_request() function using kernel
+dynamic tracing, and will need updating to match any changes to this function.
+
+Note that the flushes themselves are likely to originate from higher in the
+I/O stack, such as from the file systems.
+
+Since this uses BPF, only the root user can use this tool.
+.SH REQUIREMENTS
+CONFIG_BPF and bpftrace.
+.SH EXAMPLES
+.TP
+Trace md flush events:
+#
+.B mdflush.bt
+.SH FIELDS
+.TP
+TIME
+Time of the flush event (HH:MM:SS).
+.TP
+PID
+The process ID that was on-CPU when the event was issued. This may identify
+the cause of the flush (eg, the "sync" command), but will often identify a
+kernel worker thread that was managing I/O.
+.TP
+COMM
+The command name for the PID.
+.TP
+DEVICE
+The md device name.
+.SH OVERHEAD
+Expected to be negligible.
+.SH SOURCE
+This is from bpftrace.
+.IP
+https://github.com/iovisor/bpftrace
+.PP
+Also look in the bpftrace distribution for a companion _examples.txt file containing
+example usage, output, and commentary for this tool.
+
+This is a bpftrace version of the bcc tool of the same name. The bcc tool
+may provide more options and customizations.
+.IP
+https://github.com/iovisor/bcc
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+biosnoop(8)

tools/mdflush.bt

+/*
+ * mdflush	Trace md flush events.
+ *		For Linux, uses bpftrace and eBPF.
+ *
+ * USAGE: mdflush.bt
+ *
+ * This is a bpftrace version of the bcc tool of the same name.
+ *
+ * Copyright 2018 Netflix, Inc.
+ * Licensed under the Apache License, Version 2.0 (the "License")
+ *
+ * 08-Sep-2018	Brendan Gregg	Created this.
+ */
+
+#include <linux/genhd.h>
+#include <linux/bio.h>
+
+BEGIN
+{
+	printf("Tracing md flush events... Hit Ctrl-C to end.\n");
+	printf("%-8s %-6s %-16s %s", "TIME", "PID", "COMM", "DEVICE");
+}
+
+kprobe:md_flush_request
+{
+	time("%H:%M:%S ");
+	printf("%-6d %-16s %s\n", pid, comm, ((bio *)arg1)->bi_disk->disk_name);
+}

tools/mdflush_example.txt

+Demonstrations of mdflush, the Linux bpftrace/eBPF version.
+
+
+The mdflush tool traces flushes at the md driver level, and prints details
+including the time of the flush:
+
+# ./mdflush.bt
+Tracing md flush requests... Hit Ctrl-C to end.
+TIME     PID    COMM             DEVICE
+03:13:49 16770  sync             md0
+03:14:08 16864  sync             md0
+03:14:49 496    kworker/1:0H     md0
+03:14:49 488    xfsaild/md0      md0
+03:14:54 488    xfsaild/md0      md0
+03:15:00 488    xfsaild/md0      md0
+03:15:02 85     kswapd0          md0
+03:15:02 488    xfsaild/md0      md0
+03:15:05 488    xfsaild/md0      md0
+03:15:08 488    xfsaild/md0      md0
+03:15:10 488    xfsaild/md0      md0
+03:15:11 488    xfsaild/md0      md0
+03:15:11 488    xfsaild/md0      md0
+03:15:11 488    xfsaild/md0      md0
+03:15:11 488    xfsaild/md0      md0
+03:15:11 488    xfsaild/md0      md0
+03:15:12 488    xfsaild/md0      md0
+03:15:13 488    xfsaild/md0      md0
+03:15:15 488    xfsaild/md0      md0
+03:15:19 496    kworker/1:0H     md0
+03:15:49 496    kworker/1:0H     md0
+03:15:55 18840  sync             md0
+03:16:49 496    kworker/1:0H     md0
+03:17:19 496    kworker/1:0H     md0
+03:20:19 496    kworker/1:0H     md0
+03:21:19 496    kworker/1:0H     md0
+03:21:49 496    kworker/1:0H     md0
+03:25:19 496    kworker/1:0H     md0
+[...]
+
+This can be useful for correlation with latency outliers or spikes in disk
+latency, as measured using another tool (eg, system monitoring). If spikes in
+disk latency often coincide with md flush events, then it would make flushing
+a target for tuning.
+
+Note that the flush events are likely to originate from higher in the I/O
+stack, such as from file systems. This traces md processing them, and the
+timestamp corresponds with when md began to issue the flush to disks.
+
+There is another version of this tool in bcc: https://github.com/iovisor/bcc