Merge branch 'master' into master

README.md

@@ -65,6 +65,7 @@ Examples:
 
 Tools:
 
+- tools/[argdist](tools/argdist.py): Display function parameter values as a histogram or frequency count. [Examples](tools/argdist_example.txt).
 - tools/[bashreadline](tools/bashreadline.py): Print entered bash commands system wide. [Examples](tools/bashreadline_example.txt).
 - tools/[biolatency](tools/biolatency.py): Summarize block device I/O latency as a histogram. [Examples](tools/biolatency_example.txt).
 - tools/[biotop](tools/biotop.py): Top for disks: Summarize block device I/O by process. [Examples](tools/biotop_example.txt).

man/man8/argdist.8

+.TH argdist 8  "2016-02-11" "USER COMMANDS"
+.SH NAME
+argdist \- Trace a function and display a histogram or frequency count of its parameter values. Uses Linux eBPF/bcc.
+.SH SYNOPSIS
+.B argdist [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL] [-n COUNT] [-H specifier [specifier ...]] [-C specifier [specifier ...]]
+.SH DESCRIPTION
+argdist attaches to function entry and exit points, collects specified parameter
+values, and stores them in a histogram or a frequency collection that counts
+the number of times a parameter value occurred. It can also filter parameter
+values and instrument multiple entry points at once.
+
+This currently only works on x86_64. Check for future versions.
+.SH REQUIREMENTS
+CONFIG_BPF and bcc.
+.SH OPTIONS
+.TP
+\-h
+Print usage message.
+.TP
+\-p PID
+Trace only functions in the process PID.
+.TP
+\-z STRING_SIZE
+When collecting string arguments (of type char*), collect up to STRING_SIZE 
+characters. Longer strings will be truncated.
+.TP
+-i INTERVAL
+Print the collected data every INTERVAL seconds. The default is 1 second.
+.TP
+-n NUMBER
+Print the collected data COUNT times and then exit.
+.TP
+-H SPECIFIER, -C SPECIFIER
+One or more probe specifications that instruct argdist which functions to
+probe, which parameters to collect, how to aggregate them, and whether to perform
+any filtering. See SPECIFIER SYNTAX below.
+.SH SPECIFIER SYNTAX
+The general specifier syntax is as follows:
+
+.B {p,r}:[library]:function(signature)[:type:expr[:filter]][;label]
+.TP
+.B {p,r}
+Probe type \- "p" for function entry, "r" for function return;
+\-H for histogram collection, \-C for frequency count.
+Indicates where to place the probe and whether the probe should collect frequency
+count information, or aggregate the collected values into a histogram. Counting 
+probes will collect the number of times every parameter value was observed,
+whereas histogram probes will collect the parameter values into a histogram.
+Only integral types can be used with histogram probes; there is no such limitation
+for counting probes. Return probes can only use the pseudo-variable $retval, which
+is an alias for the function's return value.
+.TP
+.B [library]
+Library containing the probe.
+Specify the full path to the .so or executable file where the function to probe
+resides. Alternatively, you can specify just the lib name: for example, "c"
+refers to libc. If no library name is specified, the kernel is assumed.
+.TP
+.B function(signature)
+The function to probe, and its signature.
+The function name must match exactly for the probe to be placed. The signature,
+on the other hand, is only required if you plan to collect parameter values 
+based on that signature. For example, if you only want to collect the first
+parameter, you don't have to specify the rest of the parameters in the signature.
+.TP
+.B [type]
+The type of the expression to capture.
+This is the type of the keys in the histogram or raw event collection that are
+collected by the probes.
+.TP
+.B [expr]
+The expression to capture.
+These are the values that are assigned to the histogram or raw event collection.
+You may use the parameters directly, or valid C expressions that involve the
+parameters, such as "size % 10".
+.TP
+.B [filter]
+The filter applied to the captured data.
+Only parameter values that pass the filter will be collected. This is any valid
+C expression that refers to the parameter values, such as "fd == 1 && length > 16".
+.TP
+.B [label]
+The label that will be displayed when printing the probed values. By default,
+this is the probe specifier. 
+.SH EXAMPLES
+.TP
+Print a histogram of allocation sizes passed to kmalloc:
+#
+.B argdist.py -H 'p::__kmalloc(u64 size):u64:size'
+.TP
+Print a count of how many times process 1005 called malloc with an allocation size of 16 bytes:
+#
+.B argdist.py -p 1005 -C 'p:c:malloc(size_t size):size_t:size:size==16'
+.TP
+Snoop on all strings returned by gets():
+#
+.B argdist.py -C 'r:c:gets():char*:$retval'
+.TP
+Print frequency counts of how many times writes were issued to a particular file descriptor number, in process 1005:
+#
+.B argdist.py -p 1005 -C 'p:c:write(int fd):int:fd'
+.TP
+Print a histogram of error codes returned by read() in process 1005:
+#
+.B argdist.py -p 1005 -H 'r:c:read()'
+.TP
+Print a histogram of buffer sizes passed to write() across all processes, where the file descriptor was 1 (STDOUT):
+#
+.B argdist.py -H 'p:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1'
+.TP
+Count fork() calls in libc across all processes, grouped by pid:
+#
+.B argdist.py -C 'p:c:fork():int:$PID;fork per process'
+.TP
+Print histograms of sleep() and nanosleep() parameter values:
+#
+.B argdist.py -H 'p:c:sleep(u32 seconds):u32:seconds' -H 'p:c:nanosleep(struct timespec { time_t tv_sec; long tv_nsec; } *req):long:req->tv_nsec'
+.TP
+Spy on writes to STDOUT performed by process 2780, up to a string size of 120 characters:
+#
+.B argdist.py -p 2780 -z 120 -C 'p:c:write(int fd, char* buf, size_t len):char*:buf:fd==1'
+.SH SOURCE
+This is from bcc.
+.IP
+https://github.com/iovisor/bcc
+.PP
+Also look in the bcc distribution for a companion _examples.txt file containing
+example usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Sasha Goldshtein

tools/argdist.py

+#!/usr/bin/env python
+#
+# argdist.py   Trace a function and display a distribution of its
+#              parameter values as a histogram or frequency count.
+#
+# USAGE: argdist.py [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL]
+#                   [-n COUNT] [-C specifier [specifier ...]]
+#                   [-H specifier [specifier ...]]
+#
+# Licensed under the Apache License, Version 2.0 (the "License")
+# Copyright (C) 2016 Sasha Goldshtein.
+
+from bcc import BPF
+from time import sleep, strftime
+import argparse
+
+class Specifier(object):
+        text = """
+DATA_DECL
+
+int PROBENAME(struct pt_regs *ctx SIGNATURE)
+{
+        PID_FILTER
+        KEY_EXPR
+        if (!(FILTER)) return 0;
+        COLLECT
+        return 0;
+}
+"""
+        next_probe_index = 0
+        aliases = { "$PID": "bpf_get_current_pid_tgid()" }
+
+        def _substitute_aliases(self, expr):
+                if expr is None:
+                        return expr
+                for alias, subst in Specifier.aliases.items():
+                        expr = expr.replace(alias, subst)
+                return expr
+
+        def __init__(self, type, specifier, pid):
+                self.raw_spec = specifier 
+                spec_and_label = specifier.split(';')
+                self.label = spec_and_label[1] \
+                             if len(spec_and_label) == 2 else None
+                parts = spec_and_label[0].strip().split(':')
+                if len(parts) < 3 or len(parts) > 6:
+                        raise ValueError("invalid specifier format")
+                self.type = type    # hist or freq
+                self.is_ret_probe = parts[0] == "r"
+                if self.type != "hist" and self.type != "freq":
+                        raise ValueError("unrecognized probe type")
+                if parts[0] not in ["r", "p"]:
+                        raise ValueError("unrecognized probe type")
+                self.library = parts[1]
+                self.is_user = len(self.library) > 0
+                fparts = parts[2].split('(')
+                if len(fparts) != 2:
+                        raise ValueError("invalid specifier format")
+                self.function = fparts[0]
+                self.signature = fparts[1][:-1]
+                self.is_default_expr = len(parts) < 5
+                if not self.is_default_expr:
+                        self.expr_type = parts[3]
+                        self.expr = parts[4]
+                else:
+                        if not self.is_ret_probe and self.type == "hist":
+                                raise ValueError("dist probes must have expr")
+                        self.expr_type = \
+                                "u64" if not self.is_ret_probe else "int"
+                        self.expr = "1" if not self.is_ret_probe else "$retval"
+                self.expr = self.expr.replace("$retval",
+                                              "(%s)ctx->ax" % self.expr_type)
+                self.filter = None if len(parts) != 6 else parts[5]
+                if self.filter is not None:
+                        self.filter = self.filter.replace("$retval",
+                                "(%s)ctx->ax" % self.expr_type)
+                self.expr = self._substitute_aliases(self.expr)
+                self.filter = self._substitute_aliases(self.filter)
+                self.pid = pid
+                self.probe_func_name = "%s_probe%d" % \
+                        (self.function, Specifier.next_probe_index)
+                self.probe_hash_name = "%s_hash%d" % \
+                        (self.function, Specifier.next_probe_index)
+                Specifier.next_probe_index += 1
+
+        def _is_string_probe(self):
+                return self.expr_type == "char*" or self.expr_type == "char *"
+
+        def generate_text(self, string_size):
+                program = self.text.replace("PROBENAME", self.probe_func_name)
+                signature = "" if len(self.signature) == 0 \
+                               else "," + self.signature
+                program = program.replace("SIGNATURE", signature)
+                if self.pid is not None and not self.is_user:
+                        # kernel probes need to explicitly filter pid
+                        program = program.replace("PID_FILTER",
+                                "u32 pid = bpf_get_current_pid_tgid();\n" + \
+                                "if (pid != %d) { return 0; }" % self.pid)
+                else:
+                        program = program.replace("PID_FILTER", "")
+                if self._is_string_probe():
+                        decl = """
+struct %s_key_t { char key[%d]; };
+BPF_HASH(%s, struct %s_key_t, u64);
+""" \
+                        % (self.function, string_size,
+                           self.probe_hash_name, self.function)
+                        collect = "%s.increment(__key);" % self.probe_hash_name
+                        key_expr = """
+struct %s_key_t __key = {0};
+bpf_probe_read(&__key.key, sizeof(__key.key), %s);
+""" \
+                        % (self.function, self.expr)
+                elif self.type == "freq":
+                        decl = "BPF_HASH(%s, %s, u64);" % \
+                                (self.probe_hash_name, self.expr_type)
+                        collect = "%s.increment(__key);" % self.probe_hash_name
+                        key_expr = "%s __key = %s;" % \
+                                   (self.expr_type, self.expr)
+                elif self.type == "hist":
+                        decl = "BPF_HISTOGRAM(%s, %s);" % \
+                                (self.probe_hash_name, self.expr_type)
+                        collect = "%s.increment(bpf_log2l(__key));" % \
+                                  self.probe_hash_name 
+                        key_expr = "%s __key = %s;" % \
+                                   (self.expr_type, self.expr)
+                program = program.replace("DATA_DECL", decl)
+                program = program.replace("KEY_EXPR", key_expr) 
+                program = program.replace("FILTER", self.filter or "1") 
+                program = program.replace("COLLECT", collect)
+                return program
+
+        def attach(self, bpf):
+                self.bpf = bpf
+                if self.is_user:
+                        if self.is_ret_probe:
+                                bpf.attach_uretprobe(name=self.library,
+                                                  sym=self.function,
+                                                  fn_name=self.probe_func_name,
+                                                  pid=self.pid or -1)
+                        else:
+                                bpf.attach_uprobe(name=self.library,
+                                                  sym=self.function,
+                                                  fn_name=self.probe_func_name,
+                                                  pid=self.pid or -1)
+                else:
+                        if self.is_ret_probe:
+                                bpf.attach_kretprobe(event=self.function,
+                                                  fn_name=self.probe_func_name)
+                        else:
+                                bpf.attach_kprobe(event=self.function,
+                                                  fn_name=self.probe_func_name)
+
+        def display(self):
+                print(self.label or self.raw_spec)
+                data = self.bpf.get_table(self.probe_hash_name)
+                if self.type == "freq":
+                        print("\t%-10s %s" % ("COUNT", "EVENT"))
+                        for key, value in sorted(data.items(),
+                                                 key=lambda kv: kv[1].value):
+                                key_val = key.key if self._is_string_probe() \
+                                                  else str(key.value)
+                                if self.is_default_expr:
+                                        if not self.is_ret_probe:
+                                                key_str = "total calls"
+                                        else:
+                                                key_str = "retval = %s" % \
+                                                          key_val
+                                else:
+                                        key_str = "%s = %s" % \
+                                                  (self.expr, key_val)
+                                print("\t%-10s %s" % \
+                                      (str(value.value), key_str))
+                elif self.type == "hist":
+                        label = self.expr if not self.is_default_expr \
+                                          else "retval"
+                        data.print_log2_hist(val_type=label)
+
+examples = """
+Probe specifier syntax:
+        {p,r}:[library]:function(signature)[:type:expr[:filter]][;label]
+Where:
+        p,r        -- probe at function entry or at function exit
+                      in exit probes, only $retval is accessible
+        library    -- the library that contains the function
+                      (leave empty for kernel functions)
+        function   -- the function name to trace
+        signature  -- the function's parameters, as in the C header
+        type       -- the type of the expression to collect
+        expr       -- the expression to collect
+        filter     -- the filter that is applied to collected values
+        label      -- the label for this probe in the resulting output
+
+EXAMPLES:
+
+argdist.py -H 'p::__kmalloc(u64 size):u64:size'
+        Print a histogram of allocation sizes passed to kmalloc
+
+argdist.py -p 1005 -C 'p:c:malloc(size_t size):size_t:size:size==16'
+        Print a frequency count of how many times process 1005 called malloc
+        with an allocation size of 16 bytes
+
+argdist.py -C 'r:c:gets():char*:$retval;snooped strings'
+        Snoop on all strings returned by gets()
+
+argdist.py -p 1005 -C 'p:c:write(int fd):int:fd'
+        Print frequency counts of how many times writes were issued to a
+        particular file descriptor number, in process 1005
+
+argdist.py -p 1005 -H 'r:c:read()'
+        Print a histogram of error codes returned by read() in process 1005
+
+argdist.py -H \\
+        'p:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1'
+        Print a histogram of buffer sizes passed to write() across all
+        processes, where the file descriptor was 1 (STDOUT)
+
+argdist.py -C 'p:c:fork();fork calls'
+        Count fork() calls in libc across all processes
+        Can also use funccount.py, which is easier and more flexible 
+
+argdist.py \\
+        -H 'p:c:sleep(u32 seconds):u32:seconds' \\
+        -H 'p:c:nanosleep(struct timespec { time_t tv_sec; long tv_nsec; } *req):long:req->tv_nsec'
+        Print histograms of sleep() and nanosleep() parameter values
+
+argdist.py -p 2780 -z 120 \\
+        -C 'p:c:write(int fd, char* buf, size_t len):char*:buf:fd==1'
+        Spy on writes to STDOUT performed by process 2780, up to a string size
+        of 120 characters 
+"""
+
+parser = argparse.ArgumentParser(description=
+        "Trace a function and display a summary of its parameter values.",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=examples)
+parser.add_argument("-p", "--pid", type=int,
+        help="id of the process to trace (optional)")
+parser.add_argument("-z", "--string-size", default=80, type=int,
+        help="maximum string size to read from char* arguments")
+parser.add_argument("-i", "--interval", default=1, type=int,
+        help="output interval, in seconds")
+parser.add_argument("-n", "--number", type=int, dest="count",
+        help="number of outputs")
+parser.add_argument("-H", "--histogram", nargs="*", dest="histspecifier",
+        help="probe specifier to capture histogram of (see examples below)")
+parser.add_argument("-C", "--count", nargs="*", dest="countspecifier",
+        help="probe specifier to capture count of (see examples below)")
+parser.add_argument("-v", "--verbose", action="store_true",
+        help="print resulting BPF program code before executing")
+args = parser.parse_args()
+
+specifiers = []
+for specifier in (args.countspecifier or []):
+        specifiers.append(Specifier("freq", specifier, args.pid))
+for histspecifier in (args.histspecifier or []):
+        specifiers.append(Specifier("hist", histspecifier, args.pid))
+if len(specifiers) == 0:
+        print("at least one specifier is required")
+        exit(1)
+
+bpf_source = "#include <uapi/linux/ptrace.h>\n"
+for specifier in specifiers:
+        bpf_source += specifier.generate_text(args.string_size)
+
+if args.verbose:
+        print(bpf_source)
+
+bpf = BPF(text=bpf_source)
+
+for specifier in specifiers:
+        specifier.attach(bpf)
+
+count_so_far = 0
+while True:
+        try:
+                sleep(args.interval)
+        except KeyboardInterrupt:
+                exit()
+        print("[%s]" % strftime("%H:%M:%S"))
+        for specifier in specifiers:
+                specifier.display()
+        count_so_far += 1
+        if args.count is not None and count_so_far >= args.count:
+                exit()

tools/argdist_examples.txt

+Demonstrations of argdist.
+
+
+argdist probes functions you specify and collects parameter values into a
+histogram or a frequency count. This can be used to understand the distribution
+of values a certain parameter takes, filter and print interesting parameters
+without attaching a debugger, and obtain general execution statistics on
+various functions.
+
+For example, suppose you want to find what allocation sizes are common in
+your application:
+
+# ./argdist.py -p 2420 -C 'p:c:malloc(size_t size):size_t:size'
+[01:42:29]
+p:c:malloc(size_t size):size_t:size
+        COUNT      EVENT
+[01:42:30]
+p:c:malloc(size_t size):size_t:size
+        COUNT      EVENT
+[01:42:31]
+p:c:malloc(size_t size):size_t:size
+        COUNT      EVENT
+        1          size = 16
+[01:42:32]
+p:c:malloc(size_t size):size_t:size
+        COUNT      EVENT
+        2          size = 16
+[01:42:33]
+p:c:malloc(size_t size):size_t:size
+        COUNT      EVENT
+        3          size = 16
+[01:42:34]
+p:c:malloc(size_t size):size_t:size
+        COUNT      EVENT
+        4          size = 16
+^C
+
+It seems that the application is allocating blocks of size 16. The COUNT
+column contains the number of occurrences of a particular event, and the
+EVENT column describes the event. In this case, the "size" parameter was 
+probed and its value was 16, repeatedly.
+
+Now, suppose you wanted a histogram of buffer sizes passed to the write()
+function across the system:
+
+# ./argdist.py -H 'p:c:write(int fd, void *buf, size_t len):size_t:len'
+[01:45:22]
+p:c:write(int fd, void *buf, size_t len):size_t:len
+     len                 : count     distribution
+         0 -> 1          : 0        |                                        |
+         2 -> 3          : 2        |*************                           |
+         4 -> 7          : 0        |                                        |
+         8 -> 15         : 2        |*************                           |
+        16 -> 31         : 0        |                                        |
+        32 -> 63         : 6        |****************************************|
+[01:45:23]
+p:c:write(int fd, void *buf, size_t len):size_t:len
+     len                 : count     distribution
+         0 -> 1          : 0        |                                        |
+         2 -> 3          : 11       |***************                         |
+         4 -> 7          : 0        |                                        |
+         8 -> 15         : 4        |*****                                   |
+        16 -> 31         : 0        |                                        |
+        32 -> 63         : 28       |****************************************|
+        64 -> 127        : 12       |*****************                       |
+[01:45:24]
+p:c:write(int fd, void *buf, size_t len):size_t:len
+     len                 : count     distribution
+         0 -> 1          : 0        |                                        |
+         2 -> 3          : 21       |****************                        |
+         4 -> 7          : 0        |                                        |
+         8 -> 15         : 6        |****                                    |
+        16 -> 31         : 0        |                                        |
+        32 -> 63         : 52       |****************************************|
+        64 -> 127        : 26       |********************                    |
+^C
+
+It seems that most writes fall into three buckets: very small writes of 2-3
+bytes, medium writes of 32-63 bytes, and larger writes of 64-127 bytes.
+
+But these are writes across the board -- what if you wanted to focus on writes
+to STDOUT?
+
+# ./argdist.py -H 'p:c:write(int fd, void *buf, size_t len):size_t:len:fd==1'
+[01:47:17]
+p:c:write(int fd, void *buf, size_t len):size_t:len:fd==1
+     len                 : count     distribution
+         0 -> 1          : 0        |                                        |
+         2 -> 3          : 0        |                                        |
+         4 -> 7          : 0        |                                        |
+         8 -> 15         : 1        |****************************************|
+        16 -> 31         : 0        |                                        |
+        32 -> 63         : 1        |****************************************|
+[01:47:18]
+p:c:write(int fd, void *buf, size_t len):size_t:len:fd==1
+     len                 : count     distribution
+         0 -> 1          : 0        |                                        |
+         2 -> 3          : 0        |                                        |
+         4 -> 7          : 0        |                                        |
+         8 -> 15         : 2        |*************                           |
+        16 -> 31         : 0        |                                        |
+        32 -> 63         : 3        |********************                    |
+        64 -> 127        : 6        |****************************************|
+[01:47:19]
+p:c:write(int fd, void *buf, size_t len):size_t:len:fd==1
+     len                 : count     distribution
+         0 -> 1          : 0        |                                        |
+         2 -> 3          : 0        |                                        |
+         4 -> 7          : 0        |                                        |
+         8 -> 15         : 3        |*********                               |
+        16 -> 31         : 0        |                                        |
+        32 -> 63         : 5        |***************                         |
+        64 -> 127        : 13       |****************************************|
+^C
+
+The "fd==1" part is a filter that is applied to every invocation of write().
+Only if the filter condition is true, the value is recorded.
+
+You can also use argdist to trace kernel functions. For example, suppose you
+wanted a histogram of kernel allocation (kmalloc) sizes across the system,
+printed twice with 3 second intervals:
+
+# ./argdist.py -i 3 -n 2 -H 'p::__kmalloc(size_t size):size_t:size'
+[01:50:00]
+p::__kmalloc(size_t size):size_t:size
+     size                : count     distribution
+         0 -> 1          : 0        |                                        |
+         2 -> 3          : 0        |                                        |
+         4 -> 7          : 0        |                                        |
+         8 -> 15         : 6        |****************************************|
+[01:50:03]
+p::__kmalloc(size_t size):size_t:size
+     size                : count     distribution
+         0 -> 1          : 0        |                                        |
+         2 -> 3          : 0        |                                        |
+         4 -> 7          : 0        |                                        |
+         8 -> 15         : 22       |****************************************|
+        16 -> 31         : 0        |                                        |
+        32 -> 63         : 0        |                                        |
+        64 -> 127        : 5        |*********                               |
+       128 -> 255        : 2        |***                                     |
+
+Occasionally, numeric information isn't enough and you want to capture strings.
+What are the strings printed by puts() across the system?
+
+# ./argdist.py -i 10 -n 1 -C 'p:c:puts(char *str):char*:str'
+[01:53:54]
+p:c:puts(char *str):char*:str
+        COUNT      EVENT
+        2          str = Press ENTER to start.
+
+It looks like the message "Press ENTER to start." was printed twice during the
+10 seconds we were tracing.
+
+What about reads? You could trace gets() across the system and print the 
+strings input by the user (note how "r" is used instead of "p" to attach a
+probe to the function's return):
+
+# ./argdist.py -i 10 -n 1 -C 'r:c:gets():char*:$retval:$retval!=0'
+[02:12:23]
+r:c:gets():char*:$retval:$retval!=0
+        COUNT      EVENT
+        1          (char*)ctx->ax = hi there
+        3          (char*)ctx->ax = sasha
+        8          (char*)ctx->ax = hello
+
+Similarly, we could get a histogram of the error codes returned by read():
+
+# ./argdist.py -i 10 -c 1 -H 'r:c:read()'
+[02:15:36]
+r:c:read()
+     retval              : count     distribution
+         0 -> 1          : 29       |****************************************|
+         2 -> 3          : 11       |***************                         |
+         4 -> 7          : 0        |                                        |
+         8 -> 15         : 3        |****                                    |
+        16 -> 31         : 2        |**                                      |
+        32 -> 63         : 22       |******************************          |
+        64 -> 127        : 5        |******                                  |
+       128 -> 255        : 0        |                                        |
+       256 -> 511        : 1        |*                                       |
+       512 -> 1023       : 1        |*                                       |
+      1024 -> 2047       : 0        |                                        |
+      2048 -> 4095       : 2        |**                                      |
+
+Here's a final example that finds how many write() system calls are performed
+by each process on the system:
+
+# argdist.py -C 'p:c:write():int:$PID;write per process' -n 2
+[06:47:18]
+write by process
+        COUNT      EVENT
+        3          bpf_get_current_pid_tgid() = 8889
+        7          bpf_get_current_pid_tgid() = 7615
+        7          bpf_get_current_pid_tgid() = 2480
+[06:47:19]
+write by process
+        COUNT      EVENT
+        9          bpf_get_current_pid_tgid() = 8889
+        23         bpf_get_current_pid_tgid() = 7615
+        23         bpf_get_current_pid_tgid() = 2480
+
+As you see, the $PID alias is expanded to the BPF function bpf_get_current_pid_tgid(),
+which returns the current process' pid.
+
+
+USAGE message:
+
+usage: argdist.py [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL] [-n COUNT]
+                  [-H [HISTSPECIFIER [HISTSPECIFIER ...]]]
+                  [-C [COUNTSPECIFIER [COUNTSPECIFIER ...]]] [-v]
+
+Trace a function and display a summary of its parameter values.
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -p PID, --pid PID     id of the process to trace (optional)
+  -z STRING_SIZE, --string-size STRING_SIZE
+                        maximum string size to read from char* arguments
+  -i INTERVAL, --interval INTERVAL
+                        output interval, in seconds
+  -n COUNT, --number COUNT
+                        number of outputs
+  -H [HISTSPECIFIER [HISTSPECIFIER ...]], --histogram [HISTSPECIFIER [HISTSPECIFIER ...]]
+                        probe specifier to capture histogram of (see examples
+                        below)
+  -C [COUNTSPECIFIER [COUNTSPECIFIER ...]], --count [COUNTSPECIFIER [COUNTSPECIFIER ...]]
+                        probe specifier to capture count of (see examples
+                        below)
+  -v, --verbose         print resulting BPF program code before executing
+
+Probe specifier syntax:
+        {p,r}:[library]:function(signature)[:type:expr[:filter]][;label]
+Where:
+        p,r        -- probe at function entry or at function exit
+                      in exit probes, only $retval is accessible
+        library    -- the library that contains the function
+                      (leave empty for kernel functions)
+        function   -- the function name to trace
+        signature  -- the function's parameters, as in the C header
+        type       -- the type of the expression to collect
+        expr       -- the expression to collect
+        filter     -- the filter that is applied to collected values
+        label      -- the label for this probe in the resulting output
+
+EXAMPLES:
+
+argdist.py -H 'p::__kmalloc(u64 size):u64:size'
+        Print a histogram of allocation sizes passed to kmalloc
+
+argdist.py -p 1005 -C 'p:c:malloc(size_t size):size_t:size:size==16'
+        Print a frequency count of how many times process 1005 called malloc
+        with an allocation size of 16 bytes
+
+argdist.py -C 'r:c:gets():char*:$retval;snooped strings'
+        Snoop on all strings returned by gets()
+
+argdist.py -p 1005 -C 'p:c:write(int fd):int:fd'
+        Print frequency counts of how many times writes were issued to a
+        particular file descriptor number, in process 1005
+
+argdist.py -p 1005 -H 'r:c:read()'
+        Print a histogram of error codes returned by read() in process 1005
+
+argdist.py -H \
+        'p:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1'
+        Print a histogram of buffer sizes passed to write() across all
+        processes, where the file descriptor was 1 (STDOUT)
+
+argdist.py -C 'p:c:fork();fork calls'
+        Count fork() calls in libc across all processes
+        Can also use funccount.py, which is easier and more flexible
+
+argdist.py \
+        -H 'p:c:sleep(u32 seconds):u32:seconds' \
+        -H 'p:c:nanosleep(struct timespec { time_t tv_sec; long tv_nsec; } *req):long:req->tv_nsec'
+        Print histograms of sleep() and nanosleep() parameter values
+
+argdist.py -p 2780 -z 120 \
+        -C 'p:c:write(int fd, char* buf, size_t len):char*:buf:fd==1'
+        Spy on writes to STDOUT performed by process 2780, up to a string size
+        of 120 characters
+