Merge pull request #628 from brendangregg/master

tutorials: end-user, and python developer

README.md

@@ -2,17 +2,19 @@
 # BPF Compiler Collection (BCC)
 
 BCC is a toolkit for creating efficient kernel tracing and manipulation
-programs, and includes several useful tools and examples. It makes use of eBPF
-(Extended Berkeley Packet Filters), a new feature that was first added to
-Linux 3.15. Much of what BCC uses requires Linux 4.1 and above.
+programs, and includes several useful tools and examples. It makes use of 
+extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature
+that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1
+and above.
 
 eBPF was [described by](https://lkml.org/lkml/2015/4/14/232) Ingo Molnár as:
 
 > One of the more interesting features in this cycle is the ability to attach eBPF programs (user-defined, sandboxed bytecode executed by the kernel) to kprobes. This allows user-defined instrumentation on a live kernel image that can never crash, hang or interfere with the kernel negatively.
 
-BCC makes eBPF programs easier to write, with kernel instrumentation in C
-and a front-end in Python. It is suited for many tasks, including performance
-analysis and network traffic control.
+BCC makes BPF programs easier to write, with kernel instrumentation in C
+(and includes a C wrapper around LLVM), and front-ends in Python and lua.
+It is suited for many tasks, including performance analysis and network
+traffic control.
 
 ## Screenshot
 
@@ -170,46 +172,10 @@ The features of this toolkit include:
 In the future, more bindings besides python will likely be supported. Feel free
 to add support for the language of your choice and send a pull request!
 
-## Tutorial
+## Tutorials
 
-The BCC toolchain is currently composed of two parts: a C wrapper around LLVM,
-and a Python API to interact with the running program. Later, we will go into
-more detail of how this all works.
-
-### Hello, World
-
-First, we should include the BPF class from the bpf module:
-```python
-from bcc import BPF
-```
-
-Since the C code is so short, we will embed it inside the python script.
-
-The BPF program always takes at least one argument, which is a pointer to the
-context for this type of program. Different program types have different calling
-conventions, but for this one we don't care so `void *` is fine.
-```python
-BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
-```
-
-For this example, we will call the program every time `fork()` is called by a
-userspace process. Underneath the hood, fork translates to the `clone` syscall.
-BCC recognizes prefix `kprobe__`, and will auto attach our program to the kernel symbol `sys_clone`.
-
-The python process will then print the trace printk circular buffer until ctrl-c
-is pressed. The BPF program is removed from the kernel when the userspace
-process that loaded it closes the fd (or exits).
-
-Output:
-```
-bcc/examples$ sudo python hello_world.py
-          python-7282  [002] d...  3757.488508: : Hello, World!
-```
-
-For an explanation of the meaning of the printed fields, see the trace_pipe
-section of the [kernel ftrace doc](https://www.kernel.org/doc/Documentation/trace/ftrace.txt).
-
-[Source code listing](examples/hello_world.py)
+- [docs/tutorial.md](docs/tutorial.md): Using bcc tools to solve performance, troubleshooting, and networking issues.
+- [docs/tutorial_bcc_python_developer.md](docs/tutorial_bcc_python_developer.md): Developing new bcc programs using the Python interface.
 
 ### Networking
 
@@ -222,75 +188,6 @@ multiple granularities. See the code [here](examples/networking/tunnel_monitor).
 
 [![Screenshot](http://img.youtube.com/vi/yYy3Cwce02k/0.jpg)](https://youtu.be/yYy3Cwce02k)
 
-### Tracing
-
-Here is a slightly more complex tracing example than Hello World. This program
-will be invoked for every task change in the kernel, and record in a BPF map
-the new and old pids.
-
-The C program below introduces two new concepts.
-The first is the macro `BPF_TABLE`. This defines a table (type="hash"), with key
-type `key_t` and leaf type `u64` (a single counter). The table name is `stats`,
-containing 1024 entries maximum. One can `lookup`, `lookup_or_init`, `update`,
-and `delete` entries from the table.
-The second concept is the prev argument. This argument is treated specially by
-the BCC frontend, such that accesses to this variable are read from the saved
-context that is passed by the kprobe infrastructure. The prototype of the args
-starting from position 1 should match the prototype of the kernel function being
-kprobed. If done so, the program will have seamless access to the function
-parameters.
-```c
-#include <uapi/linux/ptrace.h>
-#include <linux/sched.h>
-
-struct key_t {
-  u32 prev_pid;
-  u32 curr_pid;
-};
-// map_type, key_type, leaf_type, table_name, num_entry
-BPF_TABLE("hash", struct key_t, u64, stats, 1024);
-// attach to finish_task_switch in kernel/sched/core.c, which has the following
-// prototype:
-//   struct rq *finish_task_switch(struct task_struct *prev)
-int count_sched(struct pt_regs *ctx, struct task_struct *prev) {
-  struct key_t key = {};
-  u64 zero = 0, *val;
-
-  key.curr_pid = bpf_get_current_pid_tgid();
-  key.prev_pid = prev->pid;
-
-  val = stats.lookup_or_init(&key, &zero);
-  (*val)++;
-  return 0;
-}
-```
-[Source code listing](examples/tracing/task_switch.c)
-
-The userspace component loads the file shown above, and attaches it to the
-`finish_task_switch` kernel function.
-The [] operator of the BPF object gives access to each BPF_TABLE in the
-program, allowing pass-through access to the values residing in the kernel. Use
-the object as you would any other python dict object: read, update, and deletes
-are all allowed.
-```python
-from bcc import BPF
-from time import sleep
-
-b = BPF(src_file="task_switch.c")
-b.attach_kprobe(event="finish_task_switch", fn_name="count_sched")
-
-# generate many schedule events
-for i in range(0, 100): sleep(0.01)
-
-for k, v in b["stats"].items():
-    print("task_switch[%5d->%5d]=%u" % (k.prev_pid, k.curr_pid, v.value))
-```
-[Source code listing](examples/tracing/task_switch.py)
-
-## Getting started
-
-See [INSTALL.md](INSTALL.md) for installation steps on your platform.
-
 ## Contributing
 
 Already pumped up to commit some code? Here are some resources to join the

examples/tracing/hello_fields.py

+#!/usr/bin/env python
+#
+# This is a Hello World example that formats output as fields.
+
+from bcc import BPF
+
+# define BPF program
+prog = """
+int hello(void *ctx) {
+    bpf_trace_printk("Hello, World!\\n");
+    return 0;
+}
+"""
+
+# load BPF program
+b = BPF(text=prog)
+b.attach_kprobe(event="sys_clone", fn_name="hello")
+
+# header
+print("%-18s %-16s %-6s %s" % ("TIME(s)", "COMM", "PID", "MESSAGE"))
+
+# format output
+while 1:
+    try:
+        (task, pid, cpu, flags, ts, msg) = b.trace_fields()
+    except ValueError:
+        continue
+    print("%-18.9f %-16s %-6d %s" % (ts, task, pid, msg))

examples/tracing/hello_perf_output.py

+#!/usr/bin/env python
+#
+# This is a Hello World example that uses BPF_PERF_OUTPUT.
+
+from bcc import BPF
+import ctypes as ct
+
+# define BPF program
+prog = """
+#include <linux/sched.h>
+
+// define output data structure in C
+struct data_t {
+    u32 pid;
+    u64 ts;
+    char comm[TASK_COMM_LEN];
+};
+BPF_PERF_OUTPUT(events);
+
+int hello(struct pt_regs *ctx) {
+    struct data_t data = {};
+
+    data.pid = bpf_get_current_pid_tgid();
+    data.ts = bpf_ktime_get_ns();
+    bpf_get_current_comm(&data.comm, sizeof(data.comm));
+
+    events.perf_submit(ctx, &data, sizeof(data));
+
+    return 0;
+}
+"""
+
+# load BPF program
+b = BPF(text=prog)
+b.attach_kprobe(event="sys_clone", fn_name="hello")
+
+# define output data structure in Python
+TASK_COMM_LEN = 16    # linux/sched.h
+class Data(ct.Structure):
+    _fields_ = [("pid", ct.c_ulonglong),
+                ("ts", ct.c_ulonglong),
+                ("comm", ct.c_char * TASK_COMM_LEN)]
+
+# header
+print("%-18s %-16s %-6s %s" % ("TIME(s)", "COMM", "PID", "MESSAGE"))
+
+# process event
+start = 0
+def print_event(cpu, data, size):
+    global start
+    event = ct.cast(data, ct.POINTER(Data)).contents
+    if start == 0:
+            start = event.ts
+    time_s = (float(event.ts - start)) / 1000000000
+    print("%-18.9f %-16s %-6d %s" % (time_s, event.comm, event.pid,
+        "Hello, perf_output!"))
+
+# loop with callback to print_event
+b["events"].open_perf_buffer(print_event)
+while 1:
+    b.kprobe_poll()

examples/tracing/strlen_count.py

@@ -24,13 +24,13 @@ struct key_t {
 BPF_HASH(counts, struct key_t);
 
 int count(struct pt_regs *ctx) {
-    if (!PT_REGS_PARM2(ctx))
+    if (!PT_REGS_PARM1(ctx))
         return 0;
 
     struct key_t key = {};
     u64 zero = 0, *val;
 
-    bpf_probe_read(&key.c, sizeof(key.c), (void *)PT_REGS_PARM2(ctx));
+    bpf_probe_read(&key.c, sizeof(key.c), (void *)PT_REGS_PARM1(ctx));
     val = counts.lookup_or_init(&key, &zero);
     (*val)++;
     return 0;

examples/tracing/strlen_snoop.py

@@ -26,7 +26,7 @@ pid = sys.argv[1]
 bpf_text = """
 #include <uapi/linux/ptrace.h>
 int printarg(struct pt_regs *ctx) {
-    if (!PT_REGS_PARM2(ctx))
+    if (!PT_REGS_PARM1(ctx))
         return 0;
 
     u32 pid = bpf_get_current_pid_tgid();
@@ -34,7 +34,7 @@ int printarg(struct pt_regs *ctx) {
         return 0;
 
     char str[80] = {};
-    bpf_probe_read(&str, sizeof(str), (void *)PT_REGS_PARM2(ctx));
+    bpf_probe_read(&str, sizeof(str), (void *)PT_REGS_PARM1(ctx));
     bpf_trace_printk("%s\\n", &str);
 
     return 0;

examples/tracing/sync_timing.py

+#!/usr/bin/python
+#
+# sync_timing.py    Trace time between syncs.
+#                   For Linux, uses BCC, eBPF. Embedded C.
+#
+# Written as a basic example of tracing time between events.
+#
+# Copyright 2016 Netflix, Inc.
+# Licensed under the Apache License, Version 2.0 (the "License")
+
+from __future__ import print_function
+from bcc import BPF
+
+# load BPF program
+b = BPF(text="""
+#include <uapi/linux/ptrace.h>
+#include <linux/blkdev.h>
+
+BPF_HASH(last);
+
+void do_trace(struct pt_regs *ctx) {
+    u64 ts, *tsp, delta, key = 0;
+
+    // attempt to read stored timestamp
+    tsp = last.lookup(&key);
+    if (tsp != 0) {
+        delta = bpf_ktime_get_ns() - *tsp;
+        if (delta < 1000000000) {
+            // output if time is less than 1 second
+            bpf_trace_printk("%d\\n", delta / 1000000);
+        }
+        last.delete(&key);
+    }
+
+    // update stored timestamp
+    ts = bpf_ktime_get_ns();
+    last.update(&key, &ts);
+}
+""")
+
+b.attach_kprobe(event="sys_sync", fn_name="do_trace")
+print("Tracing for quick sync's... Ctrl-C to end")
+
+# format output
+start = 0
+while 1:
+    (task, pid, cpu, flags, ts, ms) = b.trace_fields()
+    if start == 0:
+        start = ts
+    ts = ts - start
+    print("At time %.2f s: multiple syncs detected, last %s ms ago" % (ts, ms))