Merge pull request #274 from brendangregg/master

tcpconnect for IPv4 and IPv6, and make tcpv4connect a trimmed example

README.md

@@ -59,6 +59,7 @@ Examples:
 - examples/[bitehist.py](examples/bitehist.py) examples/[bitehist.c](examples/bitehist.c): Block I/O size histogram. [Examples](examples/bitehist_example.txt).
 - examples/[disksnoop.py](examples/disksnoop.py) examples/[disksnoop.c](examples/disksnoop.c): Trace block device I/O latency. [Examples](examples/disksnoop_example.txt).
 - examples/[hello_world.py](examples/hello_world.py): Prints "Hello, World!" for new processes.
+- examples/[tcpv4connect](examples/tcpv4connect): Trace TCP IPv4 active connections. [Examples](examples/tcpv4connect_example.txt).
 - examples/[trace_fields.py](examples/trace_fields.py): Simple example of printing fields from traced events.
 - examples/[vfsreadlat.py](examples/vfsreadlat.py) examples/[vfsreadlat.c](examples/vfsreadlat.c): VFS read latency distribution. [Examples](examples/vfsreadlat_example.txt).
 
@@ -73,7 +74,7 @@ Tools:
 - tools/[pidpersec](tools/pidpersec): Count new processes (via fork). [Examples](tools/pidpersec_example.txt).
 - tools/[syncsnoop](tools/syncsnoop): Trace sync() syscall. [Examples](tools/syncsnoop_example.txt).
 - tools/[tcpaccept](tools/tcpaccept): Trace TCP passive connections (accept()). [Examples](tools/tcpaccept_example.txt).
-- tools/[tcpv4connect](tools/tcpv4connect): Trace TCP IPv4 active connections (connect()). [Examples](tools/tcpv4connect_example.txt).
+- tools/[tcpconnect](tools/tcpconnect): Trace TCP active connections (connect()). [Examples](tools/tcpconnect_example.txt).
 - tools/[vfscount](tools/vfscount) tools/[vfscount.c](tools/vfscount.c): Count VFS calls. [Examples](tools/vfscount_example.txt).
 - tools/[vfsstat](tools/vfsstat) tools/[vfsstat.c](tools/vfsstat.c): Count some VFS calls, with column output. [Examples](tools/vfsstat_example.txt).
 

tools/tcpv4connect → examples/tcpv4connect

@@ -5,31 +5,17 @@
 #
 # USAGE: tcpv4connect [-h] [-t] [-p PID]
 #
+# This is provided as a basic example of TCP connection & socket tracing.
+#
+# All IPv4 connection attempts are traced, even if they ultimately fail.
+#
 # Copyright (c) 2015 Brendan Gregg.
 # Licensed under the Apache License, Version 2.0 (the "License")
 #
-# 25-Sep-2015	Brendan Gregg	Created this.
+# 15-Oct-2015	Brendan Gregg	Created this.
 
 from __future__ import print_function
 from bcc import BPF
-import argparse
-
-# arguments
-examples = """examples:
-    ./tcpv4connect           # trace all TCP IPv4 connect()s
-    ./tcpv4connect -t        # include timestamps
-    ./tcpv4connect -p 181    # only trace PID 181
-"""
-parser = argparse.ArgumentParser(
-	description="Trace TCP IPv4 connects",
-	formatter_class=argparse.RawDescriptionHelpFormatter,
-	epilog=examples)
-parser.add_argument("-t", "--timestamp", action="store_true",
-	help="include timestamp on output")
-parser.add_argument("-p", "--pid",
-	help="trace this PID only")
-args = parser.parse_args()
-debug = 0
 
 # define BPF program
 bpf_text = """
@@ -42,7 +28,6 @@ BPF_HASH(currsock, u32, struct sock *);
 int kprobe__tcp_v4_connect(struct pt_regs *ctx, struct sock *sk)
 {
 	u32 pid = bpf_get_current_pid_tgid();
-	FILTER
 
 	// stash the sock ptr for lookup on return
 	currsock.update(&pid, &sk);
@@ -62,8 +47,8 @@ int kretprobe__tcp_v4_connect(struct pt_regs *ctx)
 	}
 
 	if (ret != 0) {
-		// failed to send SYNC packet, socket __sk_common.{skc_rcv_saddr, ...}
-		// may not be populated properly.
+		// failed to send SYNC packet, may not have populated
+		// socket __sk_common.{skc_rcv_saddr, ...}
 		currsock.delete(&pid);
 		return 0;
 	}
@@ -85,26 +70,13 @@ int kretprobe__tcp_v4_connect(struct pt_regs *ctx)
 }
 """
 
-# code substitutions
-if args.pid:
-	bpf_text = bpf_text.replace('FILTER',
-	    'if (pid != %s) { return 0; }' % args.pid)
-else:
-	bpf_text = bpf_text.replace('FILTER', '')
-if debug:
-	print(bpf_text)
-
 # initialize BPF
 b = BPF(text=bpf_text)
 
 # header
-if args.timestamp:
-	print("%-9s" % ("TIME(s)"), end="")
 print("%-6s %-12s %-16s %-16s %-4s" % ("PID", "COMM", "SADDR", "DADDR",
     "DPORT"))
 
-start_ts = 0
-
 def inet_ntoa(addr):
 	dq = ''
 	for i in range(0, 4):
@@ -119,10 +91,6 @@ while 1:
 	(task, pid, cpu, flags, ts, msg) = b.trace_fields()
 	(saddr_hs, daddr_hs, dport_s) = msg.split(" ")
 
-	if args.timestamp:
-		if start_ts == 0:
-			start_ts = ts
-		print("%-9.3f" % (ts - start_ts), end="")
 	print("%-6d %-12.12s %-16s %-16s %-4s" % (pid, task,
 	    inet_ntoa(int(saddr_hs, 16)),
 	    inet_ntoa(int(daddr_hs, 16)),

tools/tcpv4connect_example.txt → examples/tcpv4connect_example.txt

 Demonstrations of tcpv4connect, the Linux eBPF/bcc version.
 
 
-This tool traces the kernel function performing active TCP IPv4 connections
+This example traces the kernel function performing active TCP IPv4 connections
 (eg, via a connect() syscall; accept() are passive connections). Some example
 output (IP addresses changed to protect the innocent):
 
@@ -19,35 +19,5 @@ The overhead of this tool should be negligible, since it is only tracing the
 kernel function performing a connect. It is not tracing every packet and then
 filtering.
 
-
-The -t option prints a timestamp column:
-
-# ./tcpv4connect -t
-TIME(s)  PID    COMM         SADDR            DADDR            DPORT
-31.871   2482   local_agent  10.103.219.236   10.251.148.38    7001
-31.874   2482   local_agent  10.103.219.236   10.101.3.132     7001
-31.878   2482   local_agent  10.103.219.236   10.171.133.98    7101
-90.917   2482   local_agent  10.103.219.236   10.251.148.38    7001
-90.928   2482   local_agent  10.103.219.236   10.102.64.230    7001
-90.938   2482   local_agent  10.103.219.236   10.115.167.169   7101
-
-The output shows some periodic connections (or attempts) from a "local_agent"
-process to various other addresses. A few connections occur every minute.
-
-
-USAGE message:
-
-# ./tcpv4connect -h
-usage: tcpv4connect [-h] [-t] [-p PID]
-
-Trace TCP IPv4 connects
-
-optional arguments:
-  -h, --help         show this help message and exit
-  -t, --timestamp    include timestamp on output
-  -p PID, --pid PID  trace this PID only
-
-examples:
-    ./tcpv4connect           # trace all TCP IPv4 connect()s
-    ./tcpv4connect -t        # include timestamps
-    ./tcpv4connect -p 181    # only trace PID 181
+This is provided as a basic example of TCP tracing. See tools/tcpconnect for a
+more featured version of this example (a tool).

man/man8/tcpaccept.8

@@ -85,4 +85,4 @@ Unstable - in development.
 .SH AUTHOR
 Brendan Gregg
 .SH SEE ALSO
-tcpv4connect(8), funccount(8), tcpdump(8)
+tcpconnect(8), funccount(8), tcpdump(8)

man/man8/tcpv4connect.8 → man/man8/tcpconnect.8

-.TH tcpv4connect 8  "2015-08-25" "USER COMMANDS"
+.TH tcpconnect 8  "2015-08-25" "USER COMMANDS"
 .SH NAME
-tcpv4connect \- Trace TCP IPv4 active connections (connect()). Uses Linux eBPF/bcc.
+tcpconnect \- Trace TCP active connections (connect()). Uses Linux eBPF/bcc.
 .SH SYNOPSIS
-.B tcpv4connect [\-h] [\-t] [\-x] [\-p PID]
+.B tcpconnect [\-h] [\-t] [\-x] [\-p PID]
 .SH DESCRIPTION
-This tool traces active TCP IPv4 connections (eg, via a connect() syscall;
+This tool traces active TCP connections (eg, via a connect() syscall;
 accept() are passive connections). This can be useful for general
 troubleshooting to see what connections are initiated by the local server.
 
-This works by tracing the kernel tcp_v4_connect() function using dynamic
-tracing, and will need updating to match any changes to this function.
+All connection attempts are traced, even if they ultimately fail.
+
+This works by tracing the kernel tcp_v4_connect() and tcp_v6_connect() functions
+using dynamic tracing, and will need updating to match any changes to these
+functions.
 
 Since this uses BPF, only the root user can use this tool.
 .SH REQUIREMENTS
@@ -26,17 +29,17 @@ Include a timestamp column.
 Trace this process ID only (filtered in-kernel).
 .SH EXAMPLES
 .TP
-Trace all active TCP IPv4 connections:
+Trace all active TCP connections:
 #
-.B tcpv4connect
+.B tcpconnect
 .TP
-Trace all TCP IPv4 connects, and include timestamps:
+Trace all TCP connects, and include timestamps:
 #
-.B tcpv4connect \-t
+.B tcpconnect \-t
 .TP
 Trace PID 181 only:
 #
-.B tcpv4connect \-p 181
+.B tcpconnect \-p 181
 .SH FIELDS
 .TP
 TIME(s)
@@ -48,20 +51,25 @@ Process ID
 COMM
 Process name
 .TP
+IP
+IP address family (4 or 6)
+.TP
 SADDR
-IPv4 source address, as a dotted quad
+Source IP address. IPv4 as a dotted quad, IPv6 shows "..." then the last 4
+bytes (check for newer versions of this tool for the full address).
 .TP
 DADDR
-IPv4 destination address, as a dotted quad
+Destination IP address. IPv4 as a dotted quad, IPv6 shows "..." then the last 4
+bytes (check for newer versions of this tool for the full address).
 .TP
 DPORT
 Destination port
 .SH OVERHEAD
-This traces the kernel tcp_v4_connect function and prints output for each event.
-As the rate of this is generally expected to be low (< 1000/s), the overhead is
-also expected to be negligible. If you have an application that is calling a high
-rate of connects()s, such as a proxy server, then test and understand this
-overhead before use.
+This traces the kernel tcp_v[46]_connect functions and prints output for each
+event. As the rate of this is generally expected to be low (< 1000/s), the
+overhead is also expected to be negligible. If you have an application that
+is calling a high rate of connects()s, such as a proxy server, then test and
+understand this overhead before use.
 .SH SOURCE
 This is from bcc.
 .IP

tools/tcpconnect

+#!/usr/bin/python
+#
+# tcpconnect	Trace TCP connect()s.
+#		For Linux, uses BCC, eBPF. Embedded C.
+#
+# USAGE: tcpconnect [-h] [-t] [-p PID]
+#
+# All connection attempts are traced, even if they ultimately fail.
+#
+# Copyright (c) 2015 Brendan Gregg.
+# Licensed under the Apache License, Version 2.0 (the "License")
+#
+# 25-Sep-2015	Brendan Gregg	Created this.
+
+from __future__ import print_function
+from bcc import BPF
+import argparse
+
+# arguments
+examples = """examples:
+    ./tcpconnect           # trace all TCP connect()s
+    ./tcpconnect -t        # include timestamps
+    ./tcpconnect -p 181    # only trace PID 181
+"""
+parser = argparse.ArgumentParser(
+	description="Trace TCP connects",
+	formatter_class=argparse.RawDescriptionHelpFormatter,
+	epilog=examples)
+parser.add_argument("-t", "--timestamp", action="store_true",
+	help="include timestamp on output")
+parser.add_argument("-p", "--pid",
+	help="trace this PID only")
+args = parser.parse_args()
+debug = 0
+
+# define BPF program
+bpf_text = """
+#include <uapi/linux/ptrace.h>
+#include <net/sock.h>
+#include <bcc/proto.h>
+
+BPF_HASH(currsock, u32, struct sock *);
+
+int trace_connect_entry(struct pt_regs *ctx, struct sock *sk)
+{
+	u32 pid = bpf_get_current_pid_tgid();
+	FILTER
+
+	// stash the sock ptr for lookup on return
+	currsock.update(&pid, &sk);
+
+	return 0;
+};
+
+static int trace_connect_return(struct pt_regs *ctx, short ipver)
+{
+	int ret = ctx->ax;
+	u32 pid = bpf_get_current_pid_tgid();
+
+	struct sock **skpp;
+	skpp = currsock.lookup(&pid);
+	if (skpp == 0) {
+		return 0;	// missed entry
+	}
+
+	if (ret != 0) {
+		// failed to send SYNC packet, may not have populated
+		// socket __sk_common.{skc_rcv_saddr, ...}
+		currsock.delete(&pid);
+		return 0;
+	}
+
+	// pull in details
+	struct sock *skp = *skpp;
+	u32 saddr = 0, daddr = 0;
+	u16 dport = 0;
+	bpf_probe_read(&dport, sizeof(dport), &skp->__sk_common.skc_dport);
+	if (ipver == 4) {
+		bpf_probe_read(&saddr, sizeof(saddr),
+		    &skp->__sk_common.skc_rcv_saddr);
+		bpf_probe_read(&daddr, sizeof(daddr),
+		    &skp->__sk_common.skc_daddr);
+
+		// output
+		bpf_trace_printk("4 %x %x %d\\n", saddr, daddr, ntohs(dport));
+	} else /* 6 */ {
+		// just grab the last 4 bytes for now
+		bpf_probe_read(&saddr, sizeof(saddr),
+		    &skp->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32[3]);
+		bpf_probe_read(&daddr, sizeof(daddr),
+		    &skp->__sk_common.skc_v6_daddr.in6_u.u6_addr32[3]);
+
+		// output and flip byte order of addresses
+		bpf_trace_printk("6 %x %x %d\\n", bpf_ntohl(saddr),
+		    bpf_ntohl(daddr), ntohs(dport));
+	}
+
+	currsock.delete(&pid);
+
+	return 0;
+}
+
+int trace_connect_v4_return(struct pt_regs *ctx)
+{
+	return trace_connect_return(ctx, 4);
+}
+
+int trace_connect_v6_return(struct pt_regs *ctx)
+{
+	return trace_connect_return(ctx, 6);
+}
+"""
+
+# code substitutions
+if args.pid:
+	bpf_text = bpf_text.replace('FILTER',
+	    'if (pid != %s) { return 0; }' % args.pid)
+else:
+	bpf_text = bpf_text.replace('FILTER', '')
+if debug:
+	print(bpf_text)
+
+# initialize BPF
+b = BPF(text=bpf_text)
+b.attach_kprobe(event="tcp_v4_connect", fn_name="trace_connect_entry")
+b.attach_kprobe(event="tcp_v6_connect", fn_name="trace_connect_entry")
+b.attach_kretprobe(event="tcp_v4_connect", fn_name="trace_connect_v4_return")
+b.attach_kretprobe(event="tcp_v6_connect", fn_name="trace_connect_v6_return")
+
+# header
+if args.timestamp:
+	print("%-9s" % ("TIME(s)"), end="")
+print("%-6s %-12s %-2s %-16s %-16s %-4s" % ("PID", "COMM", "IP", "SADDR",
+    "DADDR", "DPORT"))
+
+start_ts = 0
+
+def inet_ntoa(addr):
+	dq = ''
+	for i in range(0, 4):
+		dq = dq + str(addr & 0xff)
+		if (i != 3):
+			dq = dq + '.'
+		addr = addr >> 8
+	return dq
+
+# format output
+while 1:
+	(task, pid, cpu, flags, ts, msg) = b.trace_fields()
+	(ip_s, saddr_hs, daddr_hs, dport_s) = msg.split(" ")
+
+	if args.timestamp:
+		if start_ts == 0:
+			start_ts = ts
+		print("%-9.3f" % (ts - start_ts), end="")
+	print("%-6d %-12.12s %-2s %-16s %-16s %-4s" % (pid, task, ip_s,
+	    inet_ntoa(int(saddr_hs, 16)) if ip_s == "4" else "..." + saddr_hs,
+	    inet_ntoa(int(daddr_hs, 16)) if ip_s == "4" else "..." + daddr_hs,
+	    dport_s))

tools/tcpconnect_example.txt

+Demonstrations of tcpconnect, the Linux eBPF/bcc version.
+
+
+This tool traces the kernel function performing active TCP connections
+(eg, via a connect() syscall; accept() are passive connections). Some example
+output (IP addresses changed to protect the innocent):
+
+# ./tcpconnect 
+PID    COMM         IP SADDR            DADDR            DPORT
+1479   telnet       4  127.0.0.1        127.0.0.1        23  
+1469   curl         4  10.201.219.236   54.245.105.25    80  
+1469   curl         4  10.201.219.236   54.67.101.145    80  
+11072  ssh          6  ...fe8203ac      ...fe82abcd      22  
+
+This output shows four connections, one from a "telnet" process, two from
+"curl", and one from "ssh". The output details shows the IP version, source
+address, destination address, and destination port. This traces attempted
+connections: these may have failed.
+
+IPv4 addresses are printed as dotted quads. Only the last 4 bytes of IPv6
+addresses are printed for now (check for updated versions of this tool).
+
+The overhead of this tool should be negligible, since it is only tracing the
+kernel functions performing connect. It is not tracing every packet and then
+filtering.
+
+
+The -t option prints a timestamp column:
+
+# ./tcpconnect -t
+TIME(s)  PID    COMM         IP SADDR            DADDR            DPORT
+31.871   2482   local_agent  4  10.103.219.236   10.251.148.38    7001
+31.874   2482   local_agent  4  10.103.219.236   10.101.3.132     7001
+31.878   2482   local_agent  4  10.103.219.236   10.171.133.98    7101
+90.917   2482   local_agent  4  10.103.219.236   10.251.148.38    7001
+90.928   2482   local_agent  4  10.103.219.236   10.102.64.230    7001
+90.938   2482   local_agent  4  10.103.219.236   10.115.167.169   7101
+
+The output shows some periodic connections (or attempts) from a "local_agent"
+process to various other addresses. A few connections occur every minute.
+
+
+USAGE message:
+
+# ./tcpconnect -h
+usage: tcpconnect [-h] [-t] [-p PID]
+
+Trace TCP connects
+
+optional arguments:
+  -h, --help         show this help message and exit
+  -t, --timestamp    include timestamp on output
+  -p PID, --pid PID  trace this PID only
+
+examples:
+    ./tcpconnect           # trace all TCP connect()s
+    ./tcpconnect -t        # include timestamps
+    ./tcpconnect -p 181    # only trace PID 181