Merge pull request #1618 from rmanyari/tool-tcpsubnet

Tool tcpsubnet

README.md

@@ -82,6 +82,7 @@ pair of .c and .py files, and some are directories of files.
 #### Tools:
 <center><a href="images/bcc_tracing_tools_2017.png"><img src="images/bcc_tracing_tools_2017.png" border=0 width=700></a></center>
 
+
 - tools/[argdist](tools/argdist.py): Display function parameter values as a histogram or frequency count. [Examples](tools/argdist_example.txt).
 - tools/[bashreadline](tools/bashreadline.py): Print entered bash commands system wide. [Examples](tools/bashreadline_example.txt).
 - tools/[biolatency](tools/biolatency.py): Summarize block device I/O latency as a histogram. [Examples](tools/biolatency_example.txt).
@@ -140,6 +141,7 @@ pair of .c and .py files, and some are directories of files.
 - tools/[tcpconnlat](tools/tcpconnlat.py): Trace TCP active connection latency (connect()). [Examples](tools/tcpconnlat_example.txt).
 - tools/[tcplife](tools/tcplife.py): Trace TCP sessions and summarize lifespan. [Examples](tools/tcplife_example.txt).
 - tools/[tcpretrans](tools/tcpretrans.py): Trace TCP retransmits and TLPs. [Examples](tools/tcpretrans_example.txt).
+- tools/[tcpsubnet](tools/tcpsubnet.py): Summarize and aggregate TCP send by subnet. [Examples](tools/tcpsubnet_example.txt).
 - tools/[tcptop](tools/tcptop.py): Summarize TCP send/recv throughput by host. Top for TCP. [Examples](tools/tcptop_example.txt).
 - tools/[tcptracer](tools/tcptracer.py): Trace TCP established connections (connect(), accept(), close()). [Examples](tools/tcptracer_example.txt).
 - tools/[tplist](tools/tplist.py): Display kernel tracepoints or USDT probes and their formats. [Examples](tools/tplist_example.txt).

man/man8/tcpsubnet.8

+.TH tcpsubnet 8  "2018-03-01" "USER COMMANDS"
+.SH NAME
+tcpsubnet \- Summarize and aggregate IPv4 TCP traffic by subnet.
+.SH SYNOPSIS
+.B tcpsubnet [\-h] [\-v] [\--ebpf] [\-J] [\-f FORMAT] [\-i INTERVAL] [subnets]
+.SH DESCRIPTION
+This tool summarizes and aggregates IPv4 TCP sent to the subnets
+passed in argument and prints to stdout on a fixed interval.
+
+This uses dynamic tracing of kernel TCP send/receive functions, and will
+need to be updated to match kernel changes.
+
+The traced data is summarized in-kernel using a BPF map to reduce overhead.
+At very high TCP event rates, the overhead may still be measurable.
+See the OVERHEAD section for more details.
+
+Since this uses BPF, only the root user can use this tool.
+.SH REQUIREMENTS
+CONFIG_BPF and bcc.
+.SH OPTIONS
+.TP
+\-h
+Print USAGE message.
+.TP
+\-v
+Run in verbose mode. Will output subnet evaluation and the BPF program
+.TP
+\-J
+Format output in JSON.
+.TP
+\-i
+Interval between updates, seconds (default 1).
+.TP
+\-f
+Format output units. Supported values are bkmBKM. When using
+kmKM the output will be rounded to floor.
+.TP
+\--ebpf
+Prints the BPF program.
+.TP
+subnets
+Comma separated list of subnets. Traffic will be categorized
+in theses subnets. Order matters.
+(default 127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,0.0.0.0/0)
+.SH EXAMPLES
+.TP
+Summarize TCP traffic by the default subnets:
+#
+.B tcpsubnet
+.TP
+Summarize all TCP traffic:
+#
+.B tcpsubnet 0.0.0.0/0
+.TP
+Summarize all TCP traffic and output in JSON and Kb:
+#
+.B tcpsubnet -J -fk 0.0.0.0/0
+.SH FIELDS
+.TP
+(Standad output) Left hand side column:
+Subnet
+.TP
+(Standard output) Right hand side column:
+Aggregate traffic in units passed as argument
+.TP
+(JSON output) date
+Current date formatted in the system locale
+.TP
+(JSON output) time
+Current time formatted in the system locale
+.TP
+(JSON output) entries
+Map of subnets to aggregates. Values will be in format passed to -f
+.SH OVERHEAD
+This traces all tcp_sendmsg function calls in the TCP/IP stack.
+It summarizes data in-kernel to reduce overhead.
+A simple iperf test (v2.0.5) with the default values shows a loss
+of ~5% throughput. On 10 runs without tcpsubnet running the average
+throughput was 32.42Gb/s, with tcpsubnet enabled it was 31.26Gb/s.
+This is not meant to be used as a long running service. Use it
+for troubleshooting or for a controlled interval. As always,
+try it out in a test environment first.
+
+.SH SOURCE
+This is from bcc.
+.IP
+https://github.com/iovisor/bcc
+.PP
+Also look in the bcc distribution for a companion _examples.txt file containing
+example usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Rodrigo Manyari
+.SH INSPIRATION
+tcptop(8) by Brendan Gregg
+.SH SEE ALSO
+netlink(7)

tools/tcpsubnet.py

+#!/usr/bin/python
+# @lint-avoid-python-3-compatibility-imports
+#
+# tcpsubnet   Summarize TCP bytes sent to different subnets.
+#             For Linux, uses BCC, eBPF. Embedded C.
+#
+# USAGE: tcpsubnet [-h] [-v] [--ebpf] [-J] [-f FORMAT] [-i INTERVAL] [subnets]
+#
+# This uses dynamic tracing of kernel functions, and will need to be updated
+# to match kernel changes.
+#
+# This is an adaptation of tcptop from written by Brendan Gregg.
+#
+# WARNING: This traces all send at the TCP level, and while it
+# summarizes data in-kernel to reduce overhead, there may still be some
+# overhead at high TCP send/receive rates (eg, ~13% of one CPU at 100k TCP
+# events/sec. This is not the same as packet rate: funccount can be used to
+# count the kprobes below to find out the TCP rate). Test in a lab environment
+# first. If your send rate is low (eg, <1k/sec) then the overhead is
+# expected to be negligible.
+#
+# Copyright 2017 Rodrigo Manyari
+# Licensed under the Apache License, Version 2.0 (the "License")
+#
+# 03-Oct-2017   Rodrigo Manyari   Created this based on tcptop.
+# 13-Feb-2018   Rodrigo Manyari   Fix pep8 errors, some refactoring.
+# 05-Mar-2018   Rodrigo Manyari   Add date time to output.
+
+import argparse
+import json
+import logging
+import struct
+import socket
+from bcc import BPF
+from datetime import datetime as dt
+from time import sleep
+
+# arguments
+examples = """examples:
+    ./tcpsubnet                 # Trace TCP sent to the default subnets:
+                                # 127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,
+                                # 192.168.0.0/16,0.0.0.0/0
+    ./tcpsubnet -f K            # Trace TCP sent to the default subnets
+                                # aggregated in KBytes.
+    ./tcpsubnet 10.80.0.0/24    # Trace TCP sent to 10.80.0.0/24 only
+    ./tcpsubnet -J              # Format the output in JSON.
+"""
+
+default_subnets = "127.0.0.1/32,10.0.0.0/8," \
+    "172.16.0.0/12,192.168.0.0/16,0.0.0.0/0"
+
+parser = argparse.ArgumentParser(
+    description="Summarize TCP send and aggregate by subnet",
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    epilog=examples)
+parser.add_argument("subnets", help="comma separated list of subnets",
+    type=str, nargs="?", default=default_subnets)
+parser.add_argument("-v", "--verbose", action="store_true",
+    help="output debug statements")
+parser.add_argument("-J", "--json", action="store_true",
+    help="format output in JSON")
+parser.add_argument("--ebpf", action="store_true",
+    help=argparse.SUPPRESS)
+parser.add_argument("-f", "--format", default="B",
+    help="[bkmBKM] format to report: bits, Kbits, Mbits, bytes, " +
+    "KBytes, MBytes (default B)", choices=["b", "k", "m", "B", "K", "M"])
+parser.add_argument("-i", "--interval", default=1, type=int,
+    help="output interval, in seconds (default 1)")
+args = parser.parse_args()
+
+level = logging.INFO
+if args.verbose:
+    level = logging.DEBUG
+
+logging.basicConfig(level=level)
+
+logging.debug("Starting with the following args:")
+logging.debug(args)
+
+# args checking
+if int(args.interval) <= 0:
+    logging.error("Invalid interval, must be > 0. Exiting.")
+    exit(1)
+else:
+    args.interval = int(args.interval)
+
+# map of supported formats
+formats = {
+    "b": lambda x: (x * 8),
+    "k": lambda x: ((x * 8) / 1024),
+    "m": lambda x: ((x * 8) / pow(1024, 2)),
+    "B": lambda x: x,
+    "K": lambda x: x / 1024,
+    "M": lambda x: x / pow(1024, 2)
+}
+
+# Let's swap the string with the actual numeric value
+# once here so we don't have to do it on every interval
+formatFn = formats[args.format]
+
+# define the basic structure of the BPF program
+bpf_text = """
+#include <uapi/linux/ptrace.h>
+#include <net/sock.h>
+#include <bcc/proto.h>
+
+struct index_key_t {
+  u32 index;
+};
+
+BPF_HASH(ipv4_send_bytes, struct index_key_t);
+
+int kprobe__tcp_sendmsg(struct pt_regs *ctx, struct sock *sk,
+    struct msghdr *msg, size_t size)
+{
+    u16 family = sk->__sk_common.skc_family;
+    u64 *val, zero = 0;
+
+    if (family == AF_INET) {
+        u32 dst = sk->__sk_common.skc_daddr;
+        unsigned categorized = 0;
+        __SUBNETS__
+    }
+    return 0;
+}
+"""
+
+
+# Takes in a mask and returns the integer equivalent
+# e.g.
+# mask_to_int(8) returns 4278190080
+def mask_to_int(n):
+    return ((1 << n) - 1) << (32 - n)
+
+# Takes in a list of subnets and returns a list
+# of tuple-3 containing:
+# - The subnet info at index 0
+# - The addr portion as an int at index 1
+# - The mask portion as an int at index 2
+#
+# e.g.
+# parse_subnets([10.10.0.0/24]) returns
+# [
+#   ['10.10.0.0/24', 168427520, 4294967040],
+# ]
+def parse_subnets(subnets):
+    m = []
+    for s in subnets:
+        parts = s.split("/")
+        if len(parts) != 2:
+            msg = "Subnet [%s] is invalid, please refer to the examples." % s
+            raise ValueError(msg)
+        netaddr_int = 0
+        mask_int = 0
+        try:
+            netaddr_int = struct.unpack("!I", socket.inet_aton(parts[0]))[0]
+        except:
+            msg = ("Invalid net address in subnet [%s], " +
+                "please refer to the examples.") % s
+            raise ValueError(msg)
+        try:
+            mask_int = int(parts[1])
+        except:
+            msg = "Invalid mask in subnet [%s]. Mask must be an int" % s
+            raise ValueError(msg)
+        if mask_int < 0 or mask_int > 32:
+            msg = ("Invalid mask in subnet [%s]. Must be an " +
+                "int between 0 and 32.") % s
+            raise ValueError(msg)
+        mask_int = mask_to_int(int(parts[1]))
+        m.append([s, netaddr_int, mask_int])
+    return m
+
+def generate_bpf_subnets(subnets):
+    template = """
+        if (!categorized && (__NET_ADDR__ & __NET_MASK__) ==
+             (dst & __NET_MASK__)) {
+          struct index_key_t key = {.index = __POS__};
+          val = ipv4_send_bytes.lookup_or_init(&key, &zero);
+          categorized = 1;
+          (*val) += size;
+        }
+    """
+    bpf = ''
+    for i, s in enumerate(subnets):
+        branch = template
+        branch = branch.replace("__NET_ADDR__", str(socket.htonl(s[1])))
+        branch = branch.replace("__NET_MASK__", str(socket.htonl(s[2])))
+        branch = branch.replace("__POS__", str(i))
+        bpf += branch
+    return bpf
+
+subnets = []
+if args.subnets:
+    subnets = args.subnets.split(",")
+
+subnets = parse_subnets(subnets)
+
+logging.debug("Packets are going to be categorized in the following subnets:")
+logging.debug(subnets)
+
+bpf_subnets = generate_bpf_subnets(subnets)
+
+# initialize BPF
+bpf_text = bpf_text.replace("__SUBNETS__", bpf_subnets)
+
+logging.debug("Done preprocessing the BPF program, " +
+        "this is what will actually get executed:")
+logging.debug(bpf_text)
+
+if args.ebpf:
+    print(bpf_text)
+    exit()
+
+b = BPF(text=bpf_text)
+
+ipv4_send_bytes = b["ipv4_send_bytes"]
+
+if not args.json:
+    print("Tracing... Output every %d secs. Hit Ctrl-C to end" % args.interval)
+
+# output
+exiting = 0
+while (1):
+
+    try:
+        sleep(args.interval)
+    except KeyboardInterrupt:
+        exiting = 1
+
+    # IPv4:  build dict of all seen keys
+    keys = ipv4_send_bytes
+    for k, v in ipv4_send_bytes.items():
+        if k not in keys:
+            keys[k] = v
+
+    # to hold json data
+    data = {}
+
+    # output
+    now = dt.now()
+    data['date'] = now.strftime('%x')
+    data['time'] = now.strftime('%X')
+    data['entries'] = {}
+    if not args.json:
+        print(now.strftime('[%x %X]'))
+    for k, v in reversed(sorted(keys.items(), key=lambda keys: keys[1].value)):
+        send_bytes = 0
+        if k in ipv4_send_bytes:
+            send_bytes = int(ipv4_send_bytes[k].value)
+        subnet = subnets[k.index][0]
+        send = formatFn(send_bytes)
+        if args.json:
+            data['entries'][subnet] = send
+        else:
+            print("%-21s %6d" % (subnet, send))
+
+    if args.json:
+        print(json.dumps(data))
+
+    ipv4_send_bytes.clear()
+
+    if exiting:
+        exit(0)

tools/tcpsubnet_example.txt

+Demonstrations of tcpsubnet, the Linux eBPF/bcc version.
+
+
+tcpsubnet summarizes throughput by destination subnet.
+It works only for IPv4. Eg:
+
+# tcpsubnet
+Tracing... Output every 1 secs. Hit Ctrl-C to end
+[03/05/18 22:32:47]
+127.0.0.1/32               8
+[03/05/18 22:32:48]
+[03/05/18 22:32:49]
+[03/05/18 22:32:50]
+[03/05/18 22:32:51]
+[03/05/18 22:32:52]
+127.0.0.1/32              10
+[03/05/18 22:32:53]
+
+This example output shows the number of bytes sent to 127.0.0.1/32 (the
+loopback interface). For demo purposes, I set netcat listening on port
+8080, connected to it and sent the following payloads.
+
+# nc 127.0.0.1 8080
+1111111
+111111111
+
+The first line sends 7 digits plus the null character (8 bytes)
+The second line sends 9 digits plus the null character (10 bytes)
+
+Notice also, how tcpsubnet prints a header line with the current date
+and time formatted in the current locale.
+
+Try it yourself to get a feeling of how tcpsubnet works.
+
+By default, tcpsubnet will categorize traffic in the following subnets:
+
+- 127.0.0.1/32
+- 10.0.0.0/8
+- 172.16.0.0/12
+- 192.168.0.0/16
+- 0.0.0.0/0
+
+The last subnet is a catch-all. In other words, anything that doesn't
+match the first 4 defaults will be categorized under 0.0.0.0/0
+You can change this default behavoir by passing a comma separated list
+of subnets. Let's say we would like to know how much traffic we
+are sending to github.com. We first find out what IPs github.com resolves
+to, Eg:
+
+# dig +short github.com
+192.30.253.112
+192.30.253.113
+
+With this information, we can come up with a reasonable range of IPs
+to monitor, Eg:
+ 
+# tcpsubnet.py 192.30.253.110/27,0.0.0.0/0
+Tracing... Output every 1 secs. Hit Ctrl-C to end
+[03/05/18 22:38:58]
+0.0.0.0/0               5780
+192.30.253.110/27       2205
+[03/05/18 22:38:59]
+0.0.0.0/0               2036
+192.30.253.110/27       1183
+[03/05/18 22:39:00]
+[03/05/18 22:39:01]
+192.30.253.110/27      12537
+
+If we would like to be more accurate, we can use the two IPs returned
+by dig, Eg:
+
+# tcpsubnet 192.30.253.113/32,192.130.253.112/32,0.0.0.0/0
+Tracing... Output every 1 secs. Hit Ctrl-C to end
+[03/05/18 22:42:56]
+0.0.0.0/0               1177
+192.30.253.113/32        910
+[03/05/18 22:42:57]
+0.0.0.0/0              48704
+192.30.253.113/32        892
+[03/05/18 22:42:58]
+192.30.253.113/32        891
+0.0.0.0/0                858
+[03/05/18 22:42:59]
+0.0.0.0/0              11159
+192.30.253.113/32        894
+[03/05/18 22:43:00]
+0.0.0.0/0              60601
+
+NOTE: When used in production, it is expected that you will have full
+information about your network topology. In which case you won't need
+to approximate subnets nor need to put individual IP addresses like
+we just did.
+
+Notice that the order of the subnet matters. Say, we put 0.0.0.0/0 as
+the first element of the list and 192.130.253.112/32 as the second, all the
+traffic going to 192.130.253.112/32 will have been categorized in
+0.0.0.0/0 as 192.130.253.112/32 is contained in 0.0.0.0/0.
+
+The default ouput unit is bytes. You can change it by using the
+-f [--format] flag. tcpsubnet uses the same flags as iperf for the unit
+format and adds mM. When using kmKM, the output will be rounded to floor.
+Eg:
+
+# tcpsubnet -fK 0.0.0.0/0
+[03/05/18 22:44:04]
+0.0.0.0/0                  1
+[03/05/18 22:44:05]
+0.0.0.0/0                  5
+[03/05/18 22:44:06]
+0.0.0.0/0                 31
+
+Just like the majority of the bcc tools, tcpsubnet supports -i and --ebpf
+
+It also supports -v [--verbose] which gives useful debugging information
+on how the subnets are evaluated and the BPF program is constructed.
+
+Last but not least, it supports -J [--json] to print the output in
+JSON format. This is handy if you're calling tcpsubnet from another
+program (say a nodejs server) and would like to have a structured stdout.
+The output in JSON format will also include the date and time.
+Eg:
+
+# tcpsubnet -J -fK 192.130.253.110/27,0.0.0.0/0
+{"date": "03/05/18", "entries": {"0.0.0.0/0": 2}, "time": "22:46:27"}
+{"date": "03/05/18", "entries": {}, "time": "22:46:28"}
+{"date": "03/05/18", "entries": {}, "time": "22:46:29"}
+{"date": "03/05/18", "entries": {}, "time": "22:46:30"}
+{"date": "03/05/18", "entries": {"192.30.253.110/27": 0}, "time": "22:46:31"}
+{"date": "03/05/18", "entries": {"192.30.253.110/27": 1}, "time": "22:46:32"}
+{"date": "03/05/18", "entries": {"192.30.253.110/27": 18}, "time": "22:46:32"}
+
+
+USAGE:
+
+# ./tcpsubnet -h
+usage: tcpsubnet.py [-h] [-v] [-J] [-f {b,k,m,B,K,M}] [-i INTERVAL] [subnets]
+
+Summarize TCP send and aggregate by subnet
+
+positional arguments:
+  subnets               comma separated list of subnets
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -v, --verbose         output debug statements
+  -J, --json            format output in JSON
+  -f {b,k,m,B,K,M}, --format {b,k,m,B,K,M}
+                        [bkmBKM] format to report: bits, Kbits, Mbits, bytes,
+                        KBytes, MBytes (default B)
+  -i INTERVAL, --interval INTERVAL
+                        output interval, in seconds (default 1)
+
+examples:
+    ./tcpsubnet                 # Trace TCP sent to the default subnets:
+                                # 127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,
+                                # 192.168.0.0/16,0.0.0.0/0
+    ./tcpsubnet -f K            # Trace TCP sent to the default subnets
+                                # aggregated in KBytes.
+    ./tcpsubnet 10.80.0.0/24    # Trace TCP sent to 10.80.0.0/24 only
+    ./tcpsubnet -J              # Format the output in JSON.
+