1. 24 May, 2016 2 commits
    • Andrew Birchall's avatar
      7f0a6f80
    • Andrew Birchall's avatar
      Add user space stack traces to offcputime · 1f202e7b
      Andrew Birchall authored
      Summary:
      * generalize `KernelSymbolCache` into `SymbolCache` which supports user/kernel space symbols
      * create `BPF.sym()` for resolving user/kernel space symbols
      * `_ksym_cache` => `_sym_caches` and create `BPF._sym_cache()` to leverage the symbol caches
      * update `tools/offcputime.py` to print user space stack traces
      
      Test Plan:
      ```
      dev[bcc](abirchall_next): ~/bcc_run_tool.sh offcputime -U -k -f 1
      ERROR: Displaying user stacks for kernel threads doesn't make sense.
      devbig680[bcc](abirchall_next): ~/bcc_run_tool.sh offcputime -K -f 1 | grep python2 | head -n 1
      python2.7;system_call_fastpath;sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule 19
      dev[bcc](abirchall_next): ~/bcc_run_tool.sh offcputime -U -f 1 | grep python2 | head -n 1
      python2.7;clone;start_thread;t_bootstrap;PyEval_CallObjectWithKeywords;PyObject_Call;instancemethod_call;PyObject_Call;function_call;PyEval_EvalCodeEx;PyEval_EvalFrameEx;PyObject_Call;function_call;PyEval_EvalCodeEx;PyEval_EvalFrameEx;time_sleep;PyEval_RestoreThread 5
      devbig680[bcc](abirchall_next): ~/bcc_run_tool.sh offcputime -f 1 | grep python2 | head -n 1
      python2.7;system_call_fastpath;sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule;[unknown];__libc_start_main;Py_Main;RunModule;PyObject_Call;function_call;PyEval_EvalCodeEx;PyEval_EvalFrameEx;PyEval_EvalFrameEx;PyEval_EvalCodeEx;PyEval_EvalFrameEx;lock_PyThread_acquire_lock;PyEval_RestoreThread 39
      ```
      1f202e7b
  2. 23 May, 2016 1 commit
  3. 20 May, 2016 2 commits
  4. 19 May, 2016 1 commit
  5. 17 May, 2016 3 commits
    • Brendan Gregg's avatar
      Merge pull request #543 from abirchall/abirchall_release · c7f317de
      Brendan Gregg authored
      add --kernel-threads-only to tools/offcputime
      c7f317de
    • Andrew Birchall's avatar
      add --kernel-threads-only to tools/offcputime · 582b5dd2
      Andrew Birchall authored
      Summary:
      Adds `--kernel-threads-only` arg
      The kernel-threads-only arg is exclusive with pid/user-threads-only via `parser.add_mutually_exclusive_group`.
      The output message now indicates what we are tracing (pid/user threads/kernel threads/all threads).
      Removed the --verbose arg (unused).
      
      Test Plan:
      Run with combinations of the args; validate output looks sane:
      
      // test mutually exclusive group
      ```
      devbig680[bcc](abirchall_dev): ~/bcc_run_tool.sh offcputime -k -u 1
      [Running] /data/users/abirchall/bcc/tools/offcputime.py -k -u 1
      usage: offcputime.py [-h] [-p PID | -k | -u] [-v] [-f] [duration]
      offcputime.py: error: argument -u/--user-threads-only: not allowed with argument -k/--kernel-threads-only
      ```
      
      // kernel threads only
      ```
      devbig680[bcc](abirchall_dev): ~/bcc_run_tool.sh offcputime -f -k 1
      [Running] /data/users/abirchall/bcc/tools/offcputime.py -f -k 1
      swapper/21;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 11
      swapper/16;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 19
      swapper/22;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 20
      swapper/31;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 20
      swapper/23;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 67
      swapper/25;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 74
      ...
      ```
      `~/bcc_run_tool.sh offcputime -f --kernel-threads-only 1`
      
      // user threads only
      `~/bcc_run_tool.sh offcputime -f --user-threads-only 1`
      `~/bcc_run_tool.sh offcputime -f -u 1`
      
      // specific pid
      `~/bcc_run_tool.sh offcputime -f -p $(pidof hphpi) 1`
      ```
      devbig680[bcc](abirchall_dev): ~/bcc_run_tool.sh offcputime --pid $(pidof mcrouter) 10 | head
      [Running] /data/users/abirchall/bcc/tools/offcputime.py --pid 95929 10
      Tracing off-CPU time (us) of PID 95929 by kernel stack for 10 secs.
      ```
      
      Note that this last case (specific PID) doesn't appear to be working; I can debug that in a follow up commit.
      582b5dd2
    • 4ast's avatar
      Merge pull request #533 from abirchall/abirchall_dev · 09534d89
      4ast authored
      handle ENOMEM in tools/offcputime
      09534d89
  6. 16 May, 2016 4 commits
    • Andrew Birchall's avatar
      move ENOMEM comment; create --stack-storage-size arg; misc cleanup · 47d871f0
      Andrew Birchall authored
      Summary:
      * move ENOMEM comment from tools/offcputime.py to src/cc/export/helpers.h
      * create --stack-storage-size arg to allow the user to set the stack storage size
          * requires a positive_nonzero_int type checking function for argparse
      * clean up the rest of the args a bit
          * use parser.add_mutually_exclusive_group instead of manually handling the
            mutual exclusion
          * use positive_nonzero_int for duration arg
          * use positive_int for pid arg
      * only print a warning about increasing the storage size if at least one of the
        get_stackid errors was a -ENOMEM
      * remove the debug param (we can add this manually when testing)
      * fix a bug where all processes are traced when specifying pid of 0
      
      Test Plan:
      ```
      devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime --help
      [Running] /data/users/abirchall/bcc/tools/offcputime.py --help
      usage: offcputime.py [-h] [-u | -p PID] [-v] [-f]
                           [--stack-storage-size STACK_STORAGE_SIZE]
                           [duration]
      
      Summarize off-CPU time by kernel stack trace
      
      positional arguments:
        duration              duration of trace, in seconds
      
      optional arguments:
        -h, --help            show this help message and exit
        -u, --useronly        user threads only (no kernel threads)
        -p PID, --pid PID     trace this PID only
        -v, --verbose         show raw addresses
        -f, --folded          output folded format
        --stack-storage-size STACK_STORAGE_SIZE
                              the number of unique stack traces that can be stored
                              and displayed
      
      examples:
          ./offcputime             # trace off-CPU stack time until Ctrl-C
          ./offcputime 5           # trace for 5 seconds only
          ./offcputime -f 5        # 5 seconds, and output in folded format
          ./offcputime -u          # don't include kernel threads (user only)
          ./offcputime -p 185      # trace fo PID 185 only
      devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime -f 0
      [Running] /data/users/abirchall/bcc/tools/offcputime.py -f 0
      usage: offcputime.py [-h] [-u | -p PID] [-v] [-f]
                           [--stack-storage-size STACK_STORAGE_SIZE]
                           [duration]
      offcputime.py: error: argument duration: must be positive and nonzero
      devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime -f -1
      [Running] /data/users/abirchall/bcc/tools/offcputime.py -f -1
      usage: offcputime.py [-h] [-u | -p PID] [-v] [-f]
                           [--stack-storage-size STACK_STORAGE_SIZE]
                           [duration]
      offcputime.py: error: argument duration: must be positive and nonzero
      devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime --stack-storage-size 0 -f 1
      [Running] /data/users/abirchall/bcc/tools/offcputime.py --stack-storage-size 0 -f 1
      usage: offcputime.py [-h] [-u | -p PID] [-v] [-f]
                           [--stack-storage-size STACK_STORAGE_SIZE]
                           [duration]
      offcputime.py: error: argument --stack-storage-size: must be positive and nonzero
      devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime --stack-storage-size 1 -f 1
      [Running] /data/users/abirchall/bcc/tools/offcputime.py --stack-storage-size 1 -f 1
      swapper/27;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 7
      swapper/29;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 102
      swapper/25;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 113
      swapper/26;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 138
      swapper/28;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 164
      swapper/30;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 177
      swapper/24;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 980
      swapper/31;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1556
      swapper/1;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2038
      swapper/3;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2190
      swapper/2;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2235
      swapper/15;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2260
      swapper/13;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2450
      swapper/6;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2646
      swapper/12;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2675
      swapper/11;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2769
      swapper/8;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2885
      swapper/4;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 3134
      swapper/10;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 3416
      swapper/14;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 3702
      swapper/7;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 4343
      swapper/9;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 5397
      swapper/5;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 6836
      swapper/17;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 13792
      swapper/19;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 14488
      swapper/22;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 15298
      swapper/18;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 15735
      swapper/20;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 17333
      swapper/16;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 17645
      swapper/23;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 17681
      swapper/21;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 18607
      WARNING: 92 stack traces could not be displayed. Consider increasing --stack-storage-size.
      47d871f0
    • Andrew Birchall's avatar
      handle ENOMEM in tools/offcputime · ee7e5b46
      Andrew Birchall authored
      Summary:
      BPF_STACK_TRACE(_name, _size) will allocate space for _size stack traces
      (see https://github.com/torvalds/linux/blob/master/kernel/bpf/stackmap.c#L30-L50).
      
      If we've already used all of this space, subsequent calls to bpf_get_stackid()
      will return -ENOMEM (see https://github.com/torvalds/linux/blob/master/kernel/bpf/stackmap.c#L173-L176).
      
      This causes our BPF bytecode to store this value in key_t.stack_id and
      subsequently causes our python application to crash due to a KeyError when
      invoking stack_traces.walk(k.stack_id).
      
      Let's avoid calling stack_traces.walk(k.stack_id) with back stackid's
      
      Test Plan:
      Run offcputime.py in an extreme case; with space for only a single stack trace
      ```
      devbig680[bcc](tools): sed_in_file 's/BPF_STACK_TRACE(stack_traces, 1024)/BPF_STACK_TRACE(stack_traces, 1)/' tools/offcputime.py && \
      > ~/bcc_run_tool.sh offcputime -f 5; \
      > git reset --hard HEAD
      swapper/30;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 496
      swapper/26;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 553
      swapper/28;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 604
      swapper/31;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 692
      swapper/23;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 713
      swapper/18;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 919
      swapper/16;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1051
      swapper/20;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1056
      swapper/21;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1585
      swapper/24;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1597
      swapper/27;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1610
      swapper/17;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1674
      swapper/22;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2390
      swapper/25;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2574
      swapper/19;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2589
      swapper/29;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 8428
      swapper/8;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 15272
      swapper/15;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 15591
      swapper/11;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 17934
      swapper/9;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 18100
      swapper/14;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 18266
      swapper/10;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 20124
      swapper/12;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 20887
      swapper/13;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 23453
      swapper/3;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 27296
      swapper/5;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 29094
      swapper/6;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 29799
      swapper/7;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 31522
      swapper/1;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 32269
      swapper/4;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 35585
      swapper/2;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 37627
      WARNING: 249 stack traces could not be displayed. Consider increasing stack trace storage size.
      HEAD is now at d3365e9 [RFC] handle ENOMEM in tools/offcputime`
      ```
      ee7e5b46
    • 4ast's avatar
      Merge pull request #542 from mbertrone/master · 50a74246
      4ast authored
      http_filter fixes
      50a74246
    • Bertrone Matteo's avatar
  7. 13 May, 2016 3 commits
  8. 12 May, 2016 3 commits
  9. 11 May, 2016 5 commits
  10. 06 May, 2016 13 commits
  11. 05 May, 2016 3 commits
    • 4ast's avatar
      Merge pull request #530 from iovisor/bblanco_dev · 7575c64d
      4ast authored
      Add check for number of arguments
      7575c64d
    • Brenden Blanco's avatar
      Fixup tplist and argdist for python3 compat · bc94d4c1
      Brenden Blanco authored
      sys.exc_value and sys.exc_type should come from sys.exc_info()
      
      Convert (decode) char* return values from C functions to a native python
      str.
      Signed-off-by: default avatarBrenden Blanco <bblanco@plumgrid.com>
      bc94d4c1
    • Brenden Blanco's avatar
      Add check for number of arguments · 977091ec
      Brenden Blanco authored
      There are two problems here:
       1. bcc is leaving the extra function parameters in the prototype, when
       these really just serve as an annotation on the method being traced.
       They aren't really passed into the function. LLVM lowering phase later
       errors out on such functions.
       2. bcc doesn't limit the size of the argument list, when currently it
       just supports up to the number of arguments held in registers (6).
      
      Fix 1. by rewriting the arguments out of the prototype and into the
      preamble where they are currently being initialized.
      Fix 2. by enforcing the limit and returning a more meaningful error
      message.
      
      Added a test case.
      
      Extra fix included for string type on python3.
      
      Fixes: #497
      Signed-off-by: default avatarBrenden Blanco <bblanco@plumgrid.com>
      977091ec