Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
B
bcc
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
Kirill Smelkov
bcc
Commits
75d3e9d4
Commit
75d3e9d4
authored
Feb 07, 2016
by
Brendan Gregg
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
rename fsslower to fileslower
parent
1f1c8f80
Changes
4
Hide whitespace changes
Inline
Side-by-side
Showing
4 changed files
with
39 additions
and
39 deletions
+39
-39
README.md
README.md
+1
-1
man/man8/fileslower.8
man/man8/fileslower.8
+13
-13
tools/fileslower.py
tools/fileslower.py
+10
-10
tools/fileslower_example.txt
tools/fileslower_example.txt
+15
-15
No files found.
README.md
View file @
75d3e9d4
...
...
@@ -72,7 +72,7 @@ Tools:
-
tools/
[
bitesize
](
tools/bitesize.py
)
: Show per process I/O size histogram.
[
Examples
](
tools/bitesize_example.txt
)
.
-
tools/
[
cachestat
](
tools/cachestat.py
)
: Trace page cache hit/miss ratio.
[
Examples
](
tools/cachestat_example.txt
)
.
-
tools/
[
execsnoop
](
tools/execsnoop.py
)
: Trace new processes via exec() syscalls.
[
Examples
](
tools/execsnoop_example.txt
)
.
-
tools/
[
f
sslower
](
tools/fsslower.py
)
: Trace slow file system synchronous reads and writes.
[
Examples
](
tools/fs
slower_example.txt
)
.
-
tools/
[
f
ileslower
](
tools/fileslower.py
)
: Trace slow synchronous file reads and writes.
[
Examples
](
tools/file
slower_example.txt
)
.
-
tools/
[
funccount
](
tools/funccount.py
)
: Count kernel function calls.
[
Examples
](
tools/funccount_example.txt
)
.
-
tools/
[
funclatency
](
tools/funclatency.py
)
: Time kernel functions and show their latency distribution.
[
Examples
](
tools/funclatency_example.txt
)
.
-
tools/
[
gethostlatency
](
tools/gethostlatency.py
)
: Show latency for getaddrinfo/gethostbyname
[
2] calls. [Examples
](
tools/gethostlatency_example.txt
)
.
...
...
man/man8/f
s
slower.8
→
man/man8/f
ile
slower.8
View file @
75d3e9d4
.TH f
s
slower 8 "2016-02-07" "USER COMMANDS"
.TH f
ile
slower 8 "2016-02-07" "USER COMMANDS"
.SH NAME
f
sslower \- Summarize block device I/O latency as a histogram
.
f
ileslower \- Trace slow synchronous file reads and writes
.
.SH SYNOPSIS
.B f
s
slower [\-h] [\-p PID] [min_ms]
.B f
ile
slower [\-h] [\-p PID] [min_ms]
.SH DESCRIPTION
This script uses kernel dynamic tracing of synchronous reads and writes
at the VFS interface, to identify slow file system I/O for any file system.
at the VFS interface, to identify slow file reads and writes for any file
system.
This version traces __vfs_read() and __vfs_write() and only showing
synchronous I/O (the path to new_sync_read() and new_sync_write()), and
I/O with filenames. This approach provides a view of just two file
system request types: reads and writes. There are typically many others:
system request types:
file
reads and writes. There are typically many others:
asynchronous I/O, directory operations, file handle operations, file open()s,
fflush(), etc, that this tool does not currently instrument. This
implementation is a work around until we have suitable fs tracepoints.
fflush(), etc.
WARNING: See the OVERHEAD section.
...
...
@@ -34,17 +34,17 @@ min_ms
Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.
.SH EXAMPLES
.TP
Trace synchronous file
system
reads and writes slower than 10 ms:
Trace synchronous file reads and writes slower than 10 ms:
#
.B f
s
slower
.B f
ile
slower
.TP
Trace slower than 1 ms:
#
.B f
s
slower 1
.B f
ile
slower 1
.TP
Trace slower than 1 ms, for PID 181 only:
#
.B f
s
slower \-p 181 1
.B f
ile
slower \-p 181 1
.SH FIELDS
.TP
TIME(s)
...
...
@@ -86,13 +86,13 @@ instrumented events using the bcc funccount tool, eg:
.PP
# ./funccount.py -i 1 -r '^__vfs_(read|write)$'
.PP
This also costs overhead, but is somewhat less than f
s
slower.
This also costs overhead, but is somewhat less than f
ile
slower.
.PP
If the overhead is prohibitive for your workload, I'd recommend moving
down-stack a little from VFS into the file system functions (ext4, xfs, etc).
Look for updates to bcc for specific file system tools that do this. The
advantage of a per-file system approach is that we can trace post-cache,
greatly reducing events and overhead.
The disadvantage is needing custom
greatly reducing events and overhead. The disadvantage is needing custom
tracing approaches for each different file system (whereas VFS is generic).
.SH SOURCE
This is from bcc.
...
...
tools/f
s
slower.py
→
tools/f
ile
slower.py
View file @
75d3e9d4
#!/usr/bin/python
# @lint-avoid-python-3-compatibility-imports
#
# f
sslower Trace slow file system synchronous
reads and writes.
# For Linux, uses BCC, eBPF.
# f
ileslower Trace slow synchronous file
reads and writes.
#
For Linux, uses BCC, eBPF.
#
# USAGE: f
s
slower [-h] [-p PID] [min_ms]
# USAGE: f
ile
slower [-h] [-p PID] [min_ms]
#
# This script uses kernel dynamic tracing of synchronous reads and writes
# at the VFS interface, to identify slow file system I/O for any file system.
# at the VFS interface, to identify slow file reads and writes for any file
# system.
#
# This works by tracing __vfs_read() and __vfs_write(), and filtering for
# synchronous I/O (the path to new_sync_read() and new_sync_write()), and
# for I/O with filenames. This approach provides a view of just two file
# system request types. There are typically many others: asynchronous I/O,
# directory operations, file handle operations, etc, that this tool does not
# instrument. This implementation is a work around until we have suitable fs
# tracepoints.
# instrument.
#
# WARNING: This traces VFS reads and writes, which can be extremely frequent,
# and so the overhead of this tool can become severe depending on the
...
...
@@ -35,12 +35,12 @@ import signal
# arguments
examples
=
"""examples:
./f
sslower # trace sync
I/O slower than 10 ms (default)
./f
sslower 1 # trace sync
I/O slower than 1 ms
./f
s
slower -p 185 # trace PID 185 only
./f
ileslower # trace sync file
I/O slower than 10 ms (default)
./f
ileslower 1 # trace sync file
I/O slower than 1 ms
./f
ile
slower -p 185 # trace PID 185 only
"""
parser
=
argparse
.
ArgumentParser
(
description
=
"Trace slow
file system sync
reads and writes"
,
description
=
"Trace slow
synchronous file
reads and writes"
,
formatter_class
=
argparse
.
RawDescriptionHelpFormatter
,
epilog
=
examples
)
parser
.
add_argument
(
"-p"
,
"--pid"
,
...
...
tools/f
s
slower_example.txt
→
tools/f
ile
slower_example.txt
View file @
75d3e9d4
Demonstrations of f
s
slower, the Linux eBPF/bcc version.
Demonstrations of f
ile
slower, the Linux eBPF/bcc version.
f
sslower shows file-based synchronous reads and writes slower than a threshold.
For example:
f
ileslower shows file-based synchronous reads and writes slower than a
threshold.
For example:
# ./f
s
slower
# ./f
ile
slower
Tracing sync read/writes slower than 10 ms
TIME(s) COMM PID D BYTES LAT(ms) FILENAME
0.000 randread.pl 4762 R 8192 12.70 data1
...
...
@@ -20,13 +20,13 @@ I/O), file system CPU cycles, file system locks, run queue latency, etc. This
is a better measure of the latency suffered by applications reading from the
file system than measuring this down at the block device interface.
Note that this only traces reads and writes: other file system operations
(eg, directory operations, open(), fflush()) are not
currently
traced.
Note that this only traces
file
reads and writes: other file system operations
(eg, directory operations, open(), fflush()) are not traced.
The threshold can be provided as an argument. Eg, I/O slower than 1 ms:
# ./f
s
slower 1
# ./f
ile
slower 1
Tracing sync read/writes slower than 1 ms
TIME(s) COMM PID D BYTES LAT(ms) FILENAME
0.000 randread.pl 6925 R 8192 1.06 data1
...
...
@@ -62,10 +62,10 @@ spanned 12 seconds), and the lower threshold is catching more I/O.
In the following example, the file system caches were dropped before running
f
s
slower, and then in another session a "man ls" was executed. The command
f
ile
slower, and then in another session a "man ls" was executed. The command
and files read from disk can be seen:
# echo 3 > /proc/sys/vm/drop_caches; ./f
s
slower 1
# echo 3 > /proc/sys/vm/drop_caches; ./f
ile
slower 1
Tracing sync read/writes slower than 1 ms
TIME(s) COMM PID D BYTES LAT(ms) FILENAME
0.000 bash 9647 R 128 5.83 man
...
...
@@ -104,10 +104,10 @@ locks, run queue latency, etc. These can be explored using other commands.
USAGE message:
# ./f
s
slower -h
usage: f
s
slower [-h] [-p PID] [min_ms]
# ./f
ile
slower -h
usage: f
ile
slower [-h] [-p PID] [min_ms]
Trace slow
file system sync
reads and writes
Trace slow
synchronous file
reads and writes
positional arguments:
min_ms minimum I/O duration to trace, in ms (default 10)
...
...
@@ -117,6 +117,6 @@ optional arguments:
-p PID, --pid PID trace this PID only
examples:
./f
sslower # trace sync
I/O slower than 10 ms (default)
./f
sslower 1 # trace sync
I/O slower than 1 ms
./f
s
slower -p 185 # trace PID 185 only
./f
ileslower # trace sync file
I/O slower than 10 ms (default)
./f
ileslower 1 # trace sync file
I/O slower than 1 ms
./f
ile
slower -p 185 # trace PID 185 only
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment