Commit 10d0ee98 authored by yonghong-song's avatar yonghong-song Committed by GitHub

Merge pull request #1676 from hMcLauchlan/inject-4.17

Add features/docs for inject (4.17)
parents 5d23500a b222f00c
...@@ -23,6 +23,7 @@ This guide is incomplete. If something feels missing, check the bcc and kernel s ...@@ -23,6 +23,7 @@ This guide is incomplete. If something feels missing, check the bcc and kernel s
- [6. bpf_get_current_comm()](#6-bpf_get_current_comm) - [6. bpf_get_current_comm()](#6-bpf_get_current_comm)
- [7. bpf_get_current_task()](#7-bpf_get_current_task) - [7. bpf_get_current_task()](#7-bpf_get_current_task)
- [8. bpf_log2l()](#8-bpflog2l) - [8. bpf_log2l()](#8-bpflog2l)
- [9. bpf_get_prandom_u32()](#9-bpf_get_prandom_u32)
- [Debugging](#debugging) - [Debugging](#debugging)
- [1. bpf_override_return()](#1-bpf_override_return) - [1. bpf_override_return()](#1-bpf_override_return)
- [Output](#output) - [Output](#output)
...@@ -352,6 +353,16 @@ Examples in situ: ...@@ -352,6 +353,16 @@ Examples in situ:
[search /examples](https://github.com/iovisor/bcc/search?q=bpf_log2l+path%3Aexamples&type=Code), [search /examples](https://github.com/iovisor/bcc/search?q=bpf_log2l+path%3Aexamples&type=Code),
[search /tools](https://github.com/iovisor/bcc/search?q=bpf_log2l+path%3Atools&type=Code) [search /tools](https://github.com/iovisor/bcc/search?q=bpf_log2l+path%3Atools&type=Code)
### 9. bpf_get_prandom_u32()
Syntax: ```u32 bpf_get_prandom_u32()```
Returns a pseudo-random u32.
Example in situ:
[search /examples](https://github.com/iovisor/bcc/search?q=bpf_get_prandom_u32+path%3Aexamples&type=Code),
[search /tools](https://github.com/iovisor/bcc/search?q=bpf_get_prandom_u32+path%3Atools&type=Code)
## Debugging ## Debugging
### 1. bpf_override_return() ### 1. bpf_override_return()
......
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
inject \- injects appropriate error into function if input call chain and inject \- injects appropriate error into function if input call chain and
predicates are satisfied. Uses Linux eBPF/bcc. predicates are satisfied. Uses Linux eBPF/bcc.
.SH SYNOPSIS .SH SYNOPSIS
.B trace -h [-I header] [-v] .B inject -h [-I header] [-P probability] [-v] mode spec
.SH DESCRIPTION .SH DESCRIPTION
inject injects errors into specified kernel functionality when a given call inject injects errors into specified kernel functionality when a given call
chain and associated predicates are satsified. chain and associated predicates are satsified.
...@@ -13,13 +13,6 @@ kernel. You should know what you're doing if you're using this tool. ...@@ -13,13 +13,6 @@ kernel. You should know what you're doing if you're using this tool.
This makes use of a Linux 4.16 feature (bpf_override_return()) This makes use of a Linux 4.16 feature (bpf_override_return())
Additionally, use of the kmalloc failure mode is only possible with
commit f7174d08a5fc ("mm: make should_failslab always available for
fault injection")
which is in mm-tree but not yet in mainline (as of 4.16-rc5).
Since this uses BPF, only the root user can use this tool. Since this uses BPF, only the root user can use this tool.
.SH REQUIREMENTS .SH REQUIREMENTS
CONFIG_BPF, CONFIG_BPF_KPROBE_OVERRIDE, bcc CONFIG_BPF, CONFIG_BPF_KPROBE_OVERRIDE, bcc
...@@ -33,6 +26,9 @@ Display the generated BPF program, for debugging or modification. ...@@ -33,6 +26,9 @@ Display the generated BPF program, for debugging or modification.
.TP .TP
\-I header \-I header
Necessary headers to be included. Necessary headers to be included.
.TP
\-P probability
Optional probability of failure, default 1.
.SH EXAMPLES .SH EXAMPLES
Please see inject_example.txt Please see inject_example.txt
.SH SOURCE .SH SOURCE
......
...@@ -22,12 +22,10 @@ ...@@ -22,12 +22,10 @@
# Note: presently there are a few hacks to get around various rewriter/verifier # Note: presently there are a few hacks to get around various rewriter/verifier
# issues. # issues.
# #
# Note: this tool requires(as of v4.16-rc5): # Note: this tool requires:
# - commit f7174d08a5fc ("mm: make should_failslab always available for fault
# injection")
# - CONFIG_BPF_KPROBE_OVERRIDE # - CONFIG_BPF_KPROBE_OVERRIDE
# #
# USAGE: inject [-h] [-I header] [-v] # USAGE: inject [-h] [-I header] [-P probability] [-v] mode spec
# #
# Copyright (c) 2018 Facebook, Inc. # Copyright (c) 2018 Facebook, Inc.
# Licensed under the Apache License, Version 2.0 (the "License") # Licensed under the Apache License, Version 2.0 (the "License")
...@@ -45,8 +43,9 @@ class Probe: ...@@ -45,8 +43,9 @@ class Probe:
} }
@classmethod @classmethod
def configure(cls, mode): def configure(cls, mode, probability):
cls.mode = mode cls.mode = mode
cls.probability = probability
def __init__(self, func, preds, length, entry): def __init__(self, func, preds, length, entry):
# length of call chain # length of call chain
...@@ -68,15 +67,25 @@ class Probe: ...@@ -68,15 +67,25 @@ class Probe:
if not chk: if not chk:
return "" return ""
if Probe.probability == 1:
early_pred = "false"
else:
early_pred = "bpf_get_prandom_u32() > %s" % str(int((1<<32)*Probe.probability))
# init the map # init the map
# dont do an early exit here so the singular case works automatically # dont do an early exit here so the singular case works automatically
# have an early exit for probability option
enter = """ enter = """
/*
* Early exit for probability case
*/
if (%s)
return 0;
/* /*
* Top level function init map * Top level function init map
*/ */
struct pid_struct p_struct = {0, 0}; struct pid_struct p_struct = {0, 0};
m.insert(&pid, &p_struct); m.insert(&pid, &p_struct);
""" """ % early_pred
# kill the entry # kill the entry
exit = """ exit = """
...@@ -221,9 +230,9 @@ class Probe: ...@@ -221,9 +230,9 @@ class Probe:
if (p->conds_met == %s && %s) if (p->conds_met == %s && %s)
bpf_override_return(ctx, %s); bpf_override_return(ctx, %s);
return 0; return 0;
}""" % (self.prep, self.length, pred, self._get_err(), self.length - 1, pred, }"""
self._get_err()) return text % (self.prep, self.length, pred, self._get_err(),
return text self.length - 1, pred, self._get_err())
# presently parses and replaces STRCMP # presently parses and replaces STRCMP
# STRCMP exists because string comparison is inconvenient and somewhat buggy # STRCMP exists because string comparison is inconvenient and somewhat buggy
...@@ -302,6 +311,9 @@ class Tool: ...@@ -302,6 +311,9 @@ class Tool:
parser.add_argument("-I", "--include", action="append", parser.add_argument("-I", "--include", action="append",
metavar="header", metavar="header",
help="additional header files to include in the BPF program") help="additional header files to include in the BPF program")
parser.add_argument("-P", "--probability", default=1,
metavar="probability", type=float,
help="probability that this call chain will fail")
parser.add_argument("-v", "--verbose", action="store_true", parser.add_argument("-v", "--verbose", action="store_true",
help="print BPF program") help="print BPF program")
self.args = parser.parse_args() self.args = parser.parse_args()
...@@ -315,7 +327,7 @@ class Tool: ...@@ -315,7 +327,7 @@ class Tool:
# create_probes and associated stuff # create_probes and associated stuff
def _create_probes(self): def _create_probes(self):
self._parse_spec() self._parse_spec()
Probe.configure(self.args.mode) Probe.configure(self.args.mode, self.args.probability)
# self, func, preds, total, entry # self, func, preds, total, entry
# create all the pair probes # create all the pair probes
...@@ -425,8 +437,7 @@ struct pid_struct { ...@@ -425,8 +437,7 @@ struct pid_struct {
def _generate_program(self): def _generate_program(self):
# leave out auto includes for now # leave out auto includes for now
self.program += '#include <linux/mm.h>\n'
self.program += "#include <linux/mm.h>\n"
for include in (self.args.include or []): for include in (self.args.include or []):
self.program += "#include <%s>\n" % include self.program += "#include <%s>\n" % include
......
...@@ -5,14 +5,16 @@ mode (kmalloc,bio,etc) given a call chain and an optional set of predicates. You ...@@ -5,14 +5,16 @@ mode (kmalloc,bio,etc) given a call chain and an optional set of predicates. You
can also optionally print out the generated BPF program for can also optionally print out the generated BPF program for
modification/debugging purposes. modification/debugging purposes.
As a simple example, let's say you wanted to fail all mounts. While we cannot As a simple example, let's say you wanted to fail all mounts. As of 4.17 we can
fail the mount() syscall directly (a patch is in the works), we can easily fail syscalls directly, so let's do that:
fail do_mount() calls like so:
# ./inject.py kmalloc -v 'do_mount()' # ./inject.py kmalloc -v 'SyS_mount()'
The first argument indicates the mode (or what to fail). Appropriate headers are The first argument indicates the mode (or what to fail). Appropriate headers are
specified. The verbosity flag prints the generated program. specified, if necessary. The verbosity flag prints the generated program. Note
that some syscalls will be available as 'SyS_xyz' and some will be available as
'sys_xyz'. This is largely dependent on the number of arguments each syscall
takes.
Trying to mount various filesystems will fail and report an inability to Trying to mount various filesystems will fail and report an inability to
allocate memory, as expected. allocate memory, as expected.
...@@ -20,7 +22,7 @@ allocate memory, as expected. ...@@ -20,7 +22,7 @@ allocate memory, as expected.
Whenever a predicate is missing, an implicit "(true)" is inserted. The example Whenever a predicate is missing, an implicit "(true)" is inserted. The example
above can be explicitly written as: above can be explicitly written as:
# ./inject.py kmalloc -v '(true) => do_mount()(true)' # ./inject.py kmalloc -v '(true) => SyS_mount()(true)'
The "(true)" without an associated function is a predicate for the error The "(true)" without an associated function is a predicate for the error
injection mechanism of the current mode. In the case of kmalloc, the predicate injection mechanism of the current mode. In the case of kmalloc, the predicate
...@@ -106,9 +108,14 @@ As an extension to the above, one could easily fail all btrfs superblock writes ...@@ -106,9 +108,14 @@ As an extension to the above, one could easily fail all btrfs superblock writes
(we only fail the primary) by calculating the sector number of the mirrors and (we only fail the primary) by calculating the sector number of the mirrors and
amending the predicate accordingly. amending the predicate accordingly.
USAGE message: Inject also provides a probability option; this allows you to fail the
path+predicates some percentage of the time. For example, let's say we want to
fail our mounts half the time:
# ./inject.py kmalloc -v -P 0.01 'SyS_mount()'
usage: inject.py [-h] [-I header] [-v] mode spec USAGE message:
usage: inject.py [-h] [-I header] [-P probability] [-v] mode spec
Fail specified kernel functionality when call chain and predicates are met Fail specified kernel functionality when call chain and predicates are met
...@@ -120,5 +127,6 @@ optional arguments: ...@@ -120,5 +127,6 @@ optional arguments:
-h, --help show this help message and exit -h, --help show this help message and exit
-I header, --include header -I header, --include header
additional header files to include in the BPF program additional header files to include in the BPF program
-P probability, --probability probability
probability that this call chain will fail
-v, --verbose print BPF program -v, --verbose print BPF program
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment