Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
a2b6a6b9
Commit
a2b6a6b9
authored
Jan 18, 2022
by
Rémy Coutable
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
doc: Improve the Flaky tests page
parent
2d74977a
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
9 additions
and
19 deletions
+9
-19
doc/development/testing_guide/flaky_tests.md
doc/development/testing_guide/flaky_tests.md
+9
-19
No files found.
doc/development/testing_guide/flaky_tests.md
View file @
a2b6a6b9
...
...
@@ -14,14 +14,14 @@ eventually.
## Quarantined tests
When a test frequently fails in
`main`
,
[
a ~"master:broken" issue
](
https://about.gitlab.com/handbook/engineering/workflow/#broken-master
)
should be created.
create
[
a ~"failure::flaky-test" issue
](
https://about.gitlab.com/handbook/engineering/workflow/#broken-master
)
.
If the test cannot be fixed in a timely fashion, there is an impact on the
productivity of all the developers, so it should be
placed in quarantine
by
assigning the
`:quarantine`
metadata with the issue URL.
productivity of all the developers, so it should be
quarantined
by
assigning the
`:quarantine`
metadata with the issue URL
, and add the
`~"quarantined test"`
label to the issue
.
```
ruby
it
's
hould succeed
'
,
quarantine:
'https://gitlab.com/gitlab-org/gitlab/-/issues/12345'
do
it
's
ucceeds
'
,
quarantine:
'https://gitlab.com/gitlab-org/gitlab/-/issues/12345'
do
expect
(
response
).
to
have_gitlab_http_status
(
:ok
)
end
```
...
...
@@ -32,23 +32,13 @@ This means it is skipped unless run with `--tag quarantine`:
bin/rspec
--tag
quarantine
```
**
Before putting a test in quarantine, you should make sure that a
~"master:broken" issue exists for it so it doesn't stay in quarantine forever.
**
Once a test is in quarantine, there are 3 choices:
-
Should the test be fixed (that is, get rid of its flakiness)?
-
Should the test be moved to a lower level of testing?
-
Should the test be removed
entirely (for example, because there's already a
-
Fix the test (that is, get rid of its flakiness).
-
Move the test to a lower level of testing.
-
Remove the test
entirely (for example, because there's already a
lower-level test, or it's duplicating another same-level test, or it's testing
too much etc.)?
### Quarantine tests on the CI
Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:
-
`rspec-pg-quarantine`
(CE & EE)
-
`rspec-pg-quarantine-ee`
(EE only)
too much etc.).
## Automatic retries and flaky tests detection
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment