Commit fb0c3d82 authored by Marcel Amirault's avatar Marcel Amirault

Merge branch 'doc-improve-flaky-tests-page' into 'master'

Doc: Improve the Flaky tests page

See merge request gitlab-org/gitlab!78456
parents d7c3f9ba a2b6a6b9
...@@ -14,14 +14,14 @@ eventually. ...@@ -14,14 +14,14 @@ eventually.
## Quarantined tests ## Quarantined tests
When a test frequently fails in `main`, When a test frequently fails in `main`,
[a ~"master:broken" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) create [a ~"failure::flaky-test" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master).
should be created.
If the test cannot be fixed in a timely fashion, there is an impact on the If the test cannot be fixed in a timely fashion, there is an impact on the
productivity of all the developers, so it should be placed in quarantine by productivity of all the developers, so it should be quarantined by
assigning the `:quarantine` metadata with the issue URL. assigning the `:quarantine` metadata with the issue URL, and add the `~"quarantined test"` label to the issue.
```ruby ```ruby
it 'should succeed', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do it 'succeeds', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do
expect(response).to have_gitlab_http_status(:ok) expect(response).to have_gitlab_http_status(:ok)
end end
``` ```
...@@ -32,23 +32,13 @@ This means it is skipped unless run with `--tag quarantine`: ...@@ -32,23 +32,13 @@ This means it is skipped unless run with `--tag quarantine`:
bin/rspec --tag quarantine bin/rspec --tag quarantine
``` ```
**Before putting a test in quarantine, you should make sure that a
~"master:broken" issue exists for it so it doesn't stay in quarantine forever.**
Once a test is in quarantine, there are 3 choices: Once a test is in quarantine, there are 3 choices:
- Should the test be fixed (that is, get rid of its flakiness)? - Fix the test (that is, get rid of its flakiness).
- Should the test be moved to a lower level of testing? - Move the test to a lower level of testing.
- Should the test be removed entirely (for example, because there's already a - Remove the test entirely (for example, because there's already a
lower-level test, or it's duplicating another same-level test, or it's testing lower-level test, or it's duplicating another same-level test, or it's testing
too much etc.)? too much etc.).
### Quarantine tests on the CI
Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:
- `rspec-pg-quarantine` (CE & EE)
- `rspec-pg-quarantine-ee` (EE only)
## Automatic retries and flaky tests detection ## Automatic retries and flaky tests detection
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment