Skip to content

G
gitlab-ce
Commit 03e46cf7 authored by Mek Stittri's avatar Mek Stittri
Browse files
Options

Merge branch 'docs-quality/team-tasks-24' into 'master' 

Improve the 'Testing levels' documentation

Closes gitlab-org/quality/team-tasks#24

See merge request gitlab-org/gitlab-ce!21505
parents dbb342d4 35616708
Show whitespace changes
Inline Side-by-side
Showing with 119 additions and 52 deletions
doc/development/testing_guide/end_to_end_tests.md
View file @ 03e46cf7
# End-to-End Testing # End-to-end Testing
## What is End-to-End testing? ## What is end-to-end testing?
End-to-End testing is a strategy used to check whether your application works End-to-end testing is a strategy used to check whether your application works
as expected across entire software stack and architecture, including as expected across the entire software stack and architecture, including
integration of all microservices and components that are supposed to work integration of all micro-services and components that are supposed to work
together. together.
## How do we test GitLab? ## How do we test GitLab?
We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we
test these packages using [GitLab QA][gitlab-qa] project, which is entirely test these packages using the [GitLab QA orchestrator][gitlab-qa] tool, which is
black-box, click-driven testing framework. a black-box testing framework for the API and the UI.
### Testing nightly builds ### Testing nightly builds
We run scheduled pipeline each night to test nightly builds created by Omnibus. We run scheduled pipeline each night to test nightly builds created by Omnibus.
You can find these nightly pipelines at [GitLab QA pipelines page][gitlab-qa-pipelines]. You can find these nightly pipelines at [gitlab-org/quality/nightly/pipelines][quality-nightly-pipelines].
### Testing staging
We run scheduled pipeline each night to test staging.
You can find these nightly pipelines at [gitlab-org/quality/staging/pipelines][quality-staging-pipelines].
### Testing code in merge requests ### Testing code in merge requests
It is possible to run end-to-end tests (eventually being run within a It is possible to run end-to-end tests (eventually being run within a
[GitLab QA pipeline][gitlab-qa-pipelines]) for a merge request by triggering [GitLab QA pipeline][gitlab-qa-pipelines]) for a merge request by triggering
the `package-and-qa` manual action, that should be present in a merge request the `package-and-qa` manual action in the `test` stage, that should be present
widget. in a merge request widget (unless the merge request is from a fork).
Manual action that starts end-to-end tests is also available in merge requests Manual action that starts end-to-end tests is also available in merge requests
in Omnibus GitLab project. in [Omnibus GitLab][omnibus-gitlab].
Below you can read more about how to use it and how does it work. Below you can read more about how to use it and how does it work.
...@@ -35,46 +40,56 @@ Below you can read more about how to use it and how does it work. ...@@ -35,46 +40,56 @@ Below you can read more about how to use it and how does it work.
Currently, we are using _multi-project pipeline_-like approach to run QA Currently, we are using _multi-project pipeline_-like approach to run QA
pipelines. pipelines.
1. Developer triggers a manual action, that can be found in CE and EE merge 1. Developer triggers a manual action, that can be found in CE / EE merge
requests. This starts a chain of pipelines in multiple projects. requests. This starts a chain of pipelines in multiple projects.
1. The script being executed triggers a pipeline in GitLab Omnibus and waits 1. The script being executed triggers a pipeline in [Omnibus GitLab][omnibus-gitlab]
for the resulting status. We call this a _status attribution_. and waits for the resulting status. We call this a _status attribution_.
1. GitLab packages are being built in Omnibus pipeline. Packages are going to be 1. GitLab packages are being built in the [Omnibus GitLab][omnibus-gitlab]
pushed to Container Registry. pipeline. Packages are then pushed to its Container Registry.
1. When packages are ready, and available in the registry, a final step in the 1. When packages are ready, and available in the registry, a final step in the
pipeline, that is now running in Omnibus, triggers a new pipeline in the GitLab [Omnibus GitLab][omnibus-gitlab] pipeline, triggers a new
QA project. It also waits for a resulting status. [GitLab QA pipeline][gitlab-qa-pipelines]. It also waits for a resulting status.
1. GitLab QA pulls images from the registry, spins-up containers and runs tests 1. GitLab QA pulls images from the registry, spins-up containers and runs tests
against a test environment that has been just orchestrated by the `gitlab-qa` against a test environment that has been just orchestrated by the `gitlab-qa`
tool. tool.
1. The result of the GitLab QA pipeline is being propagated upstream, through 1. The result of the [GitLab QA pipeline][gitlab-qa-pipelines] is being
Omnibus, back to CE / EE merge request. propagated upstream, through Omnibus, back to the CE / EE merge request.
#### How do I write tests? #### How do I write tests?
In order to write new tests, you first need to learn more about GitLab QA In order to write new tests, you first need to learn more about GitLab QA
architecture. See the [documentation about it][gitlab-qa-architecture] in architecture. See the [documentation about it][gitlab-qa-architecture].
GitLab QA project.
Once you decided where to put test environment orchestration scenarios and Once you decided where to put [test environment orchestration scenarios] and
instance specs, take a look at the [relevant documentation][instance-qa-readme] [instance-level scenarios], take a look at the [GitLab QA README][instance-qa-readme],
and examples in [the `qa/` directory][instance-qa-examples]. the [GitLab QA orchestrator README][gitlab-qa-readme], and [the already existing
instance-level scenarios][instance-level scenarios].
## Where can I ask for help? ## Where can I ask for help?
You can ask question in the `#quality` channel on Slack (GitLab internal) or You can ask question in the `#quality` channel on Slack (GitLab internal) or
you can find an issue you would like to work on in you can find an issue you would like to work on in
[the issue tracker][gitlab-qa-issues] and start a new discussion there. [the `gitlab-ce` issue tracker][gitlab-ce-issues],
[the `gitlab-ee` issue tracker][gitlab-ce-issues], or
[the `gitlab-qa` issue tracker][gitlab-qa-issues].
[omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab [omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab
[gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa [gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa
[gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md
[gitlab-qa-pipelines]: https://gitlab.com/gitlab-org/gitlab-qa/pipelines [gitlab-qa-pipelines]: https://gitlab.com/gitlab-org/gitlab-qa/pipelines
[quality-nightly-pipelines]: https://gitlab.com/gitlab-org/quality/nightly/pipelines
[quality-staging-pipelines]: https://gitlab.com/gitlab-org/quality/staging/pipelines
[gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md [gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md
[gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues [gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario
[gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name[]=QA&label_name[]=test
[gitlab-ee-issues]: https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name[]=QA&label_name[]=test
[test environment orchestration scenarios]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario
[instance-level scenarios]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/specs/features
[Page objects documentation]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page/README.md
[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md [instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md
[instance-qa-examples]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa [instance-qa-examples]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa
doc/development/testing_guide/smoke.md
View file @ 03e46cf7
# Smoke Tests # Smoke Tests
It is imperative in any testing suite that we have Smoke Tests. In short, smoke tests are will run quick sanity It is imperative in any testing suite that we have Smoke Tests. In short, smoke
end-to-end functional tests from GitLab QA and are designed to run against the specified environment to ensure that tests will run quick sanity end-to-end functional tests from GitLab QA and are
basic functionality is working. designed to run against the specified environment to ensure that basic
functionality is working.
Currently, our suite consists of this basic functionality coverage: Currently, our suite consists of this basic functionality coverage:
...@@ -11,6 +12,8 @@ Currently, our suite consists of this basic functionality coverage: ...@@ -11,6 +12,8 @@ Currently, our suite consists of this basic functionality coverage:
- Issue Creation - Issue Creation
- Merge Request Creation - Merge Request Creation
Smoke tests have the `:smoke` RSpec metadata.
--- ---
[Return to Testing documentation](index.md) [Return to Testing documentation](index.md)
doc/development/testing_guide/testing_levels.md
View file @ 03e46cf7
...@@ -34,7 +34,11 @@ records should use stubs/doubles as much as possible. ...@@ -34,7 +34,11 @@ records should use stubs/doubles as much as possible.
Formal definition: https://en.wikipedia.org/wiki/Integration_testing Formal definition: https://en.wikipedia.org/wiki/Integration_testing
These kind of tests ensure that individual parts of the application work well together, without the overhead of the actual app environment (i.e. the browser). These tests should assert at the request/response level: status code, headers, body. They're useful to test permissions, redirections, what view is rendered etc. These kind of tests ensure that individual parts of the application work well
together, without the overhead of the actual app environment (i.e. the browser).
These tests should assert at the request/response level: status code, headers,
body.
They're useful to test permissions, redirections, what view is rendered etc.
| Code path | Tests path | Testing engine | Notes | | Code path | Tests path | Testing engine | Notes |
| --------- | ---------- | -------------- | ----- | | --------- | ---------- | -------------- | ----- |
...@@ -67,20 +71,40 @@ run JavaScript tests, so you can either run unit tests (e.g. test a single ...@@ -67,20 +71,40 @@ run JavaScript tests, so you can either run unit tests (e.g. test a single
JavaScript method), or integration tests (e.g. test a component that is composed JavaScript method), or integration tests (e.g. test a component that is composed
of multiple components). of multiple components).
## System tests or feature tests ## White-box tests at the system level (formerly known as System / Feature tests)
Formal definition: https://en.wikipedia.org/wiki/System_testing. Formal definitions:
These kind of tests ensure the application works as expected from a user point - https://en.wikipedia.org/wiki/System_testing
of view (aka black-box testing). These tests should test a happy path for a - https://en.wikipedia.org/wiki/White-box_testing
given page or set of pages, and a test case should be added for any regression
These kind of tests ensure the GitLab *Rails* application (i.e.
`gitlab-ce`/`gitlab-ee`) works as expected from a *browser* point of view.
Note that:
- knowledge of the internals of the application are still required
- data needed for the tests are usually created directly using RSpec factories
- expectations are often set on the database or objects state
These tests should only be used when:
- the functionality/component being tested is small
- the internal state of the objects/database *needs* to be tested
- it cannot be tested at a lower level
For instance, to test the breadcrumbs on a given page, writing a system test
makes sense since it's a small component, which cannot be tested at the unit or
controller level.
Only test the happy path, but make sure to add a test case for any regression
that couldn't have been caught at lower levels with better tests (i.e. if a that couldn't have been caught at lower levels with better tests (i.e. if a
regression is found, regression tests should be added at the lowest-level regression is found, regression tests should be added at the lowest-level
possible). possible).
| Tests path | Testing engine | Notes | | Tests path | Testing engine | Notes |
| ---------- | -------------- | ----- | | ---------- | -------------- | ----- |
| `spec/features/` | [Capybara] + [RSpec] | If your spec has the `:js` metadata, the browser driver will be [Poltergeist], otherwise it's using [RackTest]. | | `spec/features/` | [Capybara] + [RSpec] | If your test has the `:js` metadata, the browser driver will be [Poltergeist], otherwise it's using [RackTest]. |
### Consider **not** writing a system test! ### Consider **not** writing a system test!
...@@ -89,7 +113,7 @@ we have enough Unit & Integration tests), we shouldn't need to duplicate their ...@@ -89,7 +113,7 @@ we have enough Unit & Integration tests), we shouldn't need to duplicate their
thorough testing at the System test level. thorough testing at the System test level.
It's very easy to add tests, but a lot harder to remove or improve tests, so one It's very easy to add tests, but a lot harder to remove or improve tests, so one
should take care of not introducing too many (slow and duplicated) specs. should take care of not introducing too many (slow and duplicated) tests.
The reasons why we should follow these best practices are as follows: The reasons why we should follow these best practices are as follows:
...@@ -107,29 +131,33 @@ The reasons why we should follow these best practices are as follows: ...@@ -107,29 +131,33 @@ The reasons why we should follow these best practices are as follows:
[Poltergeist]: https://github.com/teamcapybara/capybara#poltergeist [Poltergeist]: https://github.com/teamcapybara/capybara#poltergeist
[RackTest]: https://github.com/teamcapybara/capybara#racktest [RackTest]: https://github.com/teamcapybara/capybara#racktest
## Black-box tests or end-to-end tests ## Black-box tests at the system level, aka end-to-end tests
Formal definitions:
- https://en.wikipedia.org/wiki/System_testing
- https://en.wikipedia.org/wiki/Black-box_testing
GitLab consists of [multiple pieces] such as [GitLab Shell], [GitLab Workhorse], GitLab consists of [multiple pieces] such as [GitLab Shell], [GitLab Workhorse],
[Gitaly], [GitLab Pages], [GitLab Runner], and GitLab Rails. All theses pieces [Gitaly], [GitLab Pages], [GitLab Runner], and GitLab Rails. All theses pieces
are configured and packaged by [GitLab Omnibus]. are configured and packaged by [GitLab Omnibus].
[GitLab QA] is a tool that allows to test that all these pieces integrate well The QA framework and instance-level scenarios are [part of GitLab Rails] so that
together by building a Docker image for a given version of GitLab Rails and they're always in-sync with the codebase (especially the views).
running feature tests (i.e. using Capybara) against it.
The actual test scenarios and steps are [part of GitLab Rails] so that they're Note that:
always in-sync with the codebase.
### Smoke tests - knowledge of the internals of the application are not required
- data needed for the tests can only be created using the GUI or the API
Smoke tests are quick tests that may be run at any time (especially after the pre-deployment migrations). - expectations can only be made against the browser page and API responses
Much like feature tests - these tests run against the UI and ensure that basic functionality is working. Every new feature should come with a [test plan].
> See [Smoke Tests](smoke.md) for more information. | Tests path | Testing engine | Notes |
| ---------- | -------------- | ----- |
| `qa/qa/specs/features/` | [Capybara] + [RSpec] + Custom QA framework | Tests should be placed under their corresponding [Product category] |
Read a separate document about [end-to-end tests](end_to_end_tests.md) to > See [end-to-end tests](end_to_end_tests.md) for more information.
learn more.
[multiple pieces]: ../architecture.md#components [multiple pieces]: ../architecture.md#components
[GitLab Shell]: https://gitlab.com/gitlab-org/gitlab-shell [GitLab Shell]: https://gitlab.com/gitlab-org/gitlab-shell
...@@ -138,8 +166,29 @@ learn more. ...@@ -138,8 +166,29 @@ learn more.
[GitLab Pages]: https://gitlab.com/gitlab-org/gitlab-pages [GitLab Pages]: https://gitlab.com/gitlab-org/gitlab-pages
[GitLab Runner]: https://gitlab.com/gitlab-org/gitlab-runner [GitLab Runner]: https://gitlab.com/gitlab-org/gitlab-runner
[GitLab Omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab [GitLab Omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab
[GitLab QA]: https://gitlab.com/gitlab-org/gitlab-qa
[part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa [part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa
[test plan]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/.gitlab/issue_templates/Test%20plan.md
[Product category]: https://about.gitlab.com/handbook/product/categories/
### Smoke tests
Smoke tests are quick tests that may be run at any time (especially after the
pre-deployment migrations).
These tests run against the UI and ensure that basic functionality is working.
> See [Smoke Tests](smoke.md) for more information.
### GitLab QA orchestrator
[GitLab QA orchestrator] is a tool that allows to test that all these pieces
integrate well together by building a Docker image for a given version of GitLab
Rails and running end-to-end tests (i.e. using Capybara) against it.
Learn more in the [GitLab QA orchestrator README][gitlab-qa-readme].
[GitLab QA orchestrator]: https://gitlab.com/gitlab-org/gitlab-qa
[gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md
## EE-specific tests ## EE-specific tests
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7