Commit 3b180468 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'eread/refine-geo-testing-topic' into 'master'

Refine new Geo testing topic

Closes #217428

See merge request gitlab-org/gitlab!31695
parents cf00e431 8062cc11
...@@ -46,6 +46,7 @@ ...@@ -46,6 +46,7 @@
"Elasticsearch", "Elasticsearch",
"Facebook", "Facebook",
"GDK", "GDK",
"Geo",
"Git LFS", "Git LFS",
"git-annex", "git-annex",
"Git", "Git",
......
...@@ -13,6 +13,7 @@ swap: ...@@ -13,6 +13,7 @@ swap:
gitlab omnibus: Omnibus GitLab gitlab omnibus: Omnibus GitLab
param: parameter param: parameter
params: parameters params: parameters
pg: PostgreSQL
postgres: PostgreSQL postgres: PostgreSQL
raketask: Rake task raketask: Rake task
raketasks: Rake tasks raketasks: Rake tasks
......
...@@ -262,7 +262,7 @@ You can login to the **secondary** node with the same credentials you used for t ...@@ -262,7 +262,7 @@ You can login to the **secondary** node with the same credentials you used for t
**secondary** Geo node and if Geo is enabled. **secondary** Geo node and if Geo is enabled.
The initial replication, or 'backfill', will probably still be in progress. You The initial replication, or 'backfill', will probably still be in progress. You
can monitor the synchronization process on each geo node from the **primary** can monitor the synchronization process on each Geo node from the **primary**
node's **Geo Nodes** dashboard in your browser. node's **Geo Nodes** dashboard in your browser.
![Geo dashboard](img/geo_node_dashboard.png) ![Geo dashboard](img/geo_node_dashboard.png)
......
# Geo validation tests # Geo validation tests
The Geo team performs manual testing and validation on common deployment configurations to ensure that Geo works when upgrading between minor GitLab versions and major PostgreSQL database versions. This section contains a journal of recent validation tests and links to the relevant issues. The Geo team performs manual testing and validation on common deployment configurations to ensure
that Geo works when upgrading between minor GitLab versions and major PostgreSQL database versions.
## GitLab Upgrades This section contains a journal of recent validation tests and links to the relevant issues.
## GitLab upgrades
The following are GitLab upgrade validation tests we performed.
### February 2020 ### February 2020
[Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/201837) [Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/201837):
- Description: Tested upgrading from GitLab 12.7.5 to the latest 12.8 package in a high availability configuration. - Description: Tested upgrading from GitLab 12.7.5 to the latest GitLab 12.8 package in a high
- Outcome: Partial success because we did not run the looping pipeline during the demo to monitor downtime. availability configuration.
- Outcome: Partial success because we did not run the looping pipeline during the demo to monitor
downtime.
### January 2020 ### January 2020
[Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/200085) [Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/200085):
- Description: Tested upgrading from GitLab 12.6.x to the latest 12.7 package in a high availability configuration. - Description: Tested upgrading from GitLab 12.6.x to the latest GitLab 12.7 package in a high
availability configuration.
- Outcome: Upgrade test was successful. - Outcome: Upgrade test was successful.
- Follow up issues: - Follow up issues:
- [Investigate Geo End to End Test Failures](https://gitlab.com/gitlab-org/gitlab/issues/201823) - [Investigate Geo end-to-end test failures](https://gitlab.com/gitlab-org/gitlab/issues/201823).
- [Add More Logging to Geo End to End Tests](https://gitlab.com/gitlab-org/gitlab/issues/201830) - [Add more logging to Geo end-to-end tests](https://gitlab.com/gitlab-org/gitlab/issues/201830).
- [Excess Service Restarts During Zero-Downtime Upgrade](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5047) - [Excess service restarts during zero-downtime upgrade](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5047).
[Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/199836) [Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/199836):
- Description: Tested upgrading from GitLab 12.5.7 to 12.6.6 in a high availability configuration - Description: Tested upgrading from GitLab 12.5.7 to GitLab 12.6.6 in a high availability
configuration.
- Outcome: Upgrade test was successful. - Outcome: Upgrade test was successful.
- Follow up issues: - Follow up issue:
- [Update docs for zero-downtime upgrades to ensure deploy node it not in-use](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5046) [Update documentation for zero-downtime upgrades to ensure deploy node it not in use](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5046).
[Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/37044) [Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/37044):
- Description: Tested upgrading from GitLab 12.4.x to the latest 12.5 package in a high availability configuration. - Description: Tested upgrading from GitLab 12.4.x to the latest GitLab 12.5 package in a high
availability configuration.
- Outcome: Upgrade test was successful. - Outcome: Upgrade test was successful.
- Follow up issues: - Follow up issues:
- [Investigate why http push spec failed on primary node](https://gitlab.com/gitlab-org/gitlab/issues/199825) - [Investigate why HTTP push spec failed on primary node](https://gitlab.com/gitlab-org/gitlab/issues/199825).
- [Investigate if docs should be modified to include refresh foreign tables task](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5041) - [Investigate if documentation should be modified to include refresh foreign tables task](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5041).
### October 2019 ### October 2019
[Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/35262) [Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/35262):
- Description: Tested uprgading from GitLab 12.3.5 to 12.4.1 in a high availability configuration. - Description: Tested upgrading from GitLab 12.3.5 to GitLab 12.4.1 in a high availability configuration.
- Outcome: Upgrade test was successful. - Outcome: Upgrade test was successful.
[Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32437) [Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32437):
- Description: Tested upgrading from GitLab 12.2.8 to 12.3.5 - Description: Tested upgrading from GitLab 12.2.8 to GitLab 12.3.5.
- Outcome: Upgrade test was successful. - Outcome: Upgrade test was successful.
[Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32435) [Upgrade Geo HA installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32435):
- Description: Tested upgrading from GitLab 12.1.9 to 12.2.8 - Description: Tested upgrading from GitLab 12.1.9 to GitLab 12.2.8.
- Outcome: Partial success due to possible misconfiguration issues. - Outcome: Partial success due to possible misconfiguration issues.
## PostgreSQL Upgrades ## PostgreSQL upgrades
The following are PostgreSQL upgrade validation tests we performed.
### April 2020 ### April 2020
[PostgreSQL 11 upgrade procedure for GEO installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4975) [PostgreSQL 11 upgrade procedure for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4975):
- Description: Prior to making PostgreSQL 11 the default version of PG in GitLab 12.10, we tested upgrading to PG 11 in Geo deployments on GitLab 12.9. - Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we
- Outcome: Partially successful. Issues were discovered in HA configurations with a separate tracking database and concerns were raised about allowing automatic upgrades when Geo enabled. tested upgrading to PostgreSQL 11 in Geo deployments on GitLab 12.9.
- Outcome: Partially successful. Issues were discovered in HA configurations with a separate
tracking database and concerns were raised about allowing automatic upgrades when Geo enabled.
- Follow up issues: - Follow up issues:
- [replicate-geo-database incorrectly tries to backup repos](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5241) - [`replicate-geo-database` incorrectly tries to back up repositories](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5241).
- [pg-upgrade fails to upgrade a standalone Geo tracking DB](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5242) - [`pg-upgrade` fails to upgrade a standalone Geo tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5242).
- [revert-pg-upgrade fails to downgrade a Geo secondary’s standalone tracking DB’s PG data](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5243) - [`revert-pg-upgrade` fails to downgrade the PostgreSQL data of a Geo secondary’s standalone tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5243).
- [Timeout error on Geo secondary read-replica near the end of `gitlab-ctl pg-upgrade`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5235) - [Timeout error on Geo secondary read-replica near the end of `gitlab-ctl pg-upgrade`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5235).
[Verify GEO installation with PostgreSQL 11](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4971) [Verify Geo installation with PostgreSQL 11](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4971):
- Description: Prior to making PostgreSQL 11 the default version of PG in GitLab 12.10, we tested fresh installations of GitLab 12.9 with Geo, installed with PG 11. - Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we
tested fresh installations of GitLab 12.9 with Geo installed with PostgreSQL 11.
- Outcome: Installation test was successful. - Outcome: Installation test was successful.
### September 2019 ### September 2019
[Test and validate PostgreSQL 10.0 upgrade for Geo](https://gitlab.com/gitlab-org/gitlab/issues/12092) [Test and validate PostgreSQL 10.0 upgrade for Geo](https://gitlab.com/gitlab-org/gitlab/issues/12092):
- Description: With the 12.0 release, GitLab required an upgrade to PostgreSQL 10.0. We tested various upgrade scenarios from GitLab 11.11.5 through to 12.1.8. - Description: With the 12.0 release, GitLab required an upgrade to PostgreSQL 10.0. We tested
various upgrade scenarios from GitLab 11.11.5 through to GitLab 12.1.8.
- Outcome: Multiple issues were found when upgrading and addressed in follow-up issues. - Outcome: Multiple issues were found when upgrading and addressed in follow-up issues.
- Follow up issues: - Follow up issues:
- [`gitlab-ctl` reconfigure fails on Redis node in HA Geo setup](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4706) - [`gitlab-ctl` reconfigure fails on Redis node in HA Geo setup](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4706).
- [HA with Geo upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4705) - [HA with Geo upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4705).
- [refresh foreign tables fails on app server in HA setup after upgrade to 12.1.9](https://gitlab.com/gitlab-org/gitlab/-/issues/32119) - [Refresh foreign tables fails on app server in HA setup after upgrade to 12.1.9](https://gitlab.com/gitlab-org/gitlab/-/issues/32119).
...@@ -73,7 +73,7 @@ from [owasp.org](https://owasp.org/). ...@@ -73,7 +73,7 @@ from [owasp.org](https://owasp.org/).
- Nothing Geo-specific. Any user where `admin: true` is set in the database is - Nothing Geo-specific. Any user where `admin: true` is set in the database is
considered an admin with super-user privileges. considered an admin with super-user privileges.
- See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-foss/issues/32730) - See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-foss/issues/32730)
(not geo-specific) (not Geo-specific).
- Much of Geo’s integration (database replication, for instance) must be - Much of Geo’s integration (database replication, for instance) must be
configured with the application, typically by system administrators. configured with the application, typically by system administrators.
......
...@@ -125,7 +125,7 @@ otherwise the networks will become a single point of failure. ...@@ -125,7 +125,7 @@ otherwise the networks will become a single point of failure.
#### Architecture #### Architecture
![PG HA Architecture](img/pg_ha_architecture.png) ![PostgreSQL HA Architecture](img/pg_ha_architecture.png)
Database nodes run two services with PostgreSQL: Database nodes run two services with PostgreSQL:
......
...@@ -33,7 +33,7 @@ This section is for links to information elsewhere in the GitLab documentation. ...@@ -33,7 +33,7 @@ This section is for links to information elsewhere in the GitLab documentation.
- [More about external PostgreSQL](../external_database.md) - [More about external PostgreSQL](../external_database.md)
- [Running GEO with external PostgreSQL](../geo/replication/external_database.md) - [Running Geo with external PostgreSQL](../geo/replication/external_database.md)
- [Upgrades when running PostgreSQL configured for HA.](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-gitlab-ha-cluster) - [Upgrades when running PostgreSQL configured for HA.](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-gitlab-ha-cluster)
...@@ -71,7 +71,7 @@ This section is for links to information elsewhere in the GitLab documentation. ...@@ -71,7 +71,7 @@ This section is for links to information elsewhere in the GitLab documentation.
HINT: Free one or increase max_replication_slots. HINT: Free one or increase max_replication_slots.
``` ```
- GEO [replication errors](../geo/replication/troubleshooting.md#fixing-replication-errors) including: - Geo [replication errors](../geo/replication/troubleshooting.md#fixing-replication-errors) including:
```plaintext ```plaintext
ERROR: replication slots can only be used if max_replication_slots > 0 ERROR: replication slots can only be used if max_replication_slots > 0
...@@ -83,11 +83,11 @@ This section is for links to information elsewhere in the GitLab documentation. ...@@ -83,11 +83,11 @@ This section is for links to information elsewhere in the GitLab documentation.
PANIC: could not write to file ‘pg_xlog/xlogtemp.123’: No space left on device PANIC: could not write to file ‘pg_xlog/xlogtemp.123’: No space left on device
``` ```
- [Checking GEO configuration](../geo/replication/troubleshooting.md#checking-configuration) including - [Checking Geo configuration](../geo/replication/troubleshooting.md#checking-configuration) including
- reconfiguring hosts/ports - reconfiguring hosts/ports
- checking and fixing user/password mappings - checking and fixing user/password mappings
- [Common GEO errors](../geo/replication/troubleshooting.md#fixing-common-errors) - [Common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors)
## Support topics ## Support topics
......
...@@ -153,7 +153,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing. ...@@ -153,7 +153,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing.
### Current versions testing ### Current versions testing
| Where? | PG version | | Where? | PostgreSQL version |
| ------ | ------ | | ------ | ------ |
| MRs | 11 | | MRs | 11 |
| `master` (non-scheduled pipelines) | 11 | | `master` (non-scheduled pipelines) | 11 |
...@@ -163,7 +163,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing. ...@@ -163,7 +163,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing.
We follow the [PostgreSQL versions shipped with Omnibus GitLab](https://docs.gitlab.com/omnibus/package-information/postgresql_versions.html): We follow the [PostgreSQL versions shipped with Omnibus GitLab](https://docs.gitlab.com/omnibus/package-information/postgresql_versions.html):
| PG version | 12.10 (April 2020) | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5 | 13.6 (November 2020) | 14.0 (May 2021?) | | PostgreSQL version | 12.10 (April 2020) | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5 | 13.6 (November 2020) | 14.0 (May 2021?) |
| ------ | ------------------ | --------------- | ---------------- | ---------------- | ------------------ | ------------ | -------------------- | ---------------- | | ------ | ------------------ | --------------- | ---------------- | ---------------- | ------------------ | ------------ | -------------------- | ---------------- |
| PG9.6 | MRs/`master`/`2-hour`/`nightly` | - | - | - | - | - | - | - | | PG9.6 | MRs/`master`/`2-hour`/`nightly` | - | - | - | - | - | - | - |
| PG10 | `nightly` | - | - | - | - | - | - | - | | PG10 | `nightly` | - | - | - | - | - | - | - |
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment