Merge branch 'mk/docs-improve-pg-upgrade' into 'master'

Geo: Improve docs to help with PG upgrades

Closes #213921

See merge request gitlab-org/gitlab!37830

doc/administration/geo/replication/multiple_servers.md

@@ -196,9 +196,27 @@ the **primary** database. Use the following as a guide.
    geo_postgresql['enable'] = false
 
    ##
-   ## Disable `geo_logcursor` service so Rails doesn't get configured here
+   ## Disable all other services that aren't needed. Note that we had to enable
+   ## geo_secondary_role to cause some configuration changes to postgresql, but
+   ## the role enables single-node services by default.
    ##
+   alertmanager['enable'] = false
+   consul['enable'] = false
    geo_logcursor['enable'] = false
+   gitaly['enable'] = false
+   gitlab_exporter['enable'] = false
+   gitlab_workhorse['enable'] = false
+   nginx['enable'] = false
+   node_exporter['enable'] = false
+   pgbouncer_exporter['enable'] = false
+   prometheus['enable'] = false
+   redis['enable'] = false
+   redis_exporter['enable'] = false
+   repmgr['enable'] = false
+   sidekiq['enable'] = false
+   sidekiq_cluster['enable'] = false
+   puma['enable'] = false
+   unicorn['enable'] = false
    ```
 
 After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect.
@@ -253,11 +271,11 @@ Configure the tracking database.
    gitlab_rails['auto_migrate'] = false
 
    ##
-   ## Disable all other services that aren't needed, since we don't have a role
-   ## that does this.
+   ## Ensure unnecessary services are disabled
    ##
    alertmanager['enable'] = false
    consul['enable'] = false
+   geo_logcursor['enable'] = false
    gitaly['enable'] = false
    gitlab_exporter['enable'] = false
    gitlab_workhorse['enable'] = false
@@ -270,7 +288,9 @@ Configure the tracking database.
    redis_exporter['enable'] = false
    repmgr['enable'] = false
    sidekiq['enable'] = false
+   sidekiq_cluster['enable'] = false
    puma['enable'] = false
+   unicorn['enable'] = false
    ```
 
 After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect.
@@ -417,6 +437,7 @@ application servers above, with some changes to run only the `sidekiq` service:
    redis_exporter['enable'] = false
    repmgr['enable'] = false
    puma['enable'] = false
+   unicorn['enable'] = false
 
    ##
    ## The unique identifier for the Geo node.

doc/administration/geo/replication/troubleshooting.md

@@ -514,6 +514,82 @@ to start again from scratch, there are a few steps that can help you:
    gitlab-rake geo:db:refresh_foreign_tables
    ```
 
+## Fixing errors during a PostgreSQL upgrade or downgrade
+
+### Message: `ERROR: psql: FATAL:  role "gitlab-consul" does not exist`
+
+When
+[upgrading PostgreSQL on a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance), you might encounter the
+following error:
+
+```plaintext
+$ sudo gitlab-ctl pg-upgrade --target-version=11
+Checking for an omnibus managed postgresql: OK
+Checking if postgresql['version'] is set: OK
+Checking if we already upgraded: NOT OK
+Checking for a newer version of PostgreSQL to install
+Upgrading PostgreSQL to 11.7
+Checking if PostgreSQL bin files are symlinked to the expected location: OK
+Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
+See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
+If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details
+
+Please hit Ctrl-C now if you want to cancel the operation.
+..............................Detected an HA cluster.
+Error running command: /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul
+ERROR: psql: FATAL:  role "gitlab-consul" does not exist
+Traceback (most recent call last):
+    10: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
+      9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
+      8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
+      7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
+      6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
+      5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:171:in `block in load_file'
+      4: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:248:in `is_master?'
+      3: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:100:in `execute_psql'
+      2: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:113:in `cmd'
+      1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:287:in `error!'
+/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '2' (Mixlib::ShellOut::ShellCommandFailed)
+---- Begin output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ----
+STDOUT:
+STDERR: psql: FATAL:  role "gitlab-consul" does not exist
+---- End output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ----
+Ran /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul returned 2
+```
+
+If you are upgrading the PostgreSQL read-replica of a Geo secondary node, and
+you are not using `consul` or `repmgr`, you may need to disable `consul` and/or
+`repmgr` services in `gitlab.rb`:
+
+```ruby
+consul['enable'] = false
+repmgr['enable'] = false
+```
+
+Then reconfigure GitLab:
+
+```shell
+sudo gitlab-ctl reconfigure
+```
+
+### Message: `ActiveRecord::StatementInvalid: PG::SqlclientUnableToEstablishSqlconnection: ERROR:  could not connect to server "gitlab_secondary"`
+
+When
+[upgrading PostgreSQL on a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance), or when downgrading, you
+might encounter the following error:
+
+```plaintext
+$ sudo gitlab-rake geo:db:refresh_foreign_tables
+
+Refreshing foreign tables for FDW: gitlab_secondary ... rake aborted!
+ActiveRecord::StatementInvalid: PG::SqlclientUnableToEstablishSqlconnection: ERROR:  could not connect to server "gitlab_secondary"
+DETAIL:  SSL error: certificate verify failed
+FATAL:  no pg_hba.conf entry for host "10.138.0.59", user "gitlab", database "gitlabhq_production", SSL off
+```
+
+You may need to `gitlab-ctl restart` the read-replica DB node for its PostgreSQL
+server to recognize recent changes.
+
 ## Fixing errors during a failover or when promoting a secondary to a primary node
 
 The following are possible errors that might be encountered during failover or

doc/administration/geo/replication/version_specific_updates.md

@@ -11,6 +11,22 @@ Check this document if it includes instructions for the version you are updating
 These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps)
 for updating Geo nodes.
 
+## Updating to GitLab 13.0
+
+Upgrading to GitLab 13.0 requires GitLab 12.10 to already be using PostgreSQL
+version 11. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
+## Updating to GitLab 12.10
+
+GitLab 12.10 does not attempt to automatically update the embedded PostgreSQL
+server when using Geo, because the PostgreSQL upgrade requires downtime on
+secondaries while reinitializing streaming replication. It must be upgraded
+manually. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
 ## Updating to GitLab 12.9
 
 CAUTION: **Warning:**
@@ -18,6 +34,32 @@ GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops
 repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523).
 The issue is fixed in GitLab 12.9.4. Please upgrade to GitLab 12.9.4 or later.
 
+By default, GitLab 12.9 will attempt to automatically update the embedded
+PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries
+while reinitializing streaming replication. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
+This can be temporarily disabled by running the following before updating:
+
+```shell
+sudo touch /etc/gitlab/disable-postgresql-upgrade
+```
+
+## Updating to GitLab 12.8
+
+By default, GitLab 12.8 will attempt to automatically update the embedded
+PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries
+while reinitializing streaming replication. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
+This can be temporarily disabled by running the following before updating:
+
+```shell
+sudo touch /etc/gitlab/disable-postgresql-upgrade
+```
+
 ## Updating to GitLab 12.7
 
 DANGER: **Danger:**
@@ -28,8 +70,92 @@ bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo
 fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was
 shipped in 12.7.5.
 
+By default, GitLab 12.7 will attempt to automatically update the embedded
+PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while
+reinitializing streaming replication. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
+This can be temporarily disabled by running the following before updating:
+
+```shell
+sudo touch /etc/gitlab/disable-postgresql-upgrade
+```
+
+## Updating to GitLab 12.6
+
+By default, GitLab 12.6 will attempt to automatically update the embedded
+PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while
+reinitializing streaming replication. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
+This can be temporarily disabled by running the following before updating:
+
+```shell
+sudo touch /etc/gitlab/disable-postgresql-upgrade
+```
+
+## Updating to GitLab 12.5
+
+By default, GitLab 12.5 will attempt to automatically update the embedded
+PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while
+reinitializing streaming replication. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
+This can be temporarily disabled by running the following before updating:
+
+```shell
+sudo touch /etc/gitlab/disable-postgresql-upgrade
+```
+
+## Updating to GitLab 12.4
+
+By default, GitLab 12.4 will attempt to automatically update the embedded
+PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while
+reinitializing streaming replication. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
+This can be temporarily disabled by running the following before updating:
+
+```shell
+sudo touch /etc/gitlab/disable-postgresql-upgrade
+```
+
+## Updating to GitLab 12.3
+
+DANGER: **Danger:**
+If the existing PostgreSQL server version is 9.6.x, it is recommended to
+upgrade to GitLab 12.4 or newer. By default, GitLab 12.3 will attempt to
+automatically update the embedded PostgreSQL server to 10.9 from 9.6. In
+certain circumstances, it will fail. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for more information.
+
+Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade
+requires downtime on secondaries while reinitializing streaming replication.
+Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
 ## Updating to GitLab 12.2
 
+DANGER: **Danger:**
+If the existing PostgreSQL server version is 9.6.x, it is recommended to
+upgrade to GitLab 12.4 or newer. By default, GitLab 12.2 will attempt to
+automatically update the embedded PostgreSQL server to 10.9 from 9.6. In
+certain circumstances, it will fail. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for more information.
+
+Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade
+requires downtime on secondaries while reinitializing streaming replication.
+Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for the recommended procedure.
+
 GitLab 12.2 includes the following minor PostgreSQL updates:
 
 - To version `9.6.14` if you run PostgreSQL 9.6.
@@ -48,17 +174,20 @@ The restart avoids a version mismatch when PostgreSQL tries to load the FDW exte
 
 ## Updating to GitLab 12.1
 
-By default, GitLab 12.1 will attempt to automatically update the
-embedded PostgreSQL server to 10.7 from 9.6. Please see
-[the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+DANGER: **Danger:**
+If the existing PostgreSQL server version is 9.6.x, it is recommended to
+upgrade to GitLab 12.4 or newer. By default, GitLab 12.1 will attempt to
+automatically update the embedded PostgreSQL server to 10.9 from 9.6. In
+certain circumstances, it will fail. Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
+for more information.
+
+Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade
+requires downtime on secondaries while reinitializing streaming replication.
+Please see
+[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
 for the recommended procedure.
 
-This can be temporarily disabled by running the following before updating:
-
-```shell
-sudo touch /etc/gitlab/disable-postgresql-upgrade
-```
-
 ## Updating to GitLab 12.0
 
 CAUTION: **Warning:**