Merge branch 'docs-jayo-geo-replicator-password-rotation' into 'master'

Documentation: Add gitlab replication password change instructions

See merge request gitlab-org/gitlab!69585

doc/administration/geo/setup/database.md

@@ -501,6 +501,79 @@ two other clusters of nodes supporting a Geo **secondary** site. One for the
 main database and the other for the tracking database. For more information,
 see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md).
 
+### Changing the replication password
+
+To change the password for the [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication)
+when using Omnibus-managed PostgreSQL instances:
+
+On the GitLab Geo **primary** server:
+
+1. The default value for the replication user is `gitlab_replicator`, but if you've set a custom replication
+   user in your `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']` setting, make sure to
+   adapt the following instructions for your own user.
+
+   Generate an MD5 hash of the desired password:
+
+   ```shell
+   sudo gitlab-ctl pg-password-md5 gitlab_replicator
+   # Enter password: <your_password_here>
+   # Confirm password: <your_password_here>
+   # 950233c0dfc2f39c64cf30457c3b7f1e
+   ```
+
+   Edit `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab_replicator`
+   postgresql['sql_replication_password'] = '<md5_hash_of_your_password>'
+   ```
+
+1. Save the file and reconfigure GitLab to change the replication user's password in PostgreSQL:
+
+   ```shell
+   sudo gitlab-ctl reconfigure
+   ```
+
+1. Restart PostgreSQL for the replication password change to take effect:
+
+   ```shell
+   sudo gitlab-ctl restart postgresql
+   ```
+
+Until the password is updated on any **secondary** servers, the [PostgreSQL log](../../logs.md#postgresql-logs) on
+the secondaries will report the following error message:
+
+```console
+FATAL:  could not connect to the primary server: FATAL:  password authentication failed for user "gitlab_replicator"
+```
+
+On all GitLab Geo **secondary** servers:
+
+1. The first step isn't necessary from a configuration perspective, since the hashed `'sql_replication_password'`
+   is not used on the GitLab Geo **secondary**. However in the event that **secondary** needs to be promoted
+   to the GitLab Geo **primary**, make sure to match the `'sql_replication_password'` in the secondary
+   server configuration.
+
+   Edit `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab_replicator` on the Geo primary
+   postgresql['sql_replication_password'] = '<md5_hash_of_your_password>'
+   ```
+
+1. During the initial replication setup, the `gitlab-ctl replicate-geo-database` command writes the plaintext
+   password for the replication user account to two locations:
+
+   - `gitlab-geo.conf`: Used by the PostgreSQL replication process, written to the PostgreSQL data
+      directory, by default at `/var/opt/gitlab/postgresql/data/gitlab-geo.conf`.
+   - `.pgpass`: Used by the `gitlab-psql` user, located by default at `/var/opt/gitlab/postgresql/.pgpass`.
+
+   Update the plaintext password in both of these files, and restart PostgreSQL:
+
+   ```shell
+   sudo gitlab-ctl restart postgresql
+   ```
+
 ## Multi-node database replication
 
 In GitLab 14.0, Patroni replaced `repmgr` as the supported