Add migrating to a new server details to backup and restore

https://gitlab.com/gitlab-org/gitlab/-/issues/344487

doc/raketasks/backup_restore.md

@@ -12,11 +12,11 @@ An application data backup creates an archive file that contains the database,
 all repositories and all attachments.
 
 You can only restore a backup to **exactly the same version and type (CE/EE)**
-of GitLab on which it was created. The best way to migrate your repositories
-from one server to another is through a backup and restore.
+of GitLab on which it was created. The best way to [migrate your projects
+from one server to another](#migrate-to-a-new-server) is through a backup and restore.
 
 WARNING:
-GitLab doesn't back up items that aren't stored in the file system. If you're
+GitLab doesn't back up items that aren't stored on the file system. If you're
 using [object storage](../administration/object_storage.md), be sure to enable
 backups with your object storage provider, if desired.
 
@@ -68,6 +68,7 @@ including:
 Backups do not include:
 
 - [Mattermost data](https://docs.mattermost.com/administration/config-settings.html#file-storage)
+- Redis (and thus Sidekiq jobs)
 
 WARNING:
 GitLab does not back up any configuration files (`/etc/gitlab`), TLS keys and certificates, or system
@@ -983,7 +984,7 @@ your installation is using PgBouncer, for either performance reasons or when usi
 Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary,
 [as previously mentioned](#restore-prerequisites).
 
-Reconfigure, restart and check GitLab:
+Reconfigure, restart and [check](../administration/raketasks/maintenance.md#check-gitlab-configuration) GitLab:
 
 ```shell
 sudo gitlab-ctl reconfigure
@@ -999,6 +1000,14 @@ the target for the restore.
 sudo gitlab-rake gitlab:doctor:secrets
 ```
 
+For added assurance, you can perform [an integrity check on the uploaded files](../administration/raketasks/check.md#uploaded-files-integrity):
+
+```shell
+sudo gitlab-rake gitlab:artifacts:check
+sudo gitlab-rake gitlab:lfs:check
+sudo gitlab-rake gitlab:uploads:check
+```
+
 ### Restore for Docker image and GitLab Helm chart installations
 
 For GitLab installations using the Docker image or the GitLab Helm chart on a
@@ -1180,7 +1189,7 @@ has a longer discussion explaining the potential problems.
 
 To prevent writes to the Git repository data, there are two possible approaches:
 
-- Use [maintenance mode](../administration/maintenance_mode/index.md) to place GitLab in a read-only state.
+- Use [maintenance mode](../administration/maintenance_mode/index.md) **(PREMIUM SELF)** to place GitLab in a read-only state.
 - Create explicit downtime by stopping all Gitaly services before backing up the repositories:
 
   ```shell
@@ -1282,6 +1291,198 @@ sudo GITLAB_BACKUP_PGHOST=192.168.1.10 GITLAB_BACKUP_PGPORT=5432 /opt/gitlab/bin
 See the [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-envars.html)
 for more details on what these parameters do.
 
+## Migrate to a new server
+
+<!-- some details borrowed from GitLab.com move from Azure to GCP detailed at https://gitlab.com/gitlab-com/migration/-/blob/master/.gitlab/issue_templates/failover.md -->
+
+You can use GitLab backup and restore to migrate your instance to a new server. This section outlines a typical procedure for a GitLab deployment running on a single server.
+If you're running GitLab Geo, an alternative option is [Geo disaster recovery for planned failover](../administration/geo/disaster_recovery/planned_failover.md).
+
+WARNING:
+Avoid uncoordinated data processing by both the new and old servers, where multiple
+servers could connect concurrently and process the same data. For example, when using
+[incoming email](../administration/incoming_email.md), if both GitLab instances are
+processing email at the same time, then both instances will end up missing some data.
+This type of problem can occur with other services as well, such as a
+[non-packaged database](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server),
+a non-packaged Redis instance, or non-packaged Sidekiq.
+
+Prerequisites:
+
+- Some time before your migration, consider notifying your users of upcoming
+  scheduled maintenance with a [broadcast message banner](../user/admin_area/broadcast_messages.md).
+- Ensure your backups are complete and current. Create a complete system-level backup, or
+  take a snapshot of all servers involved in the migration, in case destructive commands
+  (like `rm`) are run incorrectly.
+
+### Prepare the new server
+
+To prepare the new server:
+
+1. Copy the
+   [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079)
+   from the old server to avoid man-in-the-middle attack warnings.
+1. [Install and configure GitLab](https://about.gitlab.com/install) except
+   [incoming email](../administration/incoming_email.md):
+   1. Install GitLab.
+   1. Configure by copying `/etc/gitlab` files from the old server to the new server, and update as necessary.
+      Read the
+      [Omnibus configuration backup and restore instructions](https://docs.gitlab.com/omnibus/settings/backups.html) for more detail.
+   1. If applicable, disable [incoming email](../administration/incoming_email.md).
+   1. Block new CI/CD jobs from starting upon initial startup after the backup and restore.
+      Edit `/etc/gitlab/gitlab.rb` and set the following:
+
+      ```ruby
+      nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n"
+      ```
+
+   1. Reconfigure GitLab:
+
+      ```shell
+      sudo gitlab-ctl reconfigure
+      ```
+
+1. Stop GitLab to avoid any potential unnecessary and unintentional data processing:
+
+   ```shell
+   sudo gitlab-ctl stop
+   ```
+
+1. Configure the new server to allow receiving the Redis database and GitLab backup files:
+
+   ```shell
+   sudo rm -f /var/opt/gitlab/redis/dump.rdb
+   sudo chown <your-linux-username> /var/opt/gitlab/redis
+   sudo mkdir /var/opt/gitlab/backups
+   sudo chown <your-linux-username> /var/opt/gitlab/backups
+   ```
+
+### Prepare and transfer content from the old server
+
+1. Ensure you have an up-to-date system-level backup or snapshot of the old server.
+1. Enable [maintenance mode](../administration/maintenance_mode/index.md) **(PREMIUM SELF)**,
+   if supported by your GitLab edition.
+1. Block new CI/CD jobs from starting:
+   1. Edit `/etc/gitlab/gitlab.rb`, and set the following:
+
+      ```ruby
+      nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n"
+      ```
+
+   1. Reconfigure GitLab:
+
+      ```shell
+      sudo gitlab-ctl reconfigure
+      ```
+
+1. Disable periodic background jobs:
+   1. On the top bar, select **Menu > Admin**.
+   1. On the left sidebar, select **Monitoring > Background Jobs**.
+   1. Under the Sidekiq dashboard, select **Cron** tab and then
+      **Disable All**.
+1. Wait for the currently running CI/CD jobs to finish, or accept that jobs that have not completed may be lost.
+   To view jobs currently running, on the left sidebar, select **Overviews > Jobs**,
+   and then select **Running**.
+1. Wait for Sidekiq jobs to finish:
+   1. On the left sidebar, select **Monitoring > Background Jobs**.
+   1. Under the Sidekiq dashboard, select **Queues** and then **Live Poll**.
+      Wait for **Busy** and **Enqueued** to drop to 0.
+      These queues contain work that has been submitted by your users;
+      shutting down before these jobs complete may cause the work to be lost.
+      Make note of the numbers shown in the Sidekiq dashboard for post-migration verification.
+1. Flush the Redis database to disk, and stop GitLab other than the services needed for migration:
+
+   ```shell
+   sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket save && sudo gitlab-ctl stop && sudo gitlab-ctl start postgresql
+   ```
+
+1. Create a GitLab backup:
+
+   ```shell
+   sudo gitlab-backup create
+   ```
+
+1. Disable the following GitLab services and prevent unintentional restarts by adding the following to the bottom of `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   alertmanager['enable'] = false
+   gitlab_exporter['enable'] = false
+   gitlab_pages['enable'] = false
+   gitlab_workhorse['enable'] = false
+   grafana['enable'] = false
+   logrotate['enable'] = false
+   gitlab_rails['incoming_email_enabled'] = false
+   nginx['enable'] = false
+   node_exporter['enable'] = false
+   postgres_exporter['enable'] = false
+   postgresql['enable'] = false
+   prometheus['enable'] = false
+   puma['enable'] = false
+   redis['enable'] = false
+   redis_exporter['enable'] = false
+   registry['enable'] = false
+   sidekiq['enable'] = false
+   ```
+
+1. Reconfigure GitLab:
+
+   ```shell
+   sudo gitlab-ctl reconfigure
+   ```
+
+1. Verify everything is stopped, and confirm no services are running:
+
+   ```shell
+   sudo gitlab-ctl status
+   ```
+
+1. Transfer the Redis database and GitLab backups to the new server:
+
+   ```shell
+   sudo scp /var/opt/gitlab/redis/dump.rdb <your-linux-username>@new-server:/var/opt/gitlab/redis
+   sudo scp /var/opt/gitlab/backups/your-backup.tar <your-linux-username>@new-server:/var/opt/gitlab/backups
+   ```
+
+### Restore data on the new server
+
+1. Restore appropriate file system permissions:
+
+   ```shell
+   sudo chown gitlab-redis /var/opt/gitlab/redis
+   sudo chown gitlab-redis:gitlab-redis /var/opt/gitlab/redis/dump.rdb
+   sudo chown git:root /var/opt/gitlab/backups
+   sudo chown git:git /var/opt/gitlab/backups/your-backup.tar
+   ```
+
+1. [Restore the GitLab backup](#restore-gitlab).
+1. Verify that the Redis database restored correctly:
+   1. On the top bar, select **Menu > Admin**.
+   1. On the left sidebar, select **Monitoring > Background Jobs**.
+   1. Under the Sidekiq dashboard, verify that the numbers
+      match with what was shown on the old server.   
+   1. While still under the Sidekiq dashboard, select **Cron** and then **Enable All**
+      to re-enable periodic background jobs.
+1. Test that read-only operations on the GitLab instance work as expected. For example, browse through project repository files, merge requests, and issues.
+1. Disable [Maintenance Mode](../administration/maintenance_mode/index.md) **(PREMIUM SELF)**, if previously enabled.
+1. Test that the GitLab instance is working as expected.
+1. If applicable, re-enable [incoming email](../administration/incoming_email.md) and test it is working as expected.
+1. Update your DNS or load balancer to point at the new server.
+1. Unblock new CI/CD jobs from starting by removing the custom NGINX config
+   you added previously:
+
+   ```ruby
+   # The following line must be removed
+   nginx['custom_gitlab_server_config'] = "location /api/v4/jobs/request {\n deny all;\n return 503;\n}\n"
+   ```
+
+1. Reconfigure GitLab:
+
+   ```shell
+   sudo gitlab-ctl reconfigure
+   ```
+
+1. Remove the scheduled maintenance [broadcast message banner](../user/admin_area/broadcast_messages.md).
+
 ## Additional notes
 
 This documentation is for GitLab Community and Enterprise Edition. We back up