Merge branch 'parallel_backup_restore' into 'master'

Support restoring repository backups in parallel

See merge request gitlab-org/gitlab!69330

doc/raketasks/backup_restore.md

@@ -347,18 +347,21 @@ sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_
 
 #### Back up Git repositories concurrently
 
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37158) in GitLab 13.3.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37158) in GitLab 13.3.
+> - [Concurrent restore introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69330) in GitLab 14.3
 
 When using [multiple repository storages](../administration/repository_storage_paths.md),
-repositories can be backed up concurrently to help fully use CPU time. The
+repositories can be backed up or restored concurrently to help fully use CPU time. The
 following variables are available to modify the default behavior of the Rake
 task:
 
 - `GITLAB_BACKUP_MAX_CONCURRENCY`: The maximum number of projects to back up at
-  the same time. Defaults to `1`.
+  the same time. Defaults to the number of logical CPUs (in GitLab 14.1 and
+  earlier, defaults to `1`).
 - `GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY`: The maximum number of projects to
   back up at the same time on each storage. This allows the repository backups
-  to be spread across storages. Defaults to `1`.
+  to be spread across storages. Defaults to `2` (in GitLab 14.1 and earlier,
+  defaults to `1`).
 
 For example, for Omnibus GitLab installations with 4 repository storages:
 

lib/backup/gitaly_backup.rb

@@ -22,8 +22,8 @@ module Backup
                 end
 
       args = []
-      args += ['-parallel', @parallel.to_s] if type == :create && @parallel
-      args += ['-parallel-storage', @parallel_storage.to_s] if type == :create && @parallel_storage
+      args += ['-parallel', @parallel.to_s] if @parallel
+      args += ['-parallel-storage', @parallel_storage.to_s] if @parallel_storage
 
       @stdin, stdout, @thread = Open3.popen2(ENV, bin_path, command, '-path', backup_repos_path, *args)
 

spec/lib/backup/gitaly_backup_spec.rb

@@ -131,8 +131,19 @@ RSpec.describe Backup::GitalyBackup do
     context 'parallel option set' do
       let(:parallel) { 3 }
 
-      it 'does not pass parallel option through' do
-        expect(Open3).to receive(:popen2).with(ENV, anything, 'restore', '-path', anything).and_call_original
+      it 'passes parallel option through' do
+        expect(Open3).to receive(:popen2).with(ENV, anything, 'restore', '-path', anything, '-parallel', '3').and_call_original
+
+        subject.start(:restore)
+        subject.wait
+      end
+    end
+
+    context 'parallel_storage option set' do
+      let(:parallel_storage) { 3 }
+
+      it 'passes parallel option through' do
+        expect(Open3).to receive(:popen2).with(ENV, anything, 'restore', '-path', anything, '-parallel-storage', '3').and_call_original
 
         subject.start(:restore)
         subject.wait