Add rake task to pause or resume elastic indexing

In some cases Elasticsearch indexing may make an instance unstable,
bringing down Puma. This in turn prevents admins from accessing the UI
to pause indexing, leading to extended downtime.

This allows users to pause or resume indexing from the command line.

Closes https://gitlab.com/gitlab-org/gitlab/-/issues/329667.

Changelog: changed
EE: true

doc/integration/elasticsearch.md

@@ -461,6 +461,8 @@ The following are some available Rake tasks:
 | Task                                                                                                                                                    | Description                                                                                                                                                                               |
 |:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake)                            | Enables Elasticsearch indexing and run `gitlab:elastic:create_empty_index`, `gitlab:elastic:clear_index_status`, `gitlab:elastic:index_projects`, and `gitlab:elastic:index_snippets`.                          |
+| [`sudo gitlab-rake gitlab:elastic:pause_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake)                            | Pauses Elasticsearch indexing. Changes are still tracked. Useful for cluster/index migrations. |
+| [`sudo gitlab-rake gitlab:elastic:resume_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake)                            | Resumes Elasticsearch indexing. |
 | [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake)                   | Iterates over all projects and queues Sidekiq jobs to index them in the background.                                                                                                       |
 | [`sudo gitlab-rake gitlab:elastic:index_projects_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake)            | Determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. |
 | [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake)               | Deletes all instances of IndexStatus for all projects. Note that this command will result in a complete wipe of the index, and it should be used with caution.                                                                                              |

ee/lib/tasks/gitlab/elastic.rake

@@ -178,6 +178,30 @@ namespace :gitlab do
       puts 'Please note that it is possible to index only selected namespaces/projects by using Elasticsearch indexing restrictions.'
     end
 
+    desc "GitLab | Elasticsearch | Pause indexing"
+    task pause_indexing: :environment do
+      puts "Pausing indexing...".color(:green)
+
+      if ::Gitlab::CurrentSettings.elasticsearch_pause_indexing?
+        puts "Indexing is already paused.".color(:orange)
+      else
+        ::Gitlab::CurrentSettings.update!(elasticsearch_pause_indexing: true)
+        puts "Indexing is now paused.".color(:green)
+      end
+    end
+
+    desc "GitLab | Elasticsearch | Resume indexing"
+    task resume_indexing: :environment do
+      puts "Resuming indexing...".color(:green)
+
+      if ::Gitlab::CurrentSettings.elasticsearch_pause_indexing?
+        ::Gitlab::CurrentSettings.update!(elasticsearch_pause_indexing: false)
+        puts "Indexing is now running.".color(:green)
+      else
+        puts "Indexing is already running.".color(:orange)
+      end
+    end
+
     def project_id_batches(&blk)
       relation = Project.all
 

ee/spec/tasks/gitlab/elastic_rake_spec.rb

@@ -244,4 +244,60 @@ RSpec.describe 'gitlab:elastic namespace rake tasks', :elastic, :silence_stdout
       expect { subject }.to output(/your cluster size should be at least 20.5 MB/).to_stdout
     end
   end
+
+  describe 'pause_indexing' do
+    subject { run_rake_task('gitlab:elastic:pause_indexing') }
+
+    let(:settings) { ::Gitlab::CurrentSettings }
+
+    before do
+      allow(settings).to receive(:elasticsearch_pause_indexing?).and_return(indexing_paused)
+    end
+
+    context 'when indexing is already paused' do
+      let(:indexing_paused) { true }
+
+      it 'does not do anything' do
+        expect(settings).not_to receive(:update!)
+        expect { subject }.to output(/Indexing is already paused/).to_stdout
+      end
+    end
+
+    context 'when indexing is running' do
+      let(:indexing_paused) { false }
+
+      it 'pauses indexing' do
+        expect(settings).to receive(:update!).with(elasticsearch_pause_indexing: true)
+        expect { subject }.to output(/Indexing is now paused/).to_stdout
+      end
+    end
+
+    describe 'resume_indexing' do
+      subject { run_rake_task('gitlab:elastic:resume_indexing') }
+
+      let(:settings) { ::Gitlab::CurrentSettings }
+
+      before do
+        allow(settings).to receive(:elasticsearch_pause_indexing?).and_return(indexing_paused)
+      end
+
+      context 'when indexing is already running' do
+        let(:indexing_paused) { false }
+
+        it 'does not do anything' do
+          expect(settings).not_to receive(:update!)
+          expect { subject }.to output(/Indexing is already running/).to_stdout
+        end
+      end
+
+      context 'when indexing is not running' do
+        let(:indexing_paused) { true }
+
+        it 'resumes indexing' do
+          expect(settings).to receive(:update!).with(elasticsearch_pause_indexing: false)
+          expect { subject }.to output(/Indexing is now running/).to_stdout
+        end
+      end
+    end
+  end
 end