Merge branch '281696-mark-reindex-failed' into 'master'

Create a rake command to mark reindex job failed

See merge request gitlab-org/gitlab!48938

doc/integration/elasticsearch.md

@@ -441,6 +441,22 @@ After the reindexing is completed, the original index will be scheduled to be de
 
 While the reindexing is running, you will be able to follow its progress under that same section.
 
+### Mark the most recent reindex job as failed and unpause the indexing
+
+Sometimes, you might want to abandon the unfinished reindex job and unpause the indexing. You can achieve this via the following steps:
+
+1. Mark the most recent reindex job as failed:
+
+   ```shell
+   # Omnibus installations
+   sudo gitlab-rake gitlab:elastic:mark_reindex_failed
+
+   # Installations from source
+   bundle exec rake gitlab:elastic:mark_reindex_failed RAILS_ENV=production
+   ```
+
+1. Uncheck the "Pause Elasticsearch indexing" checkbox in **Admin Area > Settings > General > Advanced Search**.
+
 ## Background migrations
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/234046) in GitLab 13.6.
@@ -511,7 +527,8 @@ The following are some available Rake tasks:
 | [`sudo gitlab-rake gitlab:elastic:recreate_index[<TARGET_NAME>]`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)     | Wrapper task for `gitlab:elastic:delete_index[<TARGET_NAME>]` and `gitlab:elastic:create_empty_index[<TARGET_NAME>]`.                                                                       |
 | [`sudo gitlab-rake gitlab:elastic:index_snippets`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)                   | Performs an Elasticsearch import that indexes the snippets data.                                                                                                                          |
 | [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)             | Displays which projects are not indexed.                                                                                                                                                  |
-| [`sudo gitlab-rake gitlab:elastic:reindex_cluster`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)             | Schedules a zero-downtime cluster reindexing task. This feature should be used with an index that was created after GitLab 13.0. |
+| [`sudo gitlab-rake gitlab:elastic:reindex_cluster`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)                  | Schedules a zero-downtime cluster reindexing task. This feature should be used with an index that was created after GitLab 13.0. |
+| [`sudo gitlab-rake gitlab:elastic:mark_reindex_failed`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)`]            | Mark the most recent re-index job as failed. |
 
 NOTE:
 The `TARGET_NAME` parameter is optional and will use the default index/alias name from the current `RAILS_ENV` if not set.
@@ -789,7 +806,7 @@ There are a couple of ways to achieve that:
   This is always correctly identifying whether the current project/namespace
   being searched is using Elasticsearch.
 
-- From the admin area under **Settings > General > Elasticsearch** check that the
+- From the admin area under **Settings > General > Advanced Search** check that the
   Advanced Search settings are checked.
 
   Those same settings there can be obtained from the Rails console if necessary:

ee/changelogs/unreleased/281696-mark-reindex-failed.yml

+---
+title: Create a rake command to mark reindex job failed
+merge_request: 48938
+author:
+type: added

ee/lib/tasks/gitlab/elastic.rake

@@ -114,6 +114,16 @@ namespace :gitlab do
       end
     end
 
+    desc "GitLab | Elasticsearch | Mark last reindexing job as failed"
+    task mark_reindex_failed: :environment do
+      if Elastic::ReindexingTask.running?
+        Elastic::ReindexingTask.current.failure!
+        puts 'Marked the current reindexing job as failed.'.color(:green)
+      else
+        puts 'Did not find the current running reindexing job.'
+      end
+    end
+
     def project_id_batches(&blk)
       relation = Project.all
 

ee/spec/tasks/gitlab/elastic_rake_spec.rb

@@ -119,4 +119,28 @@ RSpec.describe 'gitlab:elastic namespace rake tasks', :elastic do
       }.from(false).to(true)
     end
   end
+
+  describe 'mark_reindex_failed' do
+    subject { run_rake_task('gitlab:elastic:mark_reindex_failed') }
+
+    context 'when there is a running reindex job' do
+      before do
+        Elastic::ReindexingTask.create!
+      end
+
+      it 'marks the current reindex job as failed' do
+        expect { subject }.to change {Elastic::ReindexingTask.running?}.from(true).to(false)
+      end
+
+      it 'prints a message after marking it as failed' do
+        expect { subject }.to output("Marked the current reindexing job as failed.\n").to_stdout
+      end
+    end
+
+    context 'when no running reindex job' do
+      it 'just prints a message' do
+        expect { subject }.to output("Did not find the current running reindexing job.\n").to_stdout
+      end
+    end
+  end
 end