Merge branch 'fj-215861-add-rake-tasks-to-migrate-snippets' into 'master'

Add snippet migration rake tasks

See merge request gitlab-org/gitlab!30489

changelogs/unreleased/fj-215861-add-rake-tasks-to-migrate-snippets.yml

+---
+title: Add snippet migration rake tasks
+merge_request: 30489
+author:
+type: other

doc/raketasks/README.md

@@ -38,3 +38,4 @@ The following are available Rake tasks:
 | [User management](user_management.md)                                                               | Perform user management tasks.                                                            |
 | [Webhooks administration](web_hooks.md)                                                             | Maintain project Webhooks.                                                                |
 | [X.509 signatures](x509_signatures.md)                                                              | Update X.509 commit signatures, useful if certificate store has changed.                  |
+| [Migrate Snippets to Git](migrate_snippets.md)                                                      | Migrate GitLab Snippets to Git repositories and show migration status                     |

doc/raketasks/migrate_snippets.md

+# Migration to Versioned Snippets **(CORE ONLY)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215861) in GitLab 13.0.
+
+In GitLab 13.0, [GitLab Snippets are backed by Git repositories](../user/snippets.md#versioned-snippets).
+This means that the snippet content will be stored in the repository
+and users can update it directly through Git.
+
+Nevertheless, existing GitLab Snippets have to be migrated to this new functionality.
+For each snippet, a new repository is created and the snippet content is committed
+to the repository inside a file whose name is the file name used in the snippet
+as well.
+
+GitLab performs this migration through a [Background Migration](../development/background_migrations.md)
+automatically when the GitLab instance is upgrade to 13.0 or a higher version.
+However, if the migration fails for any of the snippets, they still need
+to be migrated individually.
+
+The following Rake tasks will help with that process.
+
+## Migrate specific snippets to Git
+
+In case you want to migrate a range of snippets, run the tasks as described below.
+
+For Omnibus installations, run:
+
+```shell
+sudo gitlab-rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4
+```
+
+There is a default limit (100) to the number of ids supported in the migration
+process. You can modify this limit by using the env variable `LIMIT`.
+
+```shell
+sudo gitlab-rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4 LIMIT=50
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4 LIMIT=50
+```
+
+## Show whether the snippet background migration is running
+
+In case you want to check the status of the snippet background migration,
+whether it is running or not, you can use the following task.
+
+For Omnibus installations, run:
+
+```shell
+sudo gitlab-rake gitlab:snippets:migration_status
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:migration_status RAILS_ENV=production
+```
+
+## List non-migrated snippets
+
+With the following task, you can get the ids of all of the snippets
+that haven't been migrated yet or failed to migrate.
+
+For Omnibus installations, run:
+
+```shell
+sudo gitlab-rake gitlab:snippets:list_non_migrated
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:list_non_migrated RAILS_ENV=production
+```
+
+As the number of non-migrated snippets can be large, we limit
+by default the size of the number of ids returned to 100. You can
+modify this limit by using the env variable `LIMIT`.
+
+```shell
+sudo gitlab-rake gitlab:snippets:list_non_migrated LIMIT=200
+```
+
+For installations from source code, run:
+
+```shell
+bundle exec rake gitlab:snippets:list_non_migrated RAILS_ENV=production LIMIT=200
+```

lib/gitlab/background_migration/backfill_snippet_repositories.rb

@@ -8,7 +8,21 @@ module Gitlab
       MAX_RETRIES = 2
 
       def perform(start_id, stop_id)
-        Snippet.includes(:author, snippet_repository: :shard).where(id: start_id..stop_id).find_each do |snippet|
+        snippets = snippet_relation.where(id: start_id..stop_id)
+
+        migrate_snippets(snippets)
+      end
+
+      def perform_by_ids(snippet_ids)
+        snippets = snippet_relation.where(id: snippet_ids)
+
+        migrate_snippets(snippets)
+      end
+
+      private
+
+      def migrate_snippets(snippets)
+        snippets.find_each do |snippet|
           # We need to expire the exists? value for the cached method in case it was cached
           snippet.repository.expire_exists_cache
 
@@ -38,7 +52,9 @@ module Gitlab
         end
       end
 
-      private
+      def snippet_relation
+        @snippet_relation ||= Snippet.includes(:author, snippet_repository: :shard)
+      end
 
       def repository_present?(snippet)
         snippet.snippet_repository && !snippet.empty_repo?

lib/tasks/gitlab/snippets.rake

+# frozen_string_literal: true
+
+namespace :gitlab do
+  namespace :snippets do
+    DEFAULT_LIMIT = 100
+
+    # @example
+    #   bundle exec rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4
+    #   bundle exec rake gitlab:snippets:migrate SNIPPET_IDS=1,2,3,4 LIMIT=50
+    desc 'GitLab | Migrate specific snippets to git'
+    task :migrate, [:ids] => :environment do |_, args|
+      unless ENV['SNIPPET_IDS'].presence
+        raise "Please supply the list of ids through the SNIPPET_IDS env var"
+      end
+
+      raise "Invalid limit value" if limit.zero?
+
+      if migration_running?
+        raise "There are already snippet migrations running. Please wait until they are finished."
+      end
+
+      ids = parse_snippet_ids!
+
+      puts "Starting the migration..."
+      Gitlab::BackgroundMigration::BackfillSnippetRepositories.new.perform_by_ids(ids)
+
+      list_non_migrated = non_migrated_snippets.where(id: ids)
+
+      if list_non_migrated.exists?
+        puts "The following snippets couldn't be migrated:"
+        puts list_non_migrated.pluck(:id).join(',')
+      else
+        puts "All snippets were migrated successfully"
+      end
+    end
+
+    def parse_snippet_ids!
+      ids = ENV['SNIPPET_IDS'].delete(' ').split(',').map do |id|
+        id.to_i.tap do |value|
+          raise "Invalid id provided" if value.zero?
+        end
+      end
+
+      if ids.size > limit
+        raise "The number of ids provided is higher than #{limit}. You can update this limit by using the env var `LIMIT`"
+      end
+
+      ids
+    end
+
+    # @example
+    #   bundle exec rake gitlab:snippets:migration_status
+    desc 'GitLab | Show whether there are snippet background migrations running'
+    task migration_status: :environment do
+      if migration_running?
+        puts "There are snippet migrations running"
+      else
+        puts "There are no snippet migrations running"
+      end
+    end
+
+    def migration_running?
+      Sidekiq::ScheduledSet.new.any? { |r| r.klass == 'BackgroundMigrationWorker' && r.args[0] == 'BackfillSnippetRepositories' }
+    end
+
+    # @example
+    #   bundle exec rake gitlab:snippets:list_non_migrated
+    #   bundle exec rake gitlab:snippets:list_non_migrated LIMIT=50
+    desc 'GitLab | Show non migrated snippets'
+    task list_non_migrated: :environment do
+      raise "Invalid limit value" if limit.zero?
+
+      non_migrated_count = non_migrated_snippets.count
+      if non_migrated_count.zero?
+        puts "All snippets have been successfully migrated"
+      else
+        puts "There are #{non_migrated_count} snippets that haven't been migrated. Showing a batch of ids of those snippets:\n"
+        puts non_migrated_snippets.limit(limit).pluck(:id).join(',')
+      end
+    end
+
+    def non_migrated_snippets
+      @non_migrated_snippets ||= Snippet.select(:id).where.not(id: SnippetRepository.select(:snippet_id))
+    end
+
+    # There are problems with the specs if we memoize this value
+    def limit
+      ENV['LIMIT'] ? ENV['LIMIT'].to_i : DEFAULT_LIMIT
+    end
+  end
+end

spec/tasks/gitlab/snippets_rake_spec.rb

+# frozen_string_literal: true
+
+require 'rake_helper'
+
+describe 'gitlab:snippets namespace rake task' do
+  let_it_be(:user) { create(:user)}
+  let_it_be(:migrated) { create(:personal_snippet, :repository, author: user) }
+  let(:non_migrated) { create_list(:personal_snippet, 3, author: user) }
+  let(:non_migrated_ids) { non_migrated.pluck(:id) }
+
+  before :all do
+    Rake.application.rake_require 'tasks/gitlab/snippets'
+  end
+
+  describe 'migrate' do
+    subject { run_rake_task('gitlab:snippets:migrate') }
+
+    before do
+      stub_env('SNIPPET_IDS' => non_migrated_ids.join(','))
+    end
+
+    it 'can migrate specific snippets passing ids' do
+      expect_next_instance_of(Gitlab::BackgroundMigration::BackfillSnippetRepositories) do |instance|
+        expect(instance).to receive(:perform_by_ids).with(non_migrated_ids).and_call_original
+      end
+
+      expect { subject }.to output(/All snippets were migrated successfully/).to_stdout
+    end
+
+    it 'returns the ids of those snippet that failed the migration' do
+      expect_next_instance_of(Gitlab::BackgroundMigration::BackfillSnippetRepositories) do |instance|
+        expect(instance).to receive(:perform_by_ids).with(non_migrated_ids)
+      end
+
+      expect { subject }.to output(/The following snippets couldn't be migrated:\n#{non_migrated_ids.join(',')}/).to_stdout
+    end
+
+    it 'fails if the SNIPPET_IDS env var is not set' do
+      stub_env('SNIPPET_IDS' => nil)
+
+      expect { subject }.to raise_error(RuntimeError, 'Please supply the list of ids through the SNIPPET_IDS env var')
+    end
+
+    it 'fails if the number of ids provided is higher than the limit' do
+      stub_env('LIMIT' => 2)
+
+      expect { subject }.to raise_error(RuntimeError, /The number of ids provided is higher than/)
+    end
+
+    it 'fails if the env var LIMIT is invalid' do
+      stub_env('LIMIT' => 'foo')
+
+      expect { subject }.to raise_error(RuntimeError, 'Invalid limit value')
+    end
+
+    it 'fails if the ids are invalid' do
+      stub_env('SNIPPET_IDS' => '1,2,a')
+
+      expect { subject }.to raise_error(RuntimeError, 'Invalid id provided')
+    end
+
+    it 'fails if the snippet background migration is running' do
+      Sidekiq::Testing.disable! do
+        BackgroundMigrationWorker.perform_in(180, 'BackfillSnippetRepositories', [non_migrated.first.id, non_migrated.last.id])
+        expect(Sidekiq::ScheduledSet.new).to be_one
+
+        expect { subject }.to raise_error(RuntimeError, /There are already snippet migrations running/)
+
+        Sidekiq::ScheduledSet.new.clear
+      end
+    end
+  end
+
+  describe 'migration_status' do
+    subject { run_rake_task('gitlab:snippets:migration_status') }
+
+    it 'returns a message when the background migration is not running' do
+      expect { subject }.to output("There are no snippet migrations running\n").to_stdout
+    end
+
+    it 'returns a message saying that the background migration is running' do
+      Sidekiq::Testing.disable! do
+        BackgroundMigrationWorker.perform_in(180, 'BackfillSnippetRepositories', [non_migrated.first.id, non_migrated.last.id])
+        expect(Sidekiq::ScheduledSet.new).to be_one
+
+        expect { subject }.to output("There are snippet migrations running\n").to_stdout
+
+        Sidekiq::ScheduledSet.new.clear
+      end
+    end
+  end
+
+  describe 'list_non_migrated' do
+    subject { run_rake_task('gitlab:snippets:list_non_migrated') }
+
+    it 'returns a message if all snippets are migrated' do
+      expect { subject }.to output("All snippets have been successfully migrated\n").to_stdout
+    end
+
+    context 'when there are still non migrated snippets' do
+      let!(:non_migrated) { create_list(:personal_snippet, 3, author: user) }
+
+      it 'returns a message returning the non migrated snippets ids' do
+        expect { subject }.to output(/#{non_migrated_ids}/).to_stdout
+      end
+
+      it 'returns as many snippet ids as the limit set' do
+        stub_env('LIMIT' => 1)
+
+        expect { subject }.to output(/#{non_migrated_ids[0]}/).to_stdout
+      end
+    end
+  end
+end