rsync and Gitaly, gitlab:cleanup:repos is gone, attributes on Rails cheat sheet

419fae3b · Ben Prescott @bprescott_↙ ☺ · Evan Read · 157f80c6 · 419fae3b · 419fae3b
Commit 419fae3b authored Oct 29, 2020 by Ben Prescott @bprescott_↙ ☺ Committed by Evan Read Oct 29, 2020
3 changed files
--- a/doc/administration/operations/moving_repositories.md
+++ b/doc/administration/operations/moving_repositories.md
@@ -5,10 +5,35 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 type: reference
 ---

-# Moving repositories managed by GitLab
+# Moving repositories managed by GitLab **(CORE ONLY)**

 Sometimes you need to move all repositories managed by GitLab to
-another file system or another server. In this document we look
+another file system or another server.
+
+## Moving data within a GitLab instance
+
+The recommended way to move Git repositories between servers, between different storage, and
+from unclustered to clustered Gitaly (Praefect) is using the API.
+
+Read more:
+
+- [Configuring additional storage for Gitaly](../gitaly/index.md#network-architecture)
+  - Within this example, additional storage called `storage1` and `storage2` is configured.
+- [The API documentation](../../api/project_repository_storage_moves.md) details the endpoints for quering and scheduling repository moves.
+- [Migrate existing repositories to Gitaly Cluster](../gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster)
+
+### Limitations
+
+Read more in the [API documentation](../../api/project_repository_storage_moves.md#limitations).
+
+## Migrating to another GitLab instance
+
+Using the API isn't an option if you are migrating to a new GitLab environment, for example:
+
+- From a single-node GitLab to a scaled-out architecture.
+- From a GitLab instance in your private datacenter to a cloud provider.
+
+The rest of the document will look
 at some of the ways you can copy all your repositories from
 `/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`.

@@ -22,7 +47,14 @@ DANGER: **Warning:**
 Each of the approaches we list can or does overwrite data in the target directory
 `/mnt/gitlab/repositories`. Do not mix up the source and the target.

-## Target directory is empty: use a `tar` pipe
+### Recommended approach in all cases
+
+GitLab's [backup and restore capability](../../raketasks/backup_restore.md) should be used. Git repositories are accessed, managed and stored on GitLab servers by the Gitaly component of the product as a database. Data loss can result from directly accessing and copying Gitaly's files using tools like `rsync`.
+
+- From GitLab 13.3, backup performance can be improved by [processing multiple repositories concurrently](../../raketasks/backup_restore.md#back-up-git-repositories-concurrently).
+- Backups can be created of just the repositories using the [skip feature](../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup)
+
+### Target directory is empty: use a `tar` pipe

 If the target directory `/mnt/gitlab/repositories` is empty the
 simplest thing to do is to use a `tar` pipe. This method has low
@@ -37,7 +69,7 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\

 If you want to see progress, replace `-xf` with `-xvf`.

-### `tar` pipe to another server
+#### `tar` pipe to another server

 You can also use a `tar` pipe to copy data to another server. If your
 `git` user has SSH access to the new server as `git@newserver`, you
@@ -51,7 +83,12 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
 If you want to compress the data before it goes over the network
 (which costs you CPU cycles) you can replace `ssh` with `ssh -C`.

-## The target directory contains an outdated copy of the repositories: use `rsync`
+### The target directory contains an outdated copy of the repositories: use `rsync`
+
+DANGER: **Warning:**
+Using `rsync` to migrate
+Git data can cause data loss and repository corruption.
+[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422).

 If the target directory already contains a partial / outdated copy
 of the repositories it may be wasteful to copy all the data again
@@ -68,7 +105,12 @@ The `/.` in the command above is very important, without it you can
 easily get the wrong directory structure in the target directory.
 If you want to see progress, replace `-a` with `-av`.

-### Single `rsync` to another server
+#### Single `rsync` to another server
+
+DANGER: **Warning:**
+Using `rsync` to migrate
+Git data can cause data loss and repository corruption.
+[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422).

 If the `git` user on your source system has SSH access to the target
 server you can send the repositories over the network with `rsync`.
@@ -78,7 +120,12 @@ sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
  git@newserver:/mnt/gitlab/repositories'
 ```

-## Thousands of Git repositories: use one `rsync` per repository
+### Thousands of Git repositories: use one `rsync` per repository
+
+DANGER: **Warning:**
+Using `rsync` to migrate
+Git data can cause data loss and repository corruption.
+[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422).

 Every time you start an `rsync` job it has to inspect all files in
 the source directory, all files in the target directory, and then
@@ -93,11 +140,14 @@ This utility is not included in GitLab so you need to install it yourself with `
 or `yum`. Also note that the GitLab scripts we used below were added in GitLab 8.1.

 **This process does not clean up repositories at the target location that no
-longer exist at the source.** If you start using your GitLab instance with
-`/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos`
-after switching to the new repository storage directory.
+longer exist at the source.**
+
+#### Parallel `rsync` for all repositories known to GitLab

-### Parallel `rsync` for all repositories known to GitLab
+DANGER: **Warning:**
+Using `rsync` to migrate
+Git data can cause data loss and repository corruption.
+[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422).

 This syncs repositories with 10 `rsync` processes at a time. We keep
 track of progress so that the transfer can be restarted if necessary.
@@ -154,7 +204,12 @@ cat /home/git/transfer-logs/* | sort | uniq -u |\
 `
 ```

-### Parallel `rsync` only for repositories with recent activity
+#### Parallel `rsync` only for repositories with recent activity
+
+DANGER: **Warning:**
+Using `rsync` to migrate
+Git data can cause data loss and repository corruption.
+[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422).

 Suppose you have already done one sync that started after 2015-10-1 12:00 UTC.
 Then you might only want to sync repositories that were changed via GitLab

--- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
+++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
@@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 type: reference
 ---

-# GitLab Rails Console Cheat Sheet
+# GitLab Rails Console Cheat Sheet **(CORE ONLY)**

 This is the GitLab Support Team's collection of information regarding the GitLab Rails
 console, for use while troubleshooting. It is listed here for transparency,
@@ -46,6 +46,40 @@ instance_of_object.method(:foo).source_location
 project.method(:private?).source_location
 ```

+## Attributes
+
+View available attributes, formatted using pretty print (`pp`).
+
+For example, determine what attributes contain users' names and email addresses:
+
+```ruby
+u = User.find_by_username('someuser')
+pp u.attributes
+```
+
+Partial output:
+
+```plaintext
+{"id"=>1234,
+ "email"=>"someuser@example.com",
+ "sign_in_count"=>99,
+ "name"=>"S User",
+ "username"=>"someuser",
+ "first_name"=>nil,
+ "last_name"=>nil,
+ "bot_type"=>nil}
+```
+
+Then make use of the attributes, [testing SMTP, for example](https://docs.gitlab.com/omnibus/settings/smtp.html#testing-the-smtp-configuration):
+
+```ruby
+e = u.email
+n = u.name
+Notify.test_email(e, "Test email for #{n}", 'Test email').deliver_now
+#
+Notify.test_email(u.email, "Test email for #{u.name}", 'Test email').deliver_now
+```
+
 ## Query the database using an ActiveRecord Model

 ```ruby
@@ -148,7 +182,7 @@ project.repository.expire_exists_cache
 Project.update_all(visibility_level: 0)
 ```

-### Find & remove projects that are pending deletion
+### Find projects that are pending deletion

 ```ruby
 #
@@ -177,8 +211,6 @@ project = Project.find_by_full_path('group-changeme/project-changeme')
 ::Projects::DestroyService.new(project, user, {}).execute
 ```

-Next, run `sudo gitlab-rake gitlab:cleanup:repos` on the command line to finish.
-
 ### Destroy a project

 ```ruby

--- a/doc/api/project_repository_storage_moves.md
+++ b/doc/api/project_repository_storage_moves.md
@@ -30,6 +30,11 @@ read-only. Please try again later.` message if they try to push new commits.

 This API requires you to [authenticate yourself](README.md#authentication) as an administrator.

+## Limitations
+
+- [The repositories associated with snippets can't currently be moved with the API](https://gitlab.com/groups/gitlab-org/-/epics/3393).
+- [Group level wikis can't currently be moved with the API](https://gitlab.com/gitlab-org/gitlab/-/issues/219003).
+
 ## Retrieve all project repository storage moves

 ```plaintext