Commit 319864ba authored by Rémy Coutable's avatar Rémy Coutable

Revert "Merge branch 'eread/improve-rake-task-topics' into 'master'"

This reverts merge request !30124
parent 983cb647
......@@ -209,7 +209,6 @@ Piwik
PgBouncer
plaintext
PostgreSQL
precompile
preconfigure
preconfigured
preconfigures
......
# Integrity check Rake task
# Integrity Check Rake Task
GitLab provides Rake tasks to check the integrity of various components.
## Repository integrity
## Repository Integrity
Even though Git is very resilient and tries to prevent data integrity issues,
there are times when things go wrong. The following Rake tasks intend to
......@@ -45,7 +43,7 @@ sudo gitlab-rake gitlab:git:fsck
sudo -u git -H bundle exec rake gitlab:git:fsck RAILS_ENV=production
```
## Uploaded files integrity
## Uploaded Files Integrity
Various types of files can be uploaded to a GitLab installation by users.
These integrity checks can detect missing files. Additionally, for locally
......@@ -129,7 +127,7 @@ Checking integrity of Uploads
Done!
```
## LDAP check
## LDAP Check
The LDAP check Rake task will test the bind DN and password credentials
(if configured) and will list a sample of LDAP users. This task is also
......
# Geo Rake Tasks **(PREMIUM ONLY)**
The following Rake tasks are for [Geo installations](../geo/replication/index.md).
## Git housekeeping
There are few tasks you can run to schedule a Git housekeeping to start at the
next repository sync in a **secondary** node:
next repository sync in a **Secondary node**:
### Incremental Repack
......
......@@ -9,7 +9,7 @@ which will become the owner of the project. You can resume an import
with the same command.
Bear in mind that the syntax is very specific. Remove any spaces within the argument block and
before/after the brackets. Also, some shells (for example, `zsh`) can interpret the open/close brackets
before/after the brackets. Also, Some shells (e.g., zsh) can interpret the open/close brackets
(`[]`) separately. You may need to either escape the brackets or use double quotes.
## Importing multiple projects
......
# LDAP Rake tasks
The following are LDAP-related Rake tasks.
# LDAP Rake Tasks
## Check
......@@ -28,7 +26,7 @@ limit by passing a number to the check task:
rake gitlab:ldap:check[50]
```
## Run a group sync
## Run a Group Sync
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.2.
......
# Maintenance Rake tasks
# Maintenance Rake Tasks
GitLab provides Rake tasks for general maintenance.
## Gather information about GitLab and the system it runs on
## Gather GitLab and system information
This command gathers information about your GitLab installation and the system it runs on.
These may be useful when asking for help or reporting issues.
This command gathers information about your GitLab installation and the System it runs on. These may be useful when asking for help or reporting issues.
**Omnibus Installation**
......@@ -53,23 +50,20 @@ Git: /usr/bin/git
## Check GitLab configuration
The `gitlab:check` Rake task runs the following Rake tasks:
Runs the following Rake tasks:
- `gitlab:gitlab_shell:check`
- `gitlab:gitaly:check`
- `gitlab:sidekiq:check`
- `gitlab:app:check`
It will check that each component was set up according to the installation guide and suggest fixes
for issues found. This command must be run from your application server and will not work correctly on
component servers like [Gitaly](../gitaly/index.md#running-gitaly-on-its-own-server).
You may also have a look at our troubleshooting guides for:
It will check that each component was set up according to the installation guide and suggest fixes for issues found.
This command must be run from your app server and will not work correctly on component servers like [Gitaly](../gitaly/index.md#running-gitaly-on-its-own-server).
- [GitLab](../index.md#troubleshooting)
- [Omnibus GitLab](https://docs.gitlab.com/omnibus/README.html#troubleshooting)
You may also have a look at our Troubleshooting Guides:
To run `gitlab:check`, run:
- [Troubleshooting Guide (GitLab)](../index.md#troubleshooting)
- [Troubleshooting Guide (Omnibus GitLab)](https://docs.gitlab.com/omnibus/README.html#troubleshooting)
**Omnibus Installation**
......@@ -83,8 +77,7 @@ sudo gitlab-rake gitlab:check
bundle exec rake gitlab:check RAILS_ENV=production
```
NOTE: **Note:**
Use `SANITIZE=true` for `gitlab:check` if you want to omit project names from the output.
NOTE: Use `SANITIZE=true` for `gitlab:check` if you want to omit project names from the output.
Example output:
......@@ -133,7 +126,7 @@ Checking GitLab ... Finished
## Rebuild authorized_keys file
In some case it is necessary to rebuild the `authorized_keys` file. To do this, run:
In some case it is necessary to rebuild the `authorized_keys` file.
**Omnibus Installation**
......@@ -148,8 +141,6 @@ cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production
```
Example output:
```plaintext
This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.
......@@ -158,8 +149,8 @@ Do you want to continue (yes/no)? yes
## Clear Redis cache
If for some reason the dashboard displays the wrong information, you might want to
clear Redis' cache. To do this, run:
If for some reason the dashboard shows wrong information you might want to
clear Redis' cache.
**Omnibus Installation**
......@@ -179,7 +170,7 @@ sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production
Sometimes during version upgrades you might end up with some wrong CSS or
missing some icons. In that case, try to precompile the assets again.
This only applies to source installations and does NOT apply to
Note that this only applies to source installations and does NOT apply to
Omnibus packages.
**Source Installation**
......@@ -202,8 +193,6 @@ GitLab provides a Rake task that lets you track deployments in GitLab
Performance Monitoring. This Rake task simply stores the current GitLab version
in the GitLab Performance Monitoring database.
To run `gitlab:track_deployment`:
**Omnibus Installation**
```shell
......
......@@ -3,13 +3,11 @@
> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/3050) in GitLab 8.9.
> - From GitLab 11.3, import/export can use object storage automatically.
GitLab provides Rake tasks relating to project import and export. For more information, see:
See also:
- [Project import/export documentation](../../user/project/settings/import_export.md).
- [Project import/export API](../../api/project_import_export.md).
## Import/export tasks
The GitLab import/export version can be checked by using the following command:
```shell
......@@ -30,6 +28,8 @@ sudo gitlab-rake gitlab:import_export:data
bundle exec rake gitlab:import_export:data RAILS_ENV=production
```
## Important notes
Note the following:
- Importing is only possible if the version of the import and export GitLab instances are
......
# Repository storage Rake tasks
# Repository Storage Rake Tasks
This is a collection of Rake tasks you can use to help you list and migrate
existing projects and attachments associated with it from Legacy storage to
......@@ -6,7 +6,7 @@ the new Hashed storage type.
You can read more about the storage types [here](../repository_storage_types.md).
## Migrate existing projects to hashed storage
## Migrate existing projects to Hashed storage
Before migrating your existing projects, you should
[enable hashed storage](../repository_storage_types.md#how-to-migrate-to-hashed-storage) for the new projects as well.
......@@ -34,9 +34,9 @@ export ID_FROM=20
export ID_TO=50
```
You can monitor the progress in the **{admin}** **Admin Area > Monitoring > Background Jobs** page.
There is a specific queue you can watch to see how long it will take to finish:
`hashed_storage:hashed_storage_project_migrate`.
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
There is a specific Queue you can watch to see how long it will take to finish:
`hashed_storage:hashed_storage_project_migrate`
After it reaches zero, you can confirm every project has been migrated by running the commands bellow.
If you find it necessary, you can run this migration script again to schedule missing projects.
......@@ -44,18 +44,16 @@ If you find it necessary, you can run this migration script again to schedule mi
Any error or warning will be logged in Sidekiq's log file.
NOTE: **Note:**
If [Geo](../geo/replication/index.md) is enabled, each project that is successfully migrated
generates an event to replicate the changes on any **secondary** nodes.
If Geo is enabled, each project that is successfully migrated generates an event to replicate the changes on any **secondary** nodes.
You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but we have additional
commands below that helps you inspect projects and attachments in both legacy and hashed storage.
## Rollback from hashed storage to legacy storage
## Rollback from Hashed storage to Legacy storage
If you need to rollback the storage migration for any reason, you can follow the steps described here.
NOTE: **Note:**
Hashed storage will be required in future version of GitLab.
NOTE: **Note:** Hashed Storage will be required in future version of GitLab.
To prevent new projects from being created in the Hashed storage,
you need to undo the [enable hashed storage](../repository_storage_types.md#how-to-migrate-to-hashed-storage) changes.
......@@ -83,7 +81,7 @@ export ID_FROM=20
export ID_TO=50
```
You can monitor the progress in the **{admin}** **Admin Area > Monitoring > Background Jobs** page.
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish.
After it reaches zero, you can confirm every project has been rolled back by running the commands bellow.
......@@ -91,13 +89,9 @@ If some projects weren't rolled back, you can run this rollback script again to
Any error or warning will be logged in Sidekiq's log file.
## List projects
The following are Rake tasks for listing projects.
### List projects on legacy storage
## List projects on Legacy storage
To have a simple summary of projects using legacy storage:
To have a simple summary of projects using **Legacy** storage:
**Omnibus Installation**
......@@ -111,7 +105,7 @@ sudo gitlab-rake gitlab:storage:legacy_projects
sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production
```
To list projects using legacy storage:
To list projects using **Legacy** storage:
**Omnibus Installation**
......@@ -126,9 +120,9 @@ sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=pr
```
### List projects on hashed storage
## List projects on Hashed storage
To have a simple summary of projects using hashed storage:
To have a simple summary of projects using **Hashed** storage:
**Omnibus Installation**
......@@ -142,7 +136,7 @@ sudo gitlab-rake gitlab:storage:hashed_projects
sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production
```
To list projects using hashed storage:
To list projects using **Hashed** storage:
**Omnibus Installation**
......@@ -156,13 +150,9 @@ sudo gitlab-rake gitlab:storage:list_hashed_projects
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production
```
## List attachments
The following are Rake tasks for listing attachments.
### List attachments on legacy storage
## List attachments on Legacy storage
To have a simple summary of project attachments using legacy storage:
To have a simple summary of project attachments using **Legacy** storage:
**Omnibus Installation**
......@@ -176,7 +166,7 @@ sudo gitlab-rake gitlab:storage:legacy_attachments
sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production
```
To list project attachments using legacy storage:
To list project attachments using **Legacy** storage:
**Omnibus Installation**
......@@ -190,9 +180,9 @@ sudo gitlab-rake gitlab:storage:list_legacy_attachments
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production
```
### List attachments on hashed storage
## List attachments on Hashed storage
To have a simple summary of project attachments using hashed storage:
To have a simple summary of project attachments using **Hashed** storage:
**Omnibus Installation**
......@@ -206,7 +196,7 @@ sudo gitlab-rake gitlab:storage:hashed_attachments
sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production
```
To list project attachments using hashed storage:
To list project attachments using **Hashed** storage:
**Omnibus Installation**
......
# Import bare repositories
# Import bare repositories into your GitLab instance
Rake tasks are available to import bare repositories into a GitLab instance.
## Notes
Note that:
- The owner of the project will be the first admin
- The groups will be created as needed, including subgroups
- The owner of the group will be the first admin
- Existing projects will be skipped
- Projects in hashed storage may be skipped (see [Importing bare repositories from hashed storage](#importing-bare-repositories-from-hashed-storage))
- The existing Git repos will be moved from disk (removed from the original path)
- The owner of the project will be the first administrator.
- The groups will be created as needed, including subgroups.
- The owner of the group will be the first administrator.
- Existing projects will be skipped.
- Projects in hashed storage may be skipped. For more information, see
[Importing bare repositories from hashed storage](#importing-bare-repositories-from-hashed-storage).
- The existing Git repositories will be moved from disk (removed from the original path).
## How to use
To import bare repositories into a GitLab instance:
### Create a new folder to import your Git repositories from
1. Create a new folder to import your Git repositories from. The new folder needs to have Git user
ownership and read/write/execute access for Git user and its group:
The new folder needs to have Git user ownership and read/write/execute access for Git user and its group:
```shell
sudo -u git mkdir -p /var/opt/gitlab/git-data/repository-import-<date>/new_group
```
```shell
sudo -u git mkdir -p /var/opt/gitlab/git-data/repository-import-<date>/new_group
```
### Copy your bare repositories inside this newly created folder
1. Copy your bare repositories inside this newly created folder. Note:
- Any `.git` repositories found on any of the subfolders will be imported as projects
- Groups will be created as needed, these could be nested folders. Example:
- Any `.git` repositories found on any of the subfolders will be imported as projects.
- Groups will be created as needed, these could be nested folders.
If we copy the repos to `/var/opt/gitlab/git-data/repository-import-<date>`, and repo A needs to be under the groups G1 and G2, it will
have to be created under those folders: `/var/opt/gitlab/git-data/repository-import-<date>/G1/G2/A.git`.
For example, if we copy the repositories to `/var/opt/gitlab/git-data/repository-import-<date>`,
and repository `A` needs to be under the groups `G1` and `G2`, it must be created under those folders:
`/var/opt/gitlab/git-data/repository-import-<date>/G1/G2/A.git`.
```shell
sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repository-import-<date>/new_group/
```shell
sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repository-import-<date>/new_group/
# Do this once when you are done copying git repositories
sudo chown -R git:git /var/opt/gitlab/git-data/repository-import-<date>
```
# Do this once when you are done copying git repositories
sudo chown -R git:git /var/opt/gitlab/git-data/repository-import-<date>
```
`foo.git` needs to be owned by the `git` user and `git` users group.
`foo.git` needs to be owned by the `git` user and `git` users group.
If you are using an installation from source, replace `/var/opt/gitlab/` with `/home/git`.
If you are using an installation from source, replace `/var/opt/gitlab/` with `/home/git`.
### Run the command below depending on your type of installation
1. Run the following command depending on your type of installation:
#### Omnibus Installation
- Omnibus Installation
```shell
sudo gitlab-rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>']
```
```shell
sudo gitlab-rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>']
```
#### Installation from source
- Installation from source. Before running this command you need to change to the directory where
your GitLab installation is located:
Before running this command you need to change the directory to where your GitLab installation is located:
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>'] RAILS_ENV=production
```
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-<date>'] RAILS_ENV=production
```
## Example output
#### Example output
```plaintext
Processing /var/opt/gitlab/git-data/repository-import-1/a/b/c/blah.git
......@@ -75,6 +73,8 @@ Processing /var/opt/gitlab/git-data/repository-import-1/group/xyz.git
## Importing bare repositories from hashed storage
### Background
Projects in legacy storage have a directory structure that mirrors their full
project path in GitLab, including their namespace structure. This information is
leveraged by the bare repository importer to import projects into their proper
......@@ -86,17 +86,17 @@ improved performance and data integrity. See
[Repository Storage Types](../administration/repository_storage_types.md) for
more details.
The repositories that are importable depends on the version of GitLab.
### Which repositories are importable?
### GitLab 10.3 or earlier
#### GitLab 10.3 or earlier
Importing bare repositories from hashed storage is unsupported.
### GitLab 10.4 and later
#### GitLab 10.4 and later
To support importing bare repositories from hashed storage, GitLab 10.4 and
later stores the full project path with each repository, in a special section of
the Git repository's configuration file. This section is formatted as follows:
the Git repository's config file. This section is formatted as follows:
```ini
[gitlab]
......
# Listing repository directories
You can print a list of all Git repositories on disk managed by GitLab.
To print a list, run the following command:
You can print a list of all Git repositories on disk managed by
GitLab with the following command:
```shell
# Omnibus
......@@ -13,13 +12,10 @@ cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production
```
NOTE: **Note:**
The results use the default ordering of the GitLab Rails application.
## Limit search results
To list only projects with recent activity, pass a date with the `SINCE` environment variable. The
time you specify is parsed by the Rails [TimeZone#parse function](https://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html#method-i-parse).
If you only want to list projects with recent activity you can pass
a date with the 'SINCE' environment variable. The time you specify
is parsed by the Rails [TimeZone#parse
function](https://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html#method-i-parse).
```shell
# Omnibus
......@@ -29,3 +25,6 @@ sudo gitlab-rake gitlab:list_repos SINCE='Sep 1 2015'
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:list_repos RAILS_ENV=production SINCE='Sep 1 2015'
```
Note that the projects listed are NOT sorted by activity; they use
the default ordering of the GitLab Rails application.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment