Skip to content

G
gitlab-ce
Commit b695254a authored by Marcel Amirault's avatar Marcel Amirault
Browse files
Options

Merge branch 'eread/improve-more-rake-task-topics' into 'master' 

Improve more Rake task topics

See merge request gitlab-org/gitlab!30456
parents 8b5cd5fe 44a2fc43
Hide whitespace changes
Inline Side-by-side
Showing with 192 additions and 145 deletions
doc/administration/raketasks/uploads/migrate.md
View file @ b695254a
# Uploads Migrate Rake Tasks
# Uploads migrate Rake tasks
## Migrate to Object Storage
`gitlab:uploads:migrate` migrates uploads between different storage types.
After [configuring the object storage](../../uploads.md#using-object-storage-core-only) for GitLab's uploads, you may use this task to migrate existing uploads from the local storage to the remote storage.
## Migrate to object storage
>**Note:**
All of the processing will be done in a background worker and requires **no downtime**.
After [configuring the object storage](../../uploads.md#using-object-storage-core-only) for GitLab's
uploads, use this task to migrate existing uploads from the local storage to the remote storage.
Read more about using [object storage with GitLab](../../object_storage.md).
[Read more about using object storage with GitLab](../../object_storage.md).
NOTE: **Note:**
All of the processing will be done in a background worker and requires **no downtime**.
### All-in-one Rake task
GitLab provides a wrapper Rake task that migrates all uploaded files - avatars,
logos, attachments, favicon, etc. - to object storage in one go. Under the hood,
it invokes individual Rake tasks to migrate files falling under each of this
category one by one. The specifications of these individual Rake tasks are
described in the next section.
GitLab provides a wrapper Rake task that migrates all uploaded files (for example avatars, logos,
attachments, and favicon) to object storage in one step. The wrapper task invokes individual Rake
tasks to migrate files falling under each of these categories one by one.
These [individual Rake tasks](#individual-rake-tasks) are described in the next section.
To migrate all uploads from local storage to object storage, run:
**Omnibus Installation**
......@@ -31,26 +36,29 @@ sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate:all
### Individual Rake tasks
>**Note:**
If you already ran the Rake task mentioned above, no need to run these individual Rake tasks as that has been done automatically.
If you already ran the [all-in-one Rake task](#all-in-one-rake-task), there is no need to run these
individual tasks.
The Rake task uses three parameters to find uploads to migrate:
The Rake task uses 3 parameters to find uploads to migrate.
| Parameter | Type | Description |
|:-----------------|:--------------|:-------------------------------------------------------|
| `uploader_class` | string | Type of the uploader to migrate from. |
| `model_class` | string | Type of the model to migrate from. |
| `mount_point` | string/symbol | Name of the model's column the uploader is mounted on. |
Parameter | Type | Description
--------- | ---- | -----------
`uploader_class` | string | Type of the uploader to migrate from
`model_class` | string | Type of the model to migrate from
`mount_point` | string/symbol | Name of the model's column on which the uploader is mounted on.
NOTE: **Note:**
These parameters are mainly internal to GitLab's structure, you may want to refer to the task list
instead below.
>**Note:**
These parameters are mainly internal to GitLab's structure, you may want to refer to the task list instead below.
This task also accepts an environment variable which you can use to override
the default batch size:
This task also accepts some environment variables which you can use to override
certain values:
| Variable | Type | Description |
|:---------|:--------|:--------------------------------------------------|
| `BATCH` | integer | Specifies the size of the batch. Defaults to 200. |
Variable | Type | Description
-------- | ---- | -----------
`BATCH` | integer | Specifies the size of the batch. Defaults to 200.
The following shows how to run `gitlab:uploads:migrate` for individual types of uploads.
**Omnibus Installation**
......@@ -82,7 +90,7 @@ gitlab-rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, De
**Source Installation**
>**Note:**
NOTE: **Note:**
Use `RAILS_ENV=production` for every task.
```shell
......@@ -111,12 +119,10 @@ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeReque
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action]"
```
## Migrate from object storage to local storage
## Migrate to local storage
If you need to disable Object Storage for any reason, first you need to migrate
your data out of Object Storage and back into your local storage.
**Before proceeding, it is important to disable both `direct_upload` and `background_upload` under `uploads` settings in `gitlab.rb`**
If you need to disable [object storage](../../object_storage.md) for any reason, you must first
migrate your data out of object storage and back into your local storage.
CAUTION: **Warning:**
**Extended downtime is required** so no new files are created in object storage during
......@@ -126,23 +132,29 @@ To follow progress, see the [relevant issue](https://gitlab.com/gitlab-org/gitla
### All-in-one Rake task
GitLab provides a wrapper Rake task that migrates all uploaded files - avatars,
logos, attachments, favicon, etc. - to local storage in one go. Under the hood,
it invokes individual Rake tasks to migrate files falling under each of this
category one by one. For details on these Rake tasks please [refer to the section above](#individual-rake-tasks),
GitLab provides a wrapper Rake task that migrates all uploaded files (for example, avatars, logos,
attachments, and favicon) to local storage in one step. The wrapper task invokes individual Rake
tasks to migrate files falling under each of these categories one by one.
For details on these Rake tasks, refer to [Individual Rake tasks](#individual-rake-tasks),
keeping in mind the task name in this case is `gitlab:uploads:migrate_to_local`.
**Omnibus Installation**
To migrate uploads from object storage to local storage:
```shell
gitlab-rake "gitlab:uploads:migrate_to_local:all"
```
1. Disable both `direct_upload` and `background_upload` under `uploads` settings in `gitlab.rb`.
1. Run the Rake task:
**Source Installation**
**Omnibus Installation**
```shell
sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate_to_local:all
```
```shell
gitlab-rake "gitlab:uploads:migrate_to_local:all"
```
**Source Installation**
```shell
sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate_to_local:all
```
After this is done, you may disable Object Storage by undoing the changes described
in the instructions to [configure object storage](../../uploads.md#using-object-storage-core-only)
After running the Rake task, you can disable object storage by undoing the changes described
in the instructions to [configure object storage](../../uploads.md#using-object-storage-core-only).
doc/administration/raketasks/uploads/sanitize.md
View file @ b695254a
# Uploads Sanitize tasks
# Uploads sanitize Rake tasks
Since GitLab 11.9, EXIF data is automatically stripped from JPG or TIFF image uploads.
EXIF data may contain sensitive information (for example, GPS location), so you
can remove EXIF data from existing images that were uploaded to an earlier version of GitLab.
## Requirements
You need `exiftool` installed on your system. If you installed GitLab:
To run this Rake task, you need `exiftool` installed on your system. If you installed GitLab:
- Using the Omnibus package, you're all set.
- From source, make sure `exiftool` is installed:
......@@ -17,48 +22,48 @@ You need `exiftool` installed on your system. If you installed GitLab:
## Remove EXIF data from existing uploads
Since 11.9 EXIF data are automatically stripped from JPG or TIFF image uploads.
Because EXIF data may contain sensitive information (e.g. GPS location), you
can remove EXIF data also from existing images which were uploaded before
with the following command:
To remove EXIF data from existing uploads, run the following command:
```shell
sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif
```
This command by default runs in dry mode and it doesn't remove EXIF data. It can be used for
By default, this command runs in "dry run" mode and doesn't remove EXIF data. It can be used for
checking if (and how many) images should be sanitized.
The Rake task accepts following parameters.
Parameter | Type | Description
--------- | ---- | -----------
`start_id` | integer | Only uploads with equal or greater ID will be processed
`stop_id` | integer | Only uploads with equal or smaller ID will be processed
`dry_run` | boolean | Do not remove EXIF data, only check if EXIF data are present or not, default: true
`sleep_time` | float | Pause for number of seconds after processing each image, default: 0.3 seconds
`uploader` | string | Run sanitization only for uploads of the given uploader (`FileUploader`, `PersonalFileUploader`, `NamespaceFileUploader`)
`since` | date | Run sanitization only for uploads newer than given date (e.g. `2019-05-01`)
| Parameter | Type | Description |
|:-------------|:--------|:----------------------------------------------------------------------------------------------------------------------------|
| `start_id` | integer | Only uploads with equal or greater ID will be processed |
| `stop_id` | integer | Only uploads with equal or smaller ID will be processed |
| `dry_run` | boolean | Do not remove EXIF data, only check if EXIF data are present or not. Defaults to `true` |
| `sleep_time` | float | Pause for number of seconds after processing each image. Defaults to 0.3 seconds |
| `uploader` | string | Run sanitization only for uploads of the given uploader: `FileUploader`, `PersonalFileUploader`, or `NamespaceFileUploader` |
| `since` | date | Run sanitization only for uploads newer than given date. For example, `2019-05-01` |
If you have too many uploads, you can speed up sanitization by setting
`sleep_time` to a lower value or by running multiple Rake tasks in parallel,
each with a separate range of upload IDs (by setting `start_id` and `stop_id`).
If you have too many uploads, you can speed up sanitization by:
To run the command without dry mode and remove EXIF data from all uploads, you can use:
- Setting `sleep_time` to a lower value.
- Running multiple Rake tasks in parallel, each with a separate range of upload IDs (by setting
`start_id` and `stop_id`).
To remove EXIF data from all uploads, use:
```shell
sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif[,,false,] 2>&1 | tee exif.log
```
To run the command without dry mode on uploads with ID between 100 and 5000 and pause for 0.1 second, you can use:
To remove EXIF data on uploads with an ID between 100 and 5000 and pause for 0.1 second after each file, use:
```shell
sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif[100,5000,false,0.1] 2>&1 | tee exif.log
```
Because the output of commands will be probably long, the output is written also into exif.log file.
The output is written into an `exif.log` file because it will probably be long.
If sanitization fails for an upload, an error message should be in the output of the Rake task.
Typical reasons include that the file is missing in the storage or it's not a valid image.
If sanitization fails for an upload, an error message should be in the output of the Rake task (typical reasons may
be that the file is missing in the storage or it's not a valid image). Please
[report](https://gitlab.com/gitlab-org/gitlab-foss/issues/new) any issues at `gitlab.com` and use
prefix 'EXIF' in issue title with the error output and (if possible) the image.
[Report](https://gitlab.com/gitlab-org/gitlab/issues/new) any issues and use the prefix 'EXIF' in
the issue title with the error output and (if possible) the image.
doc/raketasks/user_management.md
View file @ b695254a
# User management
GitLab provides Rake tasks for user management.
## Add user as a developer to all projects
To add a user as a developer to all projects, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:import:user_to_projects[username@domain.tld]
......@@ -12,9 +16,7 @@ bundle exec rake gitlab:import:user_to_projects[username@domain.tld] RAILS_ENV=p
## Add all users to all projects
Notes:
- admin users are added as maintainers
To add all users to all projects, run:
```shell
# omnibus-gitlab
......@@ -24,8 +26,13 @@ sudo gitlab-rake gitlab:import:all_users_to_all_projects
bundle exec rake gitlab:import:all_users_to_all_projects RAILS_ENV=production
```
NOTE: **Note:**
Admin users are added as maintainers.
## Add user as a developer to all groups
To add a user as a developer to all groups, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:import:user_to_groups[username@domain.tld]
......@@ -36,9 +43,7 @@ bundle exec rake gitlab:import:user_to_groups[username@domain.tld] RAILS_ENV=pro
## Add all users to all groups
Notes:
- admin users are added as owners so they can add additional users to the group
To add all users to all groups, run:
```shell
# omnibus-gitlab
......@@ -48,19 +53,25 @@ sudo gitlab-rake gitlab:import:all_users_to_all_groups
bundle exec rake gitlab:import:all_users_to_all_groups RAILS_ENV=production
```
## Maintain tight control over the number of active users on your GitLab installation
NOTE: **Note:**
Admin users are added as owners so they can add additional users to the group.
## Control the number of active users
- Enable this setting to keep new users blocked until they have been cleared by the admin (default: false).
Enable this setting to keep new users blocked until they have been cleared by the administrator.
Defaults to `false`:
```plaintext
block_auto_created_users: false
```
## Disable Two-factor Authentication (2FA) for all users
## Disable two-factor authentication for all users
This task will disable 2FA for all users that have it enabled. This can be
This task disables two-factor authentication (2FA) for all users that have it enabled. This can be
useful if GitLab's `config/secrets.yml` file has been lost and users are unable
to login, for example.
to log in, for example.
To disable two-factor authentication for all users, run:
```shell
# omnibus-gitlab
......@@ -70,65 +81,65 @@ sudo gitlab-rake gitlab:two_factor:disable_for_all_users
bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production
```
## Rotate Two-factor Authentication (2FA) encryption key
## Rotate two-factor authentication encryption key
GitLab stores the secret data enabling 2FA to work in an encrypted database
column. The encryption key for this data is known as `otp_key_base`, and is
GitLab stores the secret data required for two-factor authentication (2FA) in an encrypted
database column. The encryption key for this data is known as `otp_key_base`, and is
stored in `config/secrets.yml`.
If that file is leaked, but the individual 2FA secrets have not, it's possible
to re-encrypt those secrets with a new encryption key. This allows you to change
the leaked key without forcing all users to change their 2FA details.
First, look up the old key. This is in the `config/secrets.yml` file, but
**make sure you're working with the production section**. The line you're
interested in will look like this:
To rotate the two-factor authentication encryption key:
```yaml
production:
otp_key_base: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
```
1. Look up the old key. This is in the `config/secrets.yml` file, but **make sure you're working
with the production section**. The line you're interested in will look like this:
Next, generate a new secret:
```yaml
production:
otp_key_base: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
```
```shell
# omnibus-gitlab
sudo gitlab-rake secret
1. Generate a new secret:
# installation from source
bundle exec rake secret RAILS_ENV=production
```
```shell
# omnibus-gitlab
sudo gitlab-rake secret
Now you need to stop the GitLab server, back up the existing secrets file and
update the database:
# installation from source
bundle exec rake secret RAILS_ENV=production
```
```shell
# omnibus-gitlab
sudo gitlab-ctl stop
sudo cp config/secrets.yml config/secrets.yml.bak
sudo gitlab-rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key>
1. Stop the GitLab server, back up the existing secrets file, and update the database:
# installation from source
sudo /etc/init.d/gitlab stop
cp config/secrets.yml config/secrets.yml.bak
bundle exec rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key> RAILS_ENV=production
```
```shell
# omnibus-gitlab
sudo gitlab-ctl stop
sudo cp config/secrets.yml config/secrets.yml.bak
sudo gitlab-rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key>
The `<old key>` value can be read from `config/secrets.yml`; `<new key>` was
generated earlier. The **encrypted** values for the user 2FA secrets will be
written to the specified `filename` - you can use this to rollback in case of
error.
# installation from source
sudo /etc/init.d/gitlab stop
cp config/secrets.yml config/secrets.yml.bak
bundle exec rake gitlab:two_factor:rotate_key:apply filename=backup.csv old_key=<old key> new_key=<new key> RAILS_ENV=production
```
Finally, change `config/secrets.yml` to set `otp_key_base` to `<new key>` and
restart. Again, make sure you're operating in the **production** section.
The `<old key>` value can be read from `config/secrets.yml` (`<new key>` was
generated earlier). The **encrypted** values for the user 2FA secrets will be
written to the specified `filename`. You can use this to rollback in case of
error.
```shell
# omnibus-gitlab
sudo gitlab-ctl start
1. Change `config/secrets.yml` to set `otp_key_base` to `<new key>` and restart. Again, make sure
you're operating in the **production** section.
# installation from source
sudo /etc/init.d/gitlab start
```
```shell
# omnibus-gitlab
sudo gitlab-ctl start
# installation from source
sudo /etc/init.d/gitlab start
```
If there are any problems (perhaps using the wrong value for `old_key`), you can
restore your backup of `config/secrets.yml` and rollback the changes:
......
doc/raketasks/web_hooks.md
View file @ b695254a
# Webhooks administration **(CORE ONLY)**
## Add a webhook for **ALL** projects
GitLab provides Rake tasks for webhooks management.
Requests to the [local network by webhooks](../security/webhooks.md) can be allowed or blocked by an
administrator.
## Add a webhook to all projects
To add a webhook to all projects, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook"
# source installations
bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production
```
## Add a webhook for projects in a given **NAMESPACE**
## Add a webhook to projects in a namespace
To add a webhook to all projects in a specific namespace, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme
sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=<namespace>
# source installations
bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production
bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=<namespace> RAILS_ENV=production
```
## Remove a webhook from **ALL** projects using
## Remove a webhook from projects
To remove a webhook from all projects, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook"
# source installations
bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production
```
## Remove a webhook from projects in a given **NAMESPACE**
## Remove a webhook from projects in a namespace
To remove a webhook from projects in a specific namespace, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme
sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=<namespace>
# source installations
bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production
bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=<namespace> RAILS_ENV=production
```
## List **ALL** webhooks
## List all webhooks
To list all webhooks, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:web_hook:list
# source installations
bundle exec rake gitlab:web_hook:list RAILS_ENV=production
```
## List the webhooks from projects in a given **NAMESPACE**
## List webhooks for projects in a namespace
To list all webhook for projects in a specified namespace, run:
```shell
# omnibus-gitlab
sudo gitlab-rake gitlab:web_hook:list NAMESPACE=acme
sudo gitlab-rake gitlab:web_hook:list NAMESPACE=<namespace>
# source installations
bundle exec rake gitlab:web_hook:list NAMESPACE=acme RAILS_ENV=production
bundle exec rake gitlab:web_hook:list NAMESPACE=<namespace> RAILS_ENV=production
```
## Local requests in webhooks
[Requests to local network by webhooks](../security/webhooks.md) can be allowed
or blocked by an administrator.
doc/raketasks/x509_signatures.md
View file @ b695254a
# X509 signatures
When [signing commits with x509](../user/project/repository/x509_signed_commits/index.md)
the trust anchor might change and the signatures stored within the database have
to be updated.
When [signing commits with x509](../user/project/repository/x509_signed_commits/index.md),
the trust anchor might change and the signatures stored within the database must be updated.
## Update all x509 signatures
This task loops through all X509 signed commits and updates their verification
based on current certificate store.
This task loops through all X509 signed commits and updates their verification based on current
certificate store.
To update all x509 signatures, run:
**Omnibus Installation**
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7