Merge branch '10187-docs-improve-to-include-using-sidekiq-cluster-via-gitlab-rb' into 'master'

Docs: Improve to include using sidekiq-cluster via gitlab.rb

Closes #10187

See merge request gitlab-org/gitlab-ee!9832

doc/administration/operations/extra_sidekiq_processes.md

-# Extra Sidekiq Processes
+# Extra Sidekiq processes
 
 GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes
 besides the default one. These processes can be used to consume a dedicated set
 of queues. This can be used to ensure certain queues always have dedicated
-workers, no matter the amount of jobs that need to be processed.
+workers, no matter the number of jobs that need to be processed.
 
-## Starting Extra Processes
+## Starting extra processes via Omnibus GitLab
+
+To enable `sidekiq-cluster`, you must apply the `sidekiq_cluster['enable'] = true`
+setting `/etc/gitlab/gitlab.rb`:
+
+```ruby
+sidekiq_cluster['enable'] = true
+```
+
+You will then specify how many additional processes to create via `sidekiq-cluster`
+as well as which queues for them to handle. This is done via the 
+`sidekiq_cluster['queue_groups']` setting. This is an array whose items contain
+which queues to process. Each item in the array will equate to one additional
+sidekiq process.
+
+As an example, to make additional sidekiq processes that process the 
+`elastic_indexer` and `mailers` queues, you would apply the following:
+
+```ruby
+sidekiq_cluster['queue_groups'] = [
+  "elastic_indexer",
+  "mailers"
+]
+```
+
+To have an additional sidekiq process handle multiple queues, you simply put a
+comma after the first queue name and then put the next queue name:
+
+```ruby
+sidekiq_cluster['queue_groups'] = [
+  "elastic_indexer,elastic_commit_indexer",
+  "mailers"
+]
+```
+
+Keep in mind, all changes must be followed by reconfiguring your GitLab
+application via `sudo gitlab-ctl reconfigure`.
+
+#### Monitoring
+
+Once the Sidekiq processes are added, you can visit the "Background Jobs"
+section under the admin area in GitLab (`/admin/background_jobs`).
+
+![Extra sidekiq processes](img/sidekiq-cluster.png)
+
+#### All queues with exceptions
+
+To have the additional sidekiq processes work on every queue EXCEPT the ones
+you list:
+
+1. Edit `/etc/gitlab/gitlab.rb` and add:
+
+   ```ruby
+   sidekiq_cluster['negate'] = true
+   ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
+
+#### Limiting concurrency
+
+1. Edit `/etc/gitlab/gitlab.rb` and add:
+
+   ```ruby
+   sidekiq_cluster['concurrency'] = 25
+   ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
+Keep in mind, this normally would not exceed the number of CPU cores available.
+
+#### Modifying the check interval
+
+To modify the check interval for the additional Sidekiq processes:
+
+1. Edit `/etc/gitlab/gitlab.rb` and add:
+
+   ```ruby
+   sidekiq_cluster['interval'] = 5
+   ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+
+This tells the additional processes how often to check for enqueued jobs.
+
+## Starting extra processes via command line
 
 Starting extra Sidekiq processes can be done using the command
-`bin/sidekiq-cluster`. This command takes arguments using the following syntax:
+`/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster`. This command
+takes arguments using the following syntax:
 
 ```bash
-sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...]
+/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...]
 ```
 
 Each separate argument denotes a group of queues that have to be processed by a
@@ -29,14 +115,14 @@ For example, say you want to start 2 extra processes: one to process the
 done as follows:
 
 ```bash
-sidekiq-cluster process_commit post_receive
+/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit post_receive
 ```
 
 If you instead want to start one process processing both queues you'd use the
 following syntax:
 
 ```bash
-sidekiq-cluster process_commit,post_receive
+/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive
 ```
 
 If you want to have one Sidekiq process process the "process_commit" and
@@ -44,26 +130,20 @@ If you want to have one Sidekiq process process the "process_commit" and
 you'd use the following:
 
 ```bash
-sidekiq-cluster process_commit,post_receive gitlab_shell
+/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive gitlab_shell
 ```
 
-## Concurrency
-
-Each process started using `sidekiq-cluster` starts with a number of threads
-that equals the number of queues, plus one spare thread. For example, a process
-that processes "process_commit" and "post_receive" will use 3 threads in total.
-
-## Monitoring
+#### Monitoring
 
 The `sidekiq-cluster` command will not terminate once it has started the desired
-amount of Sidekiq processes. Instead the process will continue running and
+amount of Sidekiq processes. Instead, the process will continue running and
 forward any signals to the child processes. This makes it easy to stop all
 Sidekiq processes as you simply send a signal to the `sidekiq-cluster` process,
 instead of having to send it to the individual processes.
 
-If the `sidekiq-cluster` process crashes or is SIGKILL'd the child processes
-will terminate themselves after a few seconds. This ensures you don't end up
-with zombie Sidekiq processes.
+If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child
+processes will terminate themselves after a few seconds. This ensures you don't
+end up with zombie Sidekiq processes.
 
 All of this makes monitoring the processes fairly easy. Simply hook up
 `sidekiq-cluster` to your supervisor of choice (e.g. runit) and you're good to
@@ -75,26 +155,26 @@ process to terminate, then terminate itself. This removes the need for
 Instead you should make sure your supervisor restarts the `sidekiq-cluster`
 process whenever necessary.
 
-## PID Files
+#### PID files
 
 The `sidekiq-cluster` command can store its PID in a file. By default no PID
 file is written, but this can be changed by passing the `--pidfile` option to
 `sidekiq-cluster`. For example:
 
 ```bash
-sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit
+/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit
 ```
 
 Keep in mind that the PID file will contain the PID of the `sidekiq-cluster`
-command, and not the PID(s) of the started Sidekiq processes.
+command and not the PID(s) of the started Sidekiq processes.
 
-## Environment
+#### Environment
 
 The Rails environment can be set by passing the `--environment` flag to the
 `sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The
 default value is "development".
 
-## All Queues With Exceptions
+#### All queues with exceptions
 
 You're able to run all queues in `sidekiq_queues.yml` file on a single or
 multiple processes with exceptions using the `--negate` flag.
@@ -112,14 +192,14 @@ For multiple processes of all queues (except "process_commit" and "post_receive"
 sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate
 ```
 
-## Limiting Concurrency
+#### Limiting concurrency
 
 By default, `sidekiq-cluster` will spin up extra Sidekiq processes that use
 one thread per queue up to a maximum of 50. If you wish to change the cap, use
 the `-m N` option. For example, this would cap the maximum number of threads to 1:
 
 ```bash
-sidekiq-cluster process_commit,post_receive -m 1
+/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive -m 1
 ```
 
 For each queue group, the concurrency factor will be set to min(number of
@@ -127,4 +207,12 @@ queues, N). Setting the value to 0 will disable the limit.
 
 Note that each thread requires a Redis connection, so adding threads may
 increase Redis latency and potentially cause client timeouts. See the [Sidekiq
-documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more details.
+documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis)
+for more details.
+
+## Number of threads
+
+Each process started using `sidekiq-cluster` (whether it be via command line or
+via the gitlab.rb file) starts with a number of threads that equals the number
+of queues, plus one spare thread. For example, a process that handles the
+"process_commit" and "post_receive" queues will use 3 threads in total.