Commit 30c20263 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis Committed by Robert Speicher

Merge branch 'sidekiq-api-metrics' into 'master'

Added API endpoint for Sidekiq Metrics.

## What does this MR do?

It adds an API endpoint to gather metrics about Sidekiq, it's jobs, queues, and processes.

## Why was this MR needed?

There was no API endpoint for Sidekiq information.

## What are the relevant issue numbers?

Fixes #7171

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] API support added
- [x] Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing

See merge request !4653
parent 9f3ca136
......@@ -43,6 +43,7 @@ v 8.9.0 (unreleased)
- Fixed alignment of download dropdown in merge requests
- Upgrade to jQuery 2
- Adds selected branch name to the dropdown toggle
- Add API endpoint for Sidekiq Metrics !4653
- Use Knapsack to evenly distribute tests across multiple nodes
- Add `sha` parameter to MR merge API, to ensure only reviewed changes are merged
- Don't allow MRs to be merged when commits were added since the last review / page load
......
# Sidekiq Metrics
>**Note:** This endpoint is only available on GitLab 8.9 and above.
This API endpoint allows you to retrieve some information about the current state
of Sidekiq, its jobs, queues, and processes.
## Get the current Queue Metrics
List information about all the registered queues, their backlog and their
latency.
```
GET /sidekiq/queue_metrics
```
```bash
curl -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/sidekiq/queue_metrics
```
Example response:
```json
{
"queues": {
"default": {
"backlog": 0,
"latency": 0
}
}
}
```
## Get the current Process Metrics
List information about all the Sidekiq workers registered to process your queues.
```
GET /sidekiq/process_metrics
```
```bash
curl -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/sidekiq/process_metrics
```
Example response:
```json
{
"processes": [
{
"hostname": "gitlab.example.com",
"pid": 5649,
"tag": "gitlab",
"started_at": "2016-06-14T10:45:07.159-05:00",
"queues": [
"post_receive",
"mailers",
"archive_repo",
"system_hook",
"project_web_hook",
"gitlab_shell",
"incoming_email",
"runner",
"common",
"default"
],
"labels": [],
"concurrency": 25,
"busy": 0
}
]
}
```
## Get the current Job Statistics
List information about the jobs that Sidekiq has performed.
```
GET /sidekiq/job_stats
```
```bash
curl -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/sidekiq/job_stats
```
Example response:
```json
{
"jobs": {
"processed": 2,
"failed": 0,
"enqueued": 0
}
}
```
## Get a compound response of all the previously mentioned metrics
List all the currently available information about Sidekiq.
```
GET /sidekiq/compound_metrics
```
```bash
curl -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/sidekiq/compound_metrics
```
Example response:
```json
{
"queues": {
"default": {
"backlog": 0,
"latency": 0
}
},
"processes": [
{
"hostname": "gitlab.example.com",
"pid": 5649,
"tag": "gitlab",
"started_at": "2016-06-14T10:45:07.159-05:00",
"queues": [
"post_receive",
"mailers",
"archive_repo",
"system_hook",
"project_web_hook",
"gitlab_shell",
"incoming_email",
"runner",
"common",
"default"
],
"labels": [],
"concurrency": 25,
"busy": 0
}
],
"jobs": {
"processed": 2,
"failed": 0,
"enqueued": 0
}
}
```
......@@ -59,5 +59,6 @@ module API
mount ::API::Licenses
mount ::API::Subscriptions
mount ::API::Gitignores
mount ::API::SidekiqMetrics
end
end
require 'sidekiq/api'
module API
class SidekiqMetrics < Grape::API
before { authenticated_as_admin! }
helpers do
def queue_metrics
Sidekiq::Queue.all.each_with_object({}) do |queue, hash|
hash[queue.name] = {
backlog: queue.size,
latency: queue.latency.to_i
}
end
end
def process_metrics
Sidekiq::ProcessSet.new.map do |process|
{
hostname: process['hostname'],
pid: process['pid'],
tag: process['tag'],
started_at: Time.at(process['started_at']),
queues: process['queues'],
labels: process['labels'],
concurrency: process['concurrency'],
busy: process['busy']
}
end
end
def job_stats
stats = Sidekiq::Stats.new
{
processed: stats.processed,
failed: stats.failed,
enqueued: stats.enqueued
}
end
end
# Get Sidekiq Queue metrics
#
# Parameters:
# None
#
# Example:
# GET /sidekiq/queue_metrics
#
get 'sidekiq/queue_metrics' do
{ queues: queue_metrics }
end
# Get Sidekiq Process metrics
#
# Parameters:
# None
#
# Example:
# GET /sidekiq/process_metrics
#
get 'sidekiq/process_metrics' do
{ processes: process_metrics }
end
# Get Sidekiq Job statistics
#
# Parameters:
# None
#
# Example:
# GET /sidekiq/job_stats
#
get 'sidekiq/job_stats' do
{ jobs: job_stats }
end
# Get Sidekiq Compound metrics. Includes all previous metrics
#
# Parameters:
# None
#
# Example:
# GET /sidekiq/compound_metrics
#
get 'sidekiq/compound_metrics' do
{ queues: queue_metrics, processes: process_metrics, jobs: job_stats }
end
end
end
require 'spec_helper'
describe API::SidekiqMetrics, api: true do
include ApiHelpers
let(:admin) { create(:user, :admin) }
describe 'GET sidekiq/*' do
it 'defines the `queue_metrics` endpoint' do
get api('/sidekiq/queue_metrics', admin)
expect(response.status).to eq(200)
expect(json_response).to be_a Hash
end
it 'defines the `process_metrics` endpoint' do
get api('/sidekiq/process_metrics', admin)
expect(response.status).to eq(200)
expect(json_response['processes']).to be_an Array
end
it 'defines the `job_stats` endpoint' do
get api('/sidekiq/job_stats', admin)
expect(response.status).to eq(200)
expect(json_response).to be_a Hash
end
it 'defines the `compound_metrics` endpoint' do
get api('/sidekiq/compound_metrics', admin)
expect(response.status).to eq(200)
expect(json_response).to be_a Hash
expect(json_response['queues']).to be_a Hash
expect(json_response['processes']).to be_an Array
expect(json_response['jobs']).to be_a Hash
end
end
end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment