Commit d345591f authored by Yorick Peterse's avatar Yorick Peterse

Tracking of custom events

GitLab Performance Monitoring is now able to track custom events not
directly related to application performance. These events include the
number of tags pushed, repositories created, builds registered, etc.

The use of these events is to get a better overview of how a GitLab
instance is used and how that may affect performance. For example, a
large number of Git pushes may have a negative impact on the underlying
storage engine.

Events are stored in the "events" measurement and are not prefixed with
"rails_" or "sidekiq_", this makes it easier to query events with the
same name triggered from different parts of the application. All events
being stored in the same measurement also makes it easier to downsample
data.

Currently the following events are tracked:

* Creating repositories
* Removing repositories
* Changing the default branch of a repository
* Pushing a new tag
* Removing an existing tag
* Pushing a commit (along with the branch being pushed to)
* Pushing a new branch
* Removing an existing branch
* Importing a repository (along with the URL we're importing)
* Forking a repository (along with the source/target path)
* CI builds registered (and when no build could be found)
* CI builds being updated
* Rails and Sidekiq exceptions

Fixes gitlab-org/gitlab-ce#13720
parent d1da2e81
......@@ -16,6 +16,7 @@ v 8.11.0 (unreleased)
- API: List access requests, request access, approve, and deny access requests to a project or a group. !4833
- Use long options for curl examples in documentation !5703 (winniehell)
- Remove magic comments (`# encoding: UTF-8`) from Ruby files. !5456 (winniehell)
- GitLab Performance Monitoring can now track custom events such as the number of tags pushed to a repository
- Add support for relative links starting with ./ or / to RelativeLinkFilter (winniehell)
- Ignore URLs starting with // in Markdown links !5677 (winniehell)
- Fix CI status icon link underline (ClemMakesApps)
......
......@@ -391,6 +391,8 @@ class Repository
expire_exists_cache
expire_root_ref_cache
expire_emptiness_caches
repository_event(:create_repository)
end
# Runs code just before a repository is deleted.
......@@ -407,6 +409,8 @@ class Repository
expire_root_ref_cache
expire_emptiness_caches
expire_exists_cache
repository_event(:remove_repository)
end
# Runs code just before the HEAD of a repository is changed.
......@@ -414,6 +418,8 @@ class Repository
# Cached divergent commit counts are based on repository head
expire_branch_cache
expire_root_ref_cache
repository_event(:change_default_branch)
end
# Runs code before pushing (= creating or removing) a tag.
......@@ -421,12 +427,16 @@ class Repository
expire_cache
expire_tags_cache
expire_tag_count_cache
repository_event(:push_tag)
end
# Runs code before removing a tag.
def before_remove_tag
expire_tags_cache
expire_tag_count_cache
repository_event(:remove_tag)
end
def before_import
......@@ -443,6 +453,8 @@ class Repository
# Runs code after a new commit has been pushed.
def after_push_commit(branch_name, revision)
expire_cache(branch_name, revision)
repository_event(:push_commit, branch: branch_name)
end
# Runs code after a new branch has been created.
......@@ -450,11 +462,15 @@ class Repository
expire_branches_cache
expire_has_visible_content_cache
expire_branch_count_cache
repository_event(:push_branch)
end
# Runs code before removing an existing branch.
def before_remove_branch
expire_branches_cache
repository_event(:remove_branch)
end
# Runs code after an existing branch has been removed.
......@@ -1059,4 +1075,8 @@ class Repository
def keep_around_ref_name(sha)
"refs/keep-around/#{sha}"
end
def repository_event(event, tags = {})
Gitlab::Metrics.add_event(event, { path: path_with_namespace }.merge(tags))
end
end
......@@ -5,6 +5,10 @@ class RepositoryForkWorker
sidekiq_options queue: :gitlab_shell
def perform(project_id, forked_from_repository_storage_path, source_path, target_path)
Gitlab::Metrics.add_event(:fork_repository,
source_path: source_path,
target_path: target_path)
project = Project.find_by_id(project_id)
unless project.present?
......
......@@ -10,6 +10,10 @@ class RepositoryImportWorker
@project = Project.find(project_id)
@current_user = @project.creator
Gitlab::Metrics.add_event(:import_repository,
import_url: @project.import_url,
path: @project.path_with_namespace)
result = Projects::ImportService.new(project, current_user).execute
if result[:status] == :error
......
......@@ -9,6 +9,7 @@ The following measurements are currently stored in InfluxDB:
- `PROCESS_object_counts`
- `PROCESS_transactions`
- `PROCESS_views`
- `events`
Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the
process type. In all series, any form of duration is stored in milliseconds.
......@@ -78,6 +79,14 @@ following value fields are available:
The `action` tag contains the action name of the transaction that rendered the
view.
## events
This measurement is used to store generic events such as the number of Git
pushes, Emails sent, etc. Each point in this measurement has a single value
field called `count`. The value of this field is simply set to `1`. Each point
also has at least one tag: `event`. This tag's value is set to the event name.
Depending on the event type additional tags may be available as well.
---
Read more on:
......
......@@ -20,8 +20,13 @@ module Ci
build = Ci::RegisterBuildService.new.execute(current_runner)
if build
Gitlab::Metrics.add_event(:build_found,
project: build.project.path_with_namespace)
present build, with: Entities::BuildDetails
else
Gitlab::Metrics.add_event(:build_not_found)
not_found!
end
end
......@@ -42,6 +47,9 @@ module Ci
build.update_attributes(trace: params[:trace]) if params[:trace]
Gitlab::Metrics.add_event(:update_build,
project: build.project.path_with_namespace)
case params[:state].to_s
when 'success'
build.success
......
......@@ -124,6 +124,15 @@ module Gitlab
trans.action = action if trans
end
# Tracks an event.
#
# See `Gitlab::Metrics::Transaction#add_event` for more details.
def self.add_event(*args)
trans = current_transaction
trans.add_event(*args) if trans
end
# Returns the prefix to use for the name of a series.
def self.series_prefix
@series_prefix ||= Sidekiq.server? ? 'sidekiq_' : 'rails_'
......
......@@ -4,15 +4,20 @@ module Gitlab
class Metric
JITTER_RANGE = 0.000001..0.001
attr_reader :series, :values, :tags
attr_reader :series, :values, :tags, :type
# series - The name of the series (as a String) to store the metric in.
# values - A Hash containing the values to store.
# tags - A Hash containing extra tags to add to the metrics.
def initialize(series, values, tags = {})
def initialize(series, values, tags = {}, type = :metric)
@values = values
@series = series
@tags = tags
@type = type
end
def event?
type == :event
end
# Returns a Hash in a format that can be directly written to InfluxDB.
......
......@@ -17,6 +17,10 @@ module Gitlab
begin
retval = trans.run { @app.call(env) }
rescue Exception => error # rubocop: disable Lint/RescueException
trans.add_event(:rails_exception)
raise error
# Even in the event of an error we want to submit any metrics we
# might've gathered up to this point.
ensure
......
......@@ -11,6 +11,10 @@ module Gitlab
# Old gitlad-shell messages don't provide enqueued_at/created_at attributes
trans.set(:sidekiq_queue_duration, Time.now.to_f - (message['enqueued_at'] || message['created_at'] || 0))
trans.run { yield }
rescue Exception => error # rubocop: disable Lint/RescueException
trans.add_event(:sidekiq_exception)
raise error
ensure
trans.finish
end
......
......@@ -4,7 +4,10 @@ module Gitlab
class Transaction
THREAD_KEY = :_gitlab_metrics_transaction
attr_reader :tags, :values, :methods
# The series to store events (e.g. Git pushes) in.
EVENT_SERIES = 'events'
attr_reader :tags, :values, :method, :metrics
attr_accessor :action
......@@ -55,6 +58,20 @@ module Gitlab
@metrics << Metric.new("#{Metrics.series_prefix}#{series}", values, tags)
end
# Tracks a business level event
#
# Business level events including events such as Git pushes, Emails being
# sent, etc.
#
# event_name - The name of the event (e.g. "git_push").
# tags - A set of tags to attach to the event.
def add_event(event_name, tags = {})
@metrics << Metric.new(EVENT_SERIES,
{ count: 1 },
{ event: event_name }.merge(tags),
:event)
end
# Returns a MethodCall object for the given name.
def method_call_for(name)
unless method = @methods[name]
......@@ -101,7 +118,7 @@ module Gitlab
submit_hashes = submit.map do |metric|
hash = metric.to_hash
hash[:tags][:action] ||= @action if @action
hash[:tags][:action] ||= @action if @action && !metric.event?
hash
end
......
......@@ -23,6 +23,24 @@ describe Gitlab::Metrics::Metric do
it { is_expected.to eq({ host: 'localtoast' }) }
end
describe '#type' do
subject { metric.type }
it { is_expected.to eq(:metric) }
end
describe '#event?' do
it 'returns false for a regular metric' do
expect(metric.event?).to eq(false)
end
it 'returns true for an event metric' do
expect(metric).to receive(:type).and_return(:event)
expect(metric.event?).to eq(true)
end
end
describe '#to_hash' do
it 'returns a Hash' do
expect(metric.to_hash).to be_an_instance_of(Hash)
......
......@@ -45,6 +45,15 @@ describe Gitlab::Metrics::RackMiddleware do
middleware.call(env)
end
it 'tracks any raised exceptions' do
expect(app).to receive(:call).with(env).and_raise(RuntimeError)
expect_any_instance_of(Gitlab::Metrics::Transaction).
to receive(:add_event).with(:rails_exception)
expect { middleware.call(env) }.to raise_error(RuntimeError)
end
end
describe '#transaction_from_env' do
......
......@@ -12,7 +12,9 @@ describe Gitlab::Metrics::SidekiqMiddleware do
with('TestWorker#perform').
and_call_original
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:set).with(:sidekiq_queue_duration, instance_of(Float))
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:set).
with(:sidekiq_queue_duration, instance_of(Float))
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:finish)
middleware.call(worker, message, :test) { nil }
......@@ -25,10 +27,28 @@ describe Gitlab::Metrics::SidekiqMiddleware do
with('TestWorker#perform').
and_call_original
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:set).with(:sidekiq_queue_duration, instance_of(Float))
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:set).
with(:sidekiq_queue_duration, instance_of(Float))
expect_any_instance_of(Gitlab::Metrics::Transaction).to receive(:finish)
middleware.call(worker, {}, :test) { nil }
end
it 'tracks any raised exceptions' do
worker = double(:worker, class: double(:class, name: 'TestWorker'))
expect_any_instance_of(Gitlab::Metrics::Transaction).
to receive(:run).and_raise(RuntimeError)
expect_any_instance_of(Gitlab::Metrics::Transaction).
to receive(:add_event).with(:sidekiq_exception)
expect_any_instance_of(Gitlab::Metrics::Transaction).
to receive(:finish)
expect { middleware.call(worker, message, :test) }.
to raise_error(RuntimeError)
end
end
end
......@@ -142,5 +142,62 @@ describe Gitlab::Metrics::Transaction do
transaction.submit
end
it 'does not add an action tag for events' do
transaction.action = 'Foo#bar'
transaction.add_event(:meow)
hash = {
series: 'events',
tags: { event: :meow },
values: { count: 1 },
timestamp: an_instance_of(Fixnum)
}
expect(Gitlab::Metrics).to receive(:submit_metrics).
with([hash])
transaction.submit
end
end
describe '#add_event' do
it 'adds a metric' do
transaction.add_event(:meow)
expect(transaction.metrics[0]).to be_an_instance_of(Gitlab::Metrics::Metric)
end
it "does not prefix the metric's series name" do
transaction.add_event(:meow)
metric = transaction.metrics[0]
expect(metric.series).to eq(described_class::EVENT_SERIES)
end
it 'tracks a counter for every event' do
transaction.add_event(:meow)
metric = transaction.metrics[0]
expect(metric.values).to eq(count: 1)
end
it 'tracks the event name' do
transaction.add_event(:meow)
metric = transaction.metrics[0]
expect(metric.tags).to eq(event: :meow)
end
it 'allows tracking of custom tags' do
transaction.add_event(:meow, animal: 'cat')
metric = transaction.metrics[0]
expect(metric.tags).to eq(event: :meow, animal: 'cat')
end
end
end
......@@ -153,4 +153,28 @@ describe Gitlab::Metrics do
expect(described_class.series_prefix).to be_an_instance_of(String)
end
end
describe '.add_event' do
context 'without a transaction' do
it 'does nothing' do
expect_any_instance_of(Gitlab::Metrics::Transaction).
not_to receive(:add_event)
Gitlab::Metrics.add_event(:meow)
end
end
context 'with a transaction' do
it 'adds an event' do
transaction = Gitlab::Metrics::Transaction.new
expect(transaction).to receive(:add_event).with(:meow)
expect(Gitlab::Metrics).to receive(:current_transaction).
and_return(transaction)
Gitlab::Metrics.add_event(:meow)
end
end
end
end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment