Commit aa7cddc4 authored by Yorick Peterse's avatar Yorick Peterse

Use more accurate timestamps for InfluxDB.

This changes the timestamp of metrics to be more accurate/unique by
using Time#to_f combined with a small random jitter value. This
combination hopefully reduces the amount of collisions, though there's
no way to fully prevent any from occurring.

Fixes gitlab-com/operations#175
parent 4d04e918
...@@ -2,6 +2,8 @@ module Gitlab ...@@ -2,6 +2,8 @@ module Gitlab
module Metrics module Metrics
# Class for storing details of a single metric (label, value, etc). # Class for storing details of a single metric (label, value, etc).
class Metric class Metric
JITTER_RANGE = 0.000001..0.001
attr_reader :series, :values, :tags, :created_at attr_reader :series, :values, :tags, :created_at
# series - The name of the series (as a String) to store the metric in. # series - The name of the series (as a String) to store the metric in.
...@@ -16,11 +18,29 @@ module Gitlab ...@@ -16,11 +18,29 @@ module Gitlab
# Returns a Hash in a format that can be directly written to InfluxDB. # Returns a Hash in a format that can be directly written to InfluxDB.
def to_hash def to_hash
# InfluxDB overwrites an existing point if a new point has the same
# series, tag set, and timestamp. In a highly concurrent environment
# this means that using the number of seconds since the Unix epoch is
# inevitably going to collide with another timestamp. For example, two
# Rails requests processed by different processes may end up generating
# metrics using the _exact_ same timestamp (in seconds).
#
# Due to the way InfluxDB is set up there's no solution to this problem,
# all we can do is lower the amount of collisions. We do this by using
# Time#to_f which returns the seconds as a Float providing greater
# accuracy. We then add a small random value that is large enough to
# distinguish most timestamps but small enough to not alter the amount
# of seconds.
#
# See https://gitlab.com/gitlab-com/operations/issues/175 for more
# information.
time = @created_at.to_f + rand(JITTER_RANGE)
{ {
series: @series, series: @series,
tags: @tags, tags: @tags,
values: @values, values: @values,
timestamp: @created_at.to_i * 1_000_000_000 timestamp: (time * 1_000_000_000).to_i
} }
end end
end end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment