Commit 52edbc6e authored by Matthias Kaeppler's avatar Matthias Kaeppler

Fix: Sidekiq workers delete each other's metrics

When we moved the logic that wipes the Prometheus metrics
dir out of the Rackup file and into the initializer, all
Sidekiq workers would call this and potentially enter a
race condition where they deleted each other's database
files.

Changelog: fixed
parent b67a72df
......@@ -9,6 +9,12 @@ def master_process?
end
warmup do |app|
# The following is necessary to ensure stale Prometheus metrics don't accumulate over time.
# It needs to be done as early as here to ensure metrics files aren't deleted.
# After we hit our app in `warmup`, first metrics and corresponding files already being created,
# for example in `lib/gitlab/metrics/requests_rack_middleware.rb`.
Prometheus::CleanupMultiprocDirService.new.execute if master_process?
client = Rack::MockRequest.new(app)
client.get('/')
end
......
......@@ -40,9 +40,6 @@ if !Rails.env.test? && Gitlab::Metrics.prometheus_metrics_enabled?
# When running Puma in a Single mode, `on_master_start` and `on_worker_start` are the same.
# Thus, we order these events to run `reinitialize_on_pid_change` with `force: true` first.
Gitlab::Cluster::LifecycleEvents.on_master_start do
# Ensure that stale Prometheus metrics don't accumulate over time
::Prometheus::CleanupMultiprocDirService.new.execute
::Prometheus::Client.reinitialize_on_pid_change(force: true)
if Gitlab::Runtime.puma?
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment