Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
22b63dc6
Commit
22b63dc6
authored
Mar 07, 2022
by
Matthias Kaeppler
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Drop prometheus_initializer_refactor FF
This was enabled by default.
parent
9a3476f2
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
39 additions
and
171 deletions
+39
-171
config/feature_flags/development/prometheus_initializer_refactor.yml
...ure_flags/development/prometheus_initializer_refactor.yml
+0
-8
config/initializers/7_prometheus_metrics.rb
config/initializers/7_prometheus_metrics.rb
+39
-46
config/initializers/7_prometheus_metrics_new.rb
config/initializers/7_prometheus_metrics_new.rb
+0
-117
No files found.
config/feature_flags/development/prometheus_initializer_refactor.yml
deleted
100644 → 0
View file @
9a3476f2
---
name
:
prometheus_initializer_refactor
introduced_by_url
:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81133
rollout_issue_url
:
https://gitlab.com/gitlab-org/gitlab/-/issues/353446
milestone
:
'
14.9'
type
:
development
group
:
group::memory
default_enabled
:
true
config/initializers/7_prometheus_metrics.rb
View file @
22b63dc6
# frozen_string_literal: true
# frozen_string_literal: true
return
if
Feature
.
feature_flags_available?
&&
Feature
.
enabled?
(
:prometheus_initializer_refactor
,
default_enabled: :yaml
)
# Keep separate directories for separate processes
# Keep separate directories for separate processes
def
prometheus_default_multiproc_dir
def
prometheus_default_multiproc_dir
return
unless
Rails
.
env
.
development?
||
Rails
.
env
.
test?
return
unless
Rails
.
env
.
development?
||
Rails
.
env
.
test?
...
@@ -18,8 +16,6 @@ end
...
@@ -18,8 +16,6 @@ end
::
Prometheus
::
Client
.
configure
do
|
config
|
::
Prometheus
::
Client
.
configure
do
|
config
|
config
.
logger
=
Gitlab
::
AppLogger
config
.
logger
=
Gitlab
::
AppLogger
config
.
initial_mmap_file_size
=
4
*
1024
config
.
multiprocess_files_dir
=
ENV
[
'prometheus_multiproc_dir'
]
||
prometheus_default_multiproc_dir
config
.
multiprocess_files_dir
=
ENV
[
'prometheus_multiproc_dir'
]
||
prometheus_default_multiproc_dir
config
.
pid_provider
=
::
Prometheus
::
PidProvider
.
method
(
:worker_id
)
config
.
pid_provider
=
::
Prometheus
::
PidProvider
.
method
(
:worker_id
)
...
@@ -30,6 +26,10 @@ Gitlab::Application.configure do |config|
...
@@ -30,6 +26,10 @@ Gitlab::Application.configure do |config|
config
.
middleware
.
insert_after
(
Labkit
::
Middleware
::
Rack
,
Gitlab
::
Metrics
::
RequestsRackMiddleware
)
config
.
middleware
.
insert_after
(
Labkit
::
Middleware
::
Rack
,
Gitlab
::
Metrics
::
RequestsRackMiddleware
)
end
end
# Any actions beyond this check should only execute outside of tests, when running in an application
# context (i.e. not in the Rails console or rspec) and when users have enabled metrics.
return
if
Rails
.
env
.
test?
||
!
Gitlab
::
Runtime
.
application?
||
!
Gitlab
::
Metrics
.
prometheus_metrics_enabled?
if
Gitlab
::
Runtime
.
sidekiq?
&&
(
!
ENV
[
'SIDEKIQ_WORKER_ID'
]
||
ENV
[
'SIDEKIQ_WORKER_ID'
]
==
'0'
)
if
Gitlab
::
Runtime
.
sidekiq?
&&
(
!
ENV
[
'SIDEKIQ_WORKER_ID'
]
||
ENV
[
'SIDEKIQ_WORKER_ID'
]
==
'0'
)
# The single worker outside of a sidekiq-cluster, or the first worker (sidekiq_0)
# The single worker outside of a sidekiq-cluster, or the first worker (sidekiq_0)
# in a cluster of processes, is responsible for serving health checks.
# in a cluster of processes, is responsible for serving health checks.
...
@@ -48,55 +48,56 @@ if Gitlab::Runtime.sidekiq? && (!ENV['SIDEKIQ_WORKER_ID'] || ENV['SIDEKIQ_WORKER
...
@@ -48,55 +48,56 @@ if Gitlab::Runtime.sidekiq? && (!ENV['SIDEKIQ_WORKER_ID'] || ENV['SIDEKIQ_WORKER
end
end
end
end
if
!
Rails
.
env
.
test?
&&
Gitlab
::
Metrics
.
prometheus_metrics_enabled?
Gitlab
::
Cluster
::
LifecycleEvents
.
on_master_start
do
# When running Puma in a Single mode, `on_master_start` and `on_worker_start` are the same.
# When running Puma in a Single mode, `on_master_start` and `on_worker_start` are the same.
# Thus, we order these events to run `reinitialize_on_pid_change` with `force: true` first.
# Thus, we order these events to run `reinitialize_on_pid_change` with `force: true` first.
Gitlab
::
Cluster
::
LifecycleEvents
.
on_master_start
do
::
Prometheus
::
Client
.
reinitialize_on_pid_change
(
force:
true
)
::
Prometheus
::
Client
.
reinitialize_on_pid_change
(
force:
true
)
if
Gitlab
::
Runtime
.
puma?
Gitlab
::
Metrics
.
gauge
(
:deployments
,
'GitLab Version'
,
{},
:max
).
set
({
version:
Gitlab
::
VERSION
,
revision:
Gitlab
.
revision
},
1
)
Gitlab
::
Metrics
::
Samplers
::
PumaSampler
.
instance
.
start
end
Gitlab
::
Metrics
.
gauge
(
:deployments
,
'GitLab Version'
,
{},
:max
).
set
({
version:
Gitlab
::
VERSION
,
revision:
Gitlab
.
revision
},
1
)
if
Gitlab
::
Runtime
.
puma?
Gitlab
::
Metrics
::
RequestsRackMiddleware
.
initialize_metrics
if
Gitlab
::
Runtime
.
puma?
Gitlab
::
Metrics
::
Samplers
::
PumaSampler
.
instance
.
start
Gitlab
::
Metrics
::
RequestsRackMiddleware
.
initialize_metrics
end
Gitlab
::
Ci
::
Parsers
.
instrument!
# Starts a metrics server to export metrics from the Puma primary.
rescue
IOError
=>
e
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
start
Gitlab
::
ErrorTracking
.
track_exception
(
e
)
Gitlab
::
Metrics
.
error_detected!
end
end
Gitlab
::
Cluster
::
LifecycleEvents
.
on_worker_start
do
Gitlab
::
Ci
::
Parsers
.
instrument!
defined?
(
::
Prometheus
::
Client
.
reinitialize_on_pid_change
)
&&
::
Prometheus
::
Client
.
reinitialize_on_pid_change
rescue
IOError
=>
e
logger
=
Gitlab
::
AppLogger
Gitlab
::
ErrorTracking
.
track_exception
(
e
)
Gitlab
::
Metrics
::
Samplers
::
RubySampler
.
initialize_instance
(
logger:
logger
).
start
Gitlab
::
Metrics
.
error_detected!
Gitlab
::
Metrics
::
Samplers
::
DatabaseSampler
.
initialize_instance
(
logger:
logger
).
start
end
Gitlab
::
Metrics
::
Samplers
::
ThreadsSampler
.
initialize_instance
(
logger:
logger
).
start
if
Gitlab
::
Runtime
.
puma?
Gitlab
::
Cluster
::
LifecycleEvents
.
on_worker_start
do
Gitlab
::
Metrics
::
Samplers
::
ActionCableSampler
.
instance
(
logger:
logger
).
start
defined?
(
::
Prometheus
::
Client
.
reinitialize_on_pid_change
)
&&
::
Prometheus
::
Client
.
reinitialize_on_pid_change
end
logger
=
Gitlab
::
AppLogger
Gitlab
::
Metrics
::
Samplers
::
RubySampler
.
initialize_instance
(
logger:
logger
).
start
Gitlab
::
Metrics
::
Samplers
::
DatabaseSampler
.
initialize_instance
(
logger:
logger
).
start
Gitlab
::
Metrics
::
Samplers
::
ThreadsSampler
.
initialize_instance
(
logger:
logger
).
start
if
Gitlab
::
Runtime
.
puma?
# Since we are running a metrics server on the Puma primary, we would inherit
# this thread after forking into workers, so we need to explicitly stop it here.
# NOTE: This will not be necessary anymore after moving to an external server
# process via https://gitlab.com/gitlab-org/gitlab/-/issues/350548
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
stop
if
Gitlab
.
ee?
&&
Gitlab
::
Runtime
.
sidekiq?
Gitlab
::
Metrics
::
Samplers
::
ActionCableSampler
.
instance
(
logger:
logger
).
start
Gitlab
::
Metrics
::
Samplers
::
GlobalSearchSampler
.
instance
(
logger:
logger
).
start
end
end
Gitlab
::
Ci
::
Parsers
.
instrument!
if
Gitlab
.
ee?
&&
Gitlab
::
Runtime
.
sidekiq?
rescue
IOError
=>
e
Gitlab
::
Metrics
::
Samplers
::
GlobalSearchSampler
.
instance
(
logger:
logger
).
start
Gitlab
::
ErrorTracking
.
track_exception
(
e
)
Gitlab
::
Metrics
.
error_detected!
end
end
Gitlab
::
Ci
::
Parsers
.
instrument!
rescue
IOError
=>
e
Gitlab
::
ErrorTracking
.
track_exception
(
e
)
Gitlab
::
Metrics
.
error_detected!
end
end
if
Gitlab
::
Runtime
.
puma?
if
Gitlab
::
Runtime
.
puma?
Gitlab
::
Cluster
::
LifecycleEvents
.
on_master_start
do
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
start
end
Gitlab
::
Cluster
::
LifecycleEvents
.
on_before_graceful_shutdown
do
Gitlab
::
Cluster
::
LifecycleEvents
.
on_before_graceful_shutdown
do
# We need to ensure that before we re-exec or shutdown server
# We need to ensure that before we re-exec or shutdown server
# we do stop the exporter
# we do stop the exporter
...
@@ -111,12 +112,4 @@ if Gitlab::Runtime.puma?
...
@@ -111,12 +112,4 @@ if Gitlab::Runtime.puma?
# but it should not be needed
# but it should not be needed
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
stop
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
stop
end
end
Gitlab
::
Cluster
::
LifecycleEvents
.
on_worker_start
do
# The `#close_on_exec=` takes effect only on `execve`
# but this does not happen for Ruby fork
#
# This does stop server, as it is running on master.
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
stop
end
end
end
config/initializers/7_prometheus_metrics_new.rb
deleted
100644 → 0
View file @
9a3476f2
# frozen_string_literal: true
return
unless
Feature
.
feature_flags_available?
&&
Feature
.
enabled?
(
:prometheus_initializer_refactor
,
default_enabled: :yaml
)
# Keep separate directories for separate processes
def
prometheus_default_multiproc_dir
return
unless
Rails
.
env
.
development?
||
Rails
.
env
.
test?
if
Gitlab
::
Runtime
.
sidekiq?
Rails
.
root
.
join
(
'tmp/prometheus_multiproc_dir/sidekiq'
)
elsif
Gitlab
::
Runtime
.
puma?
Rails
.
root
.
join
(
'tmp/prometheus_multiproc_dir/puma'
)
else
Rails
.
root
.
join
(
'tmp/prometheus_multiproc_dir'
)
end
end
::
Prometheus
::
Client
.
configure
do
|
config
|
config
.
logger
=
Gitlab
::
AppLogger
config
.
multiprocess_files_dir
=
ENV
[
'prometheus_multiproc_dir'
]
||
prometheus_default_multiproc_dir
config
.
pid_provider
=
::
Prometheus
::
PidProvider
.
method
(
:worker_id
)
end
Gitlab
::
Application
.
configure
do
|
config
|
# 0 should be Sentry to catch errors in this middleware
config
.
middleware
.
insert_after
(
Labkit
::
Middleware
::
Rack
,
Gitlab
::
Metrics
::
RequestsRackMiddleware
)
end
# Any actions beyond this check should only execute outside of tests, when running in an application
# context (i.e. not in the Rails console or rspec) and when users have enabled metrics.
return
if
Rails
.
env
.
test?
||
!
Gitlab
::
Runtime
.
application?
||
!
Gitlab
::
Metrics
.
prometheus_metrics_enabled?
if
Gitlab
::
Runtime
.
sidekiq?
&&
(
!
ENV
[
'SIDEKIQ_WORKER_ID'
]
||
ENV
[
'SIDEKIQ_WORKER_ID'
]
==
'0'
)
# The single worker outside of a sidekiq-cluster, or the first worker (sidekiq_0)
# in a cluster of processes, is responsible for serving health checks.
#
# Do not clean the metrics directory here - the supervisor script should
# have already taken care of that.
Sidekiq
.
configure_server
do
|
config
|
config
.
on
(
:startup
)
do
# In https://gitlab.com/gitlab-org/gitlab/-/issues/345804 we are looking to
# only serve health-checks from a worker process; for backwards compatibility
# we still go through the metrics exporter server, but start to configure it
# with the new settings keys.
exporter_settings
=
Settings
.
monitoring
.
sidekiq_health_checks
Gitlab
::
Metrics
::
Exporter
::
SidekiqExporter
.
instance
(
exporter_settings
).
start
end
end
end
Gitlab
::
Cluster
::
LifecycleEvents
.
on_master_start
do
# When running Puma in a Single mode, `on_master_start` and `on_worker_start` are the same.
# Thus, we order these events to run `reinitialize_on_pid_change` with `force: true` first.
::
Prometheus
::
Client
.
reinitialize_on_pid_change
(
force:
true
)
Gitlab
::
Metrics
.
gauge
(
:deployments
,
'GitLab Version'
,
{},
:max
).
set
({
version:
Gitlab
::
VERSION
,
revision:
Gitlab
.
revision
},
1
)
if
Gitlab
::
Runtime
.
puma?
Gitlab
::
Metrics
::
RequestsRackMiddleware
.
initialize_metrics
Gitlab
::
Metrics
::
Samplers
::
PumaSampler
.
instance
.
start
# Starts a metrics server to export metrics from the Puma primary.
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
start
end
Gitlab
::
Ci
::
Parsers
.
instrument!
rescue
IOError
=>
e
Gitlab
::
ErrorTracking
.
track_exception
(
e
)
Gitlab
::
Metrics
.
error_detected!
end
Gitlab
::
Cluster
::
LifecycleEvents
.
on_worker_start
do
defined?
(
::
Prometheus
::
Client
.
reinitialize_on_pid_change
)
&&
::
Prometheus
::
Client
.
reinitialize_on_pid_change
logger
=
Gitlab
::
AppLogger
Gitlab
::
Metrics
::
Samplers
::
RubySampler
.
initialize_instance
(
logger:
logger
).
start
Gitlab
::
Metrics
::
Samplers
::
DatabaseSampler
.
initialize_instance
(
logger:
logger
).
start
Gitlab
::
Metrics
::
Samplers
::
ThreadsSampler
.
initialize_instance
(
logger:
logger
).
start
if
Gitlab
::
Runtime
.
puma?
# Since we are running a metrics server on the Puma primary, we would inherit
# this thread after forking into workers, so we need to explicitly stop it here.
# NOTE: This will not be necessary anymore after moving to an external server
# process via https://gitlab.com/gitlab-org/gitlab/-/issues/350548
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
stop
Gitlab
::
Metrics
::
Samplers
::
ActionCableSampler
.
instance
(
logger:
logger
).
start
end
if
Gitlab
.
ee?
&&
Gitlab
::
Runtime
.
sidekiq?
Gitlab
::
Metrics
::
Samplers
::
GlobalSearchSampler
.
instance
(
logger:
logger
).
start
end
Gitlab
::
Ci
::
Parsers
.
instrument!
rescue
IOError
=>
e
Gitlab
::
ErrorTracking
.
track_exception
(
e
)
Gitlab
::
Metrics
.
error_detected!
end
if
Gitlab
::
Runtime
.
puma?
Gitlab
::
Cluster
::
LifecycleEvents
.
on_before_graceful_shutdown
do
# We need to ensure that before we re-exec or shutdown server
# we do stop the exporter
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
stop
end
Gitlab
::
Cluster
::
LifecycleEvents
.
on_before_master_restart
do
# We need to ensure that before we re-exec server
# we do stop the exporter
#
# We do it again, for being extra safe,
# but it should not be needed
Gitlab
::
Metrics
::
Exporter
::
WebExporter
.
instance
.
stop
end
end
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment