Commit 0384d292 authored by Job van der Voort's avatar Job van der Voort

Merge branch 'sidekiq-memory-killer-shutdown-signal' into 'master'

Add SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL env var

It looks like SIGTERM may not be enough to shut down a Sidekiq process
when its RSS has gotten too big. This change will allow us to experiment
with sending SIGKILL instead of SIGTERM to Sidekiq processes on
gitlab.com.

See merge request !1812
parents 0fc6f0b5 1c1f18b4
...@@ -47,6 +47,7 @@ v 7.11.0 (unreleased) ...@@ -47,6 +47,7 @@ v 7.11.0 (unreleased)
- Allow to use non-ASCII letters and dashes in project and namespace name. (Jakub Jirutka) - Allow to use non-ASCII letters and dashes in project and namespace name. (Jakub Jirutka)
- Add footnotes support to Markdown (Guillaume Delbergue) - Add footnotes support to Markdown (Guillaume Delbergue)
- Add current_sign_in_at to UserFull REST api. - Add current_sign_in_at to UserFull REST api.
- Make Sidekiq MemoryKiller shutdown signal configurable
v 7.10.2 v 7.10.2
- Fix CI links on MR page - Fix CI links on MR page
......
...@@ -36,3 +36,5 @@ The MemoryKiller is controlled using environment variables. ...@@ -36,3 +36,5 @@ The MemoryKiller is controlled using environment variables.
Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells
Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must
restart Sidekiq. restart Sidekiq.
- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL`: defaults to 'SIGTERM'. The name of
the final signal sent to the Sidekiq process when we want it to shut down.
...@@ -7,6 +7,7 @@ module Gitlab ...@@ -7,6 +7,7 @@ module Gitlab
GRACE_TIME = (ENV['SIDEKIQ_MEMORY_KILLER_GRACE_TIME'] || 15 * 60).to_s.to_i GRACE_TIME = (ENV['SIDEKIQ_MEMORY_KILLER_GRACE_TIME'] || 15 * 60).to_s.to_i
# Wait 30 seconds for running jobs to finish during graceful shutdown # Wait 30 seconds for running jobs to finish during graceful shutdown
SHUTDOWN_WAIT = (ENV['SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT'] || 30).to_s.to_i SHUTDOWN_WAIT = (ENV['SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT'] || 30).to_s.to_i
SHUTDOWN_SIGNAL = (ENV['SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL'] || 'SIGTERM').to_s
# Create a mutex used to ensure there will be only one thread waiting to # Create a mutex used to ensure there will be only one thread waiting to
# shut Sidekiq down # shut Sidekiq down
...@@ -24,19 +25,19 @@ module Gitlab ...@@ -24,19 +25,19 @@ module Gitlab
Sidekiq.logger.warn "current RSS #{current_rss} exceeds maximum RSS "\ Sidekiq.logger.warn "current RSS #{current_rss} exceeds maximum RSS "\
"#{MAX_RSS}" "#{MAX_RSS}"
Sidekiq.logger.warn "spawned thread that will shut down PID "\ Sidekiq.logger.warn "this thread will shut down PID #{Process.pid} "\
"#{Process.pid} in #{GRACE_TIME} seconds" "in #{GRACE_TIME} seconds"
sleep(GRACE_TIME) sleep(GRACE_TIME)
Sidekiq.logger.warn "sending SIGUSR1 to PID #{Process.pid}" Sidekiq.logger.warn "sending SIGUSR1 to PID #{Process.pid}"
Process.kill('SIGUSR1', Process.pid) Process.kill('SIGUSR1', Process.pid)
Sidekiq.logger.warn "waiting #{SHUTDOWN_WAIT} seconds before sending "\ Sidekiq.logger.warn "waiting #{SHUTDOWN_WAIT} seconds before sending "\
"SIGTERM to PID #{Process.pid}" "#{SHUTDOWN_SIGNAL} to PID #{Process.pid}"
sleep(SHUTDOWN_WAIT) sleep(SHUTDOWN_WAIT)
Sidekiq.logger.warn "sending SIGTERM to PID #{Process.pid}" Sidekiq.logger.warn "sending #{SHUTDOWN_SIGNAL} to PID #{Process.pid}"
Process.kill('SIGTERM', Process.pid) Process.kill(SHUTDOWN_SIGNAL, Process.pid)
end end
end end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment