Commit 4259b102 authored by Qingyu Zhao's avatar Qingyu Zhao

Fix daemon memory killer busy loop risk

StandardError will cause busy loop in start_working. Simple fix is
to always `sleep` first in each loop iteration
parent aff4a27c
...@@ -39,8 +39,8 @@ module Gitlab ...@@ -39,8 +39,8 @@ module Gitlab
while enabled? while enabled?
begin begin
restart_sidekiq unless rss_within_range?
sleep(CHECK_INTERVAL_SECONDS) sleep(CHECK_INTERVAL_SECONDS)
restart_sidekiq unless rss_within_range?
rescue => e rescue => e
log_exception(e, __method__) log_exception(e, __method__)
rescue Exception => e # rubocop:disable Lint/RescueException rescue Exception => e # rubocop:disable Lint/RescueException
......
...@@ -40,6 +40,7 @@ describe Gitlab::SidekiqDaemon::MemoryKiller do ...@@ -40,6 +40,7 @@ describe Gitlab::SidekiqDaemon::MemoryKiller do
message: "Exception from start_working: My Exception") message: "Exception from start_working: My Exception")
expect(memory_killer).to receive(:rss_within_range?).twice.and_raise(StandardError, 'My Exception') expect(memory_killer).to receive(:rss_within_range?).twice.and_raise(StandardError, 'My Exception')
expect(memory_killer).to receive(:sleep).twice.with(Gitlab::SidekiqDaemon::MemoryKiller::CHECK_INTERVAL_SECONDS)
expect { subject }.not_to raise_exception expect { subject }.not_to raise_exception
end end
...@@ -53,6 +54,7 @@ describe Gitlab::SidekiqDaemon::MemoryKiller do ...@@ -53,6 +54,7 @@ describe Gitlab::SidekiqDaemon::MemoryKiller do
expect(memory_killer).to receive(:rss_within_range?).once.and_raise(Exception, 'My Exception') expect(memory_killer).to receive(:rss_within_range?).once.and_raise(Exception, 'My Exception')
expect(memory_killer).to receive(:sleep).with(Gitlab::SidekiqDaemon::MemoryKiller::CHECK_INTERVAL_SECONDS)
expect(Sidekiq.logger).to receive(:warn).once expect(Sidekiq.logger).to receive(:warn).once
.with( .with(
class: described_class.to_s, class: described_class.to_s,
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment