Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
12f45fdb
Commit
12f45fdb
authored
Jul 01, 2021
by
Grzegorz Bizon
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Split ci_queuing_disaster_recovery into two feature flags
parent
f7ccd64c
Changes
9
Hide whitespace changes
Inline
Side-by-side
Showing
9 changed files
with
36 additions
and
9 deletions
+36
-9
app/services/ci/queue/builds_table_strategy.rb
app/services/ci/queue/builds_table_strategy.rb
+1
-1
app/services/ci/queue/pending_builds_strategy.rb
app/services/ci/queue/pending_builds_strategy.rb
+1
-1
config/feature_flags/ops/ci_queueing_disaster_recovery_disable_fair_scheduling.yml
...ci_queueing_disaster_recovery_disable_fair_scheduling.yml
+8
-0
config/feature_flags/ops/ci_queueing_disaster_recovery_disable_quota.yml
...flags/ops/ci_queueing_disaster_recovery_disable_quota.yml
+8
-0
doc/administration/troubleshooting/defcon.md
doc/administration/troubleshooting/defcon.md
+11
-1
ee/app/services/ee/ci/queue/build_queue_service.rb
ee/app/services/ee/ci/queue/build_queue_service.rb
+1
-1
ee/spec/services/ci/register_job_service_spec.rb
ee/spec/services/ci/register_job_service_spec.rb
+2
-2
spec/services/ci/register_job_service_spec.rb
spec/services/ci/register_job_service_spec.rb
+1
-1
spec/spec_helper.rb
spec/spec_helper.rb
+3
-2
No files found.
app/services/ci/queue/builds_table_strategy.rb
View file @
12f45fdb
...
...
@@ -18,7 +18,7 @@ module Ci
.
joins
(
'LEFT JOIN project_features ON ci_builds.project_id = project_features.project_id'
)
.
where
(
'project_features.builds_access_level IS NULL or project_features.builds_access_level > 0'
)
if
Feature
.
enabled?
(
:ci_queueing_disaster_recovery
,
runner
,
type: :ops
,
default_enabled: :yaml
)
if
Feature
.
enabled?
(
:ci_queueing_disaster_recovery
_disable_fair_scheduling
,
runner
,
type: :ops
,
default_enabled: :yaml
)
# if disaster recovery is enabled, we fallback to FIFO scheduling
relation
.
order
(
'ci_builds.id ASC'
)
else
...
...
app/services/ci/queue/pending_builds_strategy.rb
View file @
12f45fdb
...
...
@@ -18,7 +18,7 @@ module Ci
.
joins
(
'LEFT JOIN project_features ON ci_pending_builds.project_id = project_features.project_id'
)
.
where
(
'project_features.builds_access_level IS NULL or project_features.builds_access_level > 0'
)
if
Feature
.
enabled?
(
:ci_queueing_disaster_recovery
,
runner
,
type: :ops
,
default_enabled: :yaml
)
if
Feature
.
enabled?
(
:ci_queueing_disaster_recovery
_disable_fair_scheduling
,
runner
,
type: :ops
,
default_enabled: :yaml
)
# if disaster recovery is enabled, we fallback to FIFO scheduling
relation
.
order
(
'ci_pending_builds.build_id ASC'
)
else
...
...
config/feature_flags/ops/ci_queueing_disaster_recovery_disable_fair_scheduling.yml
0 → 100644
View file @
12f45fdb
---
name
:
ci_queueing_disaster_recovery_disable_fair_scheduling
introduced_by_url
:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56658
rollout_issue_url
:
milestone
:
"
13.12"
type
:
ops
group
:
group::pipeline execution
default_enabled
:
false
config/feature_flags/ops/ci_queueing_disaster_recovery_disable_quota.yml
0 → 100644
View file @
12f45fdb
---
name
:
ci_queueing_disaster_recovery_disable_quota
introduced_by_url
:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56658
rollout_issue_url
:
milestone
:
"
13.12"
type
:
ops
group
:
group::pipeline execution
default_enabled
:
false
doc/administration/troubleshooting/defcon.md
View file @
12f45fdb
...
...
@@ -10,7 +10,7 @@ type: reference
This document describes a feature that allows you to disable some important but computationally
expensive parts of the application to relieve stress on the database during an ongoing downtime.
## `ci_queueing_disaster_recovery`
## `ci_queueing_disaster_recovery
_disable_fair_scheduling
`
This feature flag, if temporarily enabled, disables fair scheduling on shared runners.
This can help to reduce system resource usage on the
`jobs/request`
endpoint
...
...
@@ -20,6 +20,16 @@ Side effects:
-
In case of a large backlog of jobs, the jobs are processed in the order
they were put in the system, instead of balancing the jobs across many projects.
## `ci_queueing_disaster_recovery_disable_quota`
This feature flag, if temporarily enabled, disables enforcing CI minutes quota
on shared runners. This can help to reduce system resource usage on the
`jobs/request`
endpoint by significantly reducing the computations being
performed.
Side effects:
-
Projects which are out of quota will be run. This affects
only jobs created during the last hour, as prior jobs are canceled
by a periodic background worker (
`StuckCiJobsWorker`
).
ee/app/services/ee/ci/queue/build_queue_service.rb
View file @
12f45fdb
...
...
@@ -10,7 +10,7 @@ module EE
override
:builds_for_shared_runner
def
builds_for_shared_runner
# if disaster recovery is enabled, we disable quota
if
::
Feature
.
enabled?
(
:ci_queueing_disaster_recovery
,
runner
,
type: :ops
,
default_enabled: :yaml
)
if
::
Feature
.
enabled?
(
:ci_queueing_disaster_recovery
_disable_quota
,
runner
,
type: :ops
,
default_enabled: :yaml
)
super
else
enforce_minutes_based_on_cost_factors
(
super
)
...
...
ee/spec/services/ci/register_job_service_spec.rb
View file @
12f45fdb
...
...
@@ -37,7 +37,7 @@ RSpec.describe Ci::RegisterJobService, '#execute' do
end
it
'when in disaster recovery it ignores quota and returns anyway'
do
stub_feature_flags
(
ci_queueing_disaster_recovery:
true
)
stub_feature_flags
(
ci_queueing_disaster_recovery
_disable_quota
:
true
)
is_expected
.
to
be_kind_of
(
Ci
::
Build
)
end
...
...
@@ -67,7 +67,7 @@ RSpec.describe Ci::RegisterJobService, '#execute' do
end
it
'when in disaster recovery it ignores quota and returns anyway'
do
stub_feature_flags
(
ci_queueing_disaster_recovery:
true
)
stub_feature_flags
(
ci_queueing_disaster_recovery
_disable_quota
:
true
)
is_expected
.
to
be_kind_of
(
Ci
::
Build
)
end
...
...
spec/services/ci/register_job_service_spec.rb
View file @
12f45fdb
...
...
@@ -145,7 +145,7 @@ module Ci
context
'when using DEFCON mode that disables fair scheduling'
do
before
do
stub_feature_flags
(
ci_queueing_disaster_recovery:
true
)
stub_feature_flags
(
ci_queueing_disaster_recovery
_disable_fair_scheduling
:
true
)
end
context
'when all builds are pending'
do
...
...
spec/spec_helper.rb
View file @
12f45fdb
...
...
@@ -260,8 +260,9 @@ RSpec.configure do |config|
# tests, until we introduce it in user settings
stub_feature_flags
(
forti_token_cloud:
false
)
# This feature flag is by default disabled and used in disaster recovery mode
stub_feature_flags
(
ci_queueing_disaster_recovery:
false
)
# These feature flag are by default disabled and used in disaster recovery mode
stub_feature_flags
(
ci_queueing_disaster_recovery_disable_fair_scheduling:
false
)
stub_feature_flags
(
ci_queueing_disaster_recovery_disable_quota:
false
)
enable_rugged
=
example
.
metadata
[
:enable_rugged
].
present?
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment