Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
dc7c04de
Commit
dc7c04de
authored
May 11, 2017
by
Douglas Barbosa Alexandre
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Rename GeoBackfillWorker to GeoRepositorySyncWorker
parent
15b47e27
Changes
6
Hide whitespace changes
Inline
Side-by-side
Showing
6 changed files
with
37 additions
and
37 deletions
+37
-37
app/workers/geo_repository_sync_worker.rb
app/workers/geo_repository_sync_worker.rb
+9
-9
config/gitlab.yml.example
config/gitlab.yml.example
+2
-2
config/initializers/1_settings.rb
config/initializers/1_settings.rb
+3
-3
lib/gitlab/geo.rb
lib/gitlab/geo.rb
+4
-4
spec/lib/gitlab/geo_spec.rb
spec/lib/gitlab/geo_spec.rb
+4
-4
spec/workers/geo/geo_repository_sync_worker_spec.rb
spec/workers/geo/geo_repository_sync_worker_spec.rb
+15
-15
No files found.
app/workers/geo_
backfill
_worker.rb
→
app/workers/geo_
repository_sync
_worker.rb
View file @
dc7c04de
class
Geo
Backfill
Worker
class
Geo
RepositorySync
Worker
include
Sidekiq
::
Worker
include
CronjobQueue
...
...
@@ -15,20 +15,20 @@ class GeoBackfillWorker
project_ids_updated_recently
=
find_synced_project_ids_updated_recently
project_ids
=
interleave
(
project_ids_not_synced
,
project_ids_updated_recently
)
logger
.
info
"Started Geo
backfill
ing for
#{
project_ids
.
length
}
project(s)"
logger
.
info
"Started Geo
repository sync
ing for
#{
project_ids
.
length
}
project(s)"
project_ids
.
each
do
|
project_id
|
begin
break
if
over_time?
(
start_time
)
break
unless
Gitlab
::
Geo
.
current_node_enabled?
# We try to obtain a lease here for the entire
backfilling process
#
because backfill
the repositories continuously at a controlled rate
# instead of hammering the primary node. Initially, we are
backfill
ing
# We try to obtain a lease here for the entire
sync process because we
#
want to sync
the repositories continuously at a controlled rate
# instead of hammering the primary node. Initially, we are
sync
ing
# one repo at a time. If we don't obtain the lease here, every 5
# minutes all of 100 projects will be synced.
try_obtain_lease
do
|
lease
|
Geo
::
Repository
Backfill
Service
.
new
(
project_id
).
execute
Geo
::
Repository
Sync
Service
.
new
(
project_id
).
execute
end
rescue
ActiveRecord
::
RecordNotFound
logger
.
error
(
"Couldn't find project with ID=
#{
project_id
}
, skipping syncing"
)
...
...
@@ -36,7 +36,7 @@ class GeoBackfillWorker
end
end
logger
.
info
"Finished Geo
backfill
ing for
#{
project_ids
.
length
}
project(s)"
logger
.
info
"Finished Geo
repository sync
ing for
#{
project_ids
.
length
}
project(s)"
end
private
...
...
@@ -86,10 +86,10 @@ class GeoBackfillWorker
end
def
lease_key
Geo
::
Repository
Backfill
Service
::
LEASE_KEY_PREFIX
Geo
::
Repository
Sync
Service
::
LEASE_KEY_PREFIX
end
def
lease_timeout
Geo
::
Repository
Backfill
Service
::
LEASE_TIMEOUT
Geo
::
Repository
Sync
Service
::
LEASE_TIMEOUT
end
end
config/gitlab.yml.example
View file @
dc7c04de
...
...
@@ -216,9 +216,9 @@ production: &base
geo_bulk_notify_worker:
cron: "*/10 * * * * *"
# GitLab Geo
backfill
worker
# GitLab Geo
repository sync
worker
# NOTE: This will only take effect if Geo is enabled
geo_
backfill
_worker:
geo_
repository_sync
_worker:
cron: "*/5 * * * *"
# GitLab Geo file download worker
...
...
config/initializers/1_settings.rb
View file @
dc7c04de
...
...
@@ -396,9 +396,9 @@ Settings.cron_jobs['ldap_group_sync_worker']['job_class'] = 'LdapGroupSyncWorker
Settings
.
cron_jobs
[
'geo_bulk_notify_worker'
]
||=
Settingslogic
.
new
({})
Settings
.
cron_jobs
[
'geo_bulk_notify_worker'
][
'cron'
]
||=
'*/10 * * * * *'
Settings
.
cron_jobs
[
'geo_bulk_notify_worker'
][
'job_class'
]
||=
'GeoBulkNotifyWorker'
Settings
.
cron_jobs
[
'geo_
backfill
_worker'
]
||=
Settingslogic
.
new
({})
Settings
.
cron_jobs
[
'geo_
backfill
_worker'
][
'cron'
]
||=
'*/5 * * * *'
Settings
.
cron_jobs
[
'geo_
backfill_worker'
][
'job_class'
]
||=
'GeoBackfill
Worker'
Settings
.
cron_jobs
[
'geo_
repository_sync
_worker'
]
||=
Settingslogic
.
new
({})
Settings
.
cron_jobs
[
'geo_
repository_sync
_worker'
][
'cron'
]
||=
'*/5 * * * *'
Settings
.
cron_jobs
[
'geo_
repository_sync_worker'
][
'job_class'
]
||=
'GeoRepositorySync
Worker'
Settings
.
cron_jobs
[
'geo_download_dispatch_worker'
]
||=
Settingslogic
.
new
({})
Settings
.
cron_jobs
[
'geo_download_dispatch_worker'
][
'cron'
]
||=
'5 * * * *'
Settings
.
cron_jobs
[
'geo_download_dispatch_worker'
][
'job_class'
]
||=
'GeoFileDownloadDispatchWorker'
...
...
lib/gitlab/geo.rb
View file @
dc7c04de
...
...
@@ -13,7 +13,7 @@ module Gitlab
)
.
freeze
PRIMARY_JOBS
=
%i(bulk_notify_job)
.
freeze
SECONDARY_JOBS
=
%i(
backfill
_job file_download_job)
.
freeze
SECONDARY_JOBS
=
%i(
repository_sync
_job file_download_job)
.
freeze
def
self
.
current_node
self
.
cache_value
(
:geo_node_current
)
do
...
...
@@ -37,7 +37,7 @@ module Gitlab
def
self
.
current_node_enabled?
# No caching of the enabled! If we cache it and an admin disables
# this node, an active Geo
Backfill
Worker would keep going for up
# this node, an active Geo
RepositorySync
Worker would keep going for up
# to max run time after the node was disabled.
Gitlab
::
Geo
.
current_node
.
reload
.
enabled?
end
...
...
@@ -74,8 +74,8 @@ module Gitlab
Sidekiq
::
Cron
::
Job
.
find
(
'geo_bulk_notify_worker'
)
end
def
self
.
backfill
_job
Sidekiq
::
Cron
::
Job
.
find
(
'geo_
backfill
_worker'
)
def
self
.
repository_sync
_job
Sidekiq
::
Cron
::
Job
.
find
(
'geo_
repository_sync
_worker'
)
end
def
self
.
file_download_job
...
...
spec/lib/gitlab/geo_spec.rb
View file @
dc7c04de
...
...
@@ -122,7 +122,7 @@ describe Gitlab::Geo, lib: true do
end
before
(
:all
)
do
jobs
=
%w(geo_bulk_notify_worker geo_
backfill
_worker)
jobs
=
%w(geo_bulk_notify_worker geo_
repository_sync
_worker)
jobs
.
each
{
|
job
|
init_cron_job
(
job
,
job
.
camelize
)
}
# TODO: Make this name consistent
...
...
@@ -134,7 +134,7 @@ describe Gitlab::Geo, lib: true do
described_class
.
configure_cron_jobs!
expect
(
described_class
.
bulk_notify_job
).
to
be_enabled
expect
(
described_class
.
backfill
_job
).
not_to
be_enabled
expect
(
described_class
.
repository_sync
_job
).
not_to
be_enabled
expect
(
described_class
.
file_download_job
).
not_to
be_enabled
end
...
...
@@ -143,7 +143,7 @@ describe Gitlab::Geo, lib: true do
described_class
.
configure_cron_jobs!
expect
(
described_class
.
bulk_notify_job
).
not_to
be_enabled
expect
(
described_class
.
backfill
_job
).
to
be_enabled
expect
(
described_class
.
repository_sync
_job
).
to
be_enabled
expect
(
described_class
.
file_download_job
).
to
be_enabled
end
...
...
@@ -151,7 +151,7 @@ describe Gitlab::Geo, lib: true do
described_class
.
configure_cron_jobs!
expect
(
described_class
.
bulk_notify_job
).
not_to
be_enabled
expect
(
described_class
.
backfill
_job
).
not_to
be_enabled
expect
(
described_class
.
repository_sync
_job
).
not_to
be_enabled
expect
(
described_class
.
file_download_job
).
not_to
be_enabled
end
end
...
...
spec/workers/geo/geo_
backfill
_worker_spec.rb
→
spec/workers/geo/geo_
repository_sync
_worker_spec.rb
View file @
dc7c04de
require
'spec_helper'
describe
Geo
::
Geo
Backfill
Worker
,
services:
true
do
describe
Geo
::
Geo
RepositorySync
Worker
,
services:
true
do
let!
(
:primary
)
{
create
(
:geo_node
,
:primary
,
host:
'primary-geo-node'
)
}
let!
(
:secondary
)
{
create
(
:geo_node
,
:current
)
}
let!
(
:project_1
)
{
create
(
:empty_project
)
}
...
...
@@ -13,25 +13,25 @@ describe Geo::GeoBackfillWorker, services: true do
allow_any_instance_of
(
Gitlab
::
ExclusiveLease
).
to
receive
(
:try_obtain
)
{
true
}
end
it
'performs Geo::Repository
Backfill
Service for each project'
do
expect
(
Geo
::
Repository
Backfill
Service
).
to
receive
(
:new
).
twice
.
and_return
(
spy
)
it
'performs Geo::Repository
Sync
Service for each project'
do
expect
(
Geo
::
Repository
Sync
Service
).
to
receive
(
:new
).
twice
.
and_return
(
spy
)
subject
.
perform
end
it
'performs Geo::Repository
BackfillService for projects where last attempt to backfill
failed'
do
it
'performs Geo::Repository
SyncService for projects where last attempt to sync
failed'
do
Geo
::
ProjectRegistry
.
create
(
project:
project_1
,
last_repository_synced_at:
DateTime
.
now
,
last_repository_successful_sync_at:
nil
)
expect
(
Geo
::
Repository
Backfill
Service
).
to
receive
(
:new
).
twice
.
and_return
(
spy
)
expect
(
Geo
::
Repository
Sync
Service
).
to
receive
(
:new
).
twice
.
and_return
(
spy
)
subject
.
perform
end
it
'performs Geo::Repository
BackfillService for backfill
ed projects updated recently'
do
it
'performs Geo::Repository
SyncService for sync
ed projects updated recently'
do
Geo
::
ProjectRegistry
.
create
(
project:
project_1
,
last_repository_synced_at:
2
.
days
.
ago
,
...
...
@@ -47,39 +47,39 @@ describe Geo::GeoBackfillWorker, services: true do
project_1
.
update_attribute
(
:last_repository_updated_at
,
2
.
days
.
ago
)
project_2
.
update_attribute
(
:last_repository_updated_at
,
10
.
minutes
.
ago
)
expect
(
Geo
::
Repository
Backfill
Service
).
to
receive
(
:new
).
once
.
and_return
(
spy
)
expect
(
Geo
::
Repository
Sync
Service
).
to
receive
(
:new
).
once
.
and_return
(
spy
)
subject
.
perform
end
it
'does not perform Geo::Repository
Backfill
Service when tracking DB is not available'
do
it
'does not perform Geo::Repository
Sync
Service when tracking DB is not available'
do
allow
(
Rails
.
configuration
).
to
receive
(
:respond_to?
).
with
(
:geo_database
)
{
false
}
expect
(
Geo
::
Repository
Backfill
Service
).
not_to
receive
(
:new
)
expect
(
Geo
::
Repository
Sync
Service
).
not_to
receive
(
:new
)
subject
.
perform
end
it
'does not perform Geo::Repository
Backfill
Service when primary node does not exists'
do
it
'does not perform Geo::Repository
Sync
Service when primary node does not exists'
do
allow
(
Gitlab
::
Geo
).
to
receive
(
:primary_node
)
{
nil
}
expect
(
Geo
::
Repository
Backfill
Service
).
not_to
receive
(
:new
)
expect
(
Geo
::
Repository
Sync
Service
).
not_to
receive
(
:new
)
subject
.
perform
end
it
'does not perform Geo::Repository
Backfill
Service when node is disabled'
do
it
'does not perform Geo::Repository
Sync
Service when node is disabled'
do
allow_any_instance_of
(
GeoNode
).
to
receive
(
:enabled?
)
{
false
}
expect
(
Geo
::
Repository
Backfill
Service
).
not_to
receive
(
:new
)
expect
(
Geo
::
Repository
Sync
Service
).
not_to
receive
(
:new
)
subject
.
perform
end
it
'does not perform Geo::Repository
Backfill
Service when can not obtain a lease'
do
it
'does not perform Geo::Repository
Sync
Service when can not obtain a lease'
do
allow_any_instance_of
(
Gitlab
::
ExclusiveLease
).
to
receive
(
:try_obtain
)
{
false
}
expect
(
Geo
::
Repository
Backfill
Service
).
not_to
receive
(
:new
)
expect
(
Geo
::
Repository
Sync
Service
).
not_to
receive
(
:new
)
subject
.
perform
end
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment