Commit f91d0e09 authored by Ruben Davila's avatar Ruben Davila

Merge branch 'master' into time-tracking-integration

Conflicts:
	app/helpers/issuables_helper.rb
parents dcb60557 da0906c3
CHANGELOG.md merge=union
CHANGELOG-EE.md merge=union
*.js.es6 gitlab-language=javascript
......@@ -2,36 +2,21 @@ Please view this file on the master branch, on stable branches it's out of date.
## 8.14.0 (2016-11-22)
- No changes.
## 8.14.0 (2016-11-22)
- No changes.
## 8.14.0 (2016-11-22)
- No changes.
## 8.14.0 (2016-11-22)
- Gracefully recover from previously failed rebase.
## 8.14.0 (2016-11-22)
- No changes.
## 8.14.0 (2016-11-22)
- Disable retries for remote mirror update worker. !848
- Fix Approvals API documentation.
- Add ability to set approvals_before_merge for project through the API.
## 8.14.0 (2016-11-22)
- gitlab:check rake task checks ES version according to requirements
- Convert ASCII-8BIT LDAP DNs to UTF-8 to avoid unnecessary user deletions
- [Fix] Only owner can see "Projects" button in group edit menu
## 8.13.6 (2016-11-17)
- No changes.
- Disable retries for remote mirror update worker. !848
- Fixed cache clearing on secondary Geo nodes. !869
- Geo: fix a problem that prevented git cloning from secondary node. !873
## 8.13.5 (2016-11-08)
- No changes
......
......@@ -4,18 +4,6 @@ entry.
## 8.14.0 (2016-11-22)
- No changes.
## 8.14.0 (2016-11-22)
- No changes.
## 8.14.0 (2016-11-22)
- No changes.
## 8.14.0 (2016-11-22)
- Centralize LDAP config/filter logic. !6606
- Show random messages when the To Do list is empty. !6818 (Josep Llaneras)
- Fix record not found error on NewNoteWorker processing. !6863 (Oswaldo Ferreira)
......@@ -37,9 +25,6 @@ entry.
- Remove additional padding on right-aligned items in MR widget. !7411 (Didem Acet)
- Fix issue causing Labels not to appear in sidebar on MR page. !7416 (Alex Sanford)
- Fix project Visibility Level selector not using default values.
## 8.14.0 (2016-11-22)
- Use separate email-token for incoming email and revert back the inactive feature. !5914
- Replace jQuery.timeago with timeago.js. !6274 (ClemMakesApps)
- Add CI notifications. Who triggered a pipeline would receive an email after the pipeline is succeeded or failed. Users could also update notification settings accordingly. !6342
......@@ -142,6 +127,18 @@ entry.
- Updated commit SHA styling on the branches page.
- Fix 404 when visit /projects page
## 8.13.6 (2016-11-17)
- No changes.
- Omniauth auto link LDAP user falls back to find by DN when user cannot be found by UID. !7002
- Fix no "Register" tab if ldap auth is enabled (#24038). !7274 (Luc Didry)
- Fix cache for commit status in commits list to respect branches. !7372
- Fix issue causing Labels not to appear in sidebar on MR page. !7416 (Alex Sanford)
- Limit labels returned for a specific project as an administrator. !7496
- Clicking "force remove source branch" label now toggles the checkbox again.
- Allow commit note to be visible if repo is visible.
- Fix project Visibility Level selector not using default values.
## 8.13.5 (2016-11-08)
- Restore unauthenticated access to public container registries
......
......@@ -39,11 +39,6 @@ module IssuablesHelper
end
end
def can_add_template?(issuable)
names = issuable_templates(issuable)
names.empty? && can?(current_user, :push_code, @project) && !@project.private?
end
def template_dropdown_tag(issuable, &block)
title = selected_template(issuable) || "Choose a template"
options = {
......
module Geo
class ScheduleRepoUpdateService
attr_reader :id, :clone_url
attr_reader :id, :clone_url, :push_data
def initialize(params)
@id = params[:project_id]
@clone_url = params[:project][:git_ssh_url]
@push_data = { 'type' => params[:object_kind], 'before' => params[:before],
'after' => params[:newref], 'ref' => params[:ref] }
end
def execute
GeoRepositoryUpdateWorker.perform_async(@id, @clone_url)
GeoRepositoryUpdateWorker.perform_async(@id, @clone_url, @push_data)
end
end
end
......@@ -14,7 +14,7 @@ module Users
private
def record_activity
user_activity.touch
user_activity.touch unless Gitlab::Geo.secondary?
Rails.logger.debug("Recorded activity: #{@activity} for User ID: #{@author.id} (username: #{@author.username}")
end
......
......@@ -3,7 +3,7 @@
Geo Nodes
%p.light
With #{link_to 'GitLab Geo', help_page_path('gitlab-geo', 'README'), class: 'vlink'} you can install a special
With #{link_to 'GitLab Geo', help_page_path('gitlab-geo/README'), class: 'vlink'} you can install a special
read-only and replicated instance anywhere.
%hr
......@@ -31,7 +31,7 @@
= fg.text_area :key, class: 'form-control thin_area', rows: 5
%p.help-block
Paste a machine public key here for the GitLab user this node runs on. Read more about how to generate it
= link_to "here", help_page_path("ssh", "README")
= link_to "here", help_page_path("ssh/README")
.form-actions
= f.submit 'Add Node', class: 'btn btn-create'
......
- form = local_assigns.fetch(:f)
- project = @target_project || @project
= form_errors(issuable)
......@@ -10,44 +11,17 @@
and make sure your changes will not unintentionally remove theirs
.form-group
= f.label :title, class: 'control-label'
= form.label :title, class: 'control-label'
= render 'shared/issuable/form/template_selector', issuable: issuable
%div{ class: issuable_templates(issuable).any? ? 'col-sm-7 col-lg-8' : 'col-sm-10' }
= f.text_field :title, maxlength: 255, autofocus: true, autocomplete: 'off',
class: 'form-control pad', required: true
- if issuable.is_a?(MergeRequest)
%p.help-block
.js-wip-explanation
%a.js-toggle-wip{href: "", tabindex: -1}
Remove the
%code WIP:
prefix from the title
to allow this
%strong Work In Progress
merge request to be merged when it's ready.
.js-no-wip-explanation
%a.js-toggle-wip{href: "", tabindex: -1}
Start the title with
%code WIP:
to prevent a
%strong Work In Progress
merge request from being merged before it's ready.
- if can_add_template?(issuable)
%p.help-block
Add
= link_to "description templates", help_page_path('user/project/description_templates'), tabindex: -1
to help your contributors communicate effectively!
= render 'shared/issuable/form/title', issuable: issuable, form: form
.form-group.detail-page-description
= f.label :description, 'Description', class: 'control-label'
= form.label :description, 'Description', class: 'control-label'
.col-sm-10
= render layout: 'projects/md_preview', locals: { preview_class: "md-preview", referenced_users: true } do
= render 'projects/zen', f: f, attr: :description,
= render 'projects/zen', f: form, attr: :description,
classes: 'note-textarea',
placeholder: "Write a comment or drag your files here...",
supports_slash_commands: !issuable.persisted?
......@@ -59,8 +33,8 @@
.form-group
.col-sm-offset-2.col-sm-10
.checkbox
= f.label :confidential do
= f.check_box :confidential
= form.label :confidential do
= form.check_box :confidential
This issue is confidential and should only be visible to team members with at least Reporter access.
- if can?(current_user, :"admin_#{issuable.to_ability_name}", issuable.project)
......@@ -69,22 +43,22 @@
.row
%div{ class: (has_due_date ? "col-lg-6" : "col-sm-12") }
.form-group.issue-assignee
= f.label :assignee_id, "Assignee", class: "control-label #{"col-lg-4" if has_due_date}"
= form.label :assignee_id, "Assignee", class: "control-label #{"col-lg-4" if has_due_date}"
.col-sm-10{ class: ("col-lg-8" if has_due_date) }
.issuable-form-select-holder
- if issuable.assignee_id
= f.hidden_field :assignee_id
= form.hidden_field :assignee_id
= dropdown_tag(user_dropdown_label(issuable.assignee_id, "Assignee"), options: { toggle_class: "js-dropdown-keep-input js-user-search js-issuable-form-dropdown js-assignee-search", title: "Select assignee", filter: true, dropdown_class: "dropdown-menu-user dropdown-menu-selectable dropdown-menu-assignee js-filter-submit",
placeholder: "Search assignee", data: { first_user: current_user.try(:username), null_user: true, current_user: true, project_id: project.try(:id), selected: issuable.assignee_id, field_name: "#{issuable.class.model_name.param_key}[assignee_id]", default_label: "Assignee"} })
.form-group.issue-milestone
= f.label :milestone_id, "Milestone", class: "control-label #{"col-lg-4" if has_due_date}"
= form.label :milestone_id, "Milestone", class: "control-label #{"col-lg-4" if has_due_date}"
.col-sm-10{ class: ("col-lg-8" if has_due_date) }
.issuable-form-select-holder
= render "shared/issuable/milestone_dropdown", selected: issuable.milestone, name: "#{issuable.class.model_name.param_key}[milestone_id]", show_any: false, show_upcoming: false, extra_class: "js-issuable-form-dropdown js-dropdown-keep-input", dropdown_title: "Select milestone"
.form-group
- has_labels = @labels && @labels.any?
= f.label :label_ids, "Labels", class: "control-label #{"col-lg-4" if has_due_date}"
= f.hidden_field :label_ids, multiple: true, value: ''
= form.label :label_ids, "Labels", class: "control-label #{"col-lg-4" if has_due_date}"
= form.hidden_field :label_ids, multiple: true, value: ''
.col-sm-10{ class: "#{"col-lg-8" if has_due_date} #{'issuable-form-padding-top' if !has_labels}" }
.issuable-form-select-holder
= render "shared/issuable/label_dropdown", classes: ["js-issuable-form-dropdown"], selected: issuable.labels, data_options: { field_name: "#{issuable.class.model_name.param_key}[label_ids][]", show_any: false, show_menu_above: 'true' }, dropdown_title: "Select label"
......@@ -94,12 +68,12 @@
- weight_options.delete(Issue::WEIGHT_ALL)
- weight_options.delete(Issue::WEIGHT_ANY)
.form-group
= f.label :label_ids, class: "control-label #{"col-lg-4" if has_due_date}" do
= form.label :label_ids, class: "control-label #{"col-lg-4" if has_due_date}" do
Weight
.col-sm-10{ class: ("col-lg-8" if has_due_date) }
.issuable-form-select-holder
- if issuable.weight
= f.hidden_field :weight
= form.hidden_field :weight
= dropdown_tag(issuable.weight || "Weight", options: { title: "Select weight", toggle_class: 'js-weight-select js-issuable-form-weight', dropdown_class: "dropdown-menu-selectable dropdown-menu-weight",
placeholder: "Search weight", data: { field_name: "#{issuable.class.model_name.param_key}[weight]" , default_label: "Weight" } }) do
%ul
......@@ -111,10 +85,10 @@
- if has_due_date
.col-lg-6
.form-group
= f.label :due_date, "Due date", class: "control-label"
= form.label :due_date, "Due date", class: "control-label"
.col-sm-10
.issuable-form-select-holder
= f.text_field :due_date, id: "issuable-due-date", class: "datepicker form-control", placeholder: "Select due date"
= form.text_field :due_date, id: "issuable-due-date", class: "datepicker form-control", placeholder: "Select due date"
- if issuable.can_move?(current_user)
%hr
......@@ -128,21 +102,21 @@
title: 'Moving an issue will copy the discussion to a different project and close it here. All participants will be notified of the new location.' }
= icon('question-circle')
= render 'shared/issuable/approvals', issuable: issuable, f: f
= render 'shared/issuable/approvals', issuable: issuable, f: form
- if issuable.is_a?(MergeRequest) && !issuable.closed_without_fork?
%hr
- if @merge_request.new_record?
.form-group
= f.label :source_branch, class: 'control-label'
= form.label :source_branch, class: 'control-label'
.col-sm-10
.issuable-form-select-holder
= f.select(:source_branch, [@merge_request.source_branch], { }, { class: 'source_branch select2 span2', disabled: true })
= form.select(:source_branch, [@merge_request.source_branch], { }, { class: 'source_branch select2 span2', disabled: true })
.form-group
= f.label :target_branch, class: 'control-label'
= form.label :target_branch, class: 'control-label'
.col-sm-10
.issuable-form-select-holder
= f.select(:target_branch, @merge_request.target_branches, { include_blank: true }, { class: 'target_branch select2 span2', disabled: @merge_request.new_record?, data: {placeholder: "Select branch"} })
= form.select(:target_branch, @merge_request.target_branches, { include_blank: true }, { class: 'target_branch select2 span2', disabled: @merge_request.new_record?, data: {placeholder: "Select branch"} })
- if @merge_request.new_record?
 
= link_to 'Change branches', mr_change_branches_path(@merge_request)
......@@ -158,9 +132,9 @@
- is_footer = !(issuable.is_a?(MergeRequest) && issuable.new_record?)
.row-content-block{class: (is_footer ? "footer-block" : "middle-block")}
- if issuable.new_record?
= f.submit "Submit #{issuable.class.model_name.human.downcase}", class: 'btn btn-create'
= form.submit "Submit #{issuable.class.model_name.human.downcase}", class: 'btn btn-create'
- else
= f.submit 'Save changes', class: 'btn btn-save'
= form.submit 'Save changes', class: 'btn btn-save'
- if !issuable.persisted? && !issuable.project.empty_repo? && (guide_url = contribution_guide_path(issuable.project))
.inline.prepend-left-10
......@@ -184,4 +158,4 @@
= icon("sign-out")
Remove
= f.hidden_field :lock_version
= form.hidden_field :lock_version
- issuable = local_assigns.fetch(:issuable)
- form = local_assigns.fetch(:form)
- no_issuable_templates = issuable_templates(issuable).empty?
- div_class = no_issuable_templates ? 'col-sm-10' : 'col-sm-7 col-lg-8'
%div{ class: div_class }
= form.text_field :title, required: true, maxlength: 255, autofocus: true,
autocomplete: 'off', class: 'form-control pad'
- if issuable.respond_to?(:work_in_progress?)
%p.help-block
.js-wip-explanation
%a.js-toggle-wip{ href: '', tabindex: -1 }
Remove the
%code WIP:
prefix from the title
to allow this
%strong Work In Progress
merge request to be merged when it's ready.
.js-no-wip-explanation
%a.js-toggle-wip{ href: '', tabindex: -1 }
Start the title with
%code WIP:
to prevent a
%strong Work In Progress
merge request from being merged before it's ready.
- if no_issuable_templates && can?(current_user, :push_code, issuable.project)
%p.help-block
Add
= link_to 'description templates', help_page_path('user/project/description_templates'), tabindex: -1
to help your contributors communicate effectively!
......@@ -5,16 +5,46 @@ class GeoRepositoryUpdateWorker
attr_accessor :project
def perform(project_id, clone_url)
def perform(project_id, clone_url, push_data = nil)
@project = Project.find(project_id)
@push_data = push_data
fetch_repository(clone_url)
process_hooks if push_data # we should be compatible with old unprocessed data
end
private
def fetch_repository(remote_url)
@project.create_repository unless @project.repository_exists?
@project.repository.after_create if @project.empty_repo?
@project.repository.fetch_geo_mirror(remote_url)
end
def process_hooks
if @push_data['type'] == 'push'
branch = Gitlab::Git.ref_name(@push_data['ref'])
process_push(branch, @push_data['after'])
end
end
def process_push(branch, revision)
@project.repository.after_push_commit(branch, revision)
if push_remove_branch?
@project.repository.after_remove_branch
elsif push_to_new_branch?
@project.repository.after_create_branch
end
ProjectCacheWorker.perform_async(@project.id)
end
def push_remove_branch?
Gitlab::Git.branch_ref?(@push_data['ref']) && Gitlab::Git.blank_ref?(@push_data['after'])
end
def push_to_new_branch?
Gitlab::Git.branch_ref?(@push_data['ref']) && Gitlab::Git.blank_ref?(@push_data['before'])
end
end
......@@ -6,6 +6,7 @@
class ProjectCacheWorker
include Sidekiq::Worker
include DedicatedSidekiqQueue
prepend EE::Workers::ProjectCacheWorker
LEASE_TIMEOUT = 15.minutes.to_i
......
......@@ -117,6 +117,41 @@ production:
[Restart GitLab][restart] for the changes to take effect.
## External Groups
>**Note:** External Groups configuration is only available in GitLab EE Version
8.9 and above.
Using the `external_groups` setting will allow you to mark all users belonging
to these groups as [external users](../../user/permissions.md). Group membership is
checked periodically through the `LdapGroupSync` background task.
**Omnibus configuration**
```ruby
gitlab_rails['ldap_servers'] = YAML.load <<-EOS
main:
# snip...
external_groups: ['interns', 'contractors']
EOS
```
[Reconfigure GitLab][reconfigure] for the changes to take effect.
**Source configuration**
```yaml
production:
ldap:
servers:
main:
# snip...
external_groups: ['interns', 'contractors']
```
[Restart GitLab][restart] for the changes to take effect.
## Group Sync Technical Details
There is a lot going on with group sync 'under the hood'. This section
......@@ -178,6 +213,8 @@ network and LDAP server response time will affect these metrics.
## Troubleshooting
### Referral Error
If you see `LDAP search error: Referral` in the logs, or when troubleshooting
LDAP Group Sync, this error may indicate a configuration problem. The LDAP
configuration `/etc/gitlab/gitlab.rb` (Omnibus) or `config/gitlab.yml` (source)
......@@ -202,3 +239,178 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server
account control attribute (`userAccountControl:1.2.840.113556.1.4.803`)
has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/
for more information.
### User DN has changed
When an LDAP user is created in GitLab, their LDAP DN is stored for later reference.
If a user's DN changes, it can cause problems for LDAP sync. Administrators can
manually update a user's stored DN in this case.
> **Note:** If GitLab cannot find a user by their DN, it will attempt to fallback
to finding the user by their email. If the lookup is successful, GitLab will
update the stored DN to the new value.
1. Sign in to GitLab as an administrator user.
1. Navigate to **Admin area -> Users**.
1. Search for the user
1. Open the user, by clicking on their name. Do not click 'Edit'.
1. Navigate to the **Identities** tab.
1. Click 'Edit' next to the LDAP identity.
1. Change the 'Identifier' to match the user's new LDAP DN.
1. Save the identity.
Now the user should sync correctly.
### User is not being added to a group
Sometimes you may think a particular user should be added to a GitLab group via
LDAP group sync, but for some reason it's not happening. There are several
things to check to debug the situation.
- Ensure LDAP configuration has a `group_base` specified. This configuration is
required for group sync to work properly.
- Ensure the correct LDAP group link is added to the GitLab group. Check group
links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**.
- Check that the user has an LDAP identity
1. Sign in to GitLab as an administrator user.
1. Navigate to **Admin area -> Users**.
1. Search for the user
1. Open the user, by clicking on their name. Do not click 'Edit'.
1. Navigate to the **Identities** tab. There should be an LDAP identity with
an LDAP DN as the 'Identifier'.
If all of the above looks good, jump in to a little more advanced debugging.
Often, the best way to learn more about why group sync is behaving a certain
way is to enable debug logging. There is verbose output that details every
step of the sync.
1. Start a Rails console
```bash
# For Omnibus installations
sudo gitlab-rails console
# For installations from source
sudo -u git -H bundle exec rails console production
```
1. Set the log level to debug (only for this session):
```ruby
Rails.logger.level = Logger::DEBUG
```
1. Choose a GitLab group to test with. This group should have an LDAP group link
already configured. If the output is `nil`, the group could not be found.
If a bunch of group attributes are output, your group was found successfully.
```ruby
group = Group.find_by(name: 'my_group')
# Output
=> #<Group:0x007fe825196558 id: 1234, name: "my_group"...>
```
1. Run a group sync for this particular group.
```ruby
EE::Gitlab::LDAP::Sync::Group.execute_all_providers(group)
```
1. Look through the output of the sync. See [example log output](#example-log-output)
below for more information about the output.
1. If you still aren't able to see why the user isn't being added, query the
LDAP group directly to see what members are listed. Still in the Rails console,
run the following query:
```ruby
adapter = Gitlab::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider
ldap_group = EE::Gitlab::LDAP::Group.find_by_cn('group_cn_here', adapter)
# Output
=> #<EE::Gitlab::LDAP::Group:0x007fcbdd0bb6d8
```
1. Query the LDAP group's member DNs and see if the user's DN is in the list.
One of the DNs here should match the 'Identifier' from the LDAP identity
checked earlier. If it doesn't, the user does not appear to be in the LDAP
group.
```ruby
ldap_group.member_dns
# Output
=> ["uid=john,ou=people,dc=example,dc=com", "uid=mary,ou=people,dc=example,dc=com"]
```
1. Some LDAP servers don't store members by DN. Rather, they use UIDs instead.
If you didn't see results from the last query, try querying by UIDs instead.
```ruby
ldap_group.member_uids
# Output
=> ['john','mary']
```
#### Example log output
The output of the last command will be very verbose, but contains lots of
helpful information. For the most part you can ignore log entries that are SQL
statements.
Indicates the point where syncing actually begins:
```bash
Started syncing all providers for 'my_group' group
```
The follow entry shows an array of all user DNs GitLab sees in the LDAP server.
Note that these are the users for a single LDAP group, not a GitLab group. If
you have multiple LDAP groups linked to this GitLab group, you will see multiple
log entries like this - one for each LDAP group. If you don't see an LDAP user
DN in this log entry, LDAP is not returning the user when we do the lookup.
Verify the user is actually in the LDAP group.
```bash
Members in 'ldap_group_1' LDAP group: ["uid=john0,ou=people,dc=example,dc=com",
"uid=mary0,ou=people,dc=example,dc=com", "uid=john1,ou=people,dc=example,dc=com",
"uid=mary1,ou=people,dc=example,dc=com", "uid=john2,ou=people,dc=example,dc=com",
"uid=mary2,ou=people,dc=example,dc=com", "uid=john3,ou=people,dc=example,dc=com",
"uid=mary3,ou=people,dc=example,dc=com", "uid=john4,ou=people,dc=example,dc=com",
"uid=mary4,ou=people,dc=example,dc=com"]
```
Shortly after each of the above entries, you will see a hash of resolved member
access levels. This hash represents all user DNs GitLab thinks should have
access to this group, and at which access level (role). This hash is additive,
and more DNs may be added, or existing entries modified, based on additional
LDAP group lookups. The very last occurrence of this entry should indicate
exactly which users GitLab believes should be added to the group.
> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Master'
and 50 is 'Owner'
```bash
Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30,
"uid=mary0,ou=people,dc=example,dc=com"=>30, "uid=john1,ou=people,dc=example,dc=com"=>30,
"uid=mary1,ou=people,dc=example,dc=com"=>30, "uid=john2,ou=people,dc=example,dc=com"=>30,
"uid=mary2,ou=people,dc=example,dc=com"=>30, "uid=john3,ou=people,dc=example,dc=com"=>30,
"uid=mary3,ou=people,dc=example,dc=com"=>30, "uid=john4,ou=people,dc=example,dc=com"=>30,
"uid=mary4,ou=people,dc=example,dc=com"=>30}
```
It's not uncommon to see warnings like the following. These indicate that GitLab
would have added the user to a group, but the user could not be found in GitLab.
Usually this is not a cause for concern.
If you think a particular user should already exist in GitLab, but you're seeing
this entry, it could be due to a mismatched DN stored in GitLab. See
[User DN has changed](#User-DN-has-changed) to update the user's LDAP identity.
```bash
User with DN `uid=john0,ou=people,dc=example,dc=com` should have access
to 'my_group' group but there is no user in GitLab with that
identity. Membership will be updated once the user signs in for
the first time.
```
Finally, the following entry says syncing has finished for this group:
```bash
Finished syncing all providers for 'my_group' group
```
......@@ -11,6 +11,10 @@ membership syncing.
The information on this page is relevent for both GitLab CE and EE. For more
details about EE-specific LDAP features, see [LDAP EE Documentation](ldap-ee.md).
[//]: # (Do *NOT* modify this file in EE documentation. All changes in this)
[//]: # (file should happen in CE, too. If the change is EE-specific, put)
[//]: # (it in `ldap-ee.md`.)
## Security
GitLab assumes that LDAP users are not able to change their LDAP 'mail', 'email'
......@@ -206,28 +210,6 @@ production:
# snip...
```
### External Groups
>**Note:** External Groups configuration is only available in GitLab EE Version
8.9 and above.
Using the `external_groups` setting will allow you to mark all users belonging
to these groups as [external users](../../user/permissions.md). Group membership is
checked periodically through the `LdapGroupSync` background task.
**Configuration**
```yaml
# An array of CNs of groups containing users that should be considered external
#
# Ex. ['interns', 'contractors']
#
# Note: Not `cn=interns` or the full DN
#
external_groups: []
```
## Using an LDAP filter to limit access to your GitLab server
If you want to limit all GitLab access to a subset of the LDAP users on your
......@@ -278,6 +260,23 @@ In other words, if an existing GitLab user wants to enable LDAP sign-in for
themselves, they should check that their GitLab email address matches their
LDAP email address, and then sign into GitLab via their LDAP credentials.
## Limitations
### TLS Client Authentication
Not implemented by `Net::LDAP`.
You should disable anonymous LDAP authentication and enable simple or SASL
authentication. The TLS client authentication setting in your LDAP server cannot
be mandatory and clients cannot be authenticated with the TLS protocol.
### TLS Server Authentication
Not supported by GitLab's configuration options.
When setting `method: ssl`, the underlying authentication method used by
`omniauth-ldap` is `simple_tls`. This method establishes TLS encryption with
the LDAP server before any LDAP-protocol data is exchanged but no validation of
the LDAP server's SSL certificate is performed.
## Troubleshooting
### Debug LDAP user filter with ldapsearch
......@@ -287,7 +286,7 @@ following query returns the login names of the users that will be allowed to
log in to GitLab if you configure your own user_filter.
```
ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "(&(ObjectClass=User)($user_filter))" sAMAccountName
ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName
```
- Variables beginning with a `$` refer to a variable from the LDAP section of
......@@ -297,6 +296,7 @@ ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$ba
port.
- We are assuming the password for the bind_dn user is in bind_dn_password.txt.
### Invalid credentials when logging in
- Make sure the user you are binding with has enough permissions to read the user's
......
......@@ -7,19 +7,10 @@ highly available.
## Architecture
### Active/Passive
For pure high-availability/failover with no scaling you can use an
active/passive configuration. This utilizes DRBD (Distributed Replicated
Block Device) to keep all data in sync. DRBD requires a low latency link to
remain in sync. It is not advisable to attempt to run DRBD between data centers
or in different cloud availability zones.
There are two kinds of setups:
Components/Servers Required:
- 2 servers/virtual machines (one active/one passive)
![Active/Passive HA Diagram](../img/high_availability/active-passive-diagram.png)
- active/active
- active/passive
### Active/Active
......@@ -28,12 +19,24 @@ user requests simultaneously. The database, Redis, and GitLab application are
all deployed on separate servers. The configuration is **only** highly-available
if the database, Redis and storage are also configured as such.
![Active/Active HA Diagram](../img/high_availability/active-active-diagram.png)
**Steps to configure active/active:**
Follow the steps below to configure an active/active setup:
1. [Configure the database](database.md)
1. [Configure Redis](redis.md)
1. [Configure NFS](nfs.md)
1. [Configure the GitLab application servers](gitlab.md)
1. [Configure the load balancers](load_balancer.md)
![Active/Active HA Diagram](../img/high_availability/active-active-diagram.png)
### Active/Passive
For pure high-availability/failover with no scaling you can use an
active/passive configuration. This utilizes DRBD (Distributed Replicated
Block Device) to keep all data in sync. DRBD requires a low latency link to
remain in sync. It is not advisable to attempt to run DRBD between data centers
or in different cloud availability zones.
Components/Servers Required: 2 servers/virtual machines (one active/one passive)
![Active/Passive HA Diagram](../img/high_availability/active-passive-diagram.png)
# Configuring Redis for GitLab HA
You can choose to install and manage Redis yourself, or you can use the one
that comes bundled with GitLab Omnibus packages.
> **Note:** Redis does not require authentication by default. See
>
Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11.
Starting with 8.14, Redis Sentinel is no longer experimental.
If you've used it with versions `< 8.14` before, please check the updated
documentation here.
High Availability with [Redis] is possible using a **Master** x **Slave**
topology with a [Redis Sentinel][sentinel] service to watch and automatically
start the failover procedure.
You can choose to install and manage Redis and Sentinel yourself, use
a hosted cloud solution or you can use the one that comes bundled with
Omnibus GitLab packages.
> **Notes:**
- Redis requires authentication for High Availability. See
[Redis Security](http://redis.io/topics/security) documentation for more
information. We recommend using a combination of a Redis password and tight
firewall rules to secure your Redis service.
- You are highly encouraged to read the [Redis Sentinel][sentinel] documentation
before configuring Redis HA with GitLab to fully understand the topology and
architecture.
- This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the [Redis HA source installation](redis_source.md) guide.
- Redis Sentinel daemon is bundled with Omnibus GitLab Enterprise Edition only.
For configuring Sentinel with the Omnibus GitLab Community Edition and
installations from source, read the
[Available configuration setups](#available-configuration-setups) section
below.
## Overview
Before diving into the details of setting up Redis and Redis Sentinel for HA,
make sure you read this Overview section to better understand how the components
are tied together.
You need at least `3` independent machines: physical, or VMs running into
distinct physical machines. It is essential that all master and slaves Redis
instances run in different machines. If you fail to provision the machines in
that specific way, any issue with the shared environment can bring your entire
setup down.
It is OK to run a Sentinel along with a master or slave Redis instance.
No more than one Sentinel in the same machine though.
You also need to take in consideration the underlying network topology,
making sure you have redundant connectivity between Redis / Sentinel and
GitLab instances, otherwise the networks will become a single point of
failure.
Make sure that you read this document once as a whole before configuring the
components below.
### High Availability with Sentinel
>**Notes:**
- Starting with GitLab `8.11`, you can configure a list of Redis Sentinel
servers that will monitor a group of Redis servers to provide failover support.
- Starting with GitLab `8.14`, the Omnibus GitLab Enterprise Edition package
comes with Redis Sentinel daemon built-in.
High Availability with Redis requires a few things:
- Multiple Redis instances
- Run Redis in a **Master** x **Slave** topology
- Multiple Sentinel instances
- Application support and visibility to all Sentinel and Redis instances
Redis Sentinel can handle the most important tasks in an HA environment and that's
to help keep servers online with minimal to no downtime. Redis Sentinel:
- Monitors **Master** and **Slaves** instances to see if they are available
- Promotes a **Slave** to **Master** when the **Master** fails
- Demotes a **Master** to **Slave** when the failed **Master** comes back online
(to prevent data-partitioning)
- Can be queried by the application to always connect to the current **Master**
server
When a **Master** fails to respond, it's the application's responsibility
(in our case GitLab) to handle timeout and reconnect (querying a **Sentinel**
for a new **Master**).
## Configure your own Redis server
To get a better understanding on how to correctly setup Sentinel, please read
the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
failing to configure it correctly can lead to data loss or can bring your
whole cluster down, invalidating the failover effort.
If you're hosting GitLab on a cloud provider, you can optionally use a
managed service for Redis. For example, AWS offers a managed ElastiCache service
that runs Redis.
### Recommended setup
## Configure Redis using Omnibus
For a minimal setup, you will install the Omnibus GitLab package in `3`
**independent** machines, both with **Redis** and **Sentinel**:
If you don't want to bother setting up your own Redis server, you can use the
one bundled with Omnibus. In this case, you should disable all services except
Redis.
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
1. Download/install GitLab Omnibus using **steps 1 and 2** from
[GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
steps on the download page.
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
Be sure to change the `external_url` to match your eventual GitLab front-end
URL:
If you are not sure or don't understand why and where the amount of nodes come
from, read [Redis setup overview](#redis-setup-overview) and
[Sentinel setup overview](#sentinel-setup-overview).
```ruby
external_url 'https://gitlab.example.com'
# Disable all services except Redis
redis['enable'] = true
bootstrap['enable'] = false
nginx['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
postgresql['enable'] = false
gitlab_workhorse['enable'] = false
mailroom['enable'] = false
# Redis configuration
redis['port'] = 6379
redis['bind'] = '0.0.0.0'
For a recommended setup that can resist more failures, you will install
the Omnibus GitLab package in `5` **independent** machines, both with
**Redis** and **Sentinel**:
# If you wish to use Redis authentication (recommended)
redis['password'] = 'Redis Password'
```
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
1. Run `sudo touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
from running on upgrade. Only the primary GitLab application server should
handle migrations.
### Redis setup overview
1. Run `sudo gitlab-ctl reconfigure` to install and configure Redis.
You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they
need to be each in a independent machine (see explanation above).
> **Note**: This `reconfigure` step will result in some errors.
That's OK - don't be alarmed.
You can have additional Redis nodes, that will help survive a situation
where more nodes goes down. Whenever there is only `2` nodes online, a failover
will not be initiated.
## Experimental Redis Sentinel support
As an example, if you have `6` Redis nodes, a maximum of `3` can be
simultaneously down.
> [Introduced][ce-1877] in GitLab 8.11.
Please note that there are different requirements for Sentinel nodes.
If you host them in the same Redis machines, you may need to take
that restrictions into consideration when calculating the amount of
nodes to be provisioned. See [Sentinel setup overview](#sentinel-setup-overview)
documentation for more information.
Since GitLab 8.11, you can configure a list of Redis Sentinel servers that
will monitor a group of Redis servers to provide you with a standard failover
support.
All Redis nodes should be configured the same way and with similar server specs, as
in a failover situation, any **Slave** can be promoted as the new **Master** by
the Sentinel servers.
There is currently one exception to the Sentinel support: `mail_room`, the
component that processes incoming emails. It doesn't support Sentinel yet, but
we hope to integrate a future release that does support it.
The replication requires authentication, so you need to define a password to
protect all Redis nodes and the Sentinels. They will all share the same
password, and all instances must be able to talk to
each other over the network.
To get a better understanding on how to correctly setup Sentinel, please read
the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
failing to configure it correctly can lead to data loss.
### Sentinel setup overview
The configuration consists of three parts:
Sentinels watch both other Sentinels and Redis nodes. Whenever a Sentinel
detects that a Redis node is not responding, it will announce that to the
other Sentinels. They have to reach the **quorum**, that is the minimum amount
of Sentinels that agrees a node is down, in order to be able to start a failover.
- Redis setup
- Sentinel setup
- GitLab setup
Whenever the **quorum** is met, the **majority** of all known Sentinel nodes
need to be available and reachable, so that they can elect the Sentinel **leader**
who will take all the decisions to restore the service availability by:
Read carefully how to configure those components below.
- Promoting a new **Master**
- Reconfiguring the other **Slaves** and make them point to the new **Master**
- Announce the new **Master** to every other Sentinel peer
- Reconfigure the old **Master** and demote to **Slave** when it comes back online
### Redis setup
You must have at least `3` Redis Sentinel servers, and they need to
be each in a independent machine (that are believed to fail independently),
ideally in different geographical areas.
You must have at least 2 Redis servers: 1 Master, 1 or more Slaves.
They should be configured the same way and with similar server specs, as
in a failover situation, any Slave can be elected as the new Master by
the Sentinel servers.
You can configure them in the same machines where you've configured the other
Redis servers, but understand that if a whole node goes down, you loose both
a Sentinel and a Redis instance.
In a minimal setup, the only required change for the slaves in `redis.conf`
is the addition of a `slaveof` line pointing to the initial master.
You can increase the security by defining a `requirepass` configuration in
the master, and `masterauth` in slaves.
The number of sentinels should ideally always be an **odd** number, for the
consensus algorithm to be effective in the case of a failure.
---
In a `3` nodes topology, you can only afford `1` Sentinel node going down.
Whenever the **majority** of the Sentinels goes down, the network partition
protection prevents destructive actions and a failover **will not be started**.
**Configuring your own Redis server**
Here are some examples:
1. Add to the slaves' `redis.conf`:
- With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin.
- With `7` sentinels, a maximum of `3` nodes can go down.
```conf
# IP and port of the master Redis server
slaveof 10.10.10.10 6379
```
The **Leader** election can sometimes fail the voting round when **consensus**
is not achieved (see the odd number of nodes requirement above). In that case,
a new attempt will be made after the amount of time defined in
`sentinel['failover_timeout']` (in milliseconds).
1. Optionally, set up password authentication for increased security.
Add the following to master's `redis.conf`:
>**Note:**
We will see where `sentinel['failover_timeout']` is defined later.
The `failover_timeout` variable has a lot of different use cases. According to
the official documentation:
- The time needed to re-start a failover after a previous failover was
already tried against the same master by a given Sentinel, is two
times the failover timeout.
- The time needed for a slave replicating to a wrong master according
to a Sentinel current configuration, to be forced to replicate
with the right master, is exactly the failover timeout (counting since
the moment a Sentinel detected the misconfiguration).
- The time needed to cancel a failover that is already in progress but
did not produced any configuration change (SLAVEOF NO ONE yet not
acknowledged by the promoted slave).
- The maximum time a failover in progress waits for all the slaves to be
reconfigured as slaves of the new master. However even after this time
the slaves will be reconfigured by the Sentinels anyway, but not with
the exact parallel-syncs progression as specified.
### Available configuration setups
Based on your infrastructure setup and how you have installed GitLab, there are
multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or
Redis Sentinel bundled with them so you only need to focus on configuration.
Pick the one that suits your needs.
- [Installations from source][source]: You need to install Redis and Sentinel
yourself. Use the [Redis HA installation from source](redis_source.md)
documentation.
- [Omnibus GitLab **Community Edition** (CE) package][ce]: Redis is bundled, so you
can use the package with only the Redis service enabled as described in steps
1 and 2 of this document (works for both master and slave setups). To install
and configure Sentinel, jump directly to the Sentinel section in the
[Redis HA installation from source](redis_source.md#step-3-configuring-the-redis-sentinel-instances) documentation.
- [Omnibus GitLab **Enterprise Edition** (EE) package][ee]: Both Redis and Sentinel
are bundled in the package, so you can use the EE package to setup the whole
Redis HA infrastructure (master, slave and Sentinel) which is described in
this document.
- If you have installed GitLab using the Omnibus GitLab packages (CE or EE),
but you want to use your own external Redis server, follow steps 1-3 in the
[Redis HA installation from source](redis_source.md) documentation, then go
straight to step 4 in this guide to
[set up the GitLab application](#step-4-configuring-the-gitlab-application).
## Configuring Redis HA
This is the section where we install and setup the new Redis instances.
>**Notes:**
- We assume that you install GitLab and all HA components from scratch. If you
already have it installed and running, read how to
[switch from a single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha).
- Redis nodes (both master and slaves) will need the same password defined in
`redis['password']`. At any time during a failover the Sentinels can
reconfigure a node and change its status from master to slave and vice versa.
### Prerequisites
The prerequisites for a HA Redis setup are the following:
1. Provision the minimum required number of instances as specified in the
[recommended setup](#recommended-setup) section.
1. **Do NOT** install Redis or Redis Sentinel in the same machines your
GitLab application is running on. You can however opt in to install Redis
and Sentinel in the same machine (each in independent ones is recommended
though).
1. All Redis nodes must be able to talk to each other and accept incoming
connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you
change the default ones).
1. The server that hosts the GitLab application must be able to access the
Redis nodes.
1. Protect the nodes from access from external networks ([Internet][it]), using
firewall.
### Step 1. Configuring the master Redis instance
1. SSH into the **master** Redis server.
1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab
package you want using **steps 1 and 2** from the GitLab downloads page.
- Make sure you select the correct Omnibus package, with the same version
and type (Community, Enterprise editions) of your current install.
- Do not complete any other steps on the download page.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```conf
# Optional password authentication for increased security
requirepass "<password>"
```ruby
# Enable the master role and disable all other services in the machine
# (you can still enable Sentinel).
redis_master_role['enable'] = true
# IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces.
# If you really need to bind to an external accessible IP, make
# sure you add extra firewall rules to prevent unauthorized access.
redis['bind'] = '10.0.0.1'
# Define a port so Redis can listen for TCP requests which will allow other
# machines to connect to it.
redis['port'] = 6379
# Set up password authentication for Redis (use the same password in all nodes).
redis['password'] = 'redis-password-goes-here'
```
1. Then add this line to all the slave servers' `redis.conf`:
1. To prevent database migrations from running on upgrade, run:
```conf
masterauth "<password>"
```
sudo touch /etc/gitlab/skip-auto-migrations
```
1. Restart the Redis services for the changes to take effect.
Only the primary GitLab application server should handle migrations.
---
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
### Step 2. Configuring the slave Redis instances
**Using Redis via Omnibus**
1. SSH into the **slave** Redis server.
1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab
package you want using **steps 1 and 2** from the GitLab downloads page.
- Make sure you select the correct Omnibus package, with the same version
and type (Community, Enterprise editions) of your current install.
- Do not complete any other steps on the download page.
1. Edit `/etc/gitlab/gitlab.rb` of a master Redis machine (usualy a single machine):
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
## Redis TCP support (will disable UNIX socket transport)
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
# Enable the slave role and disable all other services in the machine
# (you can still enable Sentinel). This will also set automatically
# `redis['master'] = false`.
redis_slave_role['enable'] = true
# IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces.
# If you really need to bind to an external accessible IP, make
# sure you add extra firewall rules to prevent unauthorized access.
redis['bind'] = '10.0.0.2'
# Define a port so Redis can listen for TCP requests which will allow other
# machines to connect to it.
redis['port'] = 6379
## Master redis instance
redis['password'] = '<huge password string here>'
```
# The same password for Redeis authentication you set up for the master node.
redis['password'] = 'redis-password-goes-here'
1. Edit `/etc/gitlab/gitlab.rb` of a slave Redis machine (should be one or more machines):
# The IP of the master Redis node.
redis['master_ip'] = '10.0.0.1'
```ruby
## Redis TCP support (will disable UNIX socket transport)
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
redis['port'] = 6379
# Port of master Redis server, uncomment to change to non default. Defaults
# to `6379`.
#redis['master_port'] = 6379
```
1. To prevent database migrations from running on upgrade, run:
## Slave redis instance
redis['master_ip'] = '10.10.10.10' # IP of master Redis server
redis['master_port'] = 6379 # Port of master Redis server
redis['master_password'] = "<huge password string here>"
```
sudo touch /etc/gitlab/skip-auto-migrations
```
Only the primary GitLab application server should handle migrations.
1. Reconfigure the GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure`
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
1. Go through the steps again for all the other slave nodes.
---
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
a failover, as the nodes will be managed by the Sentinels, and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels.
### Step 3. Configuring the Redis Sentinel instances
>**Note:**
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. The
following section assumes you are using Omnibus GitLab Enterprise Edition.
For the Omnibus Community Edition and installations from source, follow the
[Redis HA source install](redis_source.md) guide.
Now that the Redis servers are all set up, let's configure the Sentinel
servers.
### Sentinel setup
If you are not sure if your Redis servers are working and replicating
correctly, please read the [Troubleshooting Replication](#troubleshooting-replication)
and fix it before proceeding with Sentinel setup.
We don't provide yet an automated way to setup and run the Sentinel daemon
from Omnibus installation method. You must follow the instructions below and
run it by yourself.
You must have at least `3` Redis Sentinel servers, and they need to
be each in an independent machine. You can configure them in the same
machines where you've configured the other Redis servers.
The support for Sentinel in Ruby has some [caveats](https://github.com/redis/redis-rb/issues/531).
While you can give any name for the `master-group-name` part of the
configuration, as in this example:
With GitLab Enterprise Edition, you can use the Omnibus package to setup
multiple machines with the Sentinel daemon.
```conf
sentinel monitor <master-group-name> <ip> <port> <quorum>
```
---
,for it to work in Ruby, you have to use the "hostname" of the master Redis
server, otherwise you will get an error message like:
`Redis::CannotConnectError: No sentinels available.`. Read
[Sentinel troubleshooting](#sentinel-troubleshooting) for more information.
1. SSH into the server that will host Redis Sentinel.
1. **You can omit this step if the Sentinels will be hosted in the same node as
the other Redis instances.**
Here is an example configuration file (`sentinel.conf`) for a Sentinel node:
[Download/install](https://about.gitlab.com/downloads-ee) the
Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the
GitLab downloads page.
- Make sure you select the correct Omnibus package, with the same version
the GitLab application is running.
- Do not complete any other steps on the download page.
```conf
port 26379
sentinel monitor master-redis.example.com 10.10.10.10 6379 1
sentinel down-after-milliseconds master-redis.example.com 10000
sentinel config-epoch master-redis.example.com 0
sentinel leader-epoch master-redis.example.com 0
```
1. Edit `/etc/gitlab/gitlab.rb` and add the contents (if you are installing the
Sentinels in the same node as the other Redis instances, some values might
be duplicate below):
---
```ruby
redis_sentinel_role['enable'] = true
The final part is to inform the main GitLab application server of the Redis
master and the new sentinels servers.
# Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis'
### GitLab setup
# The same password for Redis authentication you set up for the master node.
redis['password'] = 'redis-password-goes-here'
You can enable or disable sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the master Redis and for a few Sentinel nodes.
# The IP of the master Redis node.
redis['master_ip'] = '10.0.0.1'
It doesn't require a list of all Sentinel nodes, as in case of a failure,
the application will need to query only one of them.
# Define a port so Redis can listen for TCP requests which will allow other
# machines to connect to it.
redis['port'] = 6379
>**Note:**
The following steps should be performed in the [GitLab application server](gitlab.md).
# Port of master Redis server, uncomment to change to non default. Defaults
# to `6379`.
#redis['master_port'] = 6379
## Configure Sentinel
sentinel['bind'] = '10.0.0.1'
# Port that Sentinel listens on, uncomment to change to non default. Defaults
# to `26379`.
# sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover.
## Value must NOT be greater then the amount of sentinels.
##
## The quorum can be used to tune Sentinel in two ways:
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
## we deploy, we are basically making Sentinel more sensible to master failures,
## triggering a failover as soon as even just a minority of Sentinels is no longer
## able to talk with the master.
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
## making Sentinel able to failover only when there are a very large number (larger
## than majority) of well connected Sentinels which agree about the master being down.s
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
# sentinel['down_after_milliseconds'] = 10000
## Specifies the failover timeout in milliseconds. It is used in many ways:
##
## - The time needed to re-start a failover after a previous failover was
## already tried against the same master by a given Sentinel, is two
## times the failover timeout.
##
## - The time needed for a slave replicating to a wrong master according
## to a Sentinel current configuration, to be forced to replicate
## with the right master, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## - The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (SLAVEOF NO ONE yet not
## acknowledged by the promoted slave).
##
## - The maximum time a failover in progress waits for all the slaves to be
## reconfigured as slaves of the new master. However even after this time
## the slaves will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
# sentinel['failover_timeout'] = 60000
```
1. To prevent database migrations from running on upgrade, run:
```
sudo touch /etc/gitlab/skip-auto-migrations
```
**For source based installations**
Only the primary GitLab application server should handle migrations.
1. Edit `/home/git/gitlab/config/resque.yml` following the example in
`/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels
line, changing to the correct server credentials.
1. Restart GitLab for the changes to take effect.
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes.
**For Omnibus installations**
### Step 4. Configuring the GitLab application
The final part is to inform the main GitLab application server of the Redis
Sentinels servers and authentication credentials.
You can enable or disable Sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the Sentinel nodes.
While it doesn't require a list of all Sentinel nodes, in case of a failure,
it needs to access at least one of the listed.
>**Note:**
The following steps should be performed in the [GitLab application server](gitlab.md)
which ideally should not have Redis or Sentinels on it for a HA setup.
1. SSH into the server where the GitLab application is installed.
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
```ruby
gitlab-rails['redis_host'] = "master-redis.example.com"
gitlab-rails['redis_port'] = 6379
gitlab-rails['redis_password'] = '<huge password string here>'
gitlab-rails['redis_sentinels'] = [
{'host' => '10.10.10.1', 'port' => 26379},
{'host' => '10.10.10.2', 'port' => 26379},
{'host' => '10.10.10.3', 'port' => 26379}
```
## Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis'
## The same password for Redis authentication you set up for the master node.
redis['password'] = 'redis-password-goes-here'
## A list of sentinels with `host` and `port`
gitlab_rails['redis_sentinels'] = [
{'host' => '10.0.0.1', 'port' => 26379},
{'host' => '10.0.0.2', 'port' => 26379},
{'host' => '10.0.0.3', 'port' => 26379}
]
```
1. [Reconfigure] the GitLab for the changes to take effect.
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
### Sentinel troubleshooting
## Switching from an existing single-machine installation to Redis HA
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this issue][gh-531] ([pull request][gh-534] that should make things better).
If you already have a single-machine GitLab install running, you will need to
replicate from this machine first, before de-activating the Redis instance
inside it.
Your single-machine install will be the initial **Master**, and the `3` others
should be configured as **Slave** pointing to this machine.
It's a bit rigid the way you have to config `resque.yml` and `sentinel.conf`,
otherwise `redis-rb` will not work properly.
After replication catches up, you will need to stop services in the
single-machine install, to rotate the **Master** to one of the new nodes.
The hostname ('my-primary-redis') of the primary Redis server (`sentinel.conf`)
**must** match the one configured in GitLab (`resque.yml` for source installations
or `gitlab-rails['redis_*']` in Omnibus) and it must be valid ex:
Make the required changes in configuration and restart the new nodes again.
```conf
# sentinel.conf:
sentinel monitor my-primary-redis 10.10.10.10 6379 1
sentinel down-after-milliseconds my-primary-redis 10000
sentinel config-epoch my-primary-redis 0
sentinel leader-epoch my-primary-redis 0
To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`:
```ruby
redis['enable'] = false
```
If you fail to replicate first, you may loose data (unprocessed background jobs).
## Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels
>**Note:**
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For
different setups, read the
[available configuration setups](#available-configuration-setups) section.
In this example we consider that all servers have an internal network
interface with IPs in the `10.0.0.x` range, and that they can connect
to each other using these IPs.
In a real world usage, you would also setup firewall rules to prevent
unauthorized access from other machines and block traffic from the
outside (Internet).
We will use the same `3` nodes with **Redis** + **Sentinel** topology
discussed in [Redis setup overview](#redis-setup-overview) and
[Sentinel setup overview](#sentinel-setup-overview) documentation.
Here is a list and description of each **machine** and the assigned **IP**:
* `10.0.0.1`: Redis Master + Sentinel 1
* `10.0.0.2`: Redis Slave 1 + Sentinel 2
* `10.0.0.3`: Redis Slave 2 + Sentinel 3
* `10.0.0.4`: GitLab application
Please note that after the initial configuration, if a failover is initiated
by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
will change permanently (including in `redis.conf`) from one node to the other,
until a new failover is initiated again.
The same thing will happen with `sentinel.conf` that will be overridden after the
initial execution, after any new sentinel node starts watching the **Master**,
or a failover promotes a different **Master** node.
### Example configuration for Redis master and Sentinel 1
In `/etc/gitlab/gitlab.rb`:
```ruby
redis_master_role['enable'] = true
redis_sentinel_role['enable'] = true
redis['bind'] = '10.0.0.1'
redis['port'] = 6379
redis['password'] = 'redis-password-goes-here'
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
sentinel['bind'] = '10.0.0.1'
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
# sentinel['down_after_milliseconds'] = 10000
# sentinel['failover_timeout'] = 60000
```
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
### Example configuration for Redis slave 1 and Sentinel 2
In `/etc/gitlab/gitlab.rb`:
```ruby
redis_slave_role['enable'] = true
redis_sentinel_role['enable'] = true
redis['bind'] = '10.0.0.2'
redis['port'] = 6379
redis['password'] = 'redis-password-goes-here'
redis['master_password'] = 'redis-password-goes-here'
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
sentinel['bind'] = '10.0.0.2'
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
# sentinel['down_after_milliseconds'] = 10000
# sentinel['failover_timeout'] = 60000
```
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
### Example configuration for Redis slave 2 and Sentinel 3
In `/etc/gitlab/gitlab.rb`:
```ruby
redis_slave_role['enable'] = true
redis_sentinel_role['enable'] = true
redis['bind'] = '10.0.0.3'
redis['port'] = 6379
redis['password'] = 'redis-password-goes-here'
redis['master_password'] = 'redis-password-goes-here'
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
sentinel['bind'] = '10.0.0.3'
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
# sentinel['down_after_milliseconds'] = 10000
# sentinel['failover_timeout'] = 60000
```
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
### Example configuration for the GitLab application
In `/etc/gitlab/gitlab.rb`:
```ruby
redis['master_name'] = 'gitlab-redis'
redis['password'] = 'redis-password-goes-here'
gitlab_rails['redis_sentinels'] = [
{'host' => '10.0.0.1', 'port' => 26379},
{'host' => '10.0.0.2', 'port' => 26379},
{'host' => '10.0.0.3', 'port' => 26379}
]
```
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
## Advanced configuration
Omnibus GitLab configures some things behind the curtains to make the sysadmins'
lives easier. If you want to know what happens underneath keep reading.
### Control running services
In the previous example, we've used `redis_sentinel_role` and
`redis_master_role` which simplifies the amount of configuration changes.
If you want more control, here is what each one sets for you automatically
when enabled:
```ruby
## Redis Sentinel Role
redis_sentinel_role['enable'] = true
# When Sentinel Role is enabled, the following services are also enabled
sentinel['enable'] = true
# The following services are disabled
redis['enable'] = false
bootstrap['enable'] = false
nginx['enable'] = false
postgresql['enable'] = false
gitlab_rails['enable'] = false
mailroom['enable'] = false
-------
## Redis master/slave Role
redis_master_role['enable'] = true # enable only one of them
redis_slave_role['enable'] = true # enable only one of them
# When Redis Master or Slave role are enabled, the following services are
# enabled/disabled. Note that if Redis and Sentinel roles are combined, both
# services will be enabled.
# The following services are disabled
sentinel['enable'] = false
bootstrap['enable'] = false
nginx['enable'] = false
postgresql['enable'] = false
gitlab_rails['enable'] = false
mailroom['enable'] = false
# For Redis Slave role, also change this setting from default 'true' to 'false':
redis['master'] = false
```
```yaml
# resque.yaml
production:
url: redis://my-primary-redis:6378
sentinels:
-
host: slave1
port: 26380 # point to sentinel, not to redis port
-
host: slave2
port: 26381 # point to sentinel, not to redis port
You can find the relevant attributes defined in [gitlab_rails.rb][omnifile].
## Troubleshooting
There are a lot of moving parts that needs to be taken care carefully
in order for the HA setup to work as expected.
Before proceeding with the troubleshooting below, check your firewall rules:
- Redis machines
- Accept TCP connection in `6379`
- Connect to the other Redis machines via TCP in `6379`
- Sentinel machines
- Accept TCP connection in `26379`
- Connect to other Sentinel machines via TCP in `26379`
- Connect to the Redis machines via TCP in `6379`
### Troubleshooting Redis replication
You can check if everything is correct by connecting to each server using
`redis-cli` application, and sending the `INFO` command.
If authentication was correctly defined, it should fail with:
`NOAUTH Authentication required` error. Try to authenticate with the
previous defined password with `AUTH redis-password-goes-here` and
try the `INFO` command again.
Look for the `# Replication` section where you should see some important
information like the `role` of the server.
When connected to a `master` redis, you will see the number of connected
`slaves`, and a list of each with connection details:
```
# Replication
role:master
connected_slaves:1
slave0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1
master_repl_offset:208037658
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:206989083
repl_backlog_histlen:1048576
```
When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
When it's a `slave`, you will see details of the master connection and if
its `up` or `down`:
```
# Replication
role:slave
master_host:10.133.1.58
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:208096498
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
```
### Troubleshooting Sentinel
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this issue][gh-531].
You must make sure you are defining the same value in `redis['master_name']`
and `redis['master_pasword']` as you defined for your sentinel node.
The way the redis connector `redis-rb` works with sentinel is a bit
non-intuitive. We try to hide the complexity in omnibus, but it still requires
a few extra configs.
---
......@@ -273,7 +788,7 @@ To make sure your configuration is correct:
sudo gitlab-rails console
# For source installations
sudo -u git rails console RAILS_ENV=production
sudo -u git rails console production
```
1. Run in the console:
......@@ -288,8 +803,8 @@ To make sure your configuration is correct:
1. To simulate a failover on master Redis, SSH into the Redis server and run:
```bash
# port must match your master redis port
redis-cli -h localhost -p 6379 DEBUG sleep 60
# port must match your master redis port, and the sleep time must be a few seconds bigger than defined one
redis-cli -h localhost -p 6379 DEBUG sleep 20
```
1. Then back in the Rails console from the first step, run:
......@@ -301,10 +816,26 @@ To make sure your configuration is correct:
You should see a different port after a few seconds delay
(the failover/reconnect time).
---
Read more on high-availability configuration:
## Changelog
Changes to Redis HA over time.
**8.14**
- Redis Sentinel support is production-ready and bundled in the Omnibus GitLab
Enterprise Edition package
- Documentation restructure for better readability
**8.11**
- Experimental Redis Sentinel support was added
## Further reading
Read more on High Availability:
1. [High Availability Overview](README.md)
1. [Configure the database](database.md)
1. [Configure NFS](nfs.md)
1. [Configure the GitLab application servers](gitlab.md)
......@@ -315,3 +846,10 @@ Read more on high-availability configuration:
[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
[gh-531]: https://github.com/redis/redis-rb/issues/531
[gh-534]: https://github.com/redis/redis-rb/issues/534
[redis]: http://redis.io/
[sentinel]: http://redis.io/topics/sentinel
[omnifile]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb
[source]: ../../install/installation.md
[ce]: https://about.gitlab.com/downloads
[ee]: https://about.gitlab.com/downloads-ee
[it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png
# Configuring non-Omnibus Redis for GitLab HA
This is the documentation for configuring a Highly Available Redis setup when
you have installed Redis all by yourself and not using the bundled one that
comes with the Omnibus packages.
We cannot stress enough the importance of reading the
[Overview section](redis.md#overview) of the Omnibus Redis HA as it provides
some invaluable information to the configuration of Redis. Please proceed to
read it before going forward with this guide.
We also highly recommend that you use the Omnibus GitLab packages, as we
optimize them specifically for GitLab, and we will take care of upgrading Redis
to the latest supported version.
If you're not sure whether this guide is for you, please refer to
[Available configuration setups](redis.md#available-configuration-setups) in
the Omnibus Redis HA documentation.
## Configuring your own Redis server
This is the section where we install and setup the new Redis instances.
### Prerequisites
- All Redis servers in this guide must be configured to use a TCP connection
instead of a socket. To configure Redis to use TCP connections you need to
define both `bind` and `port` in the Redis config file. You can bind to all
interfaces (`0.0.0.0`) or specify the IP of the desired interface
(e.g., one from an internal network).
- Since Redis 3.2, you must define a password to receive external connections
(`requirepass`).
- If you are using Redis with Sentinel, you will also need to define the same
password for the slave password definition (`masterauth`) in the same instance.
In addition, read the prerequisites as described in the
[Omnibus Redis HA document](redis.md#prerequisites) since they provide some
valuable information for the general setup.
### Step 1. Configuring the master Redis instance
Assuming that the Redis master instance IP is `10.0.0.1`:
1. [Install Redis](../../install/installation.md#6-redis)
1. Edit `/etc/redis/redis.conf`:
```conf
## Define a `bind` address pointing to a local IP that your other machines
## can reach you. If you really need to bind to an external accessible IP, make
## sure you add extra firewall rules to prevent unauthorized access:
bind 10.0.0.1
## Define a `port` to force redis to listen on TCP so other machines can
## connect to it (default port is `6379`).
port 6379
## Set up password authentication (use the same password in all nodes).
## The password should be defined equal for both `requirepass` and `masterauth`
## when setting up Redis to use with Sentinel.
requirepass redis-password-goes-here
masterauth redis-password-goes-here
```
1. Restart the Redis service for the changes to take effect.
### Step 2. Configuring the slave Redis instances
Assuming that the Redis slave instance IP is `10.0.0.2`:
1. [Install Redis](../../install/installation.md#6-redis)
1. Edit `/etc/redis/redis.conf`:
```conf
## Define a `bind` address pointing to a local IP that your other machines
## can reach you. If you really need to bind to an external accessible IP, make
## sure you add extra firewall rules to prevent unauthorized access:
bind 10.0.0.2
## Define a `port` to force redis to listen on TCP so other machines can
## connect to it (default port is `6379`).
port 6379
## Set up password authentication (use the same password in all nodes).
## The password should be defined equal for both `requirepass` and `masterauth`
## when setting up Redis to use with Sentinel.
requirepass redis-password-goes-here
masterauth redis-password-goes-here
## Define `slaveof` pointing to the Redis master instance with IP and port.
slaveof 10.0.0.1 6379
```
1. Restart the Redis service for the changes to take effect.
1. Go through the steps again for all the other slave nodes.
### Step 3. Configuring the Redis Sentinel instances
Sentinel is a special type of Redis server. It inherits most of the basic
configuration options you can define in `redis.conf`, with specific ones
starting with `sentinel` prefix.
Assuming that the Redis Sentinel is installed on the same instance as Redis
master with IP `10.0.0.1` (some settings might overlap with the master):
1. [Install Redis Sentinel](http://redis.io/topics/sentinel)
1. Edit `/etc/redis/sentinel.conf`:
```conf
## Define a `bind` address pointing to a local IP that your other machines
## can reach you. If you really need to bind to an external accessible IP, make
## sure you add extra firewall rules to prevent unauthorized access:
bind 10.0.0.1
## Define a `port` to force Sentinel to listen on TCP so other machines can
## connect to it (default port is `6379`).
port 26379
## Set up password authentication (use the same password in all nodes).
## The password should be defined equal for both `requirepass` and `masterauth`
## when setting up Redis to use with Sentinel.
requirepass redis-password-goes-here
masterauth redis-password-goes-here
## Define with `sentinel auth-pass` the same shared password you have
## defined for both Redis master and slaves instances.
sentinel auth-pass gitlab-redis redis-password-goes-here
## Define with `sentinel monitor` the IP and port of the Redis
## master node, and the quorum required to start a failover.
sentinel monitor gitlab-redis 10.0.0.1 6379 2
## Define with `sentinel down-after-milliseconds` the time in `ms`
## that an unresponsive server will be considered down.
sentinel down-after-milliseconds gitlab-redis 10000
## Define a value for `sentinel failover_timeout` in `ms`. This has multiple
## meanings:
##
## * The time needed to re-start a failover after a previous failover was
## already tried against the same master by a given Sentinel, is two
## times the failover timeout.
##
## * The time needed for a slave replicating to a wrong master according
## to a Sentinel current configuration, to be forced to replicate
## with the right master, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## * The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (SLAVEOF NO ONE yet not
## acknowledged by the promoted slave).
##
## * The maximum time a failover in progress waits for all the slaves to be
## reconfigured as slaves of the new master. However even after this time
## the slaves will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes.
### Step 4. Configuring the GitLab application
You can enable or disable Sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the Sentinel nodes.
While it doesn't require a list of all Sentinel nodes, in case of a failure,
it needs to access at least one of listed ones.
The following steps should be performed in the [GitLab application server](gitlab.md)
which ideally should not have Redis or Sentinels in the same machine for a HA
setup:
1. Edit `/home/git/gitlab/config/resque.yml` following the example in
[resque.yml.example][resque], and uncomment the Sentinel lines, pointing to
the correct server credentials:
```yaml
# resque.yaml
production:
url: redis://:redi-password-goes-here@gitlab-redis/
sentinels:
-
host: 10.0.0.1
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.2
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.3
port: 26379 # point to sentinel, not to redis port
```
1. [Restart GitLab][restart] for the changes to take effect.
## Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels
In this example we consider that all servers have an internal network
interface with IPs in the `10.0.0.x` range, and that they can connect
to each other using these IPs.
In a real world usage, you would also setup firewall rules to prevent
unauthorized access from other machines, and block traffic from the
outside ([Internet][it]).
For this example, **Sentinel 1** will be configured in the same machine as the
**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the
**Slave 1** and **Slave 2** respectively.
Here is a list and description of each **machine** and the assigned **IP**:
* `10.0.0.1`: Redis Master + Sentinel 1
* `10.0.0.2`: Redis Slave 1 + Sentinel 2
* `10.0.0.3`: Redis Slave 2 + Sentinel 3
* `10.0.0.4`: GitLab application
Please note that after the initial configuration, if a failover is initiated
by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
will change permanently (including in `redis.conf`) from one node to the other,
until a new failover is initiated again.
The same thing will happen with `sentinel.conf` that will be overridden after the
initial execution, after any new sentinel node starts watching the **Master**,
or a failover promotes a different **Master** node.
### Example configuration for Redis master and Sentinel 1
1. In `/etc/redis/redis.conf`:
```conf
bind 10.0.0.1
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
```
1. In `/etc/redis/sentinel.conf`:
```conf
bind 10.0.0.1
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
### Example configuration for Redis slave 1 and Sentinel 2
1. In `/etc/redis/redis.conf`:
```conf
bind 10.0.0.2
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
slaveof 10.0.0.1 6379
```
1. In `/etc/redis/sentinel.conf`:
```conf
bind 10.0.0.2
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
### Example configuration for Redis slave 2 and Sentinel 3
1. In `/etc/redis/redis.conf`:
```conf
bind 10.0.0.3
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
slaveof 10.0.0.1 6379
```
1. In `/etc/redis/sentinel.conf`:
```conf
bind 10.0.0.3
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
### Example configuration of the GitLab application
1. Edit `/home/git/gitlab/config/resque.yml`:
```yaml
production:
url: redis://:redi-password-goes-here@gitlab-redis/
sentinels:
-
host: 10.0.0.1
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.2
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.3
port: 26379 # point to sentinel, not to redis port
```
1. [Restart GitLab][restart] for the changes to take effect.
## Troubleshooting
We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained
in the documentation for Omnibus GitLab installations. Here we will list only
the things that are specific to a source installation.
If you get an error in GitLab like `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this upstream issue][gh-531].
You must make sure that `resque.yml` and `sentinel.conf` are configured correctly,
otherwise `redis-rb` will not work properly.
The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`)
**must** be used as the hostname in GitLab (`resque.yml`):
```conf
# sentinel.conf:
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0
```
```yaml
# resque.yaml
production:
url: redis://:myredispassword@gitlab-redis/
sentinels:
-
host: 10.0.0.1
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.2
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.3
port: 26379 # point to sentinel, not to redis port
```
When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel).
[gh-531]: https://github.com/redis/redis-rb/issues/531
[downloads]: https://about.gitlab.com/downloads
[restart]: ../restart_gitlab.md#installations-from-source
[it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png
......@@ -43,18 +43,14 @@ Keep in mind that:
## Setup instructions
GitLab Geo requires some additional work installing and configuring your
instance, than a normal setup.
There are a couple of things you need to do in order to have one or more GitLab
Geo instances. Follow the steps below in the **exact order** that they appear:
1. Follow the instructions to [install GitLab Enterprise Edition][install-ee]
on the server that will serve as the secondary Geo node, but don't further
configure GitLab as authentication will be handled by the primary node (more
on this in the configuration step).
1. [Setup a database replication](database.md) in `primary <-> secondary (read-only)` topology.
1. [Configure GitLab](configuration.md) and set the primary and secondary nodes.
In order to set up one or more GitLab Geo instances, follow the steps below in
this **exact order**:
1. Follow the first 3 steps to [install GitLab Enterprise Edition][install-ee]
on the server that will serve as the secondary Geo node. Do not login or
set up anything else in the secondary node for the moment.
1. [Setup the database replication](database.md) (`primary <-> secondary (read-only)` topology)
1. [Configure GitLab](configuration.md) to set the primary and secondary nodes.
## After setup
......
# GitLab Geo configuration
> **Important:**
Make sure you have followed the first two steps of the
[Setup instructions](README.md#setup-instructions).
This is the final step you need to follow in order to setup a Geo node.
---
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**
- [Setting up GitLab](#setting-up-gitlab)
- [Prerequisites](#prerequisites)
- [Step 1. Adding the primary GitLab node](#step-1-adding-the-primary-gitlab-node)
- [Step 2. Updating the `known_hosts` file of the secondary nodes](#step-2-updating-the-known_hosts-file-of-the-secondary-nodes)
- [Step 3. Copying the database encryption key](#step-3-copying-the-database-encryption-key)
- [Step 4. Enabling the secondary GitLab node](#step-4-enabling-the-secondary-gitlab-node)
- [Step 5. Replicating the repositories data](#step-5-replicating-the-repositories-data)
- [Step 6. Regenerating the authorized keys in the secondary node](#step-6-regenerating-the-authorized-keys-in-the-secondary-node)
- [Next steps](#next-steps)
- [Adding another secondary Geo node](#adding-another-secondary-geo-node)
- [Additional information for the SSH key pairs](#additional-information-for-the-ssh-key-pairs)
- [Troubleshooting](#troubleshooting)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## Setting up GitLab
>**Notes:**
- Don't setup any custom authentication in the secondary nodes, this will be
handled by the primary node.
- Do not add anything in the secondaries Geo nodes admin area
(**Admin Area ➔ Geo Nodes**). This is handled solely by the primary node.
---
After having installed GitLab Enterprise Edition in the instance that will serve
as a Geo node and set up the database replication, the next steps can be summed
as a Geo node and set up the [database replication](database.md), the next steps can be summed
up to:
1. Configure the primary node
......@@ -13,85 +42,159 @@ up to:
1. Start GitLab in the secondary node's machine
1. Configure every secondary node in the primary's Admin screen
After GitLab's instance is online and defined in **Geo Nodes** admin screen,
new data will start to be automatically replicated, but you still need to copy
old data from the primary machine (more information below).
### Prerequisites
## Primary node GitLab setup
This is the last step of configuring a Geo node. Make sure you have followed the
first two steps of the [Setup instructions](README.md#setup-instructions):
>**Notes:**
- You will need to setup your database into a **Primary <-> Secondary (read-only)** replication
topology, and your Primary node should always point to a database's Primary
instance. If you haven't done that already, read [database replication](./database.md).
- Only in the Geo nodes admin area of the primary node, will you be adding all
nodes' information (secondary and primary). Do not add anything in the Geo
nodes admin area of the secondaries.
To setup the primary node:
1. [Create the SSH key pair][ssh-pair] for the primary node.
1. Visit the primary node's **Admin Area > Geo Nodes** (`/admin/geo_nodes`).
1. Add your primary node by providing its full URL and the public SSH key
1. You have already installed on the secondary server the same version of
GitLab Enterprise Edition that is present on the primary server.
1. You have set up the database replication.
1. Your secondary node is allowed to communicate via HTTP/HTTPS and SSH with
your primary node (make sure your firewall is not blocking that).
Some of the following steps require to configure the primary and secondary
nodes almost at the same time. For your convenience make sure you have SSH
logins opened on all nodes as we will be moving back and forth.
### Step 1. Adding the primary GitLab node
1. SSH into the **primary** node and login as root:
```
sudo -i
```
1. Create a new SSH key pair for the primary node. Choose the default location
and leave the password blank by hitting 'Enter' three times:
```bash
sudo -u git -H ssh-keygen -b 4096 -C 'Primary GitLab Geo node'
```
Read more in [additional info for SSH key pairs](#additional-information-for-the-ssh-key-pairs).
1. Get the contents of `id_rsa.pub` the was just created:
```
# Omnibus GitLab installations
sudo -u git cat /var/opt/gitlab/.ssh/id_rsa.pub
# Installations from source
sudo -u git cat /home/git/.ssh/id_rsa.pub
```
1. Visit the primary node's **Admin Area ➔ Geo Nodes** (`/admin/geo_nodes`) in
your browser.
1. Add the primary node by providing its full URL and the public SSH key
you created previously. Make sure to check the box 'This is a primary node'
when adding it.
![Add new primary Geo node](img/geo_nodes_add_new.png)
![Add new primary Geo node](img/geo_nodes_add_new.png)
---
1. Click the **Add node** button.
>**Note:**
Don't set anything up for the `secondary` node yet, make sure to follow the
[Secondary node GitLab setup](#secondary-node-gitlab-setup) first.
### Step 2. Updating the `known_hosts` file of the secondary nodes
1. SSH into the **secondary** node and login as root:
In the following table you can see what all these settings mean:
```
sudo -i
```
| Setting | Description |
| --------- | ----------- |
| Primary | This marks a Geo Node as primary. There can be only one primary, make sure that you first add the primary node and then all the others. |
| URL | Your instance's full URL, in the same way it is configured in `gitlab.yml` (source based installations) or `/etc/gitlab/gitlab.rb` (omnibus installations). |
|Public Key | The SSH public key of the user that your GitLab instance runs on (unless changed, should be the user `git`). That means that you have to go in each Geo Node separately and create an SSH key pair. See the [SSH key creation][ssh-pair] section. |
1. The secondary nodes need to know the SSH fingerprint of the primary node that
will be used for the Git clone/fetch operations. In order to add it to the
`known_hosts` file, run the following command and type `yes` when asked:
## Secondary node GitLab setup
```
sudo -u git -H ssh git@<primary-node-url>
```
>**Note:**
The Geo nodes admin area (**Admin Area > Geo Nodes**) is not used when setting
up the secondary nodes. This is handled at the primary one.
Replace `<primary-node-url>` with the FQDN of the primary node.
To install a secondary node, you must follow the normal GitLab Enterprise
Edition installation, with some extra requirements:
1. Verify that the fingerprint was added by checking `known_hosts`:
- You should point your database connection to a [replicated instance](./database.md).
- Your secondary node should be allowed to [communicate via HTTP/HTTPS and
SSH with your primary node (make sure your firewall is not blocking that).
- Don't make any extra steps you would do for a normal new installation
- Don't setup any custom authentication (this will be handled by the `primary` node)
```
# Omnibus GitLab installations
cat /var/opt/gitlab/.ssh/known_hosts
You need to make sure you restored the database backup (that is part of setting
up replication) and that the primary node PostgreSQL instance is ready to
replicate data.
# Installations from source
cat /home/git/.ssh/known_hosts
```
### Database Encryption Key
### Step 3. Copying the database encryption key
GitLab stores a unique encryption key in disk that we use to safely store
sensitive data in the database.
sensitive data in the database. Any secondary node must have the
**exact same value** for `db_key_base` as defined in the primary one.
1. SSH into the **primary** node and login as root:
```
sudo -i
```
1. Find the value of `db_key_base` and copy it:
```
# Omnibus GitLab installations
cat /etc/gitlab/gitlab-secrets.json
# Installations from source
cat /home/git/gitlab/config/secrets.yml
```
1. SSH into the **secondary** node and login as root:
```
sudo -i
```
1. Open the secrets file and paste the value of `db_key_base` you copied in the
previous step:
```
# Omnibus GitLab installations
editor /etc/gitlab/gitlab-secrets.json
# Installations from source
editor /home/git/gitlab/config/secrets.yml
```
Any secondary node must have the **exact same value** for `db_key_base` as
defined in the primary one.
1. Save and close the file.
- For Omnibus installations it is stored at `/etc/gitlab/gitlab-secrets.json`.
- For installations from source it is stored at `/home/git/gitlab/config/secrets.yml`.
### Step 4. Enabling the secondary GitLab node
Find that key in the primary node and copy paste its value in the secondaries.
1. SSH into the **secondary** node and login as root:
### Enable the secondary GitLab instance
```
sudo -i
```
1. Create a new SSH key pair for the secondary node. Choose the default location
and leave the password blank by hitting 'Enter' three times:
```bash
sudo -u git -H ssh-keygen -b 4096 -C 'Secondary GitLab Geo node'
```
Your new GitLab secondary node can now be safely started.
Read more in [additional info for SSH key pairs](#additional-information-for-the-ssh-key-pairs).
1. Get the contents of `id_rsa.pub` the was just created:
1. [Create the SSH key pair][ssh-pair] for the secondary node.
1. Visit the primary node's **Admin Area > Geo Nodes** (`/admin/geo_nodes`).
1. Add your secondary node by providing its full URL and the public SSH key
you created previously.
1. Hit the **Add node** button.
```
# Omnibus installations
sudo -u git cat /var/opt/gitlab/.ssh/id_rsa.pub
# Installations from source
sudo -u git cat /home/git/.ssh/id_rsa.pub
```
1. Visit the **primary** node's **Admin Area ➔ Geo Nodes** (`/admin/geo_nodes`)
in your browser.
1. Add the secondary node by providing its full URL and the public SSH key
you created previously. **Do NOT** check the box 'This is a primary node'.
1. Click the **Add node** button.
---
......@@ -101,29 +204,41 @@ accessible.
The two most obvious issues that replication can have here are:
- Database replication not working well
- Instance to instance notification not working. In that case, it can be
something of the following:
- You are using a custom certificate or custom CA (see the
[Troubleshooting](#troubleshooting) section)
- Instance is firewalled (check your firewall rules)
1. Database replication not working well
1. Instance to instance notification not working. In that case, it can be
something of the following:
- You are using a custom certificate or custom CA (see the
[Troubleshooting](#troubleshooting) section)
- Instance is firewalled (check your firewall rules)
### Repositories data replication
### Step 5. Replicating the repositories data
Getting a new secondary Geo node up and running, will also require the
repositories directory to be synced from the primary node. You can use `rsync`
for that. Assuming `1.2.3.4` is the IP of the primary node, SSH into the
secondary and run:
for that.
```bash
# For Omnibus installations
rsync -guavrP root@1.2.3.4:/var/opt/gitlab/git-data/repositories/ /var/opt/gitlab/git-data/repositories/
gitlab-ctl reconfigure # to fix directory permissions
Make sure `rsync` is installed in both primary and secondary servers and root
SSH access with a password is enabled. Otherwise, you can set up an SSH key-based
connection between the servers.
# For installations from source
rsync -guavrP root@1.2.3.4:/home/git/repositories/ /home/git/repositories/
chmod ug+rwX,o-rwx /home/git/repositories
```
1. SSH into the **secondary** node and login as root:
```
sudo -i
```
1. Assuming `1.2.3.4` is the IP of the primary node, run the following command
to start the sync:
```bash
# For Omnibus installations
rsync -guavrP root@1.2.3.4:/var/opt/gitlab/git-data/repositories/ /var/opt/gitlab/git-data/repositories/
gitlab-ctl reconfigure # to fix directory permissions
# For installations from source
rsync -guavrP root@1.2.3.4:/home/git/repositories/ /home/git/repositories/
chmod ug+rwX,o-rwx /home/git/repositories
```
If this step is not followed, the secondary node will eventually clone and
fetch every missing repository as they are updated with new commits on the
......@@ -133,12 +248,12 @@ While active repositories will be eventually replicated, if you don't rsync,
the files, any archived/inactive repositories will not get in the secondary node
as Geo doesn't run any routine task to look for missing repositories.
### Authorized keys regeneration
### Step 6. Regenerating the authorized keys in the secondary node
The final step will be to regenerate the keys for `~/.ssh/authorized_keys` using
the command below (HTTPS clone will still work without this extra step).
The final step is to regenerate the keys for `~/.ssh/authorized_keys`
(HTTPS clone will still work without this extra step).
On the secondary node where the database is [already replicated](./database.md),
On the **secondary** node where the database is [already replicated](./database.md),
run:
```
......@@ -152,47 +267,28 @@ sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production
This will enable `git` operations to authorize against your existing users.
New users and SSH keys updated after this step, will be replicated automatically.
### Ready to use
### Next steps
Your instance should be ready to use. You can visit the Admin area in the
secondary node to check if it's correctly identified as a secondary Geo node and
if Geo is enabled.
Your nodes should now be ready to use. You can login to the secondary node
with the same credentials as used in the primary. Visit the secondary node's
**Admin Area ➔ Geo Nodes** (`/admin/geo_nodes`) in your browser to check if it's
correctly identified as a secondary Geo node and if Geo is enabled.
If your installation isn't working properly, check the
[troubleshooting](#troubleshooting) section.
## Create SSH key pairs for new Geo nodes
## Adding another secondary Geo node
>**Note:**
These are general instructions to create a new SSH key pair for a new Geo node,
either primary or secondary.
To add another Geo node in an already Geo configured infrastructure, just follow
[the steps starting form step 2](#step-2-updating-the-known_hosts-file-of-the-secondary-nodes).
Just omit the first step that sets up the primary node.
---
## Additional information for the SSH key pairs
When adding a new Geo node, you must provide an SSH public key of the user that
your GitLab instance runs on (unless changed, should be the user `git`). This
user will act as a "normal user" who fetches from the primary Geo node.
1. Run the command below on each server that will be a Geo node:
```bash
sudo -u git -H ssh-keygen
```
1. Get the contents of `id_rsa.pub` the was just created:
```
# Omnibus installations
sudo -u git cat /var/opt/gitlab/.ssh/id_rsa.pub
# Installations from source
sudo -u git cat /home/git/.ssh/id_rsa.pub
```
1. Copy them to the admin area of the **primary** node (**Admin Area > Geo Nodes**).
---
If for any reason you generate the key using a different name from the default
`id_rsa`, or you want to generate an extra key only for the repository
synchronization feature, you can do so, but you have to create/modify your
......@@ -213,28 +309,6 @@ Host example.com # The FQDN of the primary Geo node
IdentityFile ~/.ssh/mycustom.key # The location of your private key
```
### Add the primary node to the `known_hosts` file of the secondary nodes
>**Note:**
This operation is only needed for the secondary nodes.
---
The secondary nodes need to know the SSH fingerprint of the primary node that
will be used for the Git clone/fetch operations. In order to add it to the
`known_hosts` file, while in the terminal of a secondary node, run the
following command and type `yes` when asked:
```
sudo -u git -H ssh git@<primary-node-url>
```
Replace `<primary-node-url>` with the FQDN of the primary node. You can verify
that the fingerprint was added by checking:
- `/var/opt/gitlab/.ssh/known_hosts` for Omnibus installations or
- `/home/git/.ssh/known_hosts` for installations from source
## Troubleshooting
Setting up Geo requires careful attention to details and sometimes it's easy to
......@@ -247,13 +321,13 @@ where you have to fix (all commands and path locations are for Omnibus installs)
writing permissions.
- Any secondary nodes should point only to read-only instances.
- Can Geo detect my current node correctly?
- Geo uses your defined node from `Admin > Geo` screen, and tries to match
- Geo uses your defined node from `Admin Geo` screen, and tries to match
with the value defined in `/etc/gitlab/gitlab.rb` configuration file.
The relevant line looks like: `external_url "http://gitlab.example.com"`.
- To check if node on current machine is correctly detected type:
```
sudo gitlab-rails runner "Gitlab::Geo.current_node"
sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect"
```
and expect something like:
......
......@@ -4,53 +4,111 @@ This document describes the minimal steps you have to take in order to
replicate your GitLab database into another server. You may have to change
some values according to your database setup, how big it is, etc.
The GitLab primary node where the write operations happen will connect to
`primary` database server, and the secondary ones which are read-only will
connect to `secondary` database servers (which are read-only too).
>**Note:**
In many databases documentation you will see `primary` being references as `master`
and `secondary` as either `slave` or `standby` server (read-only).
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**
- [PostgreSQL replication](#postgresql-replication)
- [PostgreSQL - Configure the primary server](#postgresql-configure-the-primary-server)
- [PostgreSQL - Configure the secondary server](#postgresql-configure-the-secondary-server)
- [PostgreSQL - Initiate the replication process](#postgresql-initiate-the-replication-process)
- [Prerequisites](#prerequisites)
- [Step 1. Configure the primary server](#step-1-configure-the-primary-server)
- [Step 2. Configure the secondary server](#step-2-configure-the-secondary-server)
- [Step 3. Initiate the replication process](#step-3-initiate-the-replication-process)
- [Next steps](#next-steps)
- [MySQL replication](#mysql-replication)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## PostgreSQL replication
The GitLab primary node where the write operations happen will connect to
`primary` database server, and the secondary ones which are read-only will
connect to `secondary` database servers (which are read-only too).
>**Note:**
In many databases documentation you will see `primary` being references as `master`
and `secondary` as either `slave` or `standby` server (read-only).
### Prerequisites
The following guide assumes that:
- You are using PostgreSQL 9.1 or later which includes the
[`pg_basebackup` tool][pgback]. As of this writing, the latest Omnibus
packages (8.5) have version 9.2.
- You have a primary server already set up, running PostgreSQL 9.2.x, and you
- You have a primary server already set up (the GitLab server you are
replicating from), running PostgreSQL 9.2.x, and you
have a new secondary server set up on the same OS and PostgreSQL version. If
you are using Omnibus, make sure the GitLab version is the same on all nodes.
- The IP of the primary server for our examples will be `1.2.3.4`, whereas the
secondary's IP will be `5.6.7.8`.
[pgback]: http://www.postgresql.org/docs/9.2/static/app-pgbasebackup.html
### Step 1. Configure the primary server
**For Omnibus installations**
1. SSH into your GitLab **primary** server and login as root:
```
sudo -i
```
1. Omnibus GitLab has already a replication user called `gitlab_replicator`.
You must set its password manually. Replace `thepassword` with a strong
password:
```bash
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql \
-d template1 \
-c "ALTER USER gitlab_replicator WITH ENCRYPTED PASSWORD 'thepassword'"
```
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
postgresql['listen_address'] = "1.2.3.4"
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32']
postgresql['md5_auth_cidr_addresses'] = ['5.6.7.8/32']
postgresql['sql_replication_user'] = "gitlab_replicator"
postgresql['wal_level'] = "hot_standby"
postgresql['max_wal_senders'] = 10
postgresql['wal_keep_segments'] = 10
postgresql['hot_standby'] = "on"
```
Where `1.2.3.4` is the public IP address of the primary server, and `5.6.7.8`
the public IP address of the secondary one. If you want to add another
secondary, the relevant setting would look like:
```ruby
postgresql['md5_auth_cidr_addresses'] = ['5.6.7.8/32','11.22.33.44/32']
```
### PostgreSQL - Configure the primary server
Edit the `wal` values as you see fit.
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
1. Now that the PostgreSQL server is set up to accept remote connections, run
`netstat -plnt` to make sure that PostgreSQL is listening to the server's
public IP.
1. Continue to [set up the secondary server](#step-2-configure-the-secondary-server).
---
**For installations from source**
1. Login as root and create a replication user:
1. SSH into your database **primary** server and login as root:
```
sudo -i
```
1. Create a replication user named `gitlab_replicator`:
```bash
sudo -u postgres psql -c "CREATE USER gitlab_replicator REPLICATION ENCRYPTED PASSWORD 'thepassword';"
```
1. Edit `postgresql.conf` to configure the primary server for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.2/main/postgresql.conf`):
(for Debian/Ubuntu that would be `/etc/postgresql/9.x/main/postgresql.conf`):
```bash
listen_address = '1.2.3.4'
......@@ -66,7 +124,7 @@ The following guide assumes that:
1. Set the access control on the primary to allow TCP connections using the
server's public IP and set the connection from the secondary to require a
password. Edit `pg_hba.conf` (for Debian/Ubuntu that would be
`/etc/postgresql/9.2/main/pg_hba.conf`):
`/etc/postgresql/9.x/main/pg_hba.conf`):
```bash
host all all 127.0.0.1/32 trust
......@@ -75,70 +133,88 @@ The following guide assumes that:
```
Where `1.2.3.4` is the public IP address of the primary server, and `5.6.7.8`
the public IP address of the secondary one.
the public IP address of the secondary one. If you want to add another
secondary, add one more row like the replication one and change the IP
address:
1. Restart PostgreSQL for the changes to take effect
```bash
host all all 127.0.0.1/32 trust
host all all 1.2.3.4/32 trust
host replication gitlab_replicator 5.6.7.8/32 md5
host replication gitlab_replicator 11.22.33.44/32 md5
```
---
1. Restart PostgreSQL for the changes to take effect.
1. Now that the PostgreSQL server is set up to accept remote connections, run
`netstat -plnt` to make sure that PostgreSQL is listening to the server's
public IP.
### Step 2. Configure the secondary server
**For Omnibus installations**
1. Omnibus GitLab has already a replicator user called `gitlab_replicator`.
You must set its password manually:
1. SSH into your GitLab **secondary** server and login as root:
```
sudo -i
```
1. Test that the remote connection to the primary server works:
```
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h 1.2.3.4 -U gitlab_replicator -d gitlabhq_production -W
```
When prompted enter the password you set in the first step for the
`gitlab_replicator` user. If all worked correctly, you should see the
database prompt.
1. Exit the PostgreSQL console:
```bash
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql \
-d template1 \
-c "ALTER USER gitlab_replicator WITH ENCRYPTED PASSWORD 'thepassword'"
```
\q
```
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
postgresql['listen_address'] = "1.2.3.4"
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32']
postgresql['md5_auth_cidr_addresses'] = ['5.6.7.8/32']
postgresql['sql_replication_user'] = "gitlab_replicator"
postgresql['wal_level'] = "hot_standby"
postgresql['max_wal_senders'] = 10
postgresql['wal_keep_segments'] = 10
postgresql['hot_standby'] = "on"
```
Where `1.2.3.4` is the public IP address of the primary server, and `5.6.7.8`
the public IP address of the secondary one.
Edit the `wal` values as you see fit.
1. [Reconfigure GitLab][] for the changes to take effect.
1. Continue to [initiate the replication process](#step-3-initiate-the-replication-process).
---
Now that the PostgreSQL server is set up to accept remote connections, run
`netstat -plnt` to make sure that PostgreSQL is listening to the server's
public IP.
**For installations from source**
Test that the remote connection works by going to the secondary server and
running:
1. SSH into your database **secondary** server and login as root:
```
# For Omnibus installations
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h 1.2.3.4 -U gitlab_replicator -d gitlabhq_production -W
```
sudo -i
```
1. Test that the remote connection to the primary server works:
# For source installations
sudo -u postgres psql -h 1.2.3.4 -U gitlab_replicator -d gitlabhq_production -W
```
```
sudo -u postgres psql -h 1.2.3.4 -U gitlab_replicator -d gitlabhq_production -W
```
When prompted enter the password you set in the first step for the
`gitlab_replicator` user. If all worked correctly, you should see the database
prompt.
When prompted enter the password you set in the first step for the
`gitlab_replicator` user. If all worked correctly, you should see the
database prompt.
### PostgreSQL - Configure the secondary server
1. Exit the PostgreSQL console:
**For installations from source**
```
\q
```
1. Edit `postgresql.conf` to configure the secondary for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.2/main/postgresql.conf`):
(for Debian/Ubuntu that would be `/etc/postgresql/9.x/main/postgresql.conf`):
```bash
wal_level = hot_standby
......@@ -148,76 +224,93 @@ prompt.
hot_standby = on
```
1. Restart PostgreSQL for the changes to take effect
---
**For Omnibus installations**
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
postgresql['wal_level'] = "hot_standby"
postgresql['max_wal_senders'] = 10
postgresql['wal_keep_segments'] = 10
postgresql['hot_standby'] = "on"
```
1. [Reconfigure GitLab][] for the changes to take effect.
1. Restart PostgreSQL for the changes to take effect.
1. Continue to [initiate the replication process](#step-3-initiate-the-replication-process).
### PostgreSQL - Initiate the replication process
### Step 3. Initiate the replication process
Below we provide a script that connects to the primary server, replicates the
database and creates the needed files for replication.
The directories used are the defaults that are set up in Omnibus. Configure it
as you see fit replacing the directories and paths.
The directories used are the defaults that are set up in Omnibus. If you have
changed any defaults or are using a source installation, configure it as you
see fit replacing the directories and paths.
>**Warning:**
Make sure to run this on the _**secondary**_ server as it removes all PostgreSQL's
Make sure to run this on the **secondary** server as it removes all PostgreSQL's
data before running `pg_basebackup`.
```bash
#!/bin/bash
1. SSH into your GitLab **secondary** server and login as root:
```
sudo -i
```
PORT="5432"
USER="gitlab_replicator"
echo Enter ip of primary postgresql server
read HOST
echo Enter password for $USER@$HOST
read -s PASSWORD
1. Save the snippet below in a file, let's say `/tmp/replica.sh`:
echo Stopping PostgreSQL
gitlab-ctl stop
```bash
#!/bin/bash
PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this scirpt is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Stopping PostgreSQL and all GitLab services
gitlab-ctl stop
echo Backing up postgresql.conf
sudo -u gitlab-psql mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
echo Cleaning up old cluster directory
sudo -u gitlab-psql rm -rf /var/opt/gitlab/postgresql/data
rm -f /tmp/postgresql.trigger
echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
echo Writing recovery.conf file
sudo -u gitlab-psql bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD'
trigger_file = '/tmp/postgresql.trigger'
_EOF1_
"
echo Restoring postgresql.conf
sudo -u gitlab-psql mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Starting PostgreSQL and all GitLab services
gitlab-ctl start
```
echo Backup postgresql.conf
sudo -u gitlab-psql mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
1. Run it with:
echo Cleaning up old cluster directory
sudo -u gitlab-psql rm -rf /var/opt/gitlab/postgresql/data
rm -f /tmp/postgresql.trigger
```
bash /tmp/replica.sh
```
echo Starting base backup as replicator
echo Enter password for $USER@$HOST
sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
When prompted, enter the password you set up for the `gitlab_replicator`
user in the first step.
echo Writing recovery.conf file
sudo -u gitlab-psql bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD'
trigger_file = '/tmp/postgresql.trigger'
_EOF1_
"
The replication process is now over.
echo Restore postgresql.conf
sudo -u gitlab-psql mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
### Next steps
echo Starting PostgreSQL
gitlab-ctl start
```
Now that the database replication is done, the next step is to configure GitLab.
When prompted, enter the password you set up for the `gitlab_replicator` user.
[➤ GitLab Geo configuration](configuration.md)
## MySQL replication
We don't support MySQL replication for GitLab Geo.
[pgback]: http://www.postgresql.org/docs/9.2/static/app-pgbasebackup.html
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
# Geo nodes admin area
For more information about setting up GitLab Geo, read the
[Geo documentation](../../gitlab-geo/README.md).
When you're done, you can navigate to **Admin area ➔ Geo nodes** (`/admin/geo_nodes`).
In the following table you can see what all these settings mean:
| Setting | Description |
| --------- | ----------- |
| Primary | This marks a Geo Node as primary. There can be only one primary, make sure that you first add the primary node and then all the others. |
| URL | Your instance's full URL, in the same way it is configured in `/etc/gitlab/gitlab.rb` (Omnibus GitLab installations) or `gitlab.yml` (source based installations). |
| Public Key | The SSH public key of the user that your GitLab instance runs on (unless changed, should be the user `git`). |
A primary node will have a star right next to it to distinguish from the
secondaries.
module EE
module Workers
# Geo specific code for cache re-generation
#
# This module is intended to encapsulate EE-specific methods
# and be **prepended** in the `ProjectCacheWorker` class.
module ProjectCacheWorker
def update_caches(project_id)
if ::Gitlab::Geo.secondary?
update_geo_caches(project_id)
else
super
end
end
private
# Geo should only update Redis based cache, as data store in the database
# will be updated on primary and replicated to the secondaries.
def update_geo_caches(project_id)
project = Project.find(project_id)
return unless project.repository.exists?
if project.repository.root_ref
project.repository.build_cache
end
end
end
end
end
require 'spec_helper'
describe Geo::ScheduleRepoUpdateService, services: true do
include RepoHelpers
let(:user) { create :user }
let(:project) { create :project }
let(:blankrev) { Gitlab::Git::BLANK_SHA }
let(:oldrev) { sample_commit.parent_id }
let(:newrev) { sample_commit.id }
let(:ref) { 'refs/heads/master' }
let(:service) { execute_push_service(project, user, oldrev, newrev, ref) }
before do
project.team << [user, :master]
end
subject { described_class.new(service.push_data) }
context 'parsed push_data' do
it 'includes required params' do
expect(subject.push_data).to include('type', 'before', 'after', 'ref')
end
end
context '#execute' do
let(:push_data) { service.push_data }
let(:args) do
[
project.id,
push_data[:project][:git_ssh_url],
{
'type' => push_data[:object_kind],
'before' => push_data[:before],
'after' => push_data[:newref],
'ref' => push_data[:ref]
}
]
end
it 'schedule update service' do
expect(GeoRepositoryUpdateWorker).to receive(:perform_async).with(*args)
subject.execute
end
end
def execute_push_service(project, user, oldrev, newrev, ref)
service = GitPushService.new(project, user, oldrev: oldrev, newrev: newrev, ref: ref)
service.execute
service
end
end
......@@ -22,5 +22,13 @@ describe Users::ActivityService, services: true do
end
end
end
context 'when in Geo secondary node' do
before { allow(Gitlab::Geo).to receive(:secondary?).and_return(true) }
it 'does not update last_activity_at' do
expect { service.execute }.not_to change { user.reload.last_activity_at }
end
end
end
end
require 'spec_helper'
describe GeoRepositoryUpdateWorker do
include RepoHelpers
let(:user) { create :user }
let(:project) { create :project }
let(:blankrev) { Gitlab::Git::BLANK_SHA }
let(:oldrev) { sample_commit.parent_id }
let(:newrev) { sample_commit.id }
let(:ref) { 'refs/heads/master' }
let(:service) { execute_push_service(project, user, oldrev, newrev, ref) }
let(:push_data) { service.push_data }
let(:parsed_push_data) do
{
'type' => push_data[:object_kind],
'before' => push_data[:before],
'after' => push_data[:after],
'ref' => push_data[:ref]
}
end
let(:clone_url) { push_data[:project][:git_ssh_url] }
let(:performed) { subject.perform(project.id, clone_url, parsed_push_data) }
before do
project.team << [user, :master]
expect(Project).to receive(:find).at_least(:once).with(project.id) { project }
end
context 'when no repository' do
before do
allow(project.repository).to receive(:fetch_geo_mirror)
allow(project).to receive(:repository_exists?) { false }
end
it 'creates a new repository' do
expect(project).to receive(:create_repository)
performed
end
it 'executes after_create hook' do
expect(project.repository).to receive(:after_create)
performed
end
end
context 'when empty repository' do
before do
allow(project.repository).to receive(:fetch_geo_mirror)
allow(project).to receive(:empty_repo?) { true }
end
it 'executes after_create hook' do
expect(project.repository).to receive(:after_create).at_least(:once)
performed
end
end
context '#process_hooks' do
before { allow(subject).to receive(:fetch_repository) }
it 'calls if push_data is present' do
expect(subject).to receive(:process_hooks)
performed
end
context 'when no push_data is present' do
let(:parsed_push_data) { nil }
it 'skips process_hooks' do
expect(subject).not_to receive(:process_hooks)
performed
end
end
end
context '#process_push' do
before { allow(subject).to receive(:fetch_repository) }
it 'executes after_push_commit' do
expect(project.repository).to receive(:after_push_commit).at_least(:once).with('master', newrev)
performed
end
context 'when removing branch' do
it 'executes after_remove_branch' do
allow(subject).to receive(:push_remove_branch?) { true }
expect(project.repository).to receive(:after_remove_branch)
performed
end
end
context 'when updating a new branch' do
it 'executes after_create_branch' do
allow(subject).to receive(:push_to_new_branch?) { true }
expect(project.repository).to receive(:after_create_branch)
performed
end
end
end
def execute_push_service(project, user, oldrev, newrev, ref)
service = GitPushService.new(project, user, oldrev: oldrev, newrev: newrev, ref: ref)
service.execute
service
end
end
......@@ -39,6 +39,8 @@ describe ProjectCacheWorker do
expect_any_instance_of(Project).to receive(:update_repository_size)
expect_any_instance_of(Project).to receive(:update_commit_count)
expect_any_instance_of(Repository).to receive(:build_cache).and_call_original
subject.perform(project.id)
end
......@@ -48,6 +50,21 @@ describe ProjectCacheWorker do
subject.perform(project.id)
end
context 'when in Geo secondary node' do
before do
allow(Gitlab::Geo).to receive(:secondary?) { true }
end
it 'updates only non database cache' do
expect_any_instance_of(Repository).to receive(:build_cache).and_call_original
expect_any_instance_of(Project).not_to receive(:update_repository_size)
expect_any_instance_of(Project).not_to receive(:update_commit_count)
subject.perform(project.id)
end
end
end
context 'when an exclusive lease can not be obtained' do
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment