Commit c3636296 authored by Douwe Maan's avatar Douwe Maan

Merge branch 'gh-pull-requests' into 'master'

Check out GH pull requests locally where the source/target branch had been deleted

## What does this MR do?

Check out GitHub pull requests where source/target branches that had been deleted locally rather than temporarily restoring them on GitHub using their References API. This helps us to not get rate limited, and allow us to import cross-repository pull requests (those from forks).

## What are the relevant issue numbers?

Fixes #15528

Fixes #17766

Fixes #19310

Fixes #19439

Fixes #19998

Fixes #20153

Fixes #20552

Fixes https://gitlab.com/gitlab-com/support-forum/issues/801

#20385

## Screenshots (if relevant)

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [X] ~~API support added~~
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5630
parents ee721e12 0a1535a9
...@@ -49,6 +49,7 @@ v 8.11.0 (unreleased) ...@@ -49,6 +49,7 @@ v 8.11.0 (unreleased)
- Document that webhook secret token is sent in X-Gitlab-Token HTTP header !5664 (lycoperdon) - Document that webhook secret token is sent in X-Gitlab-Token HTTP header !5664 (lycoperdon)
- Gitlab::Highlight is now instrumented - Gitlab::Highlight is now instrumented
- All created issues, API or WebUI, can be submitted to Akismet for spam check !5333 - All created issues, API or WebUI, can be submitted to Akismet for spam check !5333
- Allow users to import cross-repository pull requests from GitHub
- The overhead of instrumented method calls has been reduced - The overhead of instrumented method calls has been reduced
- Remove `search_id` of labels dropdown filter to fix 'Missleading URI for labels in Merge Requests and Issues view'. !5368 (Scott Le) - Remove `search_id` of labels dropdown filter to fix 'Missleading URI for labels in Merge Requests and Issues view'. !5368 (Scott Le)
- Load project invited groups and members eagerly in `ProjectTeam#fetch_members` - Load project invited groups and members eagerly in `ProjectTeam#fetch_members`
......
...@@ -4,10 +4,6 @@ ...@@ -4,10 +4,6 @@
%i.fa.fa-github %i.fa.fa-github
Import projects from GitHub Import projects from GitHub
%p
%i.fa.fa-warning
To import GitHub pull requests, any pull request source branches that had been deleted are temporarily restored on GitHub. To prevent any connected CI services from being overloaded with dozens of irrelevant branches being created and deleted again, GitHub webhooks are temporarily disabled during the import process, but only if you have admin access to the GitHub repository.
%p.light %p.light
Select projects you want to import. Select projects you want to import.
%hr %hr
......
...@@ -18,9 +18,6 @@ At its current state, GitHub importer can import: ...@@ -18,9 +18,6 @@ At its current state, GitHub importer can import:
With GitLab 8.7+, references to pull requests and issues are preserved. With GitLab 8.7+, references to pull requests and issues are preserved.
It is not yet possible to import your cross-repository pull requests (those from
forks). We are working on improving this in the near future.
The importer page is visible when you [create a new project][new-project]. The importer page is visible when you [create a new project][new-project].
Click on the **GitHub** link and, if you are logged in via the GitHub Click on the **GitHub** link and, if you are logged in via the GitHub
integration, you will be redirected to GitHub for permission to access your integration, you will be redirected to GitHub for permission to access your
......
...@@ -7,10 +7,6 @@ module Gitlab ...@@ -7,10 +7,6 @@ module Gitlab
branch_exists? && commit_exists? branch_exists? && commit_exists?
end end
def name
@name ||= exists? ? ref : "#{ref}-#{short_id}"
end
def valid? def valid?
repo.present? repo.present?
end end
......
module Gitlab
module GithubImport
class HookFormatter
EVENTS = %w[* create delete pull_request push].freeze
attr_reader :raw
delegate :id, :name, :active, to: :raw
def initialize(raw)
@raw = raw
end
def config
raw.config.attrs
end
def valid?
(EVENTS & raw.events).any? && active
end
end
end
end
...@@ -12,7 +12,6 @@ module Gitlab ...@@ -12,7 +12,6 @@ module Gitlab
if credentials if credentials
@client = Client.new(credentials[:user]) @client = Client.new(credentials[:user])
@formatter = Gitlab::ImportFormatter.new
else else
raise Projects::ImportService::Error, "Unable to find project import data credentials for project ID: #{@project.id}" raise Projects::ImportService::Error, "Unable to find project import data credentials for project ID: #{@project.id}"
end end
...@@ -66,73 +65,45 @@ module Gitlab ...@@ -66,73 +65,45 @@ module Gitlab
end end
def import_pull_requests def import_pull_requests
disable_webhooks
pull_requests = client.pull_requests(repo, state: :all, sort: :created, direction: :asc, per_page: 100) pull_requests = client.pull_requests(repo, state: :all, sort: :created, direction: :asc, per_page: 100)
pull_requests = pull_requests.map { |raw| PullRequestFormatter.new(project, raw) }.select(&:valid?) pull_requests = pull_requests.map { |raw| PullRequestFormatter.new(project, raw) }.select(&:valid?)
source_branches_removed = pull_requests.reject(&:source_branch_exists?).map { |pr| [pr.source_branch_name, pr.source_branch_sha] }
target_branches_removed = pull_requests.reject(&:target_branch_exists?).map { |pr| [pr.target_branch_name, pr.target_branch_sha] }
branches_removed = source_branches_removed | target_branches_removed
restore_branches(branches_removed)
pull_requests.each do |pull_request| pull_requests.each do |pull_request|
begin
restore_source_branch(pull_request) unless pull_request.source_branch_exists?
restore_target_branch(pull_request) unless pull_request.target_branch_exists?
merge_request = pull_request.create! merge_request = pull_request.create!
apply_labels(merge_request) apply_labels(merge_request)
import_comments(merge_request) import_comments(merge_request)
import_comments_on_diff(merge_request) import_comments_on_diff(merge_request)
end
true
rescue ActiveRecord::RecordInvalid => e rescue ActiveRecord::RecordInvalid => e
raise Projects::ImportService::Error, e.message raise Projects::ImportService::Error, e.message
ensure ensure
clean_up_restored_branches(branches_removed) clean_up_restored_branches(pull_request)
clean_up_disabled_webhooks
end end
def disable_webhooks
update_webhooks(hooks, active: false)
end end
def clean_up_disabled_webhooks true
update_webhooks(hooks, active: true)
end
def update_webhooks(hooks, options)
hooks.each do |hook|
client.edit_hook(repo, hook.id, hook.name, hook.config, options)
end
end end
def hooks def restore_source_branch(pull_request)
@hooks ||= project.repository.fetch_ref(repo_url, "pull/#{pull_request.number}/head", pull_request.source_branch_name)
begin
client.hooks(repo).map { |raw| HookFormatter.new(raw) }.select(&:valid?)
# The GitHub Repository Webhooks API returns 404 for users
# without admin access to the repository when listing hooks.
# In this case we just want to return gracefully instead of
# spitting out an error and stop the import process.
rescue Octokit::NotFound
[]
end
end end
def restore_branches(branches) def restore_target_branch(pull_request)
branches.each do |name, sha| project.repository.create_branch(pull_request.target_branch_name, pull_request.target_branch_sha)
client.create_ref(repo, "refs/heads/#{name}", sha)
end end
project.repository.fetch_ref(repo_url, '+refs/heads/*', 'refs/heads/*') def remove_branch(name)
project.repository.delete_branch(name)
rescue Rugged::ReferenceError
nil
end end
def clean_up_restored_branches(branches) def clean_up_restored_branches(pull_request)
branches.each do |name, _| remove_branch(pull_request.source_branch_name) unless pull_request.source_branch_exists?
client.delete_ref(repo, "heads/#{name}") remove_branch(pull_request.target_branch_name) unless pull_request.target_branch_exists?
project.repository.delete_branch(name) rescue Rugged::ReferenceError
end
project.repository.after_remove_branch project.repository.after_remove_branch
end end
......
module Gitlab module Gitlab
module GithubImport module GithubImport
class PullRequestFormatter < BaseFormatter class PullRequestFormatter < BaseFormatter
delegate :exists?, :name, :project, :repo, :sha, to: :source_branch, prefix: true delegate :exists?, :project, :ref, :repo, :sha, to: :source_branch, prefix: true
delegate :exists?, :name, :project, :repo, :sha, to: :target_branch, prefix: true delegate :exists?, :project, :ref, :repo, :sha, to: :target_branch, prefix: true
def attributes def attributes
{ {
...@@ -33,17 +33,29 @@ module Gitlab ...@@ -33,17 +33,29 @@ module Gitlab
end end
def valid? def valid?
source_branch.valid? && target_branch.valid? && !cross_project? source_branch.valid? && target_branch.valid?
end end
def source_branch def source_branch
@source_branch ||= BranchFormatter.new(project, raw_data.head) @source_branch ||= BranchFormatter.new(project, raw_data.head)
end end
def source_branch_name
@source_branch_name ||= begin
source_branch_exists? ? source_branch_ref : "pull/#{number}/#{source_branch_ref}"
end
end
def target_branch def target_branch
@target_branch ||= BranchFormatter.new(project, raw_data.base) @target_branch ||= BranchFormatter.new(project, raw_data.base)
end end
def target_branch_name
@target_branch_name ||= begin
target_branch_exists? ? target_branch_ref : "pull/#{number}/#{target_branch_ref}"
end
end
private private
def assigned? def assigned?
...@@ -68,10 +80,6 @@ module Gitlab ...@@ -68,10 +80,6 @@ module Gitlab
raw_data.body || "" raw_data.body || ""
end end
def cross_project?
source_branch_repo.id != target_branch_repo.id
end
def description def description
formatter.author_line(author) + body formatter.author_line(author) + body
end end
......
...@@ -32,20 +32,6 @@ describe Gitlab::GithubImport::BranchFormatter, lib: true do ...@@ -32,20 +32,6 @@ describe Gitlab::GithubImport::BranchFormatter, lib: true do
end end
end end
describe '#name' do
it 'returns raw ref when branch exists' do
branch = described_class.new(project, double(raw))
expect(branch.name).to eq 'feature'
end
it 'returns formatted ref when branch does not exist' do
branch = described_class.new(project, double(raw.merge(ref: 'removed-branch', sha: '2e5d3239642f9161dcbbc4b70a211a68e5e45e2b')))
expect(branch.name).to eq 'removed-branch-2e5d3239'
end
end
describe '#repo' do describe '#repo' do
it 'returns raw repo' do it 'returns raw repo' do
branch = described_class.new(project, double(raw)) branch = described_class.new(project, double(raw))
......
require 'spec_helper'
describe Gitlab::GithubImport::HookFormatter, lib: true do
describe '#id' do
it 'returns raw id' do
raw = double(id: 100000)
formatter = described_class.new(raw)
expect(formatter.id).to eq 100000
end
end
describe '#name' do
it 'returns raw id' do
raw = double(name: 'web')
formatter = described_class.new(raw)
expect(formatter.name).to eq 'web'
end
end
describe '#config' do
it 'returns raw config.attrs' do
raw = double(config: double(attrs: { url: 'http://something.com/webhook' }))
formatter = described_class.new(raw)
expect(formatter.config).to eq({ url: 'http://something.com/webhook' })
end
end
describe '#valid?' do
it 'returns true when events contains the wildcard event' do
raw = double(events: ['*', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq true
end
it 'returns true when events contains the create event' do
raw = double(events: ['create', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq true
end
it 'returns true when events contains delete event' do
raw = double(events: ['delete', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq true
end
it 'returns true when events contains pull_request event' do
raw = double(events: ['pull_request', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq true
end
it 'returns false when events does not contains branch related events' do
raw = double(events: ['member', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq false
end
it 'returns false when hook is not active' do
raw = double(events: ['pull_request', 'commit_comment'], active: false)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq false
end
end
end
...@@ -9,6 +9,7 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do ...@@ -9,6 +9,7 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do
let(:source_branch) { double(ref: 'feature', repo: source_repo, sha: source_sha) } let(:source_branch) { double(ref: 'feature', repo: source_repo, sha: source_sha) }
let(:target_repo) { repository } let(:target_repo) { repository }
let(:target_branch) { double(ref: 'master', repo: target_repo, sha: target_sha) } let(:target_branch) { double(ref: 'master', repo: target_repo, sha: target_sha) }
let(:removed_branch) { double(ref: 'removed-branch', repo: source_repo, sha: '2e5d3239642f9161dcbbc4b70a211a68e5e45e2b') }
let(:octocat) { double(id: 123456, login: 'octocat') } let(:octocat) { double(id: 123456, login: 'octocat') }
let(:created_at) { DateTime.strptime('2011-01-26T19:01:12Z') } let(:created_at) { DateTime.strptime('2011-01-26T19:01:12Z') }
let(:updated_at) { DateTime.strptime('2011-01-27T19:01:12Z') } let(:updated_at) { DateTime.strptime('2011-01-27T19:01:12Z') }
...@@ -165,6 +166,42 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do ...@@ -165,6 +166,42 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do
end end
end end
describe '#source_branch_name' do
context 'when source branch exists' do
let(:raw_data) { double(base_data) }
it 'returns branch ref' do
expect(pull_request.source_branch_name).to eq 'feature'
end
end
context 'when source branch does not exist' do
let(:raw_data) { double(base_data.merge(head: removed_branch)) }
it 'prefixes branch name with pull request number' do
expect(pull_request.source_branch_name).to eq 'pull/1347/removed-branch'
end
end
end
describe '#target_branch_name' do
context 'when source branch exists' do
let(:raw_data) { double(base_data) }
it 'returns branch ref' do
expect(pull_request.target_branch_name).to eq 'master'
end
end
context 'when target branch does not exist' do
let(:raw_data) { double(base_data.merge(base: removed_branch)) }
it 'prefixes branch name with pull request number' do
expect(pull_request.target_branch_name).to eq 'pull/1347/removed-branch'
end
end
end
describe '#valid?' do describe '#valid?' do
context 'when source, and target repos are not a fork' do context 'when source, and target repos are not a fork' do
let(:raw_data) { double(base_data) } let(:raw_data) { double(base_data) }
...@@ -178,8 +215,8 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do ...@@ -178,8 +215,8 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do
let(:source_repo) { double(id: 2) } let(:source_repo) { double(id: 2) }
let(:raw_data) { double(base_data) } let(:raw_data) { double(base_data) }
it 'returns false' do it 'returns true' do
expect(pull_request.valid?).to eq false expect(pull_request.valid?).to eq true
end end
end end
...@@ -187,8 +224,8 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do ...@@ -187,8 +224,8 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do
let(:target_repo) { double(id: 2) } let(:target_repo) { double(id: 2) }
let(:raw_data) { double(base_data) } let(:raw_data) { double(base_data) }
it 'returns false' do it 'returns true' do
expect(pull_request.valid?).to eq false expect(pull_request.valid?).to eq true
end end
end end
end end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment