Commit c3636296 authored by Douwe Maan's avatar Douwe Maan

Merge branch 'gh-pull-requests' into 'master'

Check out GH pull requests locally where the source/target branch had been deleted

## What does this MR do?

Check out GitHub pull requests where source/target branches that had been deleted locally rather than temporarily restoring them on GitHub using their References API. This helps us to not get rate limited, and allow us to import cross-repository pull requests (those from forks).

## What are the relevant issue numbers?

Fixes #15528

Fixes #17766

Fixes #19310

Fixes #19439

Fixes #19998

Fixes #20153

Fixes #20552

Fixes https://gitlab.com/gitlab-com/support-forum/issues/801

#20385

## Screenshots (if relevant)

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [X] ~~API support added~~
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5630
parents ee721e12 0a1535a9
......@@ -49,6 +49,7 @@ v 8.11.0 (unreleased)
- Document that webhook secret token is sent in X-Gitlab-Token HTTP header !5664 (lycoperdon)
- Gitlab::Highlight is now instrumented
- All created issues, API or WebUI, can be submitted to Akismet for spam check !5333
- Allow users to import cross-repository pull requests from GitHub
- The overhead of instrumented method calls has been reduced
- Remove `search_id` of labels dropdown filter to fix 'Missleading URI for labels in Merge Requests and Issues view'. !5368 (Scott Le)
- Load project invited groups and members eagerly in `ProjectTeam#fetch_members`
......
......@@ -4,10 +4,6 @@
%i.fa.fa-github
Import projects from GitHub
%p
%i.fa.fa-warning
To import GitHub pull requests, any pull request source branches that had been deleted are temporarily restored on GitHub. To prevent any connected CI services from being overloaded with dozens of irrelevant branches being created and deleted again, GitHub webhooks are temporarily disabled during the import process, but only if you have admin access to the GitHub repository.
%p.light
Select projects you want to import.
%hr
......
......@@ -18,9 +18,6 @@ At its current state, GitHub importer can import:
With GitLab 8.7+, references to pull requests and issues are preserved.
It is not yet possible to import your cross-repository pull requests (those from
forks). We are working on improving this in the near future.
The importer page is visible when you [create a new project][new-project].
Click on the **GitHub** link and, if you are logged in via the GitHub
integration, you will be redirected to GitHub for permission to access your
......
......@@ -7,10 +7,6 @@ module Gitlab
branch_exists? && commit_exists?
end
def name
@name ||= exists? ? ref : "#{ref}-#{short_id}"
end
def valid?
repo.present?
end
......
module Gitlab
module GithubImport
class HookFormatter
EVENTS = %w[* create delete pull_request push].freeze
attr_reader :raw
delegate :id, :name, :active, to: :raw
def initialize(raw)
@raw = raw
end
def config
raw.config.attrs
end
def valid?
(EVENTS & raw.events).any? && active
end
end
end
end
......@@ -12,7 +12,6 @@ module Gitlab
if credentials
@client = Client.new(credentials[:user])
@formatter = Gitlab::ImportFormatter.new
else
raise Projects::ImportService::Error, "Unable to find project import data credentials for project ID: #{@project.id}"
end
......@@ -66,73 +65,45 @@ module Gitlab
end
def import_pull_requests
disable_webhooks
pull_requests = client.pull_requests(repo, state: :all, sort: :created, direction: :asc, per_page: 100)
pull_requests = pull_requests.map { |raw| PullRequestFormatter.new(project, raw) }.select(&:valid?)
source_branches_removed = pull_requests.reject(&:source_branch_exists?).map { |pr| [pr.source_branch_name, pr.source_branch_sha] }
target_branches_removed = pull_requests.reject(&:target_branch_exists?).map { |pr| [pr.target_branch_name, pr.target_branch_sha] }
branches_removed = source_branches_removed | target_branches_removed
restore_branches(branches_removed)
pull_requests.each do |pull_request|
begin
restore_source_branch(pull_request) unless pull_request.source_branch_exists?
restore_target_branch(pull_request) unless pull_request.target_branch_exists?
merge_request = pull_request.create!
apply_labels(merge_request)
import_comments(merge_request)
import_comments_on_diff(merge_request)
end
true
rescue ActiveRecord::RecordInvalid => e
raise Projects::ImportService::Error, e.message
ensure
clean_up_restored_branches(branches_removed)
clean_up_disabled_webhooks
clean_up_restored_branches(pull_request)
end
def disable_webhooks
update_webhooks(hooks, active: false)
end
def clean_up_disabled_webhooks
update_webhooks(hooks, active: true)
end
def update_webhooks(hooks, options)
hooks.each do |hook|
client.edit_hook(repo, hook.id, hook.name, hook.config, options)
end
true
end
def hooks
@hooks ||=
begin
client.hooks(repo).map { |raw| HookFormatter.new(raw) }.select(&:valid?)
# The GitHub Repository Webhooks API returns 404 for users
# without admin access to the repository when listing hooks.
# In this case we just want to return gracefully instead of
# spitting out an error and stop the import process.
rescue Octokit::NotFound
[]
end
def restore_source_branch(pull_request)
project.repository.fetch_ref(repo_url, "pull/#{pull_request.number}/head", pull_request.source_branch_name)
end
def restore_branches(branches)
branches.each do |name, sha|
client.create_ref(repo, "refs/heads/#{name}", sha)
def restore_target_branch(pull_request)
project.repository.create_branch(pull_request.target_branch_name, pull_request.target_branch_sha)
end
project.repository.fetch_ref(repo_url, '+refs/heads/*', 'refs/heads/*')
def remove_branch(name)
project.repository.delete_branch(name)
rescue Rugged::ReferenceError
nil
end
def clean_up_restored_branches(branches)
branches.each do |name, _|
client.delete_ref(repo, "heads/#{name}")
project.repository.delete_branch(name) rescue Rugged::ReferenceError
end
def clean_up_restored_branches(pull_request)
remove_branch(pull_request.source_branch_name) unless pull_request.source_branch_exists?
remove_branch(pull_request.target_branch_name) unless pull_request.target_branch_exists?
project.repository.after_remove_branch
end
......
module Gitlab
module GithubImport
class PullRequestFormatter < BaseFormatter
delegate :exists?, :name, :project, :repo, :sha, to: :source_branch, prefix: true
delegate :exists?, :name, :project, :repo, :sha, to: :target_branch, prefix: true
delegate :exists?, :project, :ref, :repo, :sha, to: :source_branch, prefix: true
delegate :exists?, :project, :ref, :repo, :sha, to: :target_branch, prefix: true
def attributes
{
......@@ -33,17 +33,29 @@ module Gitlab
end
def valid?
source_branch.valid? && target_branch.valid? && !cross_project?
source_branch.valid? && target_branch.valid?
end
def source_branch
@source_branch ||= BranchFormatter.new(project, raw_data.head)
end
def source_branch_name
@source_branch_name ||= begin
source_branch_exists? ? source_branch_ref : "pull/#{number}/#{source_branch_ref}"
end
end
def target_branch
@target_branch ||= BranchFormatter.new(project, raw_data.base)
end
def target_branch_name
@target_branch_name ||= begin
target_branch_exists? ? target_branch_ref : "pull/#{number}/#{target_branch_ref}"
end
end
private
def assigned?
......@@ -68,10 +80,6 @@ module Gitlab
raw_data.body || ""
end
def cross_project?
source_branch_repo.id != target_branch_repo.id
end
def description
formatter.author_line(author) + body
end
......
......@@ -32,20 +32,6 @@ describe Gitlab::GithubImport::BranchFormatter, lib: true do
end
end
describe '#name' do
it 'returns raw ref when branch exists' do
branch = described_class.new(project, double(raw))
expect(branch.name).to eq 'feature'
end
it 'returns formatted ref when branch does not exist' do
branch = described_class.new(project, double(raw.merge(ref: 'removed-branch', sha: '2e5d3239642f9161dcbbc4b70a211a68e5e45e2b')))
expect(branch.name).to eq 'removed-branch-2e5d3239'
end
end
describe '#repo' do
it 'returns raw repo' do
branch = described_class.new(project, double(raw))
......
require 'spec_helper'
describe Gitlab::GithubImport::HookFormatter, lib: true do
describe '#id' do
it 'returns raw id' do
raw = double(id: 100000)
formatter = described_class.new(raw)
expect(formatter.id).to eq 100000
end
end
describe '#name' do
it 'returns raw id' do
raw = double(name: 'web')
formatter = described_class.new(raw)
expect(formatter.name).to eq 'web'
end
end
describe '#config' do
it 'returns raw config.attrs' do
raw = double(config: double(attrs: { url: 'http://something.com/webhook' }))
formatter = described_class.new(raw)
expect(formatter.config).to eq({ url: 'http://something.com/webhook' })
end
end
describe '#valid?' do
it 'returns true when events contains the wildcard event' do
raw = double(events: ['*', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq true
end
it 'returns true when events contains the create event' do
raw = double(events: ['create', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq true
end
it 'returns true when events contains delete event' do
raw = double(events: ['delete', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq true
end
it 'returns true when events contains pull_request event' do
raw = double(events: ['pull_request', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq true
end
it 'returns false when events does not contains branch related events' do
raw = double(events: ['member', 'commit_comment'], active: true)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq false
end
it 'returns false when hook is not active' do
raw = double(events: ['pull_request', 'commit_comment'], active: false)
formatter = described_class.new(raw)
expect(formatter.valid?).to eq false
end
end
end
......@@ -9,6 +9,7 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do
let(:source_branch) { double(ref: 'feature', repo: source_repo, sha: source_sha) }
let(:target_repo) { repository }
let(:target_branch) { double(ref: 'master', repo: target_repo, sha: target_sha) }
let(:removed_branch) { double(ref: 'removed-branch', repo: source_repo, sha: '2e5d3239642f9161dcbbc4b70a211a68e5e45e2b') }
let(:octocat) { double(id: 123456, login: 'octocat') }
let(:created_at) { DateTime.strptime('2011-01-26T19:01:12Z') }
let(:updated_at) { DateTime.strptime('2011-01-27T19:01:12Z') }
......@@ -165,6 +166,42 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do
end
end
describe '#source_branch_name' do
context 'when source branch exists' do
let(:raw_data) { double(base_data) }
it 'returns branch ref' do
expect(pull_request.source_branch_name).to eq 'feature'
end
end
context 'when source branch does not exist' do
let(:raw_data) { double(base_data.merge(head: removed_branch)) }
it 'prefixes branch name with pull request number' do
expect(pull_request.source_branch_name).to eq 'pull/1347/removed-branch'
end
end
end
describe '#target_branch_name' do
context 'when source branch exists' do
let(:raw_data) { double(base_data) }
it 'returns branch ref' do
expect(pull_request.target_branch_name).to eq 'master'
end
end
context 'when target branch does not exist' do
let(:raw_data) { double(base_data.merge(base: removed_branch)) }
it 'prefixes branch name with pull request number' do
expect(pull_request.target_branch_name).to eq 'pull/1347/removed-branch'
end
end
end
describe '#valid?' do
context 'when source, and target repos are not a fork' do
let(:raw_data) { double(base_data) }
......@@ -178,8 +215,8 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do
let(:source_repo) { double(id: 2) }
let(:raw_data) { double(base_data) }
it 'returns false' do
expect(pull_request.valid?).to eq false
it 'returns true' do
expect(pull_request.valid?).to eq true
end
end
......@@ -187,8 +224,8 @@ describe Gitlab::GithubImport::PullRequestFormatter, lib: true do
let(:target_repo) { double(id: 2) }
let(:raw_data) { double(base_data) }
it 'returns false' do
expect(pull_request.valid?).to eq false
it 'returns true' do
expect(pull_request.valid?).to eq true
end
end
end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment