Commit f7b05616 authored by Douwe Maan's avatar Douwe Maan

Merge branch 'ericidema/gitlab-ce-import-with-github-personal-access-tokens' into 'master'

Allow importing from Github using Personal Access Tokens

_Originally opened at !4005 by @ericidema._

------

## What does this MR do?

* Made changes to `Gitlab::GithubImport::Client` so that it can be used with Github Personal Access Tokens without the need for OAuth.
* Added UI to collect Personal Access Token from user.
* Detect if the user has logged in with GitHub and use OAuth to skip the Personal Access Token form.

## Are there points in the code the reviewer needs to double check?

Twin Omnibus MR: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/774

## What are the relevant issue numbers?

Closes #13970.

## Screenshots

### GitHub import is configured

![github_import_configured](/uploads/151e4f0edf3f87bfa03c2d97dda8b3d8/github_import_configured.png)

-----

### GitHub import is not configured

![github_import_not_configured](/uploads/cb129f7e2ffe66cceb28ccd9af480284/github_import_not_configured.png)

-----

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- Tests
  - [x] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !4938
parents 82ccc0af ce663540
...@@ -18,6 +18,7 @@ v 8.10.0 (unreleased) ...@@ -18,6 +18,7 @@ v 8.10.0 (unreleased)
- Exclude email check from the standard health check - Exclude email check from the standard health check
- Fix changing issue state columns in milestone view - Fix changing issue state columns in milestone view
- Add notification settings dropdown for groups - Add notification settings dropdown for groups
- Allow importing from Github using Personal Access Tokens. (Eric K Idema)
- Fix user creation with stronger minimum password requirements !4054 (nathan-pmt) - Fix user creation with stronger minimum password requirements !4054 (nathan-pmt)
- PipelinesFinder uses git cache data - PipelinesFinder uses git cache data
- Check for conflicts with existing Project's wiki path when creating a new project. - Check for conflicts with existing Project's wiki path when creating a new project.
......
class Import::GithubController < Import::BaseController class Import::GithubController < Import::BaseController
before_action :verify_github_import_enabled before_action :verify_github_import_enabled
before_action :github_auth, except: :callback before_action :github_auth, only: [:status, :jobs, :create]
rescue_from Octokit::Unauthorized, with: :github_unauthorized rescue_from Octokit::Unauthorized, with: :github_unauthorized
helper_method :logged_in_with_github?
def new
if logged_in_with_github?
go_to_github_for_permissions
elsif session[:github_access_token]
redirect_to status_import_github_url
end
end
def callback def callback
session[:github_access_token] = client.get_token(params[:code]) session[:github_access_token] = client.get_token(params[:code])
redirect_to status_import_github_url redirect_to status_import_github_url
end end
def personal_access_token
session[:github_access_token] = params[:personal_access_token]
redirect_to status_import_github_url
end
def status def status
@repos = client.repos @repos = client.repos
@already_added_projects = current_user.created_projects.where(import_type: "github") @already_added_projects = current_user.created_projects.where(import_type: "github")
...@@ -57,10 +72,14 @@ class Import::GithubController < Import::BaseController ...@@ -57,10 +72,14 @@ class Import::GithubController < Import::BaseController
end end
def github_unauthorized def github_unauthorized
go_to_github_for_permissions session[:github_access_token] = nil
redirect_to new_import_github_url,
alert: 'Access denied to your GitHub account.'
end end
private def logged_in_with_github?
current_user.identities.exists?(provider: 'github')
end
def access_params def access_params
{ github_access_token: session[:github_access_token] } { github_access_token: session[:github_access_token] }
......
- page_title "GitHub Import"
- header_title "Projects", root_path
%h3.page-title
= icon 'github', text: 'Import Projects from GitHub'
- if github_import_configured?
%p
To import a GitHub project, you first need to authorize GitLab to access
the list of your GitHub repositories:
= link_to 'List Your GitHub Repositories', status_import_github_path, class: 'btn btn-success'
%hr
%p
- if github_import_configured?
Alternatively,
- else
To import a GitHub project,
you can use a
= succeed '.' do
= link_to 'Personal Access Token', 'https://github.com/settings/tokens'
When you create your Personal Access Token,
you will need to select the <code>repo</code> scope, so we can display a
list of your public and private repositories which are available for import.
= form_tag personal_access_token_import_github_path, method: :post, class: 'form-inline' do
.form-group
= text_field_tag :personal_access_token, '', class: 'form-control', placeholder: "Personal Access Token", size: 40
= submit_tag 'List Your GitHub Repositories', class: 'btn btn-success'
- unless github_import_configured?
%hr
%p
Note:
- if current_user.admin?
As an administrator you may like to configure
- else
Consider asking your GitLab administrator to configure
= link_to 'GitHub integration', help_page_path("integration", "github")
which will allow login via GitHub and allow importing projects without
generating a Personal Access Token.
%div#github_import_modal.modal
.modal-dialog
.modal-content
.modal-header
%a.close{href: "#", "data-dismiss" => "modal"} ×
%h3 Import projects from GitHub
.modal-body
To enable importing projects from GitHub,
- if current_user.admin?
as administrator you need to configure
- else
ask your Gitlab administrator to configure
== #{link_to 'OAuth integration', help_page_path("integration", "github")}.
...@@ -23,6 +23,7 @@ ...@@ -23,6 +23,7 @@
.input-group-addon .input-group-addon
= root_url = root_url
= f.select :namespace_id, namespaces_options(params[:namespace_id] || :current_user, display_path: true), {}, {class: 'select2 js-select-namespace', tabindex: 1} = f.select :namespace_id, namespaces_options(params[:namespace_id] || :current_user, display_path: true), {}, {class: 'select2 js-select-namespace', tabindex: 1}
- else - else
.input-group-addon.static-namespace .input-group-addon.static-namespace
#{root_url}#{current_user.username}/ #{root_url}#{current_user.username}/
...@@ -44,15 +45,8 @@ ...@@ -44,15 +45,8 @@
.col-sm-12.import-buttons .col-sm-12.import-buttons
%div %div
- if github_import_enabled? - if github_import_enabled?
- if github_import_configured? = link_to new_import_github_path, class: 'btn import_github' do
= link_to status_import_github_path, class: 'btn import_github' do = icon 'github', text: 'GitHub'
%i.fa.fa-github
GitHub
- else
= link_to '#', class: 'how_to_import_link btn import_github' do
%i.fa.fa-github
GitHub
= render 'github_import_modal'
%div %div
- if bitbucket_import_enabled? - if bitbucket_import_enabled?
- if bitbucket_import_configured? - if bitbucket_import_configured?
......
...@@ -10,7 +10,8 @@ paths_to_be_protected = [ ...@@ -10,7 +10,8 @@ paths_to_be_protected = [
"#{Rails.application.config.relative_url_root}/api/#{API::API.version}/session", "#{Rails.application.config.relative_url_root}/api/#{API::API.version}/session",
"#{Rails.application.config.relative_url_root}/users", "#{Rails.application.config.relative_url_root}/users",
"#{Rails.application.config.relative_url_root}/users/confirmation", "#{Rails.application.config.relative_url_root}/users/confirmation",
"#{Rails.application.config.relative_url_root}/unsubscribes/" "#{Rails.application.config.relative_url_root}/unsubscribes/",
"#{Rails.application.config.relative_url_root}/import/github/personal_access_token"
] ]
......
...@@ -139,6 +139,7 @@ Rails.application.routes.draw do ...@@ -139,6 +139,7 @@ Rails.application.routes.draw do
# #
namespace :import do namespace :import do
resource :github, only: [:create, :new], controller: :github do resource :github, only: [:create, :new], controller: :github do
post :personal_access_token
get :status get :status
get :callback get :callback
get :jobs get :jobs
......
# Import your project from GitHub to GitLab # Import your project from GitHub to GitLab
>**Note:** >**Note:**
In order to enable the GitHub import setting, you should first In order to enable the GitHub import setting, you may also want to
enable the [GitHub integration][gh-import] in your GitLab instance. enable the [GitHub integration][gh-import] in your GitLab instance. This
configuration is optional, you will be able import your GitHub repositories
with a Personal Access Token.
At its current state, GitHub importer can import: At its current state, GitHub importer can import:
...@@ -20,9 +22,15 @@ It is not yet possible to import your cross-repository pull requests (those from ...@@ -20,9 +22,15 @@ It is not yet possible to import your cross-repository pull requests (those from
forks). We are working on improving this in the near future. forks). We are working on improving this in the near future.
The importer page is visible when you [create a new project][new-project]. The importer page is visible when you [create a new project][new-project].
Click on the **GitHub** link and you will be redirected to GitHub for Click on the **GitHub** link and, if you are logged in via the GitHub
permission to access your projects. After accepting, you'll be automatically integration, you will be redirected to GitHub for permission to access your
redirected to the importer. projects. After accepting, you'll be automatically redirected to the importer.
If you are not using the GitHub integration, you can still perform a one-off
authorization with GitHub to access your projects.
Alternatively, you can also enter a GitHub Personal Access Token. Once you enter
your token, you'll be taken to the importer.
![New project page on GitLab](img/import_projects_from_github_new_project_page.png) ![New project page on GitLab](img/import_projects_from_github_new_project_page.png)
......
...@@ -21,7 +21,7 @@ Background: ...@@ -21,7 +21,7 @@ Background:
Scenario: I should see instructions on how to import from GitHub Scenario: I should see instructions on how to import from GitHub
Given I see "New Project" page Given I see "New Project" page
When I click on "Import project from GitHub" When I click on "Import project from GitHub"
Then I see instructions on how to import from GitHub Then I am redirected to the GitHub import page
@javascript @javascript
Scenario: I should see Google Code import page Scenario: I should see Google Code import page
......
...@@ -28,14 +28,8 @@ class Spinach::Features::NewProject < Spinach::FeatureSteps ...@@ -28,14 +28,8 @@ class Spinach::Features::NewProject < Spinach::FeatureSteps
first('.import_github').click first('.import_github').click
end end
step 'I see instructions on how to import from GitHub' do step 'I am redirected to the GitHub import page' do
github_modal = first('.modal-body') expect(current_path).to eq new_import_github_path
expect(github_modal).to be_visible
expect(github_modal).to have_content "To enable importing projects from GitHub"
page.all('.modal-body').each do |element|
expect(element).not_to be_visible unless element == github_modal
end
end end
step 'I click on "Repo by URL"' do step 'I click on "Repo by URL"' do
......
...@@ -4,26 +4,39 @@ module Gitlab ...@@ -4,26 +4,39 @@ module Gitlab
GITHUB_SAFE_REMAINING_REQUESTS = 100 GITHUB_SAFE_REMAINING_REQUESTS = 100
GITHUB_SAFE_SLEEP_TIME = 500 GITHUB_SAFE_SLEEP_TIME = 500
attr_reader :client, :api attr_reader :access_token
def initialize(access_token) def initialize(access_token)
@client = ::OAuth2::Client.new( @access_token = access_token
config.app_id,
config.app_secret,
github_options.merge(ssl: { verify: config['verify_ssl'] })
)
if access_token if access_token
::Octokit.auto_paginate = false ::Octokit.auto_paginate = false
end
end
def api
@api ||= ::Octokit::Client.new(
access_token: access_token,
api_endpoint: github_options[:site],
# If there is no config, we're connecting to github.com and we
# should verify ssl.
connection_options: {
ssl: { verify: config ? config['verify_ssl'] : true }
}
)
end
@api = ::Octokit::Client.new( def client
access_token: access_token, unless config
api_endpoint: github_options[:site], raise Projects::ImportService::Error,
connection_options: { 'OAuth configuration for GitHub missing.'
ssl: { verify: config['verify_ssl'] }
}
)
end end
@client ||= ::OAuth2::Client.new(
config.app_id,
config.app_secret,
github_options.merge(ssl: { verify: config['verify_ssl'] })
)
end end
def authorize_url(redirect_uri) def authorize_url(redirect_uri)
...@@ -56,7 +69,11 @@ module Gitlab ...@@ -56,7 +69,11 @@ module Gitlab
end end
def github_options def github_options
config["args"]["client_options"].deep_symbolize_keys if config
config["args"]["client_options"].deep_symbolize_keys
else
OmniAuth::Strategies::GitHub.default_options[:client_options].symbolize_keys
end
end end
def rate_limit def rate_limit
......
...@@ -16,6 +16,24 @@ describe Import::GithubController do ...@@ -16,6 +16,24 @@ describe Import::GithubController do
allow(controller).to receive(:github_import_enabled?).and_return(true) allow(controller).to receive(:github_import_enabled?).and_return(true)
end end
describe "GET new" do
it "redirects to GitHub for an access token if logged in with GitHub" do
allow(controller).to receive(:logged_in_with_github?).and_return(true)
expect(controller).to receive(:go_to_github_for_permissions)
get :new
end
it "redirects to status if we already have a token" do
assign_session_token
allow(controller).to receive(:logged_in_with_github?).and_return(false)
get :new
expect(controller).to redirect_to(status_import_github_url)
end
end
describe "GET callback" do describe "GET callback" do
it "updates access token" do it "updates access token" do
token = "asdasd12345" token = "asdasd12345"
...@@ -32,6 +50,20 @@ describe Import::GithubController do ...@@ -32,6 +50,20 @@ describe Import::GithubController do
end end
end end
describe "POST personal_access_token" do
it "updates access token" do
token = "asdfasdf9876"
allow_any_instance_of(Gitlab::GithubImport::Client).
to receive(:user).and_return(true)
post :personal_access_token, personal_access_token: token
expect(session[:github_access_token]).to eq(token)
expect(controller).to redirect_to(status_import_github_url)
end
end
describe "GET status" do describe "GET status" do
before do before do
@repo = OpenStruct.new(login: 'vim', full_name: 'asd/vim') @repo = OpenStruct.new(login: 'vim', full_name: 'asd/vim')
...@@ -59,6 +91,17 @@ describe Import::GithubController do ...@@ -59,6 +91,17 @@ describe Import::GithubController do
expect(assigns(:already_added_projects)).to eq([@project]) expect(assigns(:already_added_projects)).to eq([@project])
expect(assigns(:repos)).to eq([]) expect(assigns(:repos)).to eq([])
end end
it "handles an invalid access token" do
allow_any_instance_of(Gitlab::GithubImport::Client).
to receive(:repos).and_raise(Octokit::Unauthorized)
get :status
expect(session[:github_access_token]).to eq(nil)
expect(controller).to redirect_to(new_import_github_url)
expect(flash[:alert]).to eq('Access denied to your GitHub account.')
end
end end
describe "POST create" do describe "POST create" do
......
...@@ -20,6 +20,20 @@ describe Gitlab::GithubImport::Client, lib: true do ...@@ -20,6 +20,20 @@ describe Gitlab::GithubImport::Client, lib: true do
expect { client.api }.not_to raise_error expect { client.api }.not_to raise_error
end end
context 'when config is missing' do
before do
allow(Gitlab.config.omniauth).to receive(:providers).and_return([])
end
it 'is still possible to get an Octokit client' do
expect { client.api }.not_to raise_error
end
it 'is not be possible to get an OAuth2 client' do
expect { client.client }.to raise_error(Projects::ImportService::Error)
end
end
context 'allow SSL verification to be configurable on API' do context 'allow SSL verification to be configurable on API' do
before do before do
github_provider['verify_ssl'] = false github_provider['verify_ssl'] = false
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment