Merge branch 'fix/import-data-issue' into 'master'

Create import data in service and fix timing issues when scheduling job

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/17401

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/17376

See merge request !4106

CHANGELOG

 Please view this file on the master branch, on stable branches it's out of date.
 
+v 8.8.0 (unreleased)
+  - Snippets tab under user profile. !4001 (Long Nguyen)
+  - Fix error when using link to uploads in global snippets
+  - Assign labels and milestone to target project when moving issue. !3934 (Long Nguyen)
+  - Use a case-insensitive comparison in sanitizing URI schemes
+  - Toggle sign-up confirmation emails in application settings
+  - Project#open_branches has been cleaned up and no longer loads entire records into memory.
+  - Escape HTML in commit titles in system note messages
+  - Improve multiple branch push performance by memoizing permission checking
+  - Log to application.log when an admin starts and stops impersonating a user
+  - Updated gitlab_git to 10.1.0
+  - GitAccess#protected_tag? no longer loads all tags just to check if a single one exists
+  - Reduce delay in destroying a project from 1-minute to immediately
+  - Make build status canceled if any of the jobs was canceled and none failed
+  - Upgrade Sidekiq to 4.1.2
+  - Added /health_check endpoint for checking service status
+  - Make 'upcoming' filter for milestones work better across projects
+  - Sanitize repo paths in new project error message
+  - Bump mail_room to 0.7.0 to fix stuck IDLE connections
+  - Remove future dates from contribution calendar graph.
+  - Support e-mail notifications for comments on project snippets
+  - Fix API leak of notes of unauthorized issues, snippets and merge requests
+  - Use ActionDispatch Remote IP for Akismet checking
+  - Fix error when visiting commit builds page before build was updated
+  - Add 'l' shortcut to open Label dropdown on issuables and 'i' to create new issue on a project
+  - Update SVG sanitizer to conform to SVG 1.1
+  - Speed up push emails with multiple recipients by only generating the email once
+  - Updated search UI
+  - Added authentication service for Container Registry
+  - Display informative message when new milestone is created
+  - Sanitize milestones and labels titles
+  - Support multi-line tag messages. !3833 (Calin Seciu)
+  - Force users to reset their password after an admin changes it
+  - Allow "NEWS" and "CHANGES" as alternative names for CHANGELOG. !3768 (Connor Shea)
+  - Added button to toggle whitespaces changes on diff view
+  - Backport GitHub Enterprise import support from EE
+  - Create tags using Rugged for performance reasons. !3745
+  - API: Expose Issue#user_notes_count. !3126 (Anton Popov)
+  - Don't show forks button when user can't view forks
+  - Files over 5MB can only be viewed in their raw form, files over 1MB without highlighting !3718
+  - Add support for supressing text diffs using .gitattributes on the default branch (Matt Oakes)
+  - Add eager load paths to help prevent dependency load issues in Sidekiq workers. !3724
+  - Added multiple colors for labels in dropdowns when dups happen.
+  - Always group commits by server timezone, not commit timestamp
+  - Improve description for the Two-factor Authentication sign-in screen. (Connor Shea)
+  - API support for the 'since' and 'until' operators on commit requests (Paco Guzman)
+  - Fix Gravatar hint in user profile when Gravatar is disabled. !3988 (Artem Sidorenko)
+  - Expire repository exists? and has_visible_content? caches after a push if necessary
+  - Fix unintentional filtering bug in Issue/MR sorted by milestone due (Takuya Noguchi)
+  - Fix adding a todo for private group members (Ahmad Sherif)
+  - Bump ace-rails-ap gem version from 2.0.1 to 4.0.2 which upgrades Ace Editor from 1.1.2 to 1.2.3
+  - Total method execution timings are no longer tracked
+  - Allow Admins to remove the Login with buttons for OAuth services and still be able to import !4034. (Andrei Gliga)
+  - Add API endpoints for un/subscribing from/to a label. !4051 (Ahmad Sherif)
+  - Hide left sidebar on phone screens to give more space for content
+  - Redesign navigation for profile and group pages
+  - Add counter metrics for rails cache
+  - Import pull requests from GitHub where the source or target branches were removed
+  - All Grape API helpers are now instrumented
+
+v 8.7.6
+  - Fix links on wiki pages for relative url setups. !4131 (Artem Sidorenko)
+  - Fix import from gitlab.com to a private instance failure
+
+v 8.7.6
+  - Fix external imports not finding the import data
+
 v 8.7.5
   - Fix relative links in wiki pages. !4050
   - Fix always showing build notification message when switching between merge requests !4086

app/controllers/projects/imports_controller.rb

@@ -20,6 +20,7 @@ class Projects::ImportsController < Projects::ApplicationController
         @project.import_retry
       else
         @project.import_start
+        @project.add_import_job
       end
     end
 

app/models/project.rb

@@ -243,7 +243,6 @@ class Project < ActiveRecord::Base
     state :finished
     state :failed
 
-    after_transition any => :started, do: :schedule_add_import_job
     after_transition any => :finished, do: :clear_import_data
   end
 
@@ -382,10 +381,6 @@ class Project < ActiveRecord::Base
     id && persisted?
   end
 
-  def schedule_add_import_job
-    run_after_commit(:add_import_job)
-  end
-
   def add_import_job
     if forked?
       job_id = RepositoryForkWorker.perform_async(self.id, forked_from_project.path_with_namespace, self.namespace.path)

app/services/projects/create_service.rb

@@ -6,6 +6,7 @@ module Projects
 
     def execute
       forked_from_project_id = params.delete(:forked_from_project_id)
+      import_data = params.delete(:import_data)
 
       @project = Project.new(params)
 
@@ -49,16 +50,14 @@ module Projects
         @project.build_forked_project_link(forked_from_project_id: forked_from_project_id)
       end
 
-      Project.transaction do
-        @project.save
+      save_project_and_import_data(import_data)
 
-        if @project.persisted? && !@project.import?
-          raise 'Failed to create repository' unless @project.create_repository
-        end
-      end
+      @project.import_start if @project.import?
 
       after_create_actions if @project.persisted?
 
+      @project.add_import_job if @project.import?
+
       @project
     rescue => e
       message = "Unable to save project: #{e.message}"
@@ -93,8 +92,16 @@ module Projects
       unless @project.group
         @project.team << [current_user, :master, current_user]
       end
+    end
 
-      @project.import_start if @project.import?
+    def save_project_and_import_data(import_data)
+      Project.transaction do
+        @project.create_or_update_import_data(data: import_data[:data], credentials: import_data[:credentials]) if import_data
+
+        if @project.save && !@project.import?
+          raise 'Failed to create repository' unless @project.create_repository
+        end
+      end
     end
   end
 end

lib/gitlab/bitbucket_import/project_creator.rb

@@ -11,7 +11,7 @@ module Gitlab
       end
 
       def execute
-        project = ::Projects::CreateService.new(
+        ::Projects::CreateService.new(
           current_user,
           name: repo["name"],
           path: repo["slug"],
@@ -21,11 +21,8 @@ module Gitlab
           import_type: "bitbucket",
           import_source: "#{repo["owner"]}/#{repo["slug"]}",
           import_url: "ssh://git@bitbucket.org/#{repo["owner"]}/#{repo["slug"]}.git",
+          import_data: { credentials: { bb_session: session_data } }
         ).execute
-
-        project.create_or_update_import_data(credentials: { bb_session: session_data })
-
-        project
       end
     end
   end

lib/gitlab/fogbugz_import/project_creator.rb

@@ -12,7 +12,7 @@ module Gitlab
       end
 
       def execute
-        project = ::Projects::CreateService.new(
+        ::Projects::CreateService.new(
           current_user,
           name: repo.safe_name,
           path: repo.path,
@@ -21,12 +21,9 @@ module Gitlab
           visibility_level: Gitlab::VisibilityLevel::INTERNAL,
           import_type: 'fogbugz',
           import_source: repo.name,
-          import_url: Project::UNKNOWN_IMPORT_URL
+          import_url: Project::UNKNOWN_IMPORT_URL,
+          import_data: { data: { 'repo' => repo.raw_data, 'user_map' => user_map }, credentials: { fb_session: fb_session } }
         ).execute
-
-        project.create_or_update_import_data(data: { 'repo' => repo.raw_data, 'user_map' => user_map }, credentials: { fb_session: fb_session })
-
-        project
       end
     end
   end

lib/gitlab/google_code_import/project_creator.rb

@@ -11,7 +11,7 @@ module Gitlab
       end
 
       def execute
-        project = ::Projects::CreateService.new(
+        ::Projects::CreateService.new(
           current_user,
           name: repo.name,
           path: repo.name,
@@ -21,12 +21,9 @@ module Gitlab
           visibility_level: Gitlab::VisibilityLevel::PUBLIC,
           import_type: "google_code",
           import_source: repo.name,
-          import_url: repo.import_url
+          import_url: repo.import_url,
+          import_data: { data: { 'repo' => repo.raw_data, 'user_map' => user_map } }
         ).execute
-
-        project.create_or_update_import_data(data: { 'repo' => repo.raw_data, 'user_map' => user_map })
-
-        project
       end
     end
   end