Merge branch 'add_user_repo_integrity_rake_task' into 'master'

Add user repository integrity check rake task

Corrupt repositories and stuck lock files can cause weird issues in
GitLab. Often we know which user is having these problems and then we
have to go hunt down which repository is causing it. Several times
recently that involved me running queries in the rails console to get
an array of projects and then writing a quick Ruby script to loop
through and run `git fsck`. This last time I also had to check for the
existence of `config.lock` and ref lock files. 

This rake task will eliminate all of those steps and allow an admin to
simply specify a username.

I also added the lock file checks to the existing `gitlab:repo:check`
task which goes through all projects. 

See merge request !2080

doc/raketasks/README.md

 # Rake tasks
 
 - [Backup restore](backup_restore.md)
+- [Check](check.md)
 - [Cleanup](cleanup.md)
 - [Features](features.md)
 - [Maintenance](maintenance.md) and self-checks
 - [User management](user_management.md)
 - [Web hooks](web_hooks.md)
 - [Import](import.md) of git repositories in bulk
-- [Rebuild authorized_keys file](http://doc.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators
+- [Rebuild authorized_keys file](http://doc.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators
\ No newline at end of file

doc/raketasks/check.md

+# Check Rake Tasks
+
+## Repository Integrity
+
+Even though Git is very resilient and tries to prevent data integrity issues,
+there are times when things go wrong. The following Rake tasks intend to
+help GitLab administrators diagnose problem repositories so they can be fixed.
+
+There are 3 things that are checked to determine integrity.
+
+1. Git repository file system check ([git fsck](https://git-scm.com/docs/git-fsck)).
+   This step verifies the connectivity and validity of objects in the repository.
+1. Check for `config.lock` in the repository directory.
+1. Check for any branch/references lock files in `refs/heads`.
+
+It's important to note that the existence of `config.lock` or reference locks
+alone do not necessarily indicate a problem. Lock files are routinely created
+and removed as Git and GitLab perform operations on the repository. They serve
+to prevent data integrity issues. However, if a Git operation is interrupted these
+locks may not be cleaned up properly.
+
+The following symptoms may indicate a problem with repository integrity. If users
+experience these symptoms you may use the rake tasks described below to determine
+exactly which repositories are causing the trouble.
+
+- Receiving an error when trying to push code - `remote: error: cannot lock ref`
+- A 500 error when viewing the GitLab dashboard or when accessing a specific project.
+
+### Check all GitLab repositories
+
+This task loops through all repositories on the GitLab server and runs the
+3 integrity checks described previously.
+
+```
+# omnibus-gitlab
+sudo gitlab-rake gitlab:repo:check
+
+# installation from source
+bundle exec rake gitlab:repo:check RAILS_ENV=production
+```
+
+### Check repositories for a specific user
+
+This task checks all repositories that a specific user has access to. This is important
+because sometimes you know which user is experiencing trouble but you don't know
+which project might be the cause.
+
+If the rake task is executed without brackets at the end, you will be prompted
+to enter a username.
+
+```bash
+# omnibus-gitlab
+sudo gitlab-rake gitlab:user:check_repos
+sudo gitlab-rake gitlab:user:check_repos[<username>]
+
+# installation from source
+bundle exec rake gitlab:user:check_repos RAILS_ENV=production
+bundle exec rake gitlab:user:check_repos[<username>] RAILS_ENV=production
+```
+
+Example output:
+
+![gitlab:user:check_repos output](check_repos_output.png)

lib/tasks/gitlab/check.rake

@@ -822,10 +822,27 @@ namespace :gitlab do
 
       namespace_dirs.each do |namespace_dir|
         repo_dirs = Dir.glob(File.join(namespace_dir, '*'))
-        repo_dirs.each do |dir|
-          puts "\nChecking repo at #{dir}"
-          system(*%W(#{Gitlab.config.git.bin_path} fsck), chdir: dir)
-        end
+        repo_dirs.each { |repo_dir| check_repo_integrity(repo_dir) }
+      end
+    end
+  end
+
+  namespace :user do
+    desc "GitLab | Check the integrity of a specific user's repositories"
+    task :check_repos, [:username] => :environment do |t, args|
+      username = args[:username] || prompt("Check repository integrity for which username? ".blue)
+      user = User.find_by(username: username)
+      if user
+        repo_dirs = user.authorized_projects.map do |p|
+                      File.join(
+                        Gitlab.config.gitlab_shell.repos_path,
+                        "#{p.path_with_namespace}.git"
+                      )
+                    end
+
+        repo_dirs.each { |repo_dir| check_repo_integrity(repo_dir) }
+      else
+        puts "\nUser '#{username}' not found".red
       end
     end
   end
@@ -952,4 +969,35 @@ namespace :gitlab do
       false
     end
   end
+
+  def check_repo_integrity(repo_dir)
+    puts "\nChecking repo at #{repo_dir.yellow}"
+
+    git_fsck(repo_dir)
+    check_config_lock(repo_dir)
+    check_ref_locks(repo_dir)
+  end
+
+  def git_fsck(repo_dir)
+    puts "Running `git fsck`".yellow
+    system(*%W(#{Gitlab.config.git.bin_path} fsck), chdir: repo_dir)
+  end
+
+  def check_config_lock(repo_dir)
+    config_exists = File.exist?(File.join(repo_dir,'config.lock'))
+    config_output = config_exists ? 'yes'.red : 'no'.green
+    puts "'config.lock' file exists?".yellow + " ... #{config_output}"
+  end
+
+  def check_ref_locks(repo_dir)
+    lock_files = Dir.glob(File.join(repo_dir,'refs/heads/*.lock'))
+    if lock_files.present?
+      puts "Ref lock files exist:".red
+      lock_files.each do |lock_file|
+        puts "  #{lock_file}"
+      end
+    else
+      puts "No ref lock files exist".green
+    end
+  end
 end