Commit 28a8d0b5 authored by Robert Speicher's avatar Robert Speicher

Merge branch 'add_user_repo_integrity_rake_task' into 'master'

Add user repository integrity check rake task

Corrupt repositories and stuck lock files can cause weird issues in
GitLab. Often we know which user is having these problems and then we
have to go hunt down which repository is causing it. Several times
recently that involved me running queries in the rails console to get
an array of projects and then writing a quick Ruby script to loop
through and run `git fsck`. This last time I also had to check for the
existence of `config.lock` and ref lock files. 

This rake task will eliminate all of those steps and allow an admin to
simply specify a username.

I also added the lock file checks to the existing `gitlab:repo:check`
task which goes through all projects. 

See merge request !2080
parents 96a15588 f8bf6c4b
# Rake tasks # Rake tasks
- [Backup restore](backup_restore.md) - [Backup restore](backup_restore.md)
- [Check](check.md)
- [Cleanup](cleanup.md) - [Cleanup](cleanup.md)
- [Features](features.md) - [Features](features.md)
- [Maintenance](maintenance.md) and self-checks - [Maintenance](maintenance.md) and self-checks
......
# Check Rake Tasks
## Repository Integrity
Even though Git is very resilient and tries to prevent data integrity issues,
there are times when things go wrong. The following Rake tasks intend to
help GitLab administrators diagnose problem repositories so they can be fixed.
There are 3 things that are checked to determine integrity.
1. Git repository file system check ([git fsck](https://git-scm.com/docs/git-fsck)).
This step verifies the connectivity and validity of objects in the repository.
1. Check for `config.lock` in the repository directory.
1. Check for any branch/references lock files in `refs/heads`.
It's important to note that the existence of `config.lock` or reference locks
alone do not necessarily indicate a problem. Lock files are routinely created
and removed as Git and GitLab perform operations on the repository. They serve
to prevent data integrity issues. However, if a Git operation is interrupted these
locks may not be cleaned up properly.
The following symptoms may indicate a problem with repository integrity. If users
experience these symptoms you may use the rake tasks described below to determine
exactly which repositories are causing the trouble.
- Receiving an error when trying to push code - `remote: error: cannot lock ref`
- A 500 error when viewing the GitLab dashboard or when accessing a specific project.
### Check all GitLab repositories
This task loops through all repositories on the GitLab server and runs the
3 integrity checks described previously.
```
# omnibus-gitlab
sudo gitlab-rake gitlab:repo:check
# installation from source
bundle exec rake gitlab:repo:check RAILS_ENV=production
```
### Check repositories for a specific user
This task checks all repositories that a specific user has access to. This is important
because sometimes you know which user is experiencing trouble but you don't know
which project might be the cause.
If the rake task is executed without brackets at the end, you will be prompted
to enter a username.
```bash
# omnibus-gitlab
sudo gitlab-rake gitlab:user:check_repos
sudo gitlab-rake gitlab:user:check_repos[<username>]
# installation from source
bundle exec rake gitlab:user:check_repos RAILS_ENV=production
bundle exec rake gitlab:user:check_repos[<username>] RAILS_ENV=production
```
Example output:
![gitlab:user:check_repos output](check_repos_output.png)
...@@ -822,12 +822,29 @@ namespace :gitlab do ...@@ -822,12 +822,29 @@ namespace :gitlab do
namespace_dirs.each do |namespace_dir| namespace_dirs.each do |namespace_dir|
repo_dirs = Dir.glob(File.join(namespace_dir, '*')) repo_dirs = Dir.glob(File.join(namespace_dir, '*'))
repo_dirs.each do |dir| repo_dirs.each { |repo_dir| check_repo_integrity(repo_dir) }
puts "\nChecking repo at #{dir}"
system(*%W(#{Gitlab.config.git.bin_path} fsck), chdir: dir)
end end
end end
end end
namespace :user do
desc "GitLab | Check the integrity of a specific user's repositories"
task :check_repos, [:username] => :environment do |t, args|
username = args[:username] || prompt("Check repository integrity for which username? ".blue)
user = User.find_by(username: username)
if user
repo_dirs = user.authorized_projects.map do |p|
File.join(
Gitlab.config.gitlab_shell.repos_path,
"#{p.path_with_namespace}.git"
)
end
repo_dirs.each { |repo_dir| check_repo_integrity(repo_dir) }
else
puts "\nUser '#{username}' not found".red
end
end
end end
# Helper methods # Helper methods
...@@ -952,4 +969,35 @@ namespace :gitlab do ...@@ -952,4 +969,35 @@ namespace :gitlab do
false false
end end
end end
def check_repo_integrity(repo_dir)
puts "\nChecking repo at #{repo_dir.yellow}"
git_fsck(repo_dir)
check_config_lock(repo_dir)
check_ref_locks(repo_dir)
end
def git_fsck(repo_dir)
puts "Running `git fsck`".yellow
system(*%W(#{Gitlab.config.git.bin_path} fsck), chdir: repo_dir)
end
def check_config_lock(repo_dir)
config_exists = File.exist?(File.join(repo_dir,'config.lock'))
config_output = config_exists ? 'yes'.red : 'no'.green
puts "'config.lock' file exists?".yellow + " ... #{config_output}"
end
def check_ref_locks(repo_dir)
lock_files = Dir.glob(File.join(repo_dir,'refs/heads/*.lock'))
if lock_files.present?
puts "Ref lock files exist:".red
lock_files.each do |lock_file|
puts " #{lock_file}"
end
else
puts "No ref lock files exist".green
end
end
end end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment