Commit 6a928360 authored by Douwe Maan's avatar Douwe Maan

Merge branch 'fix/speed-up-commit-repo-changes' into 'master'

Speed up methods that depend on diff stats only

## What does this MR do?
Speeds up and reduces memory usage of `Commit#repo_changes`loop, `Repository#expire_avatar_cache` and `IrkerWorker`.  It's a WIP because it need a new version of `gitlab_git` released with [these changes](https://gitlab.com/gitlab-org/gitlab_git/merge_requests/109).

## Are there points in the code the reviewer needs to double check?
N/A

## Why was this MR needed?
For large files committed, this method ends up loading them in memory which can consume lots of memory.

## What are the relevant issue numbers?
https://gitlab.com/gitlab-org/gitlab-ce/issues/19441#note_13425892

## Screenshots (if relevant)
N/A

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] ~~[Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~
- [ ] ~~API support added~~
- ~~Tests~~
  - [ ] ~~Added for this feature/bug~~
  - [ ] ~~All builds are passing~~
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5568
parents f7c360ca 08c1dd34
...@@ -57,6 +57,7 @@ v 8.11.0 (unreleased) ...@@ -57,6 +57,7 @@ v 8.11.0 (unreleased)
- Sensible state specific default sort order for issues and merge requests !5453 (tomb0y) - Sensible state specific default sort order for issues and merge requests !5453 (tomb0y)
- Fix RequestProfiler::Middleware error when code is reloaded in development - Fix RequestProfiler::Middleware error when code is reloaded in development
- Catch what warden might throw when profiling requests to re-throw it - Catch what warden might throw when profiling requests to re-throw it
- Speed up and reduce memory usage of Commit#repo_changes, Repository#expire_avatar_cache and IrkerWorker
v 8.10.3 v 8.10.3
- Fix Import/Export issue importing milestones and labels not associated properly. !5426 - Fix Import/Export issue importing milestones and labels not associated properly. !5426
......
...@@ -53,7 +53,7 @@ gem 'browser', '~> 2.2' ...@@ -53,7 +53,7 @@ gem 'browser', '~> 2.2'
# Extracting information from a git repository # Extracting information from a git repository
# Provide access to Gitlab::Git library # Provide access to Gitlab::Git library
gem 'gitlab_git', '~> 10.4.2' gem 'gitlab_git', '~> 10.4.3'
# LDAP Auth # LDAP Auth
# GitLab fork with several improvements to original library. For full list of changes # GitLab fork with several improvements to original library. For full list of changes
......
...@@ -278,7 +278,7 @@ GEM ...@@ -278,7 +278,7 @@ GEM
diff-lcs (~> 1.1) diff-lcs (~> 1.1)
mime-types (>= 1.16, < 3) mime-types (>= 1.16, < 3)
posix-spawn (~> 0.3) posix-spawn (~> 0.3)
gitlab_git (10.4.2) gitlab_git (10.4.3)
activesupport (~> 4.0) activesupport (~> 4.0)
charlock_holmes (~> 0.7.3) charlock_holmes (~> 0.7.3)
github-linguist (~> 4.7.0) github-linguist (~> 4.7.0)
...@@ -870,7 +870,7 @@ DEPENDENCIES ...@@ -870,7 +870,7 @@ DEPENDENCIES
github-linguist (~> 4.7.0) github-linguist (~> 4.7.0)
github-markup (~> 1.4) github-markup (~> 1.4)
gitlab-flowdock-git-hook (~> 1.0.1) gitlab-flowdock-git-hook (~> 1.0.1)
gitlab_git (~> 10.4.2) gitlab_git (~> 10.4.3)
gitlab_meta (= 7.0) gitlab_meta (= 7.0)
gitlab_omniauth-ldap (~> 1.2.1) gitlab_omniauth-ldap (~> 1.2.1)
gollum-lib (~> 4.2) gollum-lib (~> 4.2)
......
...@@ -334,7 +334,7 @@ class Commit ...@@ -334,7 +334,7 @@ class Commit
def repo_changes def repo_changes
changes = { added: [], modified: [], removed: [] } changes = { added: [], modified: [], removed: [] }
raw_diffs.each do |diff| raw_diffs(deltas_only: true).each do |diff|
if diff.deleted_file if diff.deleted_file
changes[:removed] << diff.old_path changes[:removed] << diff.old_path
elsif diff.renamed_file || diff.new_file elsif diff.renamed_file || diff.new_file
......
...@@ -372,7 +372,7 @@ class Repository ...@@ -372,7 +372,7 @@ class Repository
# We don't want to flush the cache if the commit didn't actually make any # We don't want to flush the cache if the commit didn't actually make any
# changes to any of the possible avatar files. # changes to any of the possible avatar files.
if revision && commit = self.commit(revision) if revision && commit = self.commit(revision)
return unless commit.raw_diffs. return unless commit.raw_diffs(deltas_only: true).
any? { |diff| AVATAR_FILES.include?(diff.new_path) } any? { |diff| AVATAR_FILES.include?(diff.new_path) }
end end
......
...@@ -141,7 +141,7 @@ class IrkerWorker ...@@ -141,7 +141,7 @@ class IrkerWorker
end end
def files_count(commit) def files_count(commit)
diffs = commit.raw_diffs diffs = commit.raw_diffs(deltas_only: true)
files = "#{diffs.real_size} file" files = "#{diffs.real_size} file"
files += 's' if diffs.size > 1 files += 's' if diffs.size > 1
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment