Commit 8686e012 authored by Stan Hu's avatar Stan Hu

Avoid excessive recursive calls with Rugged TreeEntries

The Rugged implementation was recursively scanning the repository to
create `flat_path` because the post-process step was being called from
with a loop. For large repositories, this was significantly slowing
things down. Break the call to `rugged_populate_flat_path` out of this
loop to make this work efficiently.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/59759
parent afbc8274
---
title: Avoid excessive recursive calls with Rugged TreeEntries
merge_request: 26813
author:
type: fixed
...@@ -15,12 +15,23 @@ module Gitlab ...@@ -15,12 +15,23 @@ module Gitlab
override :tree_entries override :tree_entries
def tree_entries(repository, sha, path, recursive) def tree_entries(repository, sha, path, recursive)
if Feature.enabled?(:rugged_tree_entries) if Feature.enabled?(:rugged_tree_entries)
tree_entries_from_rugged(repository, sha, path, recursive) tree_entries_with_flat_path_from_rugged(repository, sha, path, recursive)
else else
super super
end end
end end
def tree_entries_with_flat_path_from_rugged(repository, sha, path, recursive)
tree_entries_from_rugged(repository, sha, path, recursive).tap do |entries|
# This was an optimization to reduce N+1 queries for Gitaly
# (https://gitlab.com/gitlab-org/gitaly/issues/530). It
# used to be done lazily in the view via
# TreeHelper#flatten_tree, so it's possible there's a
# performance impact by loading this eagerly.
rugged_populate_flat_path(repository, sha, path, entries)
end
end
def tree_entries_from_rugged(repository, sha, path, recursive) def tree_entries_from_rugged(repository, sha, path, recursive)
current_path_entries = get_tree_entries_from_rugged(repository, sha, path) current_path_entries = get_tree_entries_from_rugged(repository, sha, path)
ordered_entries = [] ordered_entries = []
...@@ -32,13 +43,6 @@ module Gitlab ...@@ -32,13 +43,6 @@ module Gitlab
ordered_entries.concat(tree_entries_from_rugged(repository, sha, entry.path, true)) ordered_entries.concat(tree_entries_from_rugged(repository, sha, entry.path, true))
end end
end end
# This was an optimization to reduce N+1 queries for Gitaly
# (https://gitlab.com/gitlab-org/gitaly/issues/530). It
# used to be done lazily in the view via
# TreeHelper#flatten_tree, so it's possible there's a
# performance impact by loading this eagerly.
rugged_populate_flat_path(repository, sha, path, ordered_entries)
end end
def rugged_populate_flat_path(repository, sha, path, entries) def rugged_populate_flat_path(repository, sha, path, entries)
......
...@@ -19,7 +19,7 @@ describe Gitlab::Git::Tree, :seed_helper do ...@@ -19,7 +19,7 @@ describe Gitlab::Git::Tree, :seed_helper do
it 'returns a list of tree objects' do it 'returns a list of tree objects' do
entries = described_class.where(repository, SeedRepo::Commit::ID, 'files', true) entries = described_class.where(repository, SeedRepo::Commit::ID, 'files', true)
expect(entries.count).to be > 10 expect(entries.count).to be >= 5
expect(entries).to all(be_a(Gitlab::Git::Tree)) expect(entries).to all(be_a(Gitlab::Git::Tree))
end end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment