Automatic merge of gitlab-org/gitlab-ce master

changelogs/unreleased/sh-rugged-get-tree-entry.yml

+---
+title: Bring back Rugged implementation of TreeEntry
+merge_request: 25706
+author:
+type: other

config/gitlab.yml.example

@@ -227,7 +227,7 @@ production: &base
         # endpoint: 'http://127.0.0.1:9000' # default: nil
         # path_style: true # Use 'host/bucket_name/object' instead of 'bucket_name.host/object'
 
-  ## Packages (maven repository so far)
+  ## Packages (maven repository, npm registry, etc...)
   packages:
     enabled: true
     # The location where build packages are stored (default: shared/packages).

doc/development/testing_guide/frontend_testing.md

@@ -13,6 +13,42 @@ in the future.
 See the [Testing Standards and Style Guidelines](index.md) page for more
 information on general testing practices at GitLab.
 
+## Jest
+
+GitLab has started to migrate tests to the (Jest)[https://jestjs.io]
+testing framework. You can read a [detailed evaluation](https://gitlab.com/gitlab-org/gitlab-ce/issues/49171)
+of Jest compared to our use of Karma and Jasmine. In summary, it will allow us
+to improve the performance and consistency of our frontend tests.
+
+Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE.
+
+It is not yet a requirement to use Jest. You can view the
+[epic](https://gitlab.com/groups/gitlab-org/-/epics/873) of issues
+we need to solve before being able to use Jest for all our needs.
+
+### Timeout error
+
+The default timeout for Jest is set in
+[`/spec/frontend/test_setup.js`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/frontend/test_setup.js).
+
+If your test exceeds that time, it will fail.
+
+If you cannot improve the performance of the tests, you can increase the timeout
+for a specific test using
+[`jest.setTimeout`](https://jestjs.io/docs/en/jest-object#jestsettimeouttimeout).
+
+```javascript
+beforeAll(() => {
+  jest.setTimeout(500);
+});
+
+describe('Component', () => {
+  // ...
+});
+```
+
+Remember that the performance of each test depends on the environment.
+
 ## Karma test suite
 
 GitLab uses the [Karma][karma] test runner with [Jasmine] as its test
@@ -134,7 +170,7 @@ placeholders, and recalling when they are called and the arguments that are
 passed to them. These tools should be used liberally, to test for expected
 behavior, to mock responses, and to block unwanted side effects (such as a
 method that would generate a network request or alter `window.location`). The
-documentation for these methods can be found in the [jasmine introduction page](https://jasmine.github.io/2.0/introduction.html#section-Spies).
+documentation for these methods can be found in the [Jasmine introduction page](https://jasmine.github.io/2.0/introduction.html#section-Spies).
 
 Sometimes you may need to spy on a method that is directly imported by another
 module. GitLab has a custom `spyOnDependency` method which utilizes
@@ -168,7 +204,7 @@ export of a module who's import you want to stub, rather than an object which
 contains a method you wish to stub (if the module does not have a default
 export, one is be generated by the babel plugin). The second parameter is the
 name of the import you wish to change. The result of the function is a Spy
-object which can be treated like any other jasmine spy object.
+object which can be treated like any other Jasmine spy object.
 
 Further documentation on the babel rewire pluign API can be found on
 [its repository Readme doc](https://github.com/speedskater/babel-plugin-rewire#babel-plugin-rewire).
@@ -177,6 +213,14 @@ Further documentation on the babel rewire pluign API can be found on
 
 If you cannot avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) in tests, please use the [Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html).
 
+#### Migrating flaky Karma tests to Jest
+
+Some of our Karma tests are flaky because they access the properties of a shared scope.
+This also means that they are not easily parallelized.
+
+Migrating flaky Karma tests to Jest will help significantly as each test is executed
+in an isolated scope, improving performance and predictability.
+
 ### Vue.js unit tests
 
 See this [section][vue-test].
@@ -194,21 +238,21 @@ is sufficient (and saves you some time).
 
 ### Live testing and focused testing
 
-While developing locally, it may be helpful to keep karma running so that you
+While developing locally, it may be helpful to keep Karma running so that you
 can get instant feedback on as you write tests and modify code. To do this
-you can start karma with `yarn run karma-start`. It will compile the javascript
+you can start Karma with `yarn run karma-start`. It will compile the javascript
 assets and run a server at `http://localhost:9876/` where it will automatically
 run the tests on any browser which connects to it. You can enter that url on
 multiple browsers at once to have it run the tests on each in parallel.
 
-While karma is running, any changes you make will instantly trigger a recompile
+While Karma is running, any changes you make will instantly trigger a recompile
 and retest of the entire test suite, so you can see instantly if you've broken
-a test with your changes. You can use [jasmine focused][jasmine-focus] or
-excluded tests (with `fdescribe` or `xdescribe`) to get karma to run only the
+a test with your changes. You can use [Jasmine focused][jasmine-focus] or
+excluded tests (with `fdescribe` or `xdescribe`) to get Karma to run only the
 tests you want while you're working on a specific feature, but make sure to
 remove these directives when you commit your code.
 
-It is also possible to only run karma on specific folders or files by filtering
+It is also possible to only run Karma on specific folders or files by filtering
 the run tests via the argument `--filter-spec` or short `-f`:
 
 ```bash

lib/gitlab/git/blob.rb

@@ -23,6 +23,10 @@ module Gitlab
 
       class << self
         def find(repository, sha, path, limit: MAX_DATA_DISPLAY_SIZE)
+          tree_entry(repository, sha, path, limit)
+        end
+
+        def tree_entry(repository, sha, path, limit)
           return unless path
 
           path = path.sub(%r{\A/*}, '')
@@ -179,3 +183,5 @@ module Gitlab
     end
   end
 end
+
+Gitlab::Git::Blob.singleton_class.prepend Gitlab::Git::RuggedImpl::Blob::ClassMethods

lib/gitlab/git/rugged_impl/blob.rb

+# frozen_string_literal: true
+
+# NOTE: This code is legacy. Do not add/modify code here unless you have
+# discussed with the Gitaly team.  See
+# https://docs.gitlab.com/ee/development/gitaly.html#legacy-rugged-code
+# for more details.
+
+module Gitlab
+  module Git
+    module RuggedImpl
+      module Blob
+        module ClassMethods
+          extend ::Gitlab::Utils::Override
+
+          override :tree_entry
+          def tree_entry(repository, sha, path, limit)
+            if Feature.enabled?(:rugged_tree_entry)
+              rugged_tree_entry(repository, sha, path, limit)
+            else
+              super
+            end
+          end
+
+          private
+
+          def rugged_tree_entry(repository, sha, path, limit)
+            return unless path
+
+            # Strip any leading / characters from the path
+            path = path.sub(%r{\A/*}, '')
+
+            rugged_commit = repository.lookup(sha)
+            root_tree = rugged_commit.tree
+
+            blob_entry = find_entry_by_path(repository, root_tree.oid, *path.split('/'))
+
+            return unless blob_entry
+
+            if blob_entry[:type] == :commit
+              submodule_blob(blob_entry, path, sha)
+            else
+              blob = repository.lookup(blob_entry[:oid])
+
+              if blob
+                new(
+                  id: blob.oid,
+                  name: blob_entry[:name],
+                  size: blob.size,
+                  # Rugged::Blob#content is expensive; don't call it if we don't have to.
+                  data: limit.zero? ? '' : blob.content(limit),
+                  mode: blob_entry[:filemode].to_s(8),
+                  path: path,
+                  commit_id: sha,
+                  binary: blob.binary?
+                )
+              end
+            end
+          rescue Rugged::ReferenceError
+            nil
+          end
+
+          # Recursive search of blob id by path
+          #
+          # Ex.
+          #   blog/            # oid: 1a
+          #     app/           # oid: 2a
+          #       models/      # oid: 3a
+          #       file.rb      # oid: 4a
+          #
+          #
+          # Blob.find_entry_by_path(repo, '1a', 'blog', 'app', 'file.rb') # => '4a'
+          #
+          def find_entry_by_path(repository, root_id, *path_parts)
+            root_tree = repository.lookup(root_id)
+
+            entry = root_tree.find do |entry|
+              entry[:name] == path_parts[0]
+            end
+
+            return unless entry
+
+            if path_parts.size > 1
+              return unless entry[:type] == :tree
+
+              path_parts.shift
+              find_entry_by_path(repository, entry[:oid], *path_parts)
+            else
+              [:blob, :commit].include?(entry[:type]) ? entry : nil
+            end
+          end
+
+          def submodule_blob(blob_entry, path, sha)
+            new(
+              id: blob_entry[:oid],
+              name: blob_entry[:name],
+              size: 0,
+              data: '',
+              path: path,
+              commit_id: sha
+            )
+          end
+        end
+      end
+    end
+  end
+end

spec/lib/gitlab/git/blob_spec.rb

@@ -18,7 +18,7 @@ describe Gitlab::Git::Blob, :seed_helper do
     end
   end
 
-  describe '.find' do
+  shared_examples '.find' do
     context 'nil path' do
       let(:blob) { Gitlab::Git::Blob.find(repository, SeedRepo::Commit::ID, nil) }
 
@@ -128,6 +128,20 @@ describe Gitlab::Git::Blob, :seed_helper do
     end
   end
 
+  describe '.find with Gitaly enabled' do
+    it_behaves_like '.find'
+  end
+
+  describe '.find with Rugged enabled', :enable_rugged do
+    it 'calls out to the Rugged implementation' do
+      allow_any_instance_of(Rugged).to receive(:rev_parse).with(SeedRepo::Commit::ID).and_call_original
+
+      described_class.find(repository, SeedRepo::Commit::ID, 'files/images/6049019_460s.jpg')
+    end
+
+    it_behaves_like '.find'
+  end
+
   describe '.raw' do
     let(:raw_blob) { Gitlab::Git::Blob.raw(repository, SeedRepo::RubyBlob::ID) }
     let(:bad_blob) { Gitlab::Git::Blob.raw(repository, SeedRepo::BigCommit::ID) }