Automatic merge of gitlab-org/gitlab master

app/views/admin/dashboard/index.html.haml

@@ -143,6 +143,11 @@
               = _('GitLab Pages')
               %span.float-right
                 = Gitlab::Pages::VERSION
+          - if Gitlab::Kas.enabled?
+            %p
+              = _('GitLab KAS')
+              %span.gl-float-right
+                = Gitlab::Kas.version
 
           = render_if_exists 'admin/dashboard/geo'
 

config/feature_flags/development/header_read_timeout_buffered_io.yml

+---
+name: header_read_timeout_buffered_io
+introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78065
+rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/350233
+milestone: '14.8'
+type: development
+group: group::integrations
+default_enabled: false

doc/development/documentation/index.md

@@ -290,9 +290,7 @@ For example, [GitLab.com's `/help`](https://gitlab.com/help).
 
 ## Docs site architecture
 
-See the [Docs site architecture](site_architecture/index.md) page to learn
-how we build and deploy the site at <https://docs.gitlab.com> and
-to review all the assets and libraries in use.
+For information on how we build and deploy <https://docs.gitlab.com>, see [Docs site architecture](site_architecture/index.md).
 
 ### Global navigation
 

doc/development/documentation/site_architecture/deployment_process.md

@@ -82,7 +82,7 @@ for the stable branch of the image to rebuild. You might do this:
 
 ## Latest documentation
 
-A Docker image (tagged `latest`) is built that contains:
+We build a Docker image (tagged `latest`) that contains:
 
 - The latest online version of the documentation.
 - The documentation from the stable branches of upstream projects.

doc/development/documentation/site_architecture/index.md

@@ -60,13 +60,12 @@ To provide an optimized site structure, design, and a search-engine friendly
 website, along with a discoverable documentation, we use a few assets for
 the GitLab Documentation website.
 
-### Libraries
+### External libraries
 
-- [Bootstrap 4.3.1 components](https://getbootstrap.com/docs/4.3/components/)
-- [Bootstrap 4.3.1 JS](https://getbootstrap.com/docs/4.3/getting-started/javascript/)
-- [jQuery](https://jquery.com/) 3.3.1
-- [Clipboard JS](https://clipboardjs.com/)
-- [Font Awesome 4.7.0](https://fontawesome.com/v4.7.0/icons/)
+GitLab Docs is built with a combination of external:
+
+- [JavaScript libraries](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/package.json).
+- [Ruby libraries](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/Gemfile).
 
 ### SEO
 

doc/user/project/merge_requests/drafts.md

@@ -17,7 +17,8 @@ the **Merge** button until you remove the **Draft** flag:
 
 ## Mark merge requests as drafts
 
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32692) in GitLab 13.2, Work-In-Progress (WIP) merge requests were renamed to **Draft**. Support for using **WIP** is scheduled for removal in GitLab 14.0.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32692) in GitLab 13.2, Work-In-Progress (WIP) merge requests were renamed to **Draft**.
+> - [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/228685) all support for using **WIP** in GitLab 14.8.
 > - **Mark as draft** and **Mark as ready** buttons [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227421) in GitLab 13.5.
 
 There are several ways to flag a merge request as a draft:
@@ -35,10 +36,6 @@ There are several ways to flag a merge request as a draft:
   is not a toggle, and adding this text again in a later commit doesn't mark the
   merge request as ready.
 
-WARNING:
-Adding `WIP:` to the start of the merge request's title still marks a merge request
-as a draft. This feature is scheduled for removal in GitLab 14.0. Use `Draft:` instead.
-
 ## Mark merge requests as ready
 
 When a merge request is ready to be merged, you can remove the `Draft` flag in several ways:

doc/user/project/merge_requests/test_coverage_visualization.md

@@ -136,6 +136,9 @@ The `source` is ignored if the path does not follow this pattern. The parser ass
 
 ## Example test coverage configurations
 
+This section provides test coverage configuration examples for different programming languages. You can also see a working example in
+the [`coverage-report`](https://gitlab.com/gitlab-org/ci-sample-projects/coverage-report/) demonstration project.
+
 ### JavaScript example
 
 The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example uses [Mocha](https://mochajs.org/)

lib/gitlab/buffered_io.rb

+# frozen_string_literal: true
+
+module Gitlab
+  # Net::BufferedIO is overwritten by webmock but in order to test this class, it needs to inherit from the original BufferedIO.
+  # https://github.com/bblimke/webmock/blob/867f4b290fd133658aa9530cba4ba8b8c52c0d35/lib/webmock/http_lib_adapters/net_http.rb#L266
+  parent_class = if const_defined?('WebMock::HttpLibAdapters::NetHttpAdapter::OriginalNetBufferedIO') && Rails.env.test?
+                   WebMock::HttpLibAdapters::NetHttpAdapter::OriginalNetBufferedIO
+                 else
+                   Net::BufferedIO
+                 end
+
+  class BufferedIo < parent_class
+    extend ::Gitlab::Utils::Override
+
+    HEADER_READ_TIMEOUT = 20
+
+    # rubocop: disable Style/RedundantReturn
+    # rubocop: disable Cop/LineBreakAfterGuardClauses
+    # rubocop: disable Layout/EmptyLineAfterGuardClause
+
+    # Original method:
+    # https://github.com/ruby/ruby/blob/cdb7d699d0641e8f081d590d06d07887ac09961f/lib/net/protocol.rb#L190-L200
+    override :readuntil
+    def readuntil(terminator, ignore_eof = false)
+      start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      check_timeout = Feature.enabled?(:header_read_timeout_buffered_io)
+
+      begin
+        until idx = @rbuf.index(terminator)
+          if check_timeout && (elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time) > HEADER_READ_TIMEOUT
+            raise Gitlab::HTTP::HeaderReadTimeout, "Request timed out after reading headers for #{elapsed} seconds"
+          end
+
+          rbuf_fill
+        end
+
+        return rbuf_consume(idx + terminator.size)
+      rescue EOFError
+        raise unless ignore_eof
+        return rbuf_consume(@rbuf.size)
+      end
+    end
+    # rubocop: enable Style/RedundantReturn
+    # rubocop: enable Cop/LineBreakAfterGuardClauses
+    # rubocop: enable Layout/EmptyLineAfterGuardClause
+  end
+end

lib/gitlab/http.rb

@@ -9,6 +9,7 @@ module Gitlab
     BlockedUrlError = Class.new(StandardError)
     RedirectionTooDeep = Class.new(StandardError)
     ReadTotalTimeout = Class.new(Net::ReadTimeout)
+    HeaderReadTimeout = Class.new(Net::ReadTimeout)
 
     HTTP_TIMEOUT_ERRORS = [
       Net::OpenTimeout, Net::ReadTimeout, Net::WriteTimeout, Gitlab::HTTP::ReadTotalTimeout

lib/gitlab/http_connection_adapter.rb

 # frozen_string_literal: true
 
-# This class is part of the Gitlab::HTTP wrapper. Depending on the value
-# of the global setting allow_local_requests_from_web_hooks_and_services this adapter
-# will allow/block connection to internal IPs and/or urls.
+# This class is part of the Gitlab::HTTP wrapper. It handles local requests and header timeouts
 #
-# This functionality can be overridden by providing the setting the option
-# allow_local_requests = true in the request. For example:
-# Gitlab::HTTP.get('http://www.gitlab.com', allow_local_requests: true)
+# 1. Local requests
+#   Depending on the value of the global setting allow_local_requests_from_web_hooks_and_services,
+#   this adapter will allow/block connection to internal IPs and/or urls.
 #
-# This option will take precedence over the global setting.
+#   This functionality can be overridden by providing the setting the option
+#   allow_local_requests = true in the request. For example:
+#   Gitlab::HTTP.get('http://www.gitlab.com', allow_local_requests: true)
+#
+#   This option will take precedence over the global setting.
+#
+# 2. Header timeouts
+#   When the use_read_total_timeout option is used, that means the receiver
+#   of the HTTP request cannot be trusted. Gitlab::BufferedIo will be used,
+#   to read header data. It is a modified version of Net::BufferedIO that
+#   raises a timeout error if reading header data takes too much time.
+
 module Gitlab
   class HTTPConnectionAdapter < HTTParty::ConnectionAdapter
     extend ::Gitlab::Utils::Override
@@ -17,9 +26,20 @@ module Gitlab
     def connection
       @uri, hostname = validate_url!(uri)
 
-      super.tap do |http|
-        http.hostname_override = hostname if hostname
+      http = super
+      http.hostname_override = hostname if hostname
+
+      if options[:use_read_total_timeout]
+        gitlab_http = Gitlab::NetHttpAdapter.new(http.address, http.port)
+
+        http.instance_variables.each do |variable|
+          gitlab_http.instance_variable_set(variable, http.instance_variable_get(variable))
+        end
+
+        return gitlab_http
       end
+
+      http
     end
 
     private

lib/gitlab/net_http_adapter.rb

+# frozen_string_literal: true
+
+module Gitlab
+  # Webmock overwrites the Net::HTTP#request method with
+  # https://github.com/bblimke/webmock/blob/867f4b290fd133658aa9530cba4ba8b8c52c0d35/lib/webmock/http_lib_adapters/net_http.rb#L74
+  # Net::HTTP#request usually calls Net::HTTP#connect but the Webmock overwrite doesn't.
+  # This makes sure that, in a test environment, the superclass is the Webmock overwrite.
+  parent_class = if defined?(WebMock) && Rails.env.test?
+                   WebMock::HttpLibAdapters::NetHttpAdapter.instance_variable_get('@webMockNetHTTP')
+                 else
+                   Net::HTTP
+                 end
+
+  class NetHttpAdapter < parent_class
+    extend ::Gitlab::Utils::Override
+
+    private
+
+    override :connect
+    def connect
+      result = super
+
+      @socket = Gitlab::BufferedIo.new(@socket.io,
+        read_timeout: @socket.read_timeout,
+        write_timeout: @socket.write_timeout,
+        continue_timeout: @socket.continue_timeout,
+        debug_output: @socket.debug_output)
+
+      result
+    end
+  end
+end

lib/tasks/gitlab/tw/codeowners.rake

+# frozen_string_literal: true
+
+require 'yaml'
+
+namespace :tw do
+  desc 'Generates a list of codeowners for documentation pages.'
+  task :codeowners do
+    CodeOwnerRule = Struct.new(:category, :writer)
+
+    CODE_OWNER_RULES = [
+      CodeOwnerRule.new('Activation', '@kpaizee'),
+      CodeOwnerRule.new("Adoption", '@kpaizee'),
+      CodeOwnerRule.new('Activation', '@kpaizee'),
+      CodeOwnerRule.new('Adoption', '@kpaizee'),
+      CodeOwnerRule.new('APM', '@ngaskill'),
+      CodeOwnerRule.new('Authentication & Authorization', '@eread'),
+      CodeOwnerRule.new('Certify', '@msedlakjakubowski'),
+      CodeOwnerRule.new('Code Review', '@aqualls'),
+      CodeOwnerRule.new('Compliance', '@eread'),
+      CodeOwnerRule.new('Composition Analysis', '@rdickenson'),
+      CodeOwnerRule.new('Configure', '@marcia'),
+      CodeOwnerRule.new('Container Security', '@ngaskill'),
+      CodeOwnerRule.new('Contributor Experience', '@eread'),
+      CodeOwnerRule.new('Conversion', '@kpaizee'),
+      CodeOwnerRule.new('Database', '@marcia'),
+      CodeOwnerRule.new('Development', '@marcia'),
+      CodeOwnerRule.new('Distribution', '@axil'),
+      CodeOwnerRule.new('Distribution (Charts)', '@axil'),
+      CodeOwnerRule.new('Distribution (Omnibus)', '@axil'),
+      CodeOwnerRule.new('Documentation Guidelines', '@cnorris'),
+      CodeOwnerRule.new('Dynamic Analysis', '@rdickenson'),
+      CodeOwnerRule.new('Ecosystem', '@kpaizee'),
+      CodeOwnerRule.new('Editor', '@aqualls'),
+      CodeOwnerRule.new('Expansion', '@kpaizee'),
+      CodeOwnerRule.new('Foundations', '@rdickenson'),
+      CodeOwnerRule.new('Fuzz Testing', '@rdickenson'),
+      CodeOwnerRule.new('Geo', '@axil'),
+      CodeOwnerRule.new('Gitaly', '@eread'),
+      CodeOwnerRule.new('Global Search', '@marcia'),
+      CodeOwnerRule.new('Health', '@ngaskill'),
+      CodeOwnerRule.new('Import', '@ngaskill'),
+      CodeOwnerRule.new('Infrastructure', '@marcia'),
+      CodeOwnerRule.new('Integrations', '@kpaizee'),
+      CodeOwnerRule.new('Knowledge', '@aqualls'),
+      CodeOwnerRule.new('License', '@sselhorn'),
+      CodeOwnerRule.new('Memory', '@marcia'),
+      CodeOwnerRule.new('Monitor', '@ngaskill'),
+      CodeOwnerRule.new('Optimize', '@fneill'),
+      CodeOwnerRule.new('Package', '@ngaskill'),
+      CodeOwnerRule.new('Pipeline Authoring', '@marcel.amirault'),
+      CodeOwnerRule.new('Pipeline Execution', '@marcel.amirault'),
+      CodeOwnerRule.new('Portfolio Management', '@msedlakjakubowski'),
+      CodeOwnerRule.new('Product Intelligence', '@fneill'),
+      CodeOwnerRule.new('Product Planning', '@msedlakjakubowski'),
+      CodeOwnerRule.new('Project Management', '@msedlakjakubowski'),
+      CodeOwnerRule.new('Provision', '@sselhorn'),
+      CodeOwnerRule.new('Purchase', '@sselhorn'),
+      CodeOwnerRule.new('Redirect', 'Redirect'),
+      CodeOwnerRule.new('Release', '@rdickenson'),
+      CodeOwnerRule.new('Runner', '@sselhorn'),
+      CodeOwnerRule.new('Sharding', '@marcia'),
+      CodeOwnerRule.new('Source Code', '@aqualls'),
+      CodeOwnerRule.new('Static Analysis', '@rdickenson'),
+      CodeOwnerRule.new('Static Site Editor', '@aqualls'),
+      CodeOwnerRule.new('Style Guide', '@sselhorn'),
+      CodeOwnerRule.new('Testing', '@eread'),
+      CodeOwnerRule.new('Threat Insights', '@fneill'),
+      CodeOwnerRule.new('Utilization', '@sselhorn'),
+      CodeOwnerRule.new('Vulnerability Research', '@fneill'),
+      CodeOwnerRule.new('Workspace', '@fneill')
+    ].freeze
+
+    Document = Struct.new(:group, :redirect) do
+      def has_a_valid_group?
+        group && !redirect
+      end
+
+      def missing_metadata?
+        !group && !redirect
+      end
+    end
+
+    def self.writer_for_group(category)
+      CODE_OWNER_RULES.find { |rule| rule.category == category }&.writer
+    end
+
+    errors = []
+
+    path = Rails.root.join("doc/**/*.md")
+    Dir.glob(path) do |file|
+      yaml_data = YAML.load_file(file)
+      document = Document.new(yaml_data['group'], yaml_data['redirect_to'])
+
+      if document.missing_metadata?
+        errors << file
+        next
+      end
+
+      writer = writer_for_group(document.group)
+      next unless writer
+
+      puts "#{file.gsub(Dir.pwd, ".")} #{writer}" if document.has_a_valid_group?
+    end
+
+    if errors.present?
+      puts "-----"
+      puts "ERRORS - the following files are missing the correct metadata:"
+      errors.map { |file| puts file.gsub(Dir.pwd, ".")}
+    end
+  end
+end

locale/gitlab.pot

@@ -16274,6 +16274,9 @@ msgstr ""
 msgid "GitLab Issue"
 msgstr ""
 
+msgid "GitLab KAS"
+msgstr ""
+
 msgid "GitLab Pages"
 msgstr ""
 

spec/lib/gitlab/buffered_io_spec.rb

+# rubocop:disable Style/FrozenStringLiteralComment
+require 'spec_helper'
+
+RSpec.describe Gitlab::BufferedIo do
+  describe '#readuntil' do
+    let(:never_ending_tcp_socket) do
+      Class.new do
+        def initialize(*_)
+          @read_counter = 0
+        end
+
+        def setsockopt(*_); end
+
+        def closed?
+          false
+        end
+
+        def close
+          true
+        end
+
+        def to_io
+          StringIO.new('Hello World!')
+        end
+
+        def write_nonblock(data, *_)
+          data.size
+        end
+
+        def read_nonblock(buffer_size, *_)
+          sleep 0.01
+          @read_counter += 1
+
+          raise 'Test did not raise HeaderReadTimeout' if @read_counter > 10
+
+          'H' * buffer_size
+        end
+      end
+    end
+
+    before do
+      stub_const('Gitlab::BufferedIo::HEADER_READ_TIMEOUT', 0.1)
+    end
+
+    subject(:readuntil) do
+      Gitlab::BufferedIo.new(never_ending_tcp_socket.new).readuntil('a')
+    end
+
+    it 'raises a timeout error' do
+      expect { readuntil }.to raise_error(Gitlab::HTTP::HeaderReadTimeout, /Request timed out after reading headers for 0\.[0-9]+ seconds/)
+    end
+
+    context 'when the header_read_timeout feature is disabled' do
+      before do
+        stub_feature_flags(header_read_timeout_buffered_io: false)
+      end
+
+      it 'does not raise a timeout error' do
+        expect { readuntil }.to raise_error(RuntimeError, 'Test did not raise HeaderReadTimeout')
+      end
+    end
+  end
+end
+# rubocop:enable Style/FrozenStringLiteralComment

spec/lib/gitlab/http_connection_adapter_spec.rb

@@ -15,6 +15,18 @@ RSpec.describe Gitlab::HTTPConnectionAdapter do
       stub_all_dns('https://example.org', ip_address: '93.184.216.34')
     end
 
+    context 'with use_read_total_timeout option' do
+      let(:options) { { use_read_total_timeout: true } }
+
+      it 'sets up the connection using the Gitlab::NetHttpAdapter' do
+        expect(connection).to be_a(Gitlab::NetHttpAdapter)
+        expect(connection.address).to eq('93.184.216.34')
+        expect(connection.hostname_override).to eq('example.org')
+        expect(connection.addr_port).to eq('example.org')
+        expect(connection.port).to eq(443)
+      end
+    end
+
     context 'when local requests are allowed' do
       let(:options) { { allow_local_requests: true } }
 

spec/lib/gitlab/http_spec.rb

@@ -28,7 +28,7 @@ RSpec.describe Gitlab::HTTP do
   end
 
   context 'when reading the response is too slow' do
-    before do
+    before(:all) do
       # Override Net::HTTP to add a delay between sending each response chunk
       mocked_http = Class.new(Net::HTTP) do
         def request(*)
@@ -51,8 +51,17 @@ RSpec.describe Gitlab::HTTP do
       end
 
       @original_net_http = Net.send(:remove_const, :HTTP)
+      @webmock_net_http = WebMock::HttpLibAdapters::NetHttpAdapter.instance_variable_get('@webMockNetHTTP')
+
       Net.send(:const_set, :HTTP, mocked_http)
+      WebMock::HttpLibAdapters::NetHttpAdapter.instance_variable_set('@webMockNetHTTP', mocked_http)
+
+      # Reload Gitlab::NetHttpAdapter
+      Gitlab.send(:remove_const, :NetHttpAdapter)
+      load "#{Rails.root}/lib/gitlab/net_http_adapter.rb"
+    end
 
+    before do
       stub_const("#{described_class}::DEFAULT_READ_TOTAL_TIMEOUT", 0.001.seconds)
 
       WebMock.stub_request(:post, /.*/).to_return do |request|
@@ -60,9 +69,14 @@ RSpec.describe Gitlab::HTTP do
       end
     end
 
-    after do
+    after(:all) do
       Net.send(:remove_const, :HTTP)
       Net.send(:const_set, :HTTP, @original_net_http)
+      WebMock::HttpLibAdapters::NetHttpAdapter.instance_variable_set('@webMockNetHTTP', @webmock_net_http)
+
+      # Reload Gitlab::NetHttpAdapter
+      Gitlab.send(:remove_const, :NetHttpAdapter)
+      load "#{Rails.root}/lib/gitlab/net_http_adapter.rb"
     end
 
     let(:options) { {} }

spec/lib/gitlab/net_http_adapter_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+RSpec.describe Gitlab::NetHttpAdapter do
+  describe '#connect' do
+    let(:url) { 'https://example.org' }
+    let(:net_http_adapter) { described_class.new(url) }
+
+    subject(:connect) { net_http_adapter.send(:connect) }
+
+    before do
+      allow(TCPSocket).to receive(:open).and_return(Socket.new(:INET, :STREAM))
+    end
+
+    it 'uses a Gitlab::BufferedIo instance as @socket' do
+      connect
+
+      expect(net_http_adapter.instance_variable_get(:@socket)).to be_a(Gitlab::BufferedIo)
+    end
+  end
+end

spec/views/admin/dashboard/index.html.haml_spec.rb

@@ -63,4 +63,33 @@ RSpec.describe 'admin/dashboard/index.html.haml' do
       expect(rendered).to have_selector('.js-gitlab-version-check')
     end
   end
+
+  describe 'GitLab KAS' do
+    before do
+      allow(Gitlab::Kas).to receive(:enabled?).and_return(enabled)
+      allow(Gitlab::Kas).to receive(:version).and_return('kas-1.2.3')
+    end
+
+    context 'KAS enabled' do
+      let(:enabled) { true }
+
+      it 'includes KAS version' do
+        render
+
+        expect(rendered).to have_content('GitLab KAS')
+        expect(rendered).to have_content('kas-1.2.3')
+      end
+    end
+
+    context 'KAS disabled' do
+      let(:enabled) { false }
+
+      it 'does not include KAS version' do
+        render
+
+        expect(rendered).not_to have_content('GitLab KAS')
+        expect(rendered).not_to have_content('kas-1.2.3')
+      end
+    end
+  end
 end