Merge branch 'a11y-ci-template' into 'master'

Initial a11y scanning CI template

See merge request gitlab-org/gitlab!25144

changelogs/unreleased/a11y-ci-template.yml

+---
+title: Add accessibility scanning CI template
+merge_request: 25144
+author:
+type: added

doc/user/project/merge_requests/accessibility_testing.md

+---
+type: reference, howto
+---
+
+# Accessibility Testing
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25144) in GitLab 12.8.
+
+If your application offers a web interface and you are using
+[GitLab CI/CD](../../../ci/README.md), you can quickly determine the accessibility
+impact of pending code changes.
+
+## Overview
+
+GitLab uses [pa11y](https://pa11y.org/), a free and open source tool for
+measuring the accessibility of web sites, and has built a simple
+[CI job template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml).
+This job outputs accessibility violations, warnings, and notices for each page
+analyzed to a file called `accessibility`.
+
+## Configure Accessibility Testing
+
+This example shows how to run [pa11y](https://pa11y.org/)
+on your code with GitLab CI/CD using a node Docker image.
+
+For GitLab 12.8 and later, to define the `a11y` job, you must
+[include](../../../ci/yaml/README.md#includetemplate) the
+[`Accessibility.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml)
+included with your GitLab installation, as shown below.
+For GitLab versions earlier than 12.8, you can copy and use the job as
+defined in that template.
+
+Add the following to your `.gitlab-ci.yml` file:
+
+```yaml
+include:
+  template: Verify/Accessibility.gitlab-ci.yml
+
+a11y:
+  variables:
+    a11y_urls: https://example.com https://example.com/another-page
+```
+
+The example above will create an `a11y` job in your CI/CD pipeline and will run
+Pa11y against the webpage you defined in `a11y_urls` to build a report.
+
+The full HTML Pa11y report will be saved as an artifact that can be [viewed directly in your browser](../pipelines/job_artifacts.md#browsing-artifacts).
+
+NOTE: **Note:**
+The job definition provided by the template does not support Kubernetes yet.
+
+It is not yet possible to pass configurations into Pa11y via CI configuration. To change anything,
+copy the template to your CI file and make the desired edits.

doc/user/project/merge_requests/index.md

@@ -91,6 +91,7 @@ or link to useful information directly in the merge request page:
 
 | Feature                                                                                                | Description                                                                                                                                                                                               |
 |--------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Accessibility Testing](accessibility_testing.md)                                                      | Automatically report A11y violations for changed pages in merge requests                                                                                                                                  |
 | [Browser Performance Testing](browser_performance_testing.md) **(PREMIUM)**                            | Quickly determine the performance impact of pending code changes.                                                                                                                                         |
 | [Code Quality](code_quality.md) **(STARTER)**                                                          | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area.                                     |
 | [Display arbitrary job artifacts](../../../ci/yaml/README.md#artifactsexpose_as)                       | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../pipelines/job_artifacts.md) in merge requests.                                                |

ee/lib/ee/gitlab/template/gitlab_ci_yml_template.rb

@@ -14,12 +14,16 @@ module EE
             super.merge(categories_ee)
           end
 
+          override :disabled_templates
+          def disabled_templates
+            []
+          end
+
           private
 
           def categories_ee
             {
-              'Security' => 'Security',
-              'Verify' => 'Verify'
+              'Security' => 'Security'
             }
           end
         end

ee/spec/lib/ee/gitlab/ci/templates/templates_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+describe "CI YML Templates" do
+  using RSpec::Parameterized::TableSyntax
+
+  subject { Gitlab::Ci::YamlProcessor.new(content) }
+
+  where(:template_name) do
+    Gitlab::Template::GitlabCiYmlTemplate.all.map(&:full_name)
+  end
+
+  with_them do
+    let(:content) do
+      <<~EOS
+        include:
+          - template: #{template_name}
+
+        concrete_build_implemented_by_a_user:
+          stage: test
+          script: do something
+      EOS
+    end
+
+    it 'is valid' do
+      expect { subject }.not_to raise_error
+    end
+
+    it 'require default stages to be included' do
+      expect(subject.stages).to include(*Gitlab::Ci::Config::Entry::Stages.default)
+    end
+  end
+end

ee/spec/lib/ee/gitlab/template/gitlab_ci_yml_template_spec.rb

@@ -14,8 +14,9 @@ describe Gitlab::Template::GitlabCiYmlTemplate do
       expect(templates).to include('SAST')
     end
 
-    it 'finds the Verify templates' do
+    it 'finds all the Verify templates' do
       expect(templates).to include('Browser-Performance')
+      expect(templates).to include('Accessibility')
     end
   end
 end

lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml

+# Read more about the feature here: https://docs.gitlab.com/ee/user/project/merge_requests/accessibility_testing.html
+
+stages:
+  - build
+  - test
+  - deploy
+  - accessibility
+
+a11y:
+  stage: accessibility
+  image: node
+  script:
+    - wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
+    - echo "deb http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google.list && \
+    - apt-get update && \
+    - apt-get install -y google-chrome-stable && \
+    - rm -rf /var/lib/apt/lists/*
+    - npm install pa11y@5.3.0 pa11y-reporter-html@1.0.0
+    - 'echo { \"chromeLaunchConfig\": { \"args\": [\"--no-sandbox\"] }, \"includeWarnings\": true, \"reporter\": \"html\" } > pa11y.json'
+    - './node_modules/.bin/pa11y $a11y_urls > accessibility.html'
+  allow_failure: true
+  artifacts:
+    when: always
+    expose_as: 'accessibility'
+    paths: ['accessibility.html']
+  rules:
+    - if: $a11y_urls

lib/gitlab/template/finders/global_template_finder.rb

@@ -5,9 +5,11 @@ module Gitlab
   module Template
     module Finders
       class GlobalTemplateFinder < BaseTemplateFinder
-        def initialize(base_dir, extension, categories = {})
+        def initialize(base_dir, extension, categories = {}, exclusions: [])
           @categories = categories
           @extension  = extension
+          @exclusions = exclusions
+
           super(base_dir)
         end
 
@@ -16,6 +18,8 @@ module Gitlab
         end
 
         def find(key)
+          return if excluded?(key)
+
           file_name = "#{key}#{@extension}"
 
           # The key is untrusted input, so ensure we can't be directed outside
@@ -28,11 +32,20 @@ module Gitlab
 
         def list_files_for(dir)
           dir = "#{dir}/" unless dir.end_with?('/')
-          Dir.glob(File.join(dir, "*#{@extension}")).select { |f| f =~ self.class.filter_regex(@extension) }
+
+          Dir.glob(File.join(dir, "*#{@extension}")).select do |f|
+            next if excluded?(f)
+
+            f =~ self.class.filter_regex(@extension)
+          end
         end
 
         private
 
+        def excluded?(file_name)
+          @exclusions.include?(file_name)
+        end
+
         def select_directory(file_name)
           @categories.keys.find do |category|
             File.exist?(File.join(category_directory(category), file_name))

lib/gitlab/template/gitlab_ci_yml_template.rb

@@ -17,16 +17,25 @@ module Gitlab
           {
             'General' => '',
             'Pages' => 'Pages',
+            'Verify' => 'Verify',
             'Auto deploy' => 'autodeploy'
           }
         end
 
+        def disabled_templates
+          %w[
+            Verify/Browser-Performance
+          ]
+        end
+
         def base_dir
           Rails.root.join('lib/gitlab/ci/templates')
         end
 
         def finder(project = nil)
-          Gitlab::Template::Finders::GlobalTemplateFinder.new(self.base_dir, self.extension, self.categories)
+          Gitlab::Template::Finders::GlobalTemplateFinder.new(
+            self.base_dir, self.extension, self.categories, exclusions: self.disabled_templates
+          )
         end
       end
     end

spec/lib/gitlab/ci/templates/templates_spec.rb

@@ -2,33 +2,43 @@
 
 require 'spec_helper'
 
-describe "CI YML Templates" do
-  using RSpec::Parameterized::TableSyntax
-
+describe 'CI YML Templates' do
   subject { Gitlab::Ci::YamlProcessor.new(content) }
 
-  where(:template_name) do
-    Gitlab::Template::GitlabCiYmlTemplate.all.map(&:full_name)
-  end
-
-  with_them do
-    let(:content) do
-      <<~EOS
-        include:
-          - template: #{template_name}
+  let(:all_templates) { Gitlab::Template::GitlabCiYmlTemplate.all.map(&:full_name) }
 
-        concrete_build_implemented_by_a_user:
-          stage: test
-          script: do something
-      EOS
+  let(:disabled_templates) do
+    Gitlab::Template::GitlabCiYmlTemplate.disabled_templates.map do |template|
+      template + Gitlab::Template::GitlabCiYmlTemplate.extension
     end
+  end
+
+  context 'included in a CI YAML configuration' do
+    using RSpec::Parameterized::TableSyntax
 
-    it 'is valid' do
-      expect { subject }.not_to raise_error
+    where(:template_name) do
+      all_templates - disabled_templates
     end
 
-    it 'require default stages to be included' do
-      expect(subject.stages).to include(*Gitlab::Ci::Config::Entry::Stages.default)
+    with_them do
+      let(:content) do
+        <<~EOS
+          include:
+            - template: #{template_name}
+
+          concrete_build_implemented_by_a_user:
+            stage: test
+            script: do something
+        EOS
+      end
+
+      it 'is valid' do
+        expect { subject }.not_to raise_error
+      end
+
+      it 'require default stages to be included' do
+        expect(subject.stages).to include(*Gitlab::Ci::Config::Entry::Stages.default)
+      end
     end
   end
 end

spec/lib/gitlab/template/finders/global_template_finder_spec.rb

@@ -15,23 +15,87 @@ describe Gitlab::Template::Finders::GlobalTemplateFinder do
     FileUtils.rm_rf(base_dir)
   end
 
-  subject(:finder) { described_class.new(base_dir, '', 'Foo' => '', 'Bar' => 'bar') }
+  subject(:finder) { described_class.new(base_dir, '', { 'General' => '', 'Bar' => 'Bar' }, exclusions: exclusions) }
+
+  let(:exclusions) { [] }
 
   describe '.find' do
-    it 'finds a template in the Foo category' do
-      create_template!('test-template')
+    context 'with a non-prefixed General template' do
+      before do
+        create_template!('test-template')
+      end
 
-      expect(finder.find('test-template')).to be_present
-    end
+      it 'finds the template with no prefix' do
+        expect(finder.find('test-template')).to be_present
+      end
+
+      it 'does not find a prefixed template' do
+        expect(finder.find('Bar/test-template')).to be_nil
+      end
+
+      it 'does not permit path traversal requests' do
+        expect { finder.find('../foo') }.to raise_error(/Invalid path/)
+      end
 
-    it 'finds a template in the Bar category' do
-      create_template!('bar/test-template')
+      context 'while listed as an exclusion' do
+        let(:exclusions) { %w[test-template] }
 
-      expect(finder.find('test-template')).to be_present
+        it 'does not find the template without a prefix' do
+          expect(finder.find('test-template')).to be_nil
+        end
+
+        it 'does not find the template with a prefix' do
+          expect(finder.find('Bar/test-template')).to be_nil
+        end
+
+        it 'finds another prefixed template with the same name' do
+          create_template!('Bar/test-template')
+
+          expect(finder.find('test-template')).to be_nil
+          expect(finder.find('Bar/test-template')).to be_present
+        end
+      end
     end
 
-    it 'does not permit path traversal requests' do
-      expect { finder.find('../foo') }.to raise_error(/Invalid path/)
+    context 'with a prefixed template' do
+      before do
+        create_template!('Bar/test-template')
+      end
+
+      it 'finds the template with a prefix' do
+        expect(finder.find('Bar/test-template')).to be_present
+      end
+
+      # NOTE: This spec fails, the template Bar/test-template is found
+      # See Gitlab issue: https://gitlab.com/gitlab-org/gitlab/issues/205719
+      xit 'does not find the template without a prefix' do
+        expect(finder.find('test-template')).to be_nil
+      end
+
+      it 'does not permit path traversal requests' do
+        expect { finder.find('../foo') }.to raise_error(/Invalid path/)
+      end
+
+      context 'while listed as an exclusion' do
+        let(:exclusions) { %w[Bar/test-template] }
+
+        it 'does not find the template with a prefix' do
+          expect(finder.find('Bar/test-template')).to be_nil
+        end
+
+        # NOTE: This spec fails, the template Bar/test-template is found
+        # See Gitlab issue: https://gitlab.com/gitlab-org/gitlab/issues/205719
+        xit 'does not find the template without a prefix' do
+          expect(finder.find('test-template')).to be_nil
+        end
+
+        it 'finds another non-prefixed template with the same name' do
+          create_template!('Bar/test-template')
+
+          expect(finder.find('test-template')).to be_present
+          expect(finder.find('Bar/test-template')).to be_nil
+        end
+      end
     end
   end
 end