Merge branch '10544-add-vulnerabilities-api' into 'master'

Add "Vulnerabilities API"

See merge request gitlab-org/gitlab-ee!11539

doc/api/README.md

@@ -70,6 +70,7 @@ The following API resources are available in the project context:
 | [Search](search.md)                                                 | `/projects/:id/search` (also available for groups and standalone)                                                                                                                                     |
 | [Services](services.md)                                             | `/projects/:id/services`                                                                                                                                                                              |
 | [Tags](tags.md)                                                     | `/projects/:id/repository/tags`                                                                                                                                                                       |
+| [Vulnerabilities](vulnerabilities.md) **[ULTIMATE]**                | `/projects/:id/vulnerabilities` (also available for groups)                                                                                                                                           |
 | [Wikis](wikis.md)                                                   | `/projects/:id/wikis`                                                                                                                                                                                 |
 
 ### Group resources

doc/api/issues.md

@@ -3,7 +3,7 @@
 Every API call to issues must be authenticated.
 
 If a user is not a member of a project and the project is private, a `GET`
-request on that project will result to a `404` status code.
+request on that project will result in a `404` status code.
 
 ## Issues pagination
 

doc/api/vulnerabilities.md

+# Vulnerabilities API
+
+Every API call to vulnerabilities must be authenticated.
+
+If a user is not a member of a project and the project is private, a `GET`
+request on that project will result in a `404` status code.
+
+## Vulnerabilities pagination
+
+By default, `GET` requests return 20 results at a time because the API results
+are paginated.
+
+Read more on [pagination](README.md#pagination).
+
+## List project vulnerabilities
+
+List all of a project's vulnerabilities.
+
+```
+GET /projects/:id/vulnerabilities
+GET /projects/:id/vulnerabilities?report_type=sast
+GET /projects/:id/vulnerabilities?report_type=container_scanning
+GET /projects/:id/vulnerabilities?report_type=sast,dast
+```
+
+| Attribute           | Type             | Required   | Description                                                                                                                                         |
+| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `id`                | integer/string   | yes        | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user.                                    |
+| `report_type`       | Array[string]    | no         | Returns vulnerabilities belonging to specified report type. Valid values: `sast`, `dast`, `dependency_scanning`, or `container_scanning`.           |
+
+```bash
+curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/vulnerabilities
+```
+
+Example response:
+
+```json
+[
+  {
+    "id": null,
+    "report_type": "dependency_scanning",
+    "name": "Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js",
+    "severity": "unknown",
+    "confidence": "undefined",
+    "scanner": {
+      "external_id": "gemnasium",
+      "name": "Gemnasium"
+    },
+    "identifiers": [
+      {
+        "external_type": "gemnasium",
+        "external_id": "9952e574-7b5b-46fa-a270-aeb694198a98",
+        "name": "Gemnasium-9952e574-7b5b-46fa-a270-aeb694198a98",
+        "url": "https://deps.sec.gitlab.com/packages/npm/saml2-js/versions/1.5.0/advisories"
+      },
+      {
+        "external_type": "cve",
+        "external_id": "CVE-2017-11429",
+        "name": "CVE-2017-11429",
+        "url": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-11429"
+      }
+    ],
+    "project_fingerprint": "fa6f5b6c5d240b834ac5e901dc69f9484cef89ec",
+    "vulnerability_feedback_issue_path": "/tests/yarn-remediation-test/vulnerability_feedback",
+    "vulnerability_feedback_merge_request_path": "/tests/yarn-remediation-test/vulnerability_feedback",
+    "vulnerability_feedback_dismissal_path": "/tests/yarn-remediation-test/vulnerability_feedback",
+    "project": {
+      "id": 31,
+      "name": "yarn-remediation-test",
+      "full_path": "/tests/yarn-remediation-test",
+      "full_name": "tests / yarn-remediation-test"
+    },
+    "dismissal_feedback": null,
+    "issue_feedback": null,
+    "merge_request_feedback": null,
+    "description": "Some XML DOM traversal and canonicalization APIs may be inconsistent in handling of comments within XML nodes. Incorrect use of these APIs by some SAML libraries results in incorrect parsing of the inner text of XML nodes such that any inner text after the comment is lost prior to cryptographically signing the SAML message. Text after the comment therefore has no impact on the signature on the SAML message.\r\n\r\nA remote attacker can modify SAML content for a SAML service provider without invalidating the cryptographic signature, which may allow attackers to bypass primary authentication for the affected SAML service provider.",
+    "links": [
+      {
+        "url": "https://github.com/Clever/saml2/commit/3546cb61fd541f219abda364c5b919633609ef3d#diff-af730f9f738de1c9ad87596df3f6de84R279"
+      },
+      {
+        "url": "https://www.kb.cert.org/vuls/id/475445"
+      },
+      {
+        "url": "https://github.com/Clever/saml2/issues/127"
+      }
+    ],
+    "location": {
+      "file": "yarn.lock",
+      "dependency": {
+        "package": {
+          "name": "saml2-js"
+        },
+        "version": "1.5.0"
+      }
+    },
+    "solution": "Upgrade to fixed version.\r\n",
+    "blob_path": "/tests/yarn-remediation-test/blob/cc6c4a0778460455ae5d16ca7025ca9ca1ca75ac/yarn.lock"
+  }
+]
+```

ee/app/finders/security/pipeline_vulnerabilities_finder.rb

+# frozen_string_literal: true
+
+# Security::PipelineVulnerabilitiesFinder
+#
+# Used to retrieve security vulnerabilities from an associated Pipeline,
+# This involves normalizing Report::Occurrence POROs to Vulnerabilities::Occurrence
+#
+# Arguments:
+#   pipeline - object to filter vulnerabilities
+#   params:
+#     report_type: Array<String>
+
+module Security
+  class PipelineVulnerabilitiesFinder
+    attr_accessor :params
+    attr_reader :pipeline
+
+    def initialize(pipeline:, params: default_params)
+      @pipeline = pipeline
+      @params = params
+    end
+
+    def execute
+      pipeline_reports.each_with_object([]) do |(type, report), occurrences|
+        next unless requested_type?(report.type)
+
+        occurrences.concat(
+          normalize_report_occurrences(report.occurrences)
+        )
+      end
+    end
+
+    private
+
+    def pipeline_reports
+      pipeline.security_reports.reports
+    end
+
+    def normalize_report_occurrences(report_occurrences)
+      report_occurrences.map do |report_occurrence|
+        occurrence_hash = report_occurrence.to_hash
+          .except(:compare_key, :identifiers, :location, :scanner) # rubocop:disable CodeReuse/ActiveRecord
+
+        occurrence = Vulnerabilities::Occurrence.new(occurrence_hash)
+
+        occurrence.project = pipeline.project
+        occurrence.build_scanner(report_occurrence.scanner.to_hash)
+        occurrence.identifiers = report_occurrence.identifiers.map do |identifier|
+          Vulnerabilities::Identifier.new(identifier.to_hash)
+        end
+
+        occurrence
+      end
+    end
+
+    def requested_type?(type)
+      Array(params[:report_type]).include?(type)
+    end
+
+    def default_params
+      { report_type: Vulnerabilities::Occurrence.report_types.keys }
+    end
+  end
+end

ee/changelogs/unreleased/10544-add-vulnerabilities-api.yml

+---
+title: Add API to retrieve security vulnerabilities
+merge_request: 11539
+author:
+type: added

ee/lib/api/grape_request_proxy.rb

+# frozen_string_literal: true
+
+module API
+  # GrapeRequestProxy provides a similar interface to ActionDispatch::Request,
+  # allowing usage of a serializer with `RequestAwareEntity` within a Grape API
+  class GrapeRequestProxy < SimpleDelegator
+    attr_reader :current_user
+
+    def initialize(req, current_user)
+      @current_user = current_user
+      super(req)
+    end
+  end
+end

ee/lib/api/vulnerabilities.rb

+# frozen_string_literal: true
+
+module API
+  class Vulnerabilities < Grape::API
+    include PaginationParams
+
+    helpers do
+      def vulnerability_occurrences_by(params)
+        pipeline = params[:project].latest_pipeline_with_security_reports
+
+        return [] unless pipeline
+
+        Security::PipelineVulnerabilitiesFinder.new(pipeline: pipeline, params: params).execute
+      end
+    end
+
+    before do
+      authenticate!
+    end
+
+    params do
+      requires :id, type: String, desc: 'The ID of a project'
+    end
+
+    resource :projects, requirements: API::NAMESPACE_OR_PROJECT_REQUIREMENTS do
+      desc 'Get a list of project vulnerabilities' do
+        success ::Vulnerabilities::OccurrenceEntity
+      end
+
+      params do
+        optional :report_type, type: Array[String], desc: 'The type of report vulnerability belongs to', default: ::Vulnerabilities::Occurrence.report_types.keys
+        use :pagination
+      end
+
+      get ':id/vulnerabilities' do
+        project = Project.find(params[:id])
+
+        not_found!('Project') unless project && can?(current_user, :read_project_security_dashboard, project)
+
+        vulnerability_occurrences = Kaminari.paginate_array(
+          vulnerability_occurrences_by(declared_params.merge(project: project))
+        )
+
+        present paginate(vulnerability_occurrences),
+          with: ::Vulnerabilities::OccurrenceEntity,
+          request: GrapeRequestProxy.new(request, current_user)
+      end
+    end
+  end
+end

ee/lib/ee/api/endpoints.rb

@@ -29,6 +29,7 @@ module EE
         mount ::API::Scim
         mount ::API::ManagedLicenses
         mount ::API::ProjectApprovals
+        mount ::API::Vulnerabilities
 
         version 'v3', using: :path do
           # Although the following endpoints are kept behind V3 namespace,

ee/spec/finders/security/pipeline_vulnerabilities_finder_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+describe Security::PipelineVulnerabilitiesFinder do
+  describe '#execute' do
+    set(:project) { create(:project, :repository) }
+    set(:pipeline) { create(:ci_pipeline, :success, project: project) }
+
+    set(:build_cs) { create(:ci_build, :success, name: 'cs_job', pipeline: pipeline, project: project) }
+    set(:build_dast) { create(:ci_build, :success, name: 'dast_job', pipeline: pipeline, project: project) }
+    set(:build_ds) { create(:ci_build, :success, name: 'ds_job', pipeline: pipeline, project: project) }
+    set(:build_sast) { create(:ci_build, :success, name: 'sast_job', pipeline: pipeline, project: project) }
+
+    set(:artifact_cs) { create(:ee_ci_job_artifact, :container_scanning, job: build_cs, project: project) }
+    set(:artifact_dast) { create(:ee_ci_job_artifact, :dast, job: build_dast, project: project) }
+    set(:artifact_ds) { create(:ee_ci_job_artifact, :dependency_scanning, job: build_ds, project: project) }
+    set(:artifact_sast) { create(:ee_ci_job_artifact, :sast, job: build_sast, project: project) }
+
+    let(:cs_count) { read_fixture(artifact_cs)['unapproved'].count }
+    let(:dast_count) { read_fixture(artifact_dast).dig('site', 'alerts').first['instances'].count }
+    let(:ds_count) { read_fixture(artifact_ds)['vulnerabilities'].count }
+    let(:sast_count) { read_fixture(artifact_sast)['vulnerabilities'].count }
+
+    before do
+      stub_licensed_features(sast: true, dependency_scanning: true, container_scanning: true, dast: true)
+    end
+
+    subject { described_class.new(pipeline: pipeline, params: params).execute }
+
+    context 'by report type' do
+      context 'when sast' do
+        let(:params) { { report_type: %w[sast] } }
+
+        it 'includes only sast' do
+          expect(subject.count).to eq sast_count
+        end
+      end
+
+      context 'when dependency_scanning' do
+        let(:params) { { report_type: %w[dependency_scanning] } }
+
+        it 'includes only dependency_scanning' do
+          expect(subject.count).to eq ds_count
+        end
+      end
+
+      context 'when dast' do
+        let(:params) { { report_type: %w[dast] } }
+
+        it 'includes only dast' do
+          # binding.pry
+          expect(subject.count).to eq dast_count
+        end
+      end
+
+      context 'when container_scanning' do
+        let(:params) { { report_type: %w[container_scanning] } }
+
+        it 'includes only container_scanning' do
+          expect(subject.count).to eq cs_count
+        end
+      end
+    end
+
+    context 'by all filters' do
+      context 'with found entity' do
+        let(:params) { { report_type: %w[sast dast container_scanning dependency_scanning] } }
+
+        it 'filters by all params' do
+          expect(subject.count).to eq cs_count + dast_count + ds_count + sast_count
+        end
+      end
+
+      context 'without found entity' do
+        let(:params) { { report_type: %w[code_quality] } }
+
+        it 'did not find anything' do
+          is_expected.to be_empty
+        end
+      end
+    end
+
+    context 'without params' do
+      subject { described_class.new(pipeline: pipeline).execute }
+
+      it 'returns all report_types' do
+        expect(subject.count).to eq cs_count + dast_count + ds_count + sast_count
+      end
+    end
+
+    def read_fixture(fixture)
+      JSON.parse(File.read(fixture.file.path))
+    end
+  end
+end

ee/spec/fixtures/api/schemas/vulnerabilities/occurrence.json

@@ -55,8 +55,8 @@
       { "type": "null" },
       { "$ref": "../vulnerability_feedback.json" }
     ]},
-    "description": { "type": "string" },
-    "solution": { "type": "string" },
+    "description": { "type": ["string", "null"] },
+    "solution": { "type": ["string", "null"] },
     "location" : {
       "class" : { "type": "string" },
       "method" : { "type": "string" },

ee/spec/requests/api/vulnerabilities_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+describe API::Vulnerabilities do
+  set(:project) { create(:project, :public) }
+  set(:user) { create(:user) }
+
+  let(:pipeline) do
+    create(:ci_empty_pipeline, status: :created, project: project)
+  end
+
+  let(:build_ds) { create(:ci_build, :success, name: 'ds_job', pipeline: pipeline, project: project) }
+  let(:build_sast) { create(:ci_build, :success, name: 'sast_job', pipeline: pipeline, project: project) }
+
+  before do
+    create(:ee_ci_job_artifact, :dependency_scanning, job: build_ds, project: project)
+    create(:ee_ci_job_artifact, :sast, job: build_sast, project: project)
+  end
+
+  describe "GET /projects/:id/vulnerabilities" do
+    context 'with an authorized user with proper permissions' do
+      before do
+        project.add_developer(user)
+        stub_licensed_features(security_dashboard: true, sast: true, dependency_scanning: true, container_scanning: true)
+      end
+
+      it 'returns all vulnerabilities' do
+        get api("/projects/#{project.id}/vulnerabilities", user)
+
+        expect(response).to have_gitlab_http_status(200)
+        expect(response).to include_pagination_headers
+        expect(response).to match_response_schema('vulnerabilities/occurrence_list', dir: 'ee')
+
+        expect(response.headers['X-Total']).to eq('37')
+        expect(response.headers['X-Total-Pages']).to eql('2')
+
+        expect(json_response.count).to eq 20
+        expect(json_response.map { |v| v['report_type'] }.uniq).to match_array %w[dependency_scanning sast]
+
+        expect(json_response.first['name']).to eq 'DoS by CPU exhaustion when using malicious SSL packets'
+      end
+
+      describe 'filtering' do
+        it 'returns vulnerabilities with sast report_type' do
+          get api("/projects/#{project.id}/vulnerabilities", user), params: { report_type: 'sast' }
+
+          expect(response).to have_gitlab_http_status(200)
+
+          expect(json_response.count).to eq 20
+          expect(json_response.map { |v| v['report_type'] }.uniq).to match_array %w[sast]
+
+          expect(json_response.first['name']).to eq 'Probable insecure usage of temp file/directory.'
+        end
+
+        it 'returns vulnerabilities with dependency_scanning report_type' do
+          get api("/projects/#{project.id}/vulnerabilities", user), params: { report_type: 'dependency_scanning' }
+
+          expect(response).to have_gitlab_http_status(200)
+
+          expect(json_response.count).to eq 4
+          expect(json_response.map { |v| v['report_type'] }.uniq).to match_array %w[dependency_scanning]
+
+          expect(json_response.first['name']).to eq 'DoS by CPU exhaustion when using malicious SSL packets'
+        end
+      end
+    end
+
+    context 'with authorized user without read permissions' do
+      before do
+        project.add_reporter(user)
+        stub_licensed_features(security_dashboard: false, sast: true, dependency_scanning: true, container_scanning: true)
+      end
+
+      it 'responds with 404 Not Found' do
+        get api("/projects/#{project.id}/vulnerabilities", user)
+
+        expect(response).to have_gitlab_http_status(404)
+      end
+    end
+
+    context 'with unauthorized user' do
+      it 'responds with 404 Not Found' do
+        get api("/projects/#{project.id}/vulnerabilities", user)
+
+        expect(response).to have_gitlab_http_status(404)
+      end
+    end
+
+    context 'with unknown project' do
+      it 'responds with 404 Not Found' do
+        get api("/projects/0/vulnerabilities", user)
+
+        expect(response).to have_gitlab_http_status(404)
+      end
+    end
+  end
+end