Merge branch '212388-docs-and-follow-up-spdx' into 'master'

Follow-up from SPDX list for offline instance

See merge request gitlab-org/gitlab!39463

changelogs/unreleased/212388-docs-and-follow-up-spdx.yml

+---
+title: Enable read SPDX catalogue from local copy
+merge_request: 39463
+author:
+type: added

doc/raketasks/README.md

@@ -35,6 +35,7 @@ The following are available Rake tasks:
 | [Praefect Rake tasks](../administration/raketasks/praefect.md)                                      | [Praefect](../administration/gitaly/praefect.md)-related tasks.                           |
 | [Project import/export](../administration/raketasks/project_import_export.md)                       | Prepare for [project exports and imports](../user/project/settings/import_export.md).     |
 | [Sample Prometheus data](generate_sample_prometheus_data.md)                                        | Generate sample Prometheus data.                                                          |
+| [SPDX license list import](spdx.md) **(PREMIUM ONLY)**                                              | Import a local copy of the [SPDX license list](https://spdx.org/licenses/) for matching [License Compliance policies](../user/compliance/license_compliance/index.md).|                                                     |
 | [Repository storage](../administration/raketasks/storage.md)                                        | List and migrate existing projects and attachments from legacy storage to hashed storage. |
 | [Uploads migrate](../administration/raketasks/uploads/migrate.md)                                   | Migrate uploads between storage local and object storage.                                 |
 | [Uploads sanitize](../administration/raketasks/uploads/sanitize.md)                                 | Remove EXIF data from images uploaded to earlier versions of GitLab.                      |

doc/raketasks/spdx.md

+# SPDX license list import **(PREMIUM ONLY)**
+
+GitLab provides a Rake task for uploading a fresh copy of the [SPDX license list](https://spdx.org/licenses/)
+to a GitLab instance. This list is needed for matching the names of [License Compliance policies](../user/compliance/license_compliance/index.md).
+
+To import a fresh copy of the PDX license list, run:
+
+```shell
+# omnibus-gitlab
+sudo gitlab-rake gitlab:spdx:import
+
+# source installations
+bundle exec rake gitlab:spdx:import RAILS_ENV=production
+```
+
+To perform this task in the [offline environment](../user/application_security/offline_deployments/#defining-offline-environments),
+an outbound connection to [`licenses.json`](https://spdx.org/licenses/licenses.json) should be
+allowed.

doc/user/compliance/license_compliance/index.md

@@ -695,6 +695,16 @@ Additional configuration may be needed for connecting to
 [private Python repositories](#using-private-python-repos),
 and [private Yarn registries](#using-private-yarn-registries).
 
+### SPDX license list name matching
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212388) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.3.
+
+Prior to GitLab 13.3, offline environments required an exact name match for [project policies](#policies).
+In GitLab 13.3 and later, GitLab matches the name of [project policies](#policies)
+with identifiers from the [SPDX license list](https://spdx.org/licenses/).
+A local copy of the SPDX license list is distributed with the GitLab instance. If needed, the GitLab
+instance's administrator can manually update it with a [Rake task](../../../raketasks/spdx.md).
+
 Exact name matches are required for [project policies](#policies)
 when running in an offline environment ([see related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/212388)).
 

ee/config/feature_flags/development/offline_spdx_catalogue.yml

@@ -4,4 +4,4 @@ introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38691
 rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/212388
 group: group::composition analysis
 type: development
-default_enabled: false
+default_enabled: true

ee/lib/gitlab/spdx/catalogue_gateway.rb

@@ -7,7 +7,7 @@ module Gitlab
       OFFLINE_CATALOGUE = Rails.root.join('vendor/spdx.json').freeze
 
       def fetch
-        return offline_catalogue if Feature.enabled?(:offline_spdx_catalogue)
+        return offline_catalogue if Feature.enabled?(:offline_spdx_catalogue, default_enabled: true)
 
         response = ::Gitlab::HTTP.get(URL)
 

ee/lib/tasks/gitlab/spdx.rake

@@ -6,12 +6,15 @@ require 'gitlab/json'
 namespace :gitlab do
   namespace :spdx do
     desc 'GitLab | SPDX | Import copy of the catalogue to store it offline'
-    task :import do
-      spdx_url = Gitlab::SPDX::CatalogueGateway::URL
-      resp = Net::HTTP.get_response(URI.parse(spdx_url))
-      data = Gitlab::Json.parse(resp.body)
+    task import: :environment do
+      spdx_url = ::Gitlab::SPDX::CatalogueGateway::URL
+      resp = Gitlab::HTTP.get(URI.parse(spdx_url))
 
-      path = Gitlab::SPDX::CatalogueGateway::OFFLINE_CATALOGUE
+      raise 'Network failure' if resp.code != 200
+
+      data = ::Gitlab::Json.parse(resp.body)
+
+      path = ::Gitlab::SPDX::CatalogueGateway::OFFLINE_CATALOGUE
       IO.write(path, data.to_json, mode: 'w')
 
       puts "Local copy of SPDX catalogue is saved to #{path}"

ee/spec/tasks/gitlab/spdx_rake_spec.rb

+# frozen_string_literal: true
+
+require 'rake_helper'
+
+RSpec.describe 'gitlab:rake tasks' do
+  before do
+    Rake.application.rake_require 'tasks/gitlab/spdx'
+  end
+
+  describe 'import' do
+    subject { run_rake_task 'gitlab:spdx:import' }
+
+    let(:path) { Gitlab::SPDX::CatalogueGateway::OFFLINE_CATALOGUE }
+    let(:data) { { license1: 'test', license2: 'test2' } }
+
+    context 'with successful download of the catalogue' do
+      before do
+        stub_request(:get, Gitlab::SPDX::CatalogueGateway::URL).to_return(status: 200, body: data.to_json)
+        expect(IO).to receive(:write).with(path, anything, mode: 'w')
+      end
+
+      it 'saves the catalogue to the file' do
+        expect { subject }.to output("Local copy of SPDX catalogue is saved to #{path}\n").to_stdout
+      end
+    end
+
+    context 'when downloaded catalogue is broken' do
+      before do
+        stub_request(:get, Gitlab::SPDX::CatalogueGateway::URL).to_return(status: 200, body: data.inspect)
+      end
+
+      it 'raises parsing failure' do
+        expect { subject }.to output(/Import of SPDX catalogue failed: unexpected colon \(\)/).to_stdout
+      end
+    end
+
+    context 'with network failure' do
+      before do
+        stub_request(:get, Gitlab::SPDX::CatalogueGateway::URL).to_return(status: 404)
+      end
+
+      it 'raises network failure error' do
+        expect { subject }.to output("Import of SPDX catalogue failed: Network failure\n").to_stdout
+      end
+    end
+  end
+end