Skip to content

G
gitlab-ce
Commit 5117aeb7 authored by Tetiana Chupryna's avatar Tetiana Chupryna
Browse files
Options

Merge branch 'bwill/default-branch-image-fingerprint' into 'master' 

Add default_branch_image to container scanning location

See merge request gitlab-org/gitlab!73486
parents bab05c7e 27409c6d
Hide whitespace changes
Inline Side-by-side
Showing with 274 additions and 52 deletions
config/feature_flags/development/improved_container_scan_matching.yml 0 → 100644
View file @ 5117aeb7
---
name: improved_container_scan_matching
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73486
rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/344534
milestone: '14.6'
type: development
group: group::container security
default_enabled: false
doc/user/application_security/container_scanning/index.md
View file @ 5117aeb7
......@@ -134,6 +134,7 @@ You can [configure](#customizing-the-container-scanning-settings) analyzers by u
| `CI_APPLICATION_REPOSITORY` | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG` | Docker repository URL for the image to be scanned. | All |
| `CI_APPLICATION_TAG` | `$CI_COMMIT_SHA` | Docker repository tag for the image to be scanned. | All |
| `CS_ANALYZER_IMAGE` | `registry.gitlab.com/security-products/container-scanning:4` | Docker image of the analyzer. | All |
| `CS_DEFAULT_BRANCH_IMAGE` | `""` | The name of the `DOCKER_IMAGE` on the default branch. See [Setting the default branch image](#setting-the-default-branch-image) for more details. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338877) in GitLab 14.5. | All |
| `CS_DOCKER_INSECURE` | `"false"` | Allow access to secure Docker registries using HTTPS without validating the certificates. | All |
| `CS_REGISTRY_INSECURE` | `"false"` | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. Works with all scanners, but the registry must listen on port `80/tcp` for Trivy to work. | All |
| `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy |
......@@ -229,6 +230,51 @@ Prior to the GitLab 14.0 release, any variable defined under the scope `containe
considered for scanners other than Clair. In GitLab 14.0 and later, all variables can be defined
either as a global variable or under `container_scanning`.
### Setting the default branch image
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338877) in GitLab 14.5.
By default, container scanning assumes that the image naming convention stores any branch-specific
identifiers in the image tag rather than the image name. When the image name differs between the
default branch and the non-default branch, previously-detected vulnerabilities show up as newly
detected in merge requests.
When the same image has different names on the default branch and a non-default branch, you can use
the `CS_DEFAULT_BRANCH_IMAGE` variable to indicate what that image's name is on the default branch.
GitLab then correctly determines if a vulnerability already exists when running scans on non-default
branches.
As an example, suppose the following:
- Non-default branches publish images with the naming convention
`$CI_REGISTRY_IMAGE/$CI_COMMIT_BRANCH:$CI_COMMIT_SHA`.
- The default branch publishes images with the naming convention
`$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA`.
In this example, you can use the following CI/CD configuration to ensure that vulnerabilities aren't
duplicated:
```yaml
include:
- template: Security/Container-Scanning.gitlab-ci.yml
container_scanning:
variables:
CS_DEFAULT_BRANCH_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
before_script:
- export DOCKER_IMAGE="$CI_REGISTRY_IMAGE/$CI_COMMIT_BRANCH:$CI_COMMIT_SHA"
- |
if [ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]; then
export DOCKER_IMAGE="$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
fi
```
`CS_DEFAULT_BRANCH_IMAGE` should remain the same for a given `DOCKER_IMAGE`. If it changes, then a
duplicate set of vulnerabilities are created, which must be manually dismissed.
When using [Auto DevOps](../../../topics/autodevops/index.md), `CS_DEFAULT_BRANCH_IMAGE` is
automatically set to `$CI_REGISTRY_IMAGE/$CI_DEFAULT_BRANCH:$CI_APPLICATION_TAG`.
### Using a custom SSL CA certificate authority
You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority, which is used to verify the peer when fetching Docker images from a registry which uses HTTPS. The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the [text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1). For example, to configure this value in the `.gitlab-ci.yml` file, use the following:
......@@ -500,7 +546,7 @@ Here's an example container scanning report:
```json-doc
{
"version": "3.0.0",
"version": "14.0.0",
"vulnerabilities": [
{
"id": "df52bc8ce9a2ae56bbcb0c4ecda62123fbd6f69b",
......@@ -522,7 +568,8 @@ Here's an example container scanning report:
"version": "1.4.8"
},
"operating_system": "debian:9.4",
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
"default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
},
"identifiers": [
{
......
ee/lib/gitlab/ci/parsers/security/container_scanning.rb
View file @ 5117aeb7
......@@ -12,7 +12,22 @@ module Gitlab
image: location_data['image'],
operating_system: location_data['operating_system'],
package_name: location_data.dig('dependency', 'package', 'name'),
package_version: location_data.dig('dependency', 'version'))
package_version: location_data.dig('dependency', 'version'),
default_branch_image: default_branch_image(location_data),
improved_container_scan_matching_enabled: improved_container_scan_matching_enabled?
)
end
def default_branch_image(location_data)
return unless improved_container_scan_matching_enabled?
return if @report.pipeline.default_branch?
location_data['default_branch_image']
end
def improved_container_scan_matching_enabled?
Feature.enabled?(:improved_container_scan_matching, @report.pipeline.project, default_enabled: :yaml)
end
end
end
......
ee/lib/gitlab/ci/reports/security/locations/container_scanning.rb
View file @ 5117aeb7
......@@ -6,28 +6,49 @@ module Gitlab
module Security
module Locations
class ContainerScanning < Base
attr_reader :default_branch_image
attr_reader :image
attr_reader :operating_system
attr_reader :package_name
attr_reader :package_version
def initialize(image:, operating_system:, package_name: nil, package_version: nil)
def initialize(
image:,
operating_system:,
package_name: nil,
package_version: nil,
default_branch_image: nil,
improved_container_scan_matching_enabled: false
)
@image = image
@operating_system = operating_system
@package_name = package_name
@package_version = package_version
@default_branch_image = default_branch_image
@improved_container_scan_matching_enabled = improved_container_scan_matching_enabled
end
def fingerprint_data
"#{docker_image_name_without_tag}:#{package_name}"
end
def improved_container_scan_matching_enabled?
@improved_container_scan_matching_enabled
end
private
def docker_image_name_without_tag
base_name, version = image.split(':')
if improved_container_scan_matching_enabled?
image_name = default_branch_image.presence || image
base_name, _, version = image_name.rpartition(':')
return image if version_semver_like?(version)
return image_name if version_semver_like?(version)
else
base_name, version = image.split(':')
return image if version_semver_like?(version)
end
base_name
end
......
ee/spec/fixtures/security_reports/master/gl-container-scanning-report.json
View file @ 5117aeb7
......@@ -21,7 +21,8 @@
"version": "2.24-11+deb9u3"
},
"operating_system": "debian:9",
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
"default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
},
"identifiers": [
{
......@@ -57,7 +58,8 @@
"version": "2.24-11+deb9u3"
},
"operating_system": "debian:9",
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
"default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
},
"identifiers": [
{
......@@ -92,7 +94,8 @@
"version": "2.24-11+deb9u3"
},
"operating_system": "debian:9",
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
"default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
},
"identifiers": [
{
......@@ -127,7 +130,8 @@
"version": "2.24-11+deb9u3"
},
"operating_system": "debian:9",
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
"default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
},
"identifiers": [
{
......@@ -162,7 +166,8 @@
"version": "0.168-1"
},
"operating_system": "debian:9",
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
"default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
},
"identifiers": [
{
......@@ -197,7 +202,8 @@
"version": "2.24-11+deb9u3"
},
"operating_system": "debian:9",
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
"default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
},
"identifiers": [
{
......@@ -232,7 +238,8 @@
"version": "3.3-1"
},
"operating_system": "debian:9",
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
"default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
},
"identifiers": [
{
......@@ -268,7 +275,8 @@
"version": "5.24.1-3+deb9u3"
},
"operating_system": "debian:9",
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
"image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
"default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
},
"identifiers": [
{
......@@ -299,4 +307,4 @@
"type": "container_scanning",
"status": "success"
}
}
\ No newline at end of file
}
ee/spec/lib/gitlab/ci/parsers/security/container_scanning_spec.rb
View file @ 5117aeb7
......@@ -3,18 +3,16 @@
require 'spec_helper'
RSpec.describe Gitlab::Ci::Parsers::Security::ContainerScanning do
let(:project) { artifact.project }
let(:pipeline) { artifact.job.pipeline }
let(:project) { create(:project, :repository) }
let(:current_branch) { project.default_branch }
let(:pipeline) { create(:ci_pipeline, ref: current_branch, project: project) }
let(:job) { create(:ci_build, pipeline: pipeline)}
let(:artifact) { create(:ee_ci_job_artifact, :container_scanning, job: job) }
let(:report) { Gitlab::Ci::Reports::Security::Report.new(artifact.file_type, pipeline, 2.weeks.ago) }
let(:image) { 'registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e' }
let(:default_branch_image) { 'registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest' }
before do
artifact.each_blob { |blob| described_class.parse!(blob, report) }
end
describe '#parse!' do
let(:artifact) { create(:ee_ci_job_artifact, :container_scanning) }
let(:image) { 'registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e' }
shared_examples 'report' do
it "parses all identifiers and findings for unapproved vulnerabilities" do
expect(report.findings.length).to eq(8)
expect(report.identifiers.length).to eq(8)
......@@ -42,4 +40,66 @@ RSpec.describe Gitlab::Ci::Parsers::Security::ContainerScanning do
expect(Gitlab::Json.parse(report.findings.first.raw_metadata).dig('location', 'image')).to eq(image)
end
end
describe '#parse!' do
context 'when improved_container_scan_matching is disabled' do
before do
stub_feature_flags(improved_container_scan_matching: false)
artifact.each_blob { |blob| described_class.parse!(blob, report) }
end
it_behaves_like 'report'
context 'when not on default branch' do
let(:current_branch) { 'not-default' }
it 'does not include default_branch_image' do
location = report.findings.first.location
expect(location).to be_a(::Gitlab::Ci::Reports::Security::Locations::ContainerScanning)
expect(location).to have_attributes(
default_branch_image: nil,
improved_container_scan_matching_enabled?: false
)
end
end
end
context 'when improved_container_scan_matching is enabled' do
before do
stub_feature_flags(improved_container_scan_matching: true)
artifact.each_blob { |blob| described_class.parse!(blob, report) }
end
it_behaves_like 'report'
context 'when on default branch' do
let(:current_branch) { project.default_branch }
it 'does not include default_branch_image in location' do
location = report.findings.first.location
expect(location).to be_a(::Gitlab::Ci::Reports::Security::Locations::ContainerScanning)
expect(location).to have_attributes(
default_branch_image: nil,
improved_container_scan_matching_enabled?: true
)
end
end
context 'when not on default branch' do
let(:current_branch) { 'not-default' }
it 'includes default_branch_image in location' do
location = report.findings.first.location
expect(location).to be_a(::Gitlab::Ci::Reports::Security::Locations::ContainerScanning)
expect(location).to have_attributes(
default_branch_image: default_branch_image,
improved_container_scan_matching_enabled?: true
)
end
end
end
end
end
ee/spec/lib/gitlab/ci/reports/security/locations/container_scanning_spec.rb
View file @ 5117aeb7
......@@ -18,44 +18,107 @@ RSpec.describe Gitlab::Ci::Reports::Security::Locations::ContainerScanning do
it_behaves_like 'vulnerability location'
subject { described_class.new(**params) }
describe 'fingerprint' do
sha1_of = -> (input) { Digest::SHA1.hexdigest(input) }
subject { described_class.new(**params) }
context 'with feature enabled' do
where(:image, :default_branch_image, :expected_fingerprint_input) do
[
['alpine:3.7.3', nil, 'alpine:3.7.3:glibc'],
['alpine:3.7', nil, 'alpine:3.7:glibc'],
['alpine:8101518288111119448185914762536722131810', nil, 'alpine:glibc'],
['alpine:1.0.0-beta', nil, 'alpine:1.0.0-beta:glibc'],
[
'gdk.local:5000/group/project/branch:307e0a35643f63652a713d0820db7c388012f724',
nil,
'gdk.local:5000/group/project/branch:glibc'
],
[
'registry.gitlab.com/group/project/tmp:af864bd61230d3d694eb01d6205b268b4ad63ac0',
nil,
'registry.gitlab.com/group/project/tmp:glibc'
],
[
'registry.gitlab.com/group/project/feature:5b1a4a921d7a50c3757aae3f7df2221878775af4',
'registry.gitlab.com/group/project/master:ec301f43f14a2b477806875e49cfc4d3fa0d22c3',
'registry.gitlab.com/group/project/master:glibc'
],
[
'registry.gitlab.com/group/project/feature:d6704dc0b8e33fb550a86f7847d6a3036d4f8bd5',
'registry.gitlab.com/group/project:latest',
'registry.gitlab.com/group/project:glibc'
],
[
'registry.gitlab.com/group/project@sha256:a418bbb80b9411f9a08025baa4681e192aaafd16505039bdcb113ccdb90a88fd',
'registry.gitlab.com/group/project:latest',
'registry.gitlab.com/group/project:glibc'
],
[
'registry.gitlab.com/group/project/feature:latest',
'registry.gitlab.com/group/project:1.0.0',
'registry.gitlab.com/group/project:1.0.0:glibc'
]
]
end
specify do
params[:image] = 'alpine:3.7.3'
expect(subject.fingerprint).to eq(sha1_of.call('alpine:3.7.3:glibc'))
end
with_them do
let(:params) do
{
image: image,
default_branch_image: default_branch_image,
operating_system: 'debian:9',
package_name: 'glibc',
package_version: '1.2.3',
improved_container_scan_matching_enabled: true
}
end
specify do
params[:image] = 'alpine:3.7'
expect(subject.fingerprint).to eq(sha1_of.call('alpine:3.7:glibc'))
specify { expect(subject.fingerprint).to eq(sha1_of.call(expected_fingerprint_input)) }
end
end
specify do
params[:image] = 'alpine:8101518288111119448185914762536722131810'
expect(subject.fingerprint).to eq(sha1_of.call('alpine:glibc'))
end
context 'with feature disabled' do
let(:params) do
{
image: 'registry.gitlab.com/group/project/feature:ec301f43f14a2b477806875e49cfc4d3fa0d22c3',
default_branch_image: 'registry.gitlab.com/group/project/master:ec301f43f14a2b477806875e49cfc4d3fa0d22c3',
operating_system: 'debian:9',
package_name: 'glibc',
package_version: '1.2.3'
}
end
specify do
params[:image] = 'alpine:1.0.0-beta'
expect(subject.fingerprint).to eq(sha1_of.call('alpine:1.0.0-beta:glibc'))
end
it 'ignores default_branch_image' do
expect(subject.fingerprint).to eq(sha1_of.call('registry.gitlab.com/group/project/feature:glibc'))
end
specify do
params[:image] = 'registry.gitlab.com/gitlab-org/security-products/analyzers/container-scanning/tmp:af864bd61230d3d694eb01d6205b268b4ad63ac0'
expect(subject.fingerprint).to eq(sha1_of.call('registry.gitlab.com/gitlab-org/security-products/analyzers/container-scanning/tmp:glibc'))
end
where(:image, :expected_fingerprint_input) do
[
['alpine:3.7.3', 'alpine:3.7.3:glibc'],
['alpine:3.7', 'alpine:3.7:glibc'],
['alpine:8101518288111119448185914762536722131810', 'alpine:glibc'],
['alpine:1.0.0-beta', 'alpine:1.0.0-beta:glibc'],
[
'registry.gitlab.com/group/project/tmp:af864bd61230d3d694eb01d6205b268b4ad63ac0',
'registry.gitlab.com/group/project/tmp:glibc'
]
]
end
specify do
params[:image] = 'registry.gitlab.com/gitlab-org/security-products/tests/container-scanning/master:ec301f43f14a2b477806875e49cfc4d3fa0d22c3'
expect(subject.fingerprint).to eq(sha1_of.call('registry.gitlab.com/gitlab-org/security-products/tests/container-scanning/master:glibc'))
end
with_them do
let(:params) do
{
image: image,
operating_system: 'debian:9',
package_name: 'glibc',
package_version: '1.2.3'
}
end
specify do
params[:image] = 'registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e'
expect(subject.fingerprint).to eq(sha1_of.call('registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:glibc'))
specify { expect(subject.fingerprint).to eq(sha1_of.call(expected_fingerprint_input)) }
end
end
end
end
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7