Commit a20de8bc authored by Dmytro Zaporozhets's avatar Dmytro Zaporozhets

Merge branch 'add-sec-binaries-template' into 'master'

Add secure binaries template

See merge request gitlab-org/gitlab!28566
parents e0b8d5ec 32ab96d5
---
title: Add secure binaries template
merge_request: 28566
author:
type: added
......@@ -30,7 +30,73 @@ example of such a transfer:
1. Transfer images to offline environment.
1. Load transferred images into offline Docker registry.
### Example image packager script
### Using the official GitLab template
GitLab provides a [vendored template](../../ci/yaml/README.md#includetemplate)
to ease this process.
This template should be used in a new, empty project, with a `gitlab-ci.yml` file containing:
```yaml
include:
- template: Secure-Binaries.gitlab-ci.yml
```
The pipeline downloads the Docker images needed for the Security Scanners and saves them as
[job artifacts](../../ci/pipelines/job_artifacts.md) or pushes them to the [Container Registry](../../user/packages/container_registry/index.md)
of the project where the pipeline is executed. These archives can be transferred to another location
and [loaded](https://docs.docker.com/engine/reference/commandline/load/) in a Docker daemon.
This method requires a GitLab Runner with access to both `gitlab.com` (including
`registry.gitlab.com`) and the local offline instance. This runner must run in
[privileged mode](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode)
to be able to use the `docker` command inside the jobs. This runner can be installed in a DMZ or on
a bastion, and used only for this specific project.
#### Scheduling the updates
By default, this project's pipeline will run only once, when the `.gitlab-ci.yml` is added to the
repo. To update the GitLab security scanners and signatures, it's necessary to run this pipeline
regularly. GitLab provides a way to [schedule pipelines](../../ci/pipelines/schedules.md). For
example, you can set this up to download and store the Docker images every week.
Some images can be updated more frequently than others. For example, the [vulnerability database](https://hub.docker.com/r/arminc/clair-db/tags)
for Container Scanning is updated daily. To update this single image, create a new Scheduled
Pipeline that runs daily and set `SECURE_BINARIES_ANALYZERS` to `clair-vulnerabilities-db`. Only
this job will be triggered, and the image will be updated daily and made available in the project
registry.
#### Using the secure bundle created
The project using the `Secure-Binaries.gitlab-ci.yml` template should now host all the required
images and resources needed to run GitLab Security features.
The next step is to tell the offline instance to use these resources instead of the default ones on
`gitlab.com`. This can be done by setting the right environment variables:
`SAST_ANALYZER_IMAGE_PREFIX` for SAST analyzers, `DS_ANALYZER_IMAGE_PREFIX` for Dependency Scanning,
and so on.
You can set these variables in the project's `.gitlab-ci.yml` files by using the bundle directly, or
in the GitLab UI at the project or group level. See the [GitLab CI/CD environment variables page](../../ci/variables/README.md#creating-a-custom-environment-variable)
for more information.
#### Variables
The following table shows which variables you can use with the `Secure-Binaries.gitlab-ci.yml`
template:
| VARIABLE | Description | Default value |
|-------------------------------------------|-----------------------------------------------|-----------------------------------|
| `SECURE_BINARIES_ANALYZERS` | Comma-separated list of analyzers to download | `"bandit, brakeman, gosec, and so on..."` |
| `SECURE_BINARIES_DOWNLOAD_IMAGES` | Used to disable jobs | `"true"` |
| `SECURE_BINARIES_PUSH_IMAGES` | Push files to the project registry | `"true"` |
| `SECURE_BINARIES_SAVE_ARTIFACTS` | Also save image archives as artifacts | `"false"` |
| `SECURE_BINARIES_ANALYZER_VERSION` | Default analyzer version (docker tag) | `"2"` |
### Alternate way without the official template
If it's not possible to follow the above method, the images can be transferred manually instead:
#### Example image packager script
```sh
#!/bin/bash
......@@ -49,7 +115,7 @@ do
done
```
### Example image loader script
#### Example image loader script
This example loads the images from a bastion host to an offline host. In certain configurations,
physical media may be needed for such a transfer:
......
# This template should be used when Security Products (https://about.gitlab.com/handbook/engineering/development/secure/#security-products)
# have to be downloaded and stored locally.
#
# Usage:
#
# ```
# include:
# - template: Secure-Binaries.gitlab-ci.yml
# ```
#
# Docs: https://docs.gitlab.com/ee/topics/airgap/
variables:
SECURE_BINARIES_ANALYZERS: >-
bandit, brakeman, gosec, spotbugs, flawfinder, phpcs-security-audit, security-code-scan, nodejs-scan, eslint, tslint, secrets, sobelow, pmd-apex, kubesec,
bundler-audit, retire.js, gemnasium, gemnasium-maven, gemnasium-python,
klar, clair-vulnerabilities-db,
license-management,
dast
SECURE_BINARIES_DOWNLOAD_IMAGES: "true"
SECURE_BINARIES_PUSH_IMAGES: "true"
SECURE_BINARIES_SAVE_ARTIFACTS: "false"
SECURE_BINARIES_ANALYZER_VERSION: "2"
.download_images:
allow_failure: true
image: docker:stable
only:
refs:
- branches
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
services:
- docker:stable-dind
script:
- docker info
- env
- if [ -z "$SECURE_BINARIES_IMAGE" ]; then export SECURE_BINARIES_IMAGE=${SECURE_BINARIES_IMAGE:-"registry.gitlab.com/gitlab-org/security-products/${CI_JOB_NAME}:${SECURE_BINARIES_ANALYZER_VERSION}"}; fi
- docker pull ${SECURE_BINARIES_IMAGE}
- mkdir -p output/$(dirname ${CI_JOB_NAME})
- |
if [ "$SECURE_BINARIES_SAVE_ARTIFACTS" = "true" ]; then
docker save ${SECURE_BINARIES_IMAGE} -o output/${CI_JOB_NAME}_${SECURE_BINARIES_ANALYZER_VERSION}.tar
gzip output/${CI_JOB_NAME}_${SECURE_BINARIES_ANALYZER_VERSION}.tar
sha256sum output/${CI_JOB_NAME}_${SECURE_BINARIES_ANALYZER_VERSION}.tar.gz > output/${CI_JOB_NAME}_${SECURE_BINARIES_ANALYZER_VERSION}.tag.gz.sha256sum
fi
- |
if [ "$SECURE_BINARIES_PUSH_IMAGES" = "true" ]; then
docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
docker tag ${SECURE_BINARIES_IMAGE} ${CI_REGISTRY_IMAGE}/${CI_JOB_NAME}:${SECURE_BINARIES_ANALYZER_VERSION}
docker push ${CI_REGISTRY_IMAGE}/${CI_JOB_NAME}:${SECURE_BINARIES_ANALYZER_VERSION}
fi
artifacts:
paths:
- output/
#
# SAST jobs
#
analyzers/bandit:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bbandit\b/
analyzers/brakeman:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bbrakeman\b/
analyzers/gosec:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bgosec\b/
analyzers/spotbugs:
extends: .download_images
variables:
# TODO: Spotbugs is > 1GB, disabling for now
SECURE_BINARIES_SAVE_ARTIFACTS: "false"
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bspotbugs\b/
analyzers/flawfinder:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bflawfinder\b/
analyzers/phpcs-security-audit:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bphpcs-security-audit\b/
analyzers/security-code-scan:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bsecurity-code-scan\b/
analyzers/nodejs-scan:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bnodejs-scan\b/
analyzers/eslint:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\beslint\b/
analyzers/tslint:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\btslint\b/
analyzers/secrets:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bsecrets\b/
analyzers/sobelow:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bsobelow\b/
analyzers/pmd-apex:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bsecrets\b/
analyzers/kubesec:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bkubesec\b/
#
# Container Scanning jobs
#
analyzers/klar:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bklar\b/
analyzers/clair-vulnerabilities-db:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bclair-vulnerabilities-db\b/
variables:
SECURE_BINARIES_IMAGE: arminc/clair-db
SECURE_BINARIES_ANALYZER_VERSION: latest
#
# Dependency Scanning jobs
#
analyzers/bundler-audit:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bbundler-audit\b/
analyzers/retire.js:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bretire\.js\b/
analyzers/gemnasium:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bgemnasium\b/
analyzers/gemnasium-maven:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bgemnasium-maven\b/
analyzers/gemnasium-python:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bgemnasium-python\b/
#
# License Scanning
#
license-management:
extends: .download_images
variables:
SECURE_BINARIES_ANALYZER_VERSION: "${CI_SERVER_VERSION_MAJOR}-${CI_SERVER_VERSION_MINOR}-stable"
# TODO: license-management is > 1GB, disabling for now
SECURE_BINARIES_SAVE_ARTIFACTS: "false"
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\blicense-management\b/
#
# DAST
#
dast:
extends: .download_images
only:
variables:
- $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
$SECURE_BINARIES_ANALYZERS =~ /\bdast\b/
variables:
SECURE_BINARIES_ANALYZER_VERSION: 1
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment