Merge branch 'add-sec-binaries-template' into 'master'

Add secure binaries template

See merge request gitlab-org/gitlab!28566

changelogs/unreleased/28566-add-sec-binaries-template.yml

+---
+title: Add secure binaries template
+merge_request: 28566
+author:
+type: added

doc/topics/airgap/index.md

@@ -30,7 +30,73 @@ example of such a transfer:
 1. Transfer images to offline environment.
 1. Load transferred images into offline Docker registry.
 
-### Example image packager script
+### Using the official GitLab template
+
+GitLab provides a [vendored template](../../ci/yaml/README.md#includetemplate)
+to ease this process.
+
+This template should be used in a new, empty project, with a `gitlab-ci.yml` file containing:
+
+```yaml
+include:
+  - template: Secure-Binaries.gitlab-ci.yml
+```
+
+The pipeline downloads the Docker images needed for the Security Scanners and saves them as
+[job artifacts](../../ci/pipelines/job_artifacts.md) or pushes them to the [Container Registry](../../user/packages/container_registry/index.md)
+of the project where the pipeline is executed. These archives can be transferred to another location
+and [loaded](https://docs.docker.com/engine/reference/commandline/load/) in a Docker daemon.
+This method requires a GitLab Runner with access to both `gitlab.com` (including
+`registry.gitlab.com`) and the local offline instance. This runner must run in
+[privileged mode](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode)
+to be able to use the `docker` command inside the jobs. This runner can be installed in a DMZ or on
+a bastion, and used only for this specific project.
+
+#### Scheduling the updates
+
+By default, this project's pipeline will run only once, when the `.gitlab-ci.yml` is added to the
+repo. To update the GitLab security scanners and signatures, it's necessary to run this pipeline
+regularly. GitLab provides a way to [schedule pipelines](../../ci/pipelines/schedules.md). For
+example, you can set this up to download and store the Docker images every week.
+
+Some images can be updated more frequently than others. For example, the [vulnerability database](https://hub.docker.com/r/arminc/clair-db/tags)
+for Container Scanning is updated daily. To update this single image, create a new Scheduled
+Pipeline that runs daily and set `SECURE_BINARIES_ANALYZERS` to `clair-vulnerabilities-db`. Only
+this job will be triggered, and the image will be updated daily and made available in the project
+registry.
+
+#### Using the secure bundle created
+
+The project using the `Secure-Binaries.gitlab-ci.yml` template should now host all the required
+images and resources needed to run GitLab Security features.
+
+The next step is to tell the offline instance to use these resources instead of the default ones on
+`gitlab.com`. This can be done by setting the right environment variables:
+`SAST_ANALYZER_IMAGE_PREFIX` for SAST analyzers, `DS_ANALYZER_IMAGE_PREFIX` for Dependency Scanning,
+and so on.
+
+You can set these variables in the project's `.gitlab-ci.yml` files by using the bundle directly, or
+in the GitLab UI at the project or group level. See the [GitLab CI/CD environment variables page](../../ci/variables/README.md#creating-a-custom-environment-variable)
+for more information.
+
+#### Variables
+
+The following table shows which variables you can use with the `Secure-Binaries.gitlab-ci.yml`
+template:
+
+| VARIABLE                                  | Description                                   | Default value                     |
+|-------------------------------------------|-----------------------------------------------|-----------------------------------|
+| `SECURE_BINARIES_ANALYZERS`               | Comma-separated list of analyzers to download | `"bandit, brakeman, gosec, and so on..."` |
+| `SECURE_BINARIES_DOWNLOAD_IMAGES`         | Used to disable jobs                          | `"true"`                          |
+| `SECURE_BINARIES_PUSH_IMAGES`             | Push files to the project registry            | `"true"`                          |
+| `SECURE_BINARIES_SAVE_ARTIFACTS`          | Also save image archives as artifacts         | `"false"`                         |
+| `SECURE_BINARIES_ANALYZER_VERSION`        | Default analyzer version (docker tag)         | `"2"`                             |
+
+### Alternate way without the official template
+
+If it's not possible to follow the above method, the images can be transferred manually instead:
+
+#### Example image packager script
 
 ```sh
 #!/bin/bash
@@ -49,7 +115,7 @@ do
 done
 ```
 
-### Example image loader script
+#### Example image loader script
 
 This example loads the images from a bastion host to an offline host. In certain configurations,
 physical media may be needed for such a transfer:

lib/gitlab/ci/templates/Security/Secure-Binaries.gitlab-ci.yml

+# This template should be used when Security Products (https://about.gitlab.com/handbook/engineering/development/secure/#security-products)
+# have to be downloaded and stored locally.
+#
+# Usage:
+#
+# ```
+#   include:
+#     - template: Secure-Binaries.gitlab-ci.yml
+# ```
+#
+# Docs: https://docs.gitlab.com/ee/topics/airgap/
+
+
+variables:
+  SECURE_BINARIES_ANALYZERS: >-
+    bandit, brakeman, gosec, spotbugs, flawfinder, phpcs-security-audit, security-code-scan, nodejs-scan, eslint, tslint, secrets, sobelow, pmd-apex, kubesec,
+    bundler-audit, retire.js, gemnasium, gemnasium-maven, gemnasium-python,
+    klar, clair-vulnerabilities-db,
+    license-management,
+    dast
+
+  SECURE_BINARIES_DOWNLOAD_IMAGES: "true"
+  SECURE_BINARIES_PUSH_IMAGES: "true"
+  SECURE_BINARIES_SAVE_ARTIFACTS: "false"
+
+  SECURE_BINARIES_ANALYZER_VERSION: "2"
+
+.download_images:
+  allow_failure: true
+  image: docker:stable
+  only:
+    refs:
+      - branches
+  variables:
+    DOCKER_DRIVER: overlay2
+    DOCKER_TLS_CERTDIR: ""
+  services:
+    - docker:stable-dind
+  script:
+    - docker info
+    - env
+    - if [ -z "$SECURE_BINARIES_IMAGE" ]; then export SECURE_BINARIES_IMAGE=${SECURE_BINARIES_IMAGE:-"registry.gitlab.com/gitlab-org/security-products/${CI_JOB_NAME}:${SECURE_BINARIES_ANALYZER_VERSION}"}; fi
+    - docker pull ${SECURE_BINARIES_IMAGE}
+    - mkdir -p output/$(dirname ${CI_JOB_NAME})
+    - |
+      if [ "$SECURE_BINARIES_SAVE_ARTIFACTS" = "true" ]; then
+        docker save ${SECURE_BINARIES_IMAGE} -o output/${CI_JOB_NAME}_${SECURE_BINARIES_ANALYZER_VERSION}.tar
+        gzip output/${CI_JOB_NAME}_${SECURE_BINARIES_ANALYZER_VERSION}.tar
+        sha256sum output/${CI_JOB_NAME}_${SECURE_BINARIES_ANALYZER_VERSION}.tar.gz > output/${CI_JOB_NAME}_${SECURE_BINARIES_ANALYZER_VERSION}.tag.gz.sha256sum
+      fi
+    - |
+      if [ "$SECURE_BINARIES_PUSH_IMAGES" = "true" ]; then
+        docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
+        docker tag ${SECURE_BINARIES_IMAGE} ${CI_REGISTRY_IMAGE}/${CI_JOB_NAME}:${SECURE_BINARIES_ANALYZER_VERSION}
+        docker push ${CI_REGISTRY_IMAGE}/${CI_JOB_NAME}:${SECURE_BINARIES_ANALYZER_VERSION}
+      fi
+
+  artifacts:
+    paths:
+      - output/
+
+#
+# SAST jobs
+#
+
+analyzers/bandit:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bbandit\b/
+
+analyzers/brakeman:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bbrakeman\b/
+
+analyzers/gosec:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bgosec\b/
+
+analyzers/spotbugs:
+  extends: .download_images
+  variables:
+    # TODO: Spotbugs is > 1GB, disabling for now
+    SECURE_BINARIES_SAVE_ARTIFACTS: "false"
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bspotbugs\b/
+
+analyzers/flawfinder:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bflawfinder\b/
+
+analyzers/phpcs-security-audit:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bphpcs-security-audit\b/
+
+analyzers/security-code-scan:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bsecurity-code-scan\b/
+
+analyzers/nodejs-scan:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bnodejs-scan\b/
+
+analyzers/eslint:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\beslint\b/
+
+analyzers/tslint:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\btslint\b/
+
+analyzers/secrets:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bsecrets\b/
+
+analyzers/sobelow:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bsobelow\b/
+
+analyzers/pmd-apex:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bsecrets\b/
+
+analyzers/kubesec:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bkubesec\b/
+#
+# Container Scanning jobs
+#
+
+analyzers/klar:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bklar\b/
+
+analyzers/clair-vulnerabilities-db:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bclair-vulnerabilities-db\b/
+  variables:
+    SECURE_BINARIES_IMAGE: arminc/clair-db
+    SECURE_BINARIES_ANALYZER_VERSION: latest
+
+#
+# Dependency Scanning jobs
+#
+
+analyzers/bundler-audit:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bbundler-audit\b/
+
+analyzers/retire.js:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bretire\.js\b/
+
+analyzers/gemnasium:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bgemnasium\b/
+
+analyzers/gemnasium-maven:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bgemnasium-maven\b/
+
+analyzers/gemnasium-python:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bgemnasium-python\b/
+
+#
+# License Scanning
+#
+
+license-management:
+  extends: .download_images
+  variables:
+    SECURE_BINARIES_ANALYZER_VERSION: "${CI_SERVER_VERSION_MAJOR}-${CI_SERVER_VERSION_MINOR}-stable"
+    # TODO: license-management is > 1GB, disabling for now
+    SECURE_BINARIES_SAVE_ARTIFACTS: "false"
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\blicense-management\b/
+
+#
+# DAST
+#
+
+dast:
+  extends: .download_images
+  only:
+    variables:
+      - $SECURE_BINARIES_DOWNLOAD_IMAGES == "true" &&
+          $SECURE_BINARIES_ANALYZERS =~ /\bdast\b/
+  variables:
+    SECURE_BINARIES_ANALYZER_VERSION: 1