Commit fbf2cde3 authored by Kamil Trzciński's avatar Kamil Trzciński

Merge branch '207125-ci-jwt-auth' into 'master'

Generate JWT for authentication and provide it to CI jobs

See merge request gitlab-org/gitlab!28063
parents ff103cb8 562e5c40
......@@ -525,6 +525,7 @@ module Ci
strong_memoize(:variables) do
Gitlab::Ci::Variables::Collection.new
.concat(persisted_variables)
.concat(job_jwt_variables)
.concat(scoped_variables)
.concat(job_variables)
.concat(environment_changed_page_variables)
......@@ -974,6 +975,15 @@ module Ci
def has_expiring_artifacts?
artifacts_expire_at.present? && artifacts_expire_at > Time.now
end
def job_jwt_variables
Gitlab::Ci::Variables::Collection.new.tap do |variables|
break variables unless Feature.enabled?(:ci_job_jwt, project, default_enabled: true)
jwt = Gitlab::Ci::Jwt.for_build(self)
variables.append(key: 'CI_JOB_JWT', value: jwt, public: false, masked: true)
end
end
end
end
......
---
title: Generate JWT and provide it to CI jobs for integration with other systems
merge_request: 28063
author:
type: added
......@@ -159,6 +159,11 @@ Rails.application.routes.draw do
# Spam reports
resources :abuse_reports, only: [:new, :create]
# JWKS (JSON Web Key Set) endpoint
# Used by third parties to verify CI_JOB_JWT, placeholder route
# in case we decide to move away from doorkeeper-openid_connect
get 'jwks' => 'doorkeeper/openid_connect/discovery#keys'
end
# End of the /-/ scope.
......
......@@ -21,7 +21,7 @@ Examples are available in several forms. As a collection of:
The following table lists examples with step-by-step tutorials that are contained in this section.
| Use case | Resource |
|:----------------------------|:---------------------------------------------------------------------------------------------------------------------------|
|:------------------------------|:---------------------------------------------------------------------------------------------------------------------------|
| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](../../user/project/merge_requests/browser_performance_testing.md). |
| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). |
| Deployment with Dpl | [Using `dpl` as deployment tool](deployment/README.md). |
......@@ -38,6 +38,7 @@ The following table lists examples with step-by-step tutorials that are containe
| Ruby on Heroku | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). |
| Scala on Heroku | [Test and deploy a Scala application to Heroku](test-scala-application.md). |
| Parallel testing Ruby & JS | [GitLab CI/CD parallel jobs testing for Ruby & JavaScript projects](https://docs.knapsackpro.com/2019/how-to-run-parallel-jobs-for-rspec-tests-on-gitlab-ci-pipeline-and-speed-up-ruby-javascript-testing). |
| Secrets management with Vault | [Authenticating and Reading Secrets With Hashicorp Vault](authenticating-with-hashicorp-vault/index.md). |
### Contributing examples
......
---
type: tutorial
---
# Authenticating and Reading Secrets With Hashicorp Vault
This tutorial demonstrates how to authenticate, configure, and read secrets with HashiCorp's Vault from GitLab CI/CD.
## Requirements
This tutorial assumes you are familiar with GitLab CI/CD and Vault.
To follow along, you will need:
- An account on GitLab.
- A running Vault server and the access required to configure authentication and create roles and policies.
NOTE: **Note:**
You will need to replace the `vault.example.com` URL below with the URL of your Vault server and `gitlab.example.com` with the URL of your GitLab instance.
## How it works
Each job has JSON Web Token (JWT) provided as environment variable named `CI_JOB_JWT`. This JWT can be used to authenticate with Vault using the [JWT Auth](https://www.vaultproject.io/docs/auth/jwt/#jwt-authentication) method.
The JWT's payload looks like this:
```json
{
"jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558", # Unique identifier for this token
"iss": "gitlab.example.com", # Issuer, the domain of your GitLab instance
"iat": 1585710286, # Issued at
"nbf": 1585798372, # Not valid before
"exp": 1585713886, # Expire at
"sub": "job_1212", # Subject (job id)
"namespace_id": "1",
"namespace_path": "mygroup",
"project_id": "22",
"project_path": "mygroup/myproject",
"user_id": "42",
"user_login": "myuser",
"user_email": "myuser@example.com"
"pipeline_id": "1212",
"job_id": "1212",
"ref": "auto-deploy-2020-04-01", # Git ref for this job
"ref_type": "branch", # Git ref type, branch or tag
"ref_protected": "true" # true if this git ref is protected, false otherwise
}
```
The JWT is encoded by using RS256 and signed with your GitLab instance's OpenID Connect private key. The expire time for the token will be set to job's timeout, if specifed, or 5 minutes if it is not. The key used to sign this token may change without any notice. In such case retrying the job will generate new JWT using the current signing key.
You can use this JWT and your instance's JWKS endpoint (`https://gitlab.example.com/-/jwks`) to authenticate with a Vault server that is configured to allow the JWT Authentication method for authentication.
When configuring roles in Vault, you can use [bound_claims](https://www.vaultproject.io/docs/auth/jwt/#bound-claims) to match against the JWT's claims and restrict which secrets each CI job has access to.
To communicate with Vault, you can use either its CLI client or perform API requests (using `curl` or another client).
## Example
CAUTION: **Caution**:
JWTs are credentials, which can grant access to resources. Be careful where you paste them!
Let's say you have the passwords for your staging and production databases stored in a Vault server that is running on `http://vault.example.com:8200`. Your staging password is `pa$$w0rd` and your production password is `real-pa$$w0rd`.
```shell
$ vault kv get -field=password secret/myproject/staging/db
pa$$w0rd
$ vault kv get -field=password secret/myproject/production/db
real-pa$$w0rd
```
To configure your Vault server, start by enabling the [JWT Auth](https://www.vaultproject.io/docs/auth/jwt/) method:
```shell
$ vault auth enable jwt
Success! Enabled jwt auth method at: jwt/
```
Then create policies that allow you to read these secrets (one for each secret):
```shell
$ vault policy write myproject-staging - <<EOF
# Policy name: myproject-staging
#
# Read-only permission on 'secret/data/myproject/staging/*' path
path "secret/data/myproject/staging/*" {
capabilities = [ "read" ]
}
EOF
Success! Uploaded policy: myproject-staging
$ vault policy write myproject-production - <<EOF
# Policy name: myproject-production
#
# Read-only permission on 'secret/data/myproject/production/*' path
path "secret/data/myproject/production/*" {
capabilities = [ "read" ]
}
EOF
Success! Uploaded policy: myproject-production
```
You'll also need roles that will link the JWT with these policies.
One for staging named `myproject-staging`:
```shell
$ vault write auth/jwt/role/myproject-staging - <<EOF
{
"role_type": "jwt",
"policies": ["myproject-staging"],
"token_explicit_max_ttl": 60,
"user_claim": "user_email",
"bound_claims": {
"project_id": "22",
"ref": "master",
"ref_type": "branch"
}
}
EOF
```
And one for production named `myproject-production`:
```shell
$ vault write auth/jwt/role/myproject-production - <<EOF
{
"role_type": "jwt",
"policies": ["myproject-production"],
"token_explicit_max_ttl": 60,
"user_claim": "user_email",
"bound_claims_type": "glob",
"bound_claims": {
"project_id": "22",
"ref_protected": "true",
"ref_type": "branch",
"ref": "auto-deploy-*"
}
}
EOF
```
This example uses [bound_claims](https://www.vaultproject.io/api/auth/jwt#bound_claims) to specify that only a JWT with matching values for the specified claims will be allowed to authenticate.
Combined with GitLab's [protected branches](../../../user/project/protected_branches.md), you can restrict who is able to authenticate and read the secrets.
[token_explicit_max_ttl](https://www.vaultproject.io/api/auth/jwt#token_explicit_max_ttl) specifies that the token issued by Vault, upon successful authentication, has a hard lifetime limit of 60 seconds.
[user_claim](https://www.vaultproject.io/api/auth/jwt#user_claim) specifies the name for the Identity alias created by Vault upon a successful login.
[bound_claims_type](https://www.vaultproject.io/api-docs/auth/jwt#bound_claims_type) configures the interpretation of the `bound_claims` values. If set to `glob`, the values will be interpreted as globs, with `*` matching any number of characters.
For the full list of options, see Vault's [Create Role documentation](https://www.vaultproject.io/api/auth/jwt#create-role).
CAUTION: **Caution**:
Always restrict your roles to project or namespace by using one of the provided claims (e.g. `project_id` or `namespace_id`). Otherwise any JWT generated by this instance may be allowed to authenticate using this role.
Now, configure the JWT Authentication method:
```shell
$ vault write auth/jwt/config \
jwks_url="https://gitlab.example.com/-/jwks" \
bound_issuer="gitlab.example.com"
```
[bound_issuer](https://www.vaultproject.io/api/auth/jwt#inlinecode-bound_issuer) specifies that only a JWT with the issuer (that is, the `iss` claim) set to `gitlab.example.com` can use this method to authenticate, and that the JWKS endpoint (`https://gitlab.example.com/-/jwks`) should be used to validate the token.
For the full list of available configuration options, see Vault's [API documentation](https://www.vaultproject.io/api/auth/jwt#configure).
The following job, when run for the `master` branch, will be able to read secrets under `secret/myproject/staging/`, but not the secrets under `secret/myproject/production/`:
```yaml
read_secrets:
script:
# Check job's ref name
- echo $CI_COMMIT_REF_NAME
# and is this ref protected
- echo $CI_COMMIT_REF_PROTECTED
# Vault's address can be provided here or as CI variable
- export VAULT_ADDR=http://vault.example.com:8200
# Authenticate and get token. Token expiry time and other properties can be configured
# when configuring JWT Auth - https://www.vaultproject.io/api/auth/jwt#parameters-1
- export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-staging jwt=$CI_JOB_JWT)"
# Now use the VAULT_TOKEN to read the secret and store it in an environment variable
- export PASSWORD="$(vault kv get -field=password secret/myproject/staging/db)"
# Use the secret
- echo $PASSWORD
# This will fail because the role myproject-staging can not read secrets from secret/myproject/production/*
- export PASSWORD="$(vault kv get -field=password secret/myproject/production/db)"
```
![read_secrets staging](img/vault-read-secrets-staging.png)
The following job will be able to authenticate using the `myproject-production` role and read secrets under `/secret/myproject/production/`:
```yaml
read_secrets:
script:
# Check job's ref name
- echo $CI_COMMIT_REF_NAME
# and is this ref protected
- echo $CI_COMMIT_REF_PROTECTED
# Vault's address can be provided here or as CI variable
- export VAULT_ADDR=http://vault.example.com:8200
# Authenticate and get token. Token expiry time and other properties can be configured
# when configuring JWT Auth - https://www.vaultproject.io/api/auth/jwt#parameters-1
- export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-production jwt=$CI_JOB_JWT)"
# Now use the VAULT_TOKEN to read the secret and store it in environment variable
- export PASSWORD="$(vault kv get -field=password secret/myproject/production/db)"
# Use the secret
- echo $PASSWORD
```
![read_secrets production](img/vault-read-secrets-production.png)
......@@ -62,6 +62,7 @@ future GitLab releases.**
| `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` |
| `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` |
| `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry](../../user/packages/container_registry/index.md) and downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories) |
| `CI_JOB_JWT` | 12.10 | all | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../examples/authenticating-with-hashicorp-vault). |
| `CI_JOB_URL` | 11.1 | 0.5 | Job details URL |
| `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of username(s) of assignee(s) for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created. |
| `CI_MERGE_REQUEST_CHANGED_PAGE_PATHS` | 12.9 | all | Comma-separated list of paths of changed pages in a deployed [Review App](../review_apps/index.md) for a [Merge Request](../merge_request_pipelines/index.md). A [Route Map](../review_apps/index.md#route-maps) must be configured. |
......
# frozen_string_literal: true
module Gitlab
module Ci
class Jwt
NOT_BEFORE_TIME = 5
DEFAULT_EXPIRE_TIME = 60 * 5
def self.for_build(build)
self.new(build, ttl: build.metadata_timeout).encoded
end
def initialize(build, ttl: nil)
@build = build
@ttl = ttl
end
def payload
custom_claims.merge(reserved_claims)
end
def encoded
headers = { kid: kid, typ: 'JWT' }
JWT.encode(payload, key, 'RS256', headers)
end
private
attr_reader :build, :ttl, :key_data
def reserved_claims
now = Time.now.to_i
{
jti: SecureRandom.uuid,
iss: Settings.gitlab.host,
iat: now,
nbf: now - NOT_BEFORE_TIME,
exp: now + (ttl || DEFAULT_EXPIRE_TIME),
sub: "job_#{build.id}"
}
end
def custom_claims
{
namespace_id: namespace.id.to_s,
namespace_path: namespace.full_path,
project_id: project.id.to_s,
project_path: project.full_path,
user_id: user&.id.to_s,
user_login: user&.username,
user_email: user&.email,
pipeline_id: build.pipeline.id.to_s,
job_id: build.id.to_s,
ref: source_ref,
ref_type: ref_type,
ref_protected: build.protected.to_s
}
end
def key
@key ||= OpenSSL::PKey::RSA.new(Rails.application.secrets.openid_connect_signing_key)
end
def public_key
key.public_key
end
def kid
public_key.to_jwk[:kid]
end
def project
build.project
end
def namespace
project.namespace
end
def user
build.user
end
def source_ref
build.pipeline.source_ref
end
def ref_type
::Ci::BuildRunnerPresenter.new(build).ref_type
end
end
end
end
# frozen_string_literal: true
require 'spec_helper'
describe Gitlab::Ci::Jwt do
let(:namespace) { build_stubbed(:namespace) }
let(:project) { build_stubbed(:project, namespace: namespace) }
let(:user) { build_stubbed(:user) }
let(:pipeline) { build_stubbed(:ci_pipeline, ref: 'auto-deploy-2020-03-19') }
let(:build) do
build_stubbed(
:ci_build,
project: project,
user: user,
pipeline: pipeline
)
end
describe '#payload' do
subject(:payload) { described_class.new(build, ttl: 30).payload }
it 'has correct values for the standard JWT attributes' do
Timecop.freeze do
now = Time.now.to_i
aggregate_failures do
expect(payload[:iss]).to eq(Settings.gitlab.host)
expect(payload[:iat]).to eq(now)
expect(payload[:exp]).to eq(now + 30)
expect(payload[:sub]).to eq("job_#{build.id}")
end
end
end
it 'has correct values for the custom attributes' do
aggregate_failures do
expect(payload[:namespace_id]).to eq(namespace.id.to_s)
expect(payload[:namespace_path]).to eq(namespace.full_path)
expect(payload[:project_id]).to eq(project.id.to_s)
expect(payload[:project_path]).to eq(project.full_path)
expect(payload[:user_id]).to eq(user.id.to_s)
expect(payload[:user_email]).to eq(user.email)
expect(payload[:user_login]).to eq(user.username)
expect(payload[:pipeline_id]).to eq(pipeline.id.to_s)
expect(payload[:job_id]).to eq(build.id.to_s)
expect(payload[:ref]).to eq(pipeline.source_ref)
end
end
it 'skips user related custom attributes if build has no user assigned' do
allow(build).to receive(:user).and_return(nil)
expect { payload }.not_to raise_error
end
describe 'ref type' do
context 'branches' do
it 'is "branch"' do
expect(payload[:ref_type]).to eq('branch')
end
end
context 'tags' do
let(:build) { build_stubbed(:ci_build, :on_tag, project: project) }
it 'is "tag"' do
expect(payload[:ref_type]).to eq('tag')
end
end
context 'merge requests' do
let(:pipeline) { build_stubbed(:ci_pipeline, :detached_merge_request_pipeline) }
it 'is "branch"' do
expect(payload[:ref_type]).to eq('branch')
end
end
end
describe 'ref_protected' do
it 'is false when ref is not protected' do
expect(build).to receive(:protected).and_return(false)
expect(payload[:ref_protected]).to eq('false')
end
it 'is true when ref is protected' do
expect(build).to receive(:protected).and_return(true)
expect(payload[:ref_protected]).to eq('true')
end
end
end
describe '.for_build' do
let(:rsa_key) { OpenSSL::PKey::RSA.new(Rails.application.secrets.openid_connect_signing_key) }
subject(:jwt) { described_class.for_build(build) }
it 'generates JWT with key id' do
_payload, headers = JWT.decode(jwt, rsa_key.public_key, true, { algorithm: 'RS256' })
expect(headers['kid']).to eq(rsa_key.public_key.to_jwk['kid'])
end
it 'generates JWT for the given job with ttl equal to build timeout' do
expect(build).to receive(:metadata_timeout).and_return(3_600)
payload, _headers = JWT.decode(jwt, rsa_key.public_key, true, { algorithm: 'RS256' })
ttl = payload["exp"] - payload["iat"]
expect(ttl).to eq(3_600)
end
it 'generates JWT for the given job with default ttl if build timeout is not set' do
expect(build).to receive(:metadata_timeout).and_return(nil)
payload, _headers = JWT.decode(jwt, rsa_key.public_key, true, { algorithm: 'RS256' })
ttl = payload["exp"] - payload["iat"]
expect(ttl).to eq(5.minutes.to_i)
end
end
end
......@@ -2280,6 +2280,7 @@ describe Ci::Build do
{ key: 'CI_REGISTRY_USER', value: 'gitlab-ci-token', public: true, masked: false },
{ key: 'CI_REGISTRY_PASSWORD', value: 'my-token', public: false, masked: true },
{ key: 'CI_REPOSITORY_URL', value: build.repo_url, public: false, masked: false },
{ key: 'CI_JOB_JWT', value: 'ci.job.jwt', public: false, masked: true },
{ key: 'CI_JOB_NAME', value: 'test', public: true, masked: false },
{ key: 'CI_JOB_STAGE', value: 'test', public: true, masked: false },
{ key: 'CI_NODE_TOTAL', value: '1', public: true, masked: false },
......@@ -2332,23 +2333,36 @@ describe Ci::Build do
end
before do
allow(Gitlab::Ci::Jwt).to receive(:for_build).with(build).and_return('ci.job.jwt')
build.set_token('my-token')
build.yaml_variables = []
end
it { is_expected.to eq(predefined_variables) }
context 'when ci_job_jwt feature flag is disabled' do
before do
stub_feature_flags(ci_job_jwt: false)
end
it 'CI_JOB_JWT is not included' do
expect(subject.pluck(:key)).not_to include('CI_JOB_JWT')
end
end
describe 'variables ordering' do
context 'when variables hierarchy is stubbed' do
let(:build_pre_var) { { key: 'build', value: 'value', public: true, masked: false } }
let(:project_pre_var) { { key: 'project', value: 'value', public: true, masked: false } }
let(:pipeline_pre_var) { { key: 'pipeline', value: 'value', public: true, masked: false } }
let(:build_yaml_var) { { key: 'yaml', value: 'value', public: true, masked: false } }
let(:job_jwt_var) { { key: 'CI_JOB_JWT', value: 'ci.job.jwt', public: false, masked: true } }
before do
allow(build).to receive(:predefined_variables) { [build_pre_var] }
allow(build).to receive(:yaml_variables) { [build_yaml_var] }
allow(build).to receive(:persisted_variables) { [] }
allow(build).to receive(:job_jwt_variables) { [job_jwt_var] }
allow_any_instance_of(Project)
.to receive(:predefined_variables) { [project_pre_var] }
......@@ -2361,7 +2375,8 @@ describe Ci::Build do
it 'returns variables in order depending on resource hierarchy' do
is_expected.to eq(
[build_pre_var,
[job_jwt_var,
build_pre_var,
project_pre_var,
pipeline_pre_var,
build_yaml_var,
......
......@@ -3,6 +3,7 @@
require 'spec_helper'
# oauth_discovery_keys GET /oauth/discovery/keys(.:format) doorkeeper/openid_connect/discovery#keys
# jwks GET /-/jwks(.:format) doorkeeper/openid_connect/discovery#keys
# oauth_discovery_provider GET /.well-known/openid-configuration(.:format) doorkeeper/openid_connect/discovery#provider
# oauth_discovery_webfinger GET /.well-known/webfinger(.:format) doorkeeper/openid_connect/discovery#webfinger
describe Doorkeeper::OpenidConnect::DiscoveryController, 'routing' do
......@@ -17,6 +18,10 @@ describe Doorkeeper::OpenidConnect::DiscoveryController, 'routing' do
it "to #keys" do
expect(get('/oauth/discovery/keys')).to route_to('doorkeeper/openid_connect/discovery#keys')
end
it "/-/jwks" do
expect(get('/-/jwks')).to route_to('doorkeeper/openid_connect/discovery#keys')
end
end
# oauth_userinfo GET /oauth/userinfo(.:format) doorkeeper/openid_connect/userinfo#show
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment