Make the variable type for the GitLab CI secret configurable

When developing the initial GitLab CI secrets support with HashiCorp
Vault as the first secret engine, we've choosen to pass the resolved
secrets to job scope as a "file type" variables.

This however quickly became a limitation as some software is unable
to work with this type of the variables nor it can be adjusted
to it. And this adds additional steps that a user needs to define in
the job to make things working.

With this change we're adding a syntax to make this configurable
and we're updating the job payload API structure.

For the full support an updated version of GitLab Runner will be
required. Support on the Runner side is being added separately.

Changelog: added
EE: true

app/validators/json_schemas/build_metadata_secrets.json

@@ -23,7 +23,8 @@
             }
           },
           "additionalProperties": false
-        }
+        },
+        "^file$": { "type": "boolean" }
       },
       "additionalProperties": false
     }

doc/ci/secrets/index.md

@@ -8,6 +8,7 @@ type: concepts, howto
 # Using external secrets in CI
 
 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218746) in GitLab 13.4 and GitLab Runner 13.4.
+> - `file` setting [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250695) in GitLab 14.1 and GitLab Runner 14.1.
 
 Secrets represent sensitive information your CI job needs to complete work. This
 sensitive information can be items like API tokens, database credentials, or private keys.
@@ -116,6 +117,18 @@ After GitLab fetches the secret from Vault, the value is saved in a temporary fi
 The path to this file is stored in a CI/CD variable named `DATABASE_PASSWORD`,
 similar to [variables of type `file`](../variables/index.md#cicd-variable-types).
 
+To overwrite the default behavior, set the `file` option explicitly:
+
+```yaml
+secrets:
+  DATABASE_PASSWORD:
+    vault: production/db/password@ops
+    file: false
+```
+
+In this example, the secret value is put directly in the `DATABASE_PASSWORD` variable
+instead of pointing to a file that holds it.
+
 For more information about the supported syntax, read the
 [`.gitlab-ci.yml` reference](../yaml/index.md#secretsvault).
 

doc/ci/yaml/index.md

@@ -4230,7 +4230,7 @@ variables.
 
 #### `secrets:vault` **(PREMIUM)**
 
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4 and GitLab Runner 13.4.
 
 Use `vault` to specify secrets provided by [Hashicorp's Vault](https://www.vaultproject.io/).
 
@@ -4269,6 +4269,31 @@ job:
         field: password
 ```
 
+#### `secrets:file` **(PREMIUM)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250695) in GitLab 14.1 and GitLab Runner 14.1.
+
+By default, the secret is passed to the job context as a variable of type
+[`file`](../variables/index.md#cicd-variable-types). The value of the
+secret is stored in a file and the variable `DATABASE_PASSWORD` contains a path to the file.
+
+However, some software does not work with file variables and might require the secret value to be stored
+directly in the environment variable. For that case, define a `file` setting:
+
+```yaml
+job:
+  secrets:
+    DATABASE_PASSWORD:
+      vault: production/db/password@ops
+      file: false
+```
+
+When you set `file: false`, no files are created for that variable. It contains the secret
+itself instead.
+
+The `file` is a setting of the secret, so it belongs directly under the variable
+name level and not in the `vault` section.
+
 ### `pages`
 
 Use `pages` to upload static content to GitLab. The content

ee/lib/gitlab/ci/config/entry/secret.rb

@@ -11,14 +11,16 @@ module Gitlab
           include ::Gitlab::Config::Entry::Configurable
           include ::Gitlab::Config::Entry::Attributable
 
-          ALLOWED_KEYS = %i[vault].freeze
+          REQUIRED_KEYS = %i[vault].freeze
+          ALLOWED_KEYS = (REQUIRED_KEYS + %i[file]).freeze
 
           attributes ALLOWED_KEYS
 
           entry :vault, Entry::Vault::Secret, description: 'Vault secrets engine configuration'
+          entry :file, ::Gitlab::Config::Entry::Boolean, description: 'Should the created variable be of file type'
 
           validations do
-            validates :config, allowed_keys: ALLOWED_KEYS, required_keys: ALLOWED_KEYS
+            validates :config, allowed_keys: ALLOWED_KEYS, required_keys: REQUIRED_KEYS
           end
         end
       end

ee/spec/lib/gitlab/ci/config/entry/secret_spec.rb

@@ -11,26 +11,47 @@ RSpec.describe Gitlab::Ci::Config::Entry::Secret do
     end
 
     context 'when entry config value is correct' do
-      let(:config) do
-        {
-          vault: {
-            engine: { name: 'kv-v2', path: 'kv-v2' },
-            path: 'production/db',
-            field: 'password'
-          }
-        }
+      shared_examples 'configures secrets' do
+        describe '#value' do
+          it 'returns secret configuration' do
+            expect(entry.value).to eq(config)
+          end
+        end
+
+        describe '#valid?' do
+          it 'is valid' do
+            expect(entry).to be_valid
+          end
+        end
       end
 
-      describe '#value' do
-        it 'returns secret configuration' do
-          expect(entry.value).to eq(config)
+      context 'when file setting is not defined' do
+        let(:config) do
+          {
+            vault: {
+              engine: { name: 'kv-v2', path: 'kv-v2' },
+              path: 'production/db',
+              field: 'password'
+            }
+          }
         end
+
+        it_behaves_like 'configures secrets'
       end
 
-      describe '#valid?' do
-        it 'is valid' do
-          expect(entry).to be_valid
+      context 'when file setting is defined' do
+        let(:config) do
+          {
+            vault: {
+              engine: { name: 'kv-v2', path: 'kv-v2' },
+              path: 'production/db',
+              field: 'password'
+            },
+            file: true
+          }
         end
+
+        it_behaves_like 'configures secrets'
       end
     end
   end

ee/spec/presenters/ci/build_runner_presenter_spec.rb

@@ -19,6 +19,7 @@ RSpec.describe Ci::BuildRunnerPresenter do
       let(:secrets) do
         {
           DATABASE_PASSWORD: {
+            file: true,
             vault: {
               engine: { name: 'kv-v2', path: 'kv-v2' },
               path: 'production/db',
@@ -81,6 +82,14 @@ RSpec.describe Ci::BuildRunnerPresenter do
           end
         end
       end
+
+      context 'File variable configuration' do
+        subject { presenter.secrets_configuration.dig('DATABASE_PASSWORD') }
+
+        it 'contains the file configuration directive' do
+          expect(subject.fetch('file')).to be_truthy
+        end
+      end
     end
   end
 end

ee/spec/requests/api/ci/runner_spec.rb

@@ -18,7 +18,8 @@ RSpec.describe API::Ci::Runner do
                 engine: { name: 'kv-v2', path: 'kv-v2' },
                 path: 'production/db',
                 field: 'password'
-              }
+              },
+              file: true
             }
           }
         end
@@ -70,7 +71,8 @@ RSpec.describe API::Ci::Runner do
                         'engine' => { 'name' => 'kv-v2', 'path' => 'kv-v2' },
                         'path' => 'production/db',
                         'field' => 'password'
-                      }
+                      },
+                      'file' => true
                     }
                   }
                 )