Add internal api for getting personal access tokens from gitlab-shell

This implements the internal api side of things to support the feature
requested in gitlab-org/gitlab#19672

The actual user-facing command in gitlab-shell for obtaining tokens is
being implemented but this endpoint is its hard dependency and must be
merged first.

changelogs/unreleased/36302-add-internal-api-for-getting-personal-access-tokens.yml

+---
+title: "Add internal api for getting personal access tokens from gitlab-shell"
+merge_request: 36302
+author: Taylan Develioglu @tdevelioglu
+type: added

doc/development/internal_api.md

+---
+stage: Create
+group: Source Code
+info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers"
+type: reference, api
+---
+
 # Internal API
 
 The internal API is used by different GitLab components, it can not be
@@ -24,10 +31,11 @@ authentication.
 
 ## Git Authentication
 
-This is called by Gitaly and GitLab-shell to check access to a
+This is called by [Gitaly](https://gitlab.com/gitlab-org/gitaly) and
+[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) to check access to a
 repository.
 
-When called from GitLab-shell no changes are passed and the internal
+When called from GitLab Shell no changes are passed and the internal
 API replies with the information needed to pass the request on to
 Gitaly.
 
@@ -40,13 +48,13 @@ POST /internal/allowed
 
 | Attribute | Type   | Required | Description |
 |:----------|:-------|:---------|:------------|
-| `key_id`  | string | no       | ID of the SSH-key used to connect to GitLab-shell |
-| `username` | string | no      | Username from the certificate used to connect to GitLab-Shell |
+| `key_id`  | string | no       | ID of the SSH-key used to connect to GitLab Shell |
+| `username` | string | no      | Username from the certificate used to connect to GitLab Shell |
 | `project`  | string | no (if `gl_repository` is passed) | Path to the project |
 | `gl_repository`  | string | no (if `project` is passed) | Repository identifier (e.g. `project-7`) |
 | `protocol` | string | yes     | SSH when called from GitLab-shell, HTTP or SSH when called from Gitaly |
 | `action`   | string | yes     | Git command being run (`git-upload-pack`, `git-receive-pack`, `git-upload-archive`) |
-| `changes`  | string | yes     | `<oldrev> <newrev> <refname>` when called from Gitaly, The magic string `_any` when called from GitLab Shell |
+| `changes`  | string | yes     | `<oldrev> <newrev> <refname>` when called from Gitaly, the magic string `_any` when called from GitLab Shell |
 | `check_ip` | string | no     | IP address from which call to GitLab Shell was made |
 
 Example request:
@@ -84,17 +92,17 @@ Example response:
 ### Known consumers
 
 - Gitaly
-- GitLab-shell
+- GitLab Shell
 
 ## LFS Authentication
 
-This is the endpoint that gets called from GitLab-shell to provide
+This is the endpoint that gets called from GitLab Shell to provide
 information for LFS clients when the repository is accessed over SSH.
 
 | Attribute | Type   | Required | Description |
 |:----------|:-------|:---------|:------------|
-| `key_id`  | string | no       | ID of the SSH-key used to connect to GitLab-shell |
-| `username`| string | no       | Username from the certificate used to connect to GitLab-Shell |
+| `key_id`  | string | no       | ID of the SSH-key used to connect to GitLab Shell |
+| `username`| string | no       | Username from the certificate used to connect to GitLab Shell |
 | `project` | string | no       | Path to the project |
 
 Example request:
@@ -114,17 +122,17 @@ curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" --da
 
 ### Known consumers
 
-- GitLab-shell
+- GitLab Shell
 
 ## Authorized Keys Check
 
-This endpoint is called by the GitLab-shell authorized keys
+This endpoint is called by the GitLab Shell authorized keys
 check. Which is called by OpenSSH for [fast SSH key
 lookup](../administration/operations/fast_ssh_key_lookup.md).
 
 | Attribute | Type   | Required | Description |
 |:----------|:-------|:---------|:------------|
-| `key`     | string | yes      | SSH key as passed by OpenSSH to GitLab-shell |
+| `key`     | string | yes      | SSH key as passed by OpenSSH to GitLab Shell |
 
 ```plaintext
 GET /internal/authorized_keys
@@ -149,7 +157,7 @@ Example response:
 
 ### Known consumers
 
-- GitLab-shell
+- GitLab Shell
 
 ## Get user for user ID or key
 
@@ -159,7 +167,7 @@ discovers the user associated with an SSH key.
 | Attribute | Type   | Required | Description |
 |:----------|:-------|:---------|:------------|
 | `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check |
-| `username` | string | no | Username of the user being looked up, used by GitLab-shell when authenticating using a certificate |
+| `username` | string | no | Username of the user being looked up, used by GitLab Shell when authenticating using a certificate |
 
 ```plaintext
 GET /internal/discover
@@ -183,12 +191,12 @@ Example response:
 
 ### Known consumers
 
-- GitLab-shell
+- GitLab Shell
 
 ## Instance information
 
 This gets some generic information about the instance. This is used
-by Geo nodes to get information about each other
+by Geo nodes to get information about each other.
 
 ```plaintext
 GET /internal/check
@@ -214,12 +222,12 @@ Example response:
 ### Known consumers
 
 - GitLab Geo
-- GitLab-shell's `bin/check`
+- GitLab Shell's `bin/check`
 
 ## Get new 2FA recovery codes using an SSH key
 
-This is called from GitLab-shell and allows users to get new 2FA
-recovery codes based on their SSH key
+This is called from GitLab Shell and allows users to get new 2FA
+recovery codes based on their SSH key.
 
 | Attribute | Type   | Required | Description |
 |:----------|:-------|:---------|:------------|
@@ -258,7 +266,45 @@ Example response:
 
 ### Known consumers
 
-- GitLab-shell
+- GitLab Shell
+
+## Get new personal access-token
+
+This is called from GitLab Shell and allows users to generate a new
+personal access token.
+
+| Attribute | Type   | Required | Description |
+|:----------|:-------|:---------|:------------|
+| `name` | string | yes | The name of the new token |
+| `scopes` | string array | yes | The authorization scopes for the new token, these must be valid token scopes |
+| `expires_at` | string | no | The expiry date for the new token |
+| `key_id`  | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check |
+| `user_id` | integer | no | User\_id for which to generate the new token |
+
+```plaintext
+POST /internal/personal_access_token
+```
+
+Example request:
+
+```shell
+curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" --data "user_id=29&name=mytokenname&scopes[]=read_user&scopes[]=read_repository&expires_at=2020-07-24" http://localhost:3001/api/v4/internal/personal_access_token
+```
+
+Example response:
+
+```json
+{
+  "success": true,
+  "token": "Hf_79B288hRv_3-TSD1R",
+  "scopes": ["read_user","read_repository"],
+  "expires_at": "2020-07-24"
+}
+```
+
+### Known consumers
+
+- GitLab Shell
 
 ## Incrementing counter on pre-receive
 

lib/api/internal/base.rb

@@ -18,6 +18,10 @@ module API
 
       UNKNOWN_CHECK_RESULT_ERROR = 'Unknown check result'.freeze
 
+      VALID_PAT_SCOPES = Set.new(
+        Gitlab::Auth::API_SCOPES + Gitlab::Auth::REPOSITORY_SCOPES + Gitlab::Auth::REGISTRY_SCOPES
+      ).freeze
+
       helpers do
         def response_with_status(code: 200, success: true, message: nil, **extra_options)
           status code
@@ -194,6 +198,60 @@ module API
           { success: true, recovery_codes: codes }
         end
 
+        post '/personal_access_token' do
+          status 200
+
+          actor.update_last_used_at!
+          user = actor.user
+
+          if params[:key_id]
+            unless actor.key
+              break { success: false, message: 'Could not find the given key' }
+            end
+
+            if actor.key.is_a?(DeployKey)
+              break { success: false, message: 'Deploy keys cannot be used to create personal access tokens' }
+            end
+
+            unless user
+              break { success: false, message: 'Could not find a user for the given key' }
+            end
+          elsif params[:user_id] && user.nil?
+            break { success: false, message: 'Could not find the given user' }
+          end
+
+          if params[:name].blank?
+            break { success: false, message: "No token name specified" }
+          end
+
+          if params[:scopes].blank?
+            break { success: false, message: "No token scopes specified" }
+          end
+
+          invalid_scope = params[:scopes].find { |scope| VALID_PAT_SCOPES.exclude?(scope.to_sym) }
+
+          if invalid_scope
+            valid_scopes = VALID_PAT_SCOPES.map(&:to_s).sort
+            break { success: false, message: "Invalid scope: '#{invalid_scope}'. Valid scopes are: #{valid_scopes}" }
+          end
+
+          begin
+            expires_at = params[:expires_at].presence && Date.parse(params[:expires_at])
+          rescue ArgumentError
+            break { success: false, message: "Invalid token expiry date: '#{params[:expires_at]}'" }
+          end
+
+          access_token = nil
+
+          ::Users::UpdateService.new(current_user, user: user).execute! do |user|
+            access_token = user.personal_access_tokens.create!(
+              name: params[:name], scopes: params[:scopes], expires_at: expires_at
+            )
+          end
+
+          { success: true, token: access_token.token, scopes: access_token.scopes, expires_at: access_token.expires_at }
+        end
+
         post '/pre_receive' do
           status 200
 

spec/requests/api/internal/base_spec.rb

@@ -120,6 +120,138 @@ RSpec.describe API::Internal::Base do
     end
   end
 
+  describe 'POST /internal/personal_access_token' do
+    it 'returns an error message when the key does not exist' do
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             key_id: non_existing_record_id
+           }
+
+      expect(json_response['success']).to be_falsey
+      expect(json_response['message']).to eq('Could not find the given key')
+    end
+
+    it 'returns an error message when the key is a deploy key' do
+      deploy_key = create(:deploy_key)
+
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             key_id: deploy_key.id
+           }
+
+      expect(json_response['success']).to be_falsey
+      expect(json_response['message']).to eq('Deploy keys cannot be used to create personal access tokens')
+    end
+
+    it 'returns an error message when the user does not exist' do
+      key_without_user = create(:key, user: nil)
+
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             key_id: key_without_user.id
+           }
+
+      expect(json_response['success']).to be_falsey
+      expect(json_response['message']).to eq('Could not find a user for the given key')
+      expect(json_response['token']).to be_nil
+    end
+
+    it 'returns an error message when given an non existent user' do
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             user_id: 0
+           }
+
+      expect(json_response['success']).to be_falsey
+      expect(json_response['message']).to eq("Could not find the given user")
+    end
+
+    it 'returns an error message when no name parameter is received' do
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             key_id:  key.id
+           }
+
+      expect(json_response['success']).to be_falsey
+      expect(json_response['message']).to eq("No token name specified")
+    end
+
+    it 'returns an error message when no scopes parameter is received' do
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             key_id:  key.id,
+             name: 'newtoken'
+           }
+
+      expect(json_response['success']).to be_falsey
+      expect(json_response['message']).to eq("No token scopes specified")
+    end
+
+    it 'returns an error message when expires_at contains an invalid date' do
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             key_id:  key.id,
+             name: 'newtoken',
+             scopes: ['api'],
+             expires_at: 'invalid-date'
+           }
+
+      expect(json_response['success']).to be_falsey
+      expect(json_response['message']).to eq("Invalid token expiry date: 'invalid-date'")
+    end
+
+    it 'returns an error message when it receives an invalid scope' do
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             key_id:  key.id,
+             name: 'newtoken',
+             scopes: %w(read_api badscope read_repository)
+           }
+
+      expect(json_response['success']).to be_falsey
+      expect(json_response['message']).to match(/\AInvalid scope: 'badscope'. Valid scopes are: /)
+    end
+
+    it 'returns a token without expiry when the expires_at parameter is missing' do
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             key_id:  key.id,
+             name: 'newtoken',
+             scopes: %w(read_api read_repository)
+           }
+
+      expect(json_response['success']).to be_truthy
+      expect(json_response['token']).to match(/\A\S{20}\z/)
+      expect(json_response['scopes']).to match_array(%w(read_api read_repository))
+      expect(json_response['expires_at']).to be_nil
+    end
+
+    it 'returns a token with expiry when it receives a valid expires_at parameter' do
+      post api('/internal/personal_access_token'),
+           params: {
+             secret_token: secret_token,
+             key_id:  key.id,
+             name: 'newtoken',
+             scopes: %w(read_api read_repository),
+             expires_at: '9001-11-17'
+           }
+
+      expect(json_response['success']).to be_truthy
+      expect(json_response['token']).to match(/\A\S{20}\z/)
+      expect(json_response['scopes']).to match_array(%w(read_api read_repository))
+      expect(json_response['expires_at']).to eq('9001-11-17')
+    end
+  end
+
   describe "POST /internal/lfs_authenticate" do
     before do
       project.add_developer(user)