Squashed commit representing changes in gitlab-ce!12300.

- There were conflicting changes in `master` that were fixed in
  94258a65. This made rebasing the commits from gitlab-ce!12300
  problematic, due to conflicts.

- Instead, I squashed all !12300 commits into a single commit, and
  cherry-picked that onto 33580-fix-api-scoping-ee, which resulted
  in this commit.

Original commit messages below
==============================

Initial attempt at refactoring API scope declarations.

- Declaring an endpoint's scopes in a `before` block has proved to be
  unreliable. For example, if we're accessing the `API::Users` endpoint - code
  in a `before` block in `API::API` wouldn't be able to see the scopes set in
  `API::Users` since the `API::API` `before` block runs first.

- This commit moves these declarations to the class level, since they don't need
  to change once set.

Allow API scope declarations to be applied conditionally.

- Scope declarations of the form:

    allow_access_with_scope :read_user, if: -> (request) { request.get? }

  will only apply for `GET` requests

- Add a negative test to a `POST` endpoint in the `users` API to test this. Also
  test for this case in the `AccessTokenValidationService` unit tests.

Test `/users` endpoints for the `read_user` scope.

- Test `GET` endpoints to check that the scope is allowed.
- Test `POST` endpoints to check that the scope is disallowed.
- Test both `v3` and `v4` endpoints.

When verifying scopes, manually include scopes from `API::API`.

- They are not included automatically since `API::Users` does not inherit from
  `API::API`, as I initially assumed.

- Scopes declared in `API::API` are considered global (to the API), and need to
  be included in all cases.

Test OAuth token scope verification in the `API::Users` endpoint

Add CHANGELOG entry for CE MR 12300

Fix remaining spec failures for !12300.

1. Get the spec for `lib/gitlab/auth.rb` passing.

  - Make the `request` argument to `AccessTokenValidationService` optional -
  `auth.rb` doesn't need to pass in a request.

  - Pass in scopes in the format `[{ name: 'api' }]` rather than `['api']`, which
  is what `AccessTokenValidationService` now expects.

2. Get the spec for `API::V3::Users` passing

2. Get the spec for `AccessTokenValidationService` passing

Implement review comments from @dbalexandre for !12300.

Implement review comments from @DouweM for !12300.

- Use a struct for scopes, so we can call `scope.if` instead of `scope[:if]`

- Refactor the "remove scopes whose :if condition returns false" logic to use a
  `select` rather than a `reject`.

Extract a `Gitlab::Scope` class.

- To represent an authorization scope, such as `api` or `read_user`
- This is a better abstraction than the hash we were previously using.

`AccessTokenValidationService` accepts `String` or `API::Scope` scopes.

- There's no need to use `API::Scope` for scopes that don't have `if`
  conditions, such as in `lib/gitlab/auth.rb`.

Fix build for !12300.

- The `/users` and `/users/:id` APIs are now accessible without
  authentication (!12445), and so scopes are not relevant for these endpoints.

- Previously, we were testing our scope declaration against these two methods.
  This commit moves these tests to other `GET` user endpoints which still
  require authentication.

app/services/access_token_validation_service.rb

@@ -5,10 +5,11 @@ class AccessTokenValidationService
   REVOKED = :revoked
   INSUFFICIENT_SCOPE = :insufficient_scope
 
-  attr_reader :token
+  attr_reader :token, :request
 
-  def initialize(token)
+  def initialize(token, request: nil)
     @token = token
+    @request = request
   end
 
   def validate(scopes: [])
@@ -27,12 +28,23 @@ class AccessTokenValidationService
   end
 
   # True if the token's scope contains any of the passed scopes.
-  def include_any_scope?(scopes)
-    if scopes.blank?
+  def include_any_scope?(required_scopes)
+    if required_scopes.blank?
       true
     else
-      # Check whether the token is allowed access to any of the required scopes.
-      Set.new(scopes).intersection(Set.new(token.scopes)).present?
+      # We're comparing each required_scope against all token scopes, which would
+      # take quadratic time. This consideration is irrelevant here because of the
+      # small number of records involved.
+      # https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12300/#note_33689006
+      token_scopes = token.scopes.map(&:to_sym)
+
+      required_scopes.any? do |scope|
+        if scope.respond_to?(:sufficient?)
+          scope.sufficient?(token_scopes, request)
+        else
+          API::Scope.new(scope).sufficient?(token_scopes, request)
+        end
+      end
     end
   end
 end

changelogs/unreleased/33580-fix-api-scoping.yml

+---
+title: Fix API Scoping
+merge_request: 12300
+author:

lib/api/api.rb

@@ -2,6 +2,8 @@ module API
   class API < Grape::API
     include APIGuard
 
+    allow_access_with_scope :api
+
     version %w(v3 v4), using: :path
 
     version 'v3', using: :path do
@@ -48,7 +50,6 @@ module API
       mount ::API::V3::Variables
     end
 
-    before { allow_access_with_scope :api }
     before { header['X-Frame-Options'] = 'SAMEORIGIN' }
     before { Gitlab::I18n.locale = current_user&.preferred_language }
 

lib/api/api_guard.rb

@@ -23,6 +23,23 @@ module API
       install_error_responders(base)
     end
 
+    class_methods do
+      # Set the authorization scope(s) allowed for an API endpoint.
+      #
+      # A call to this method maps the given scope(s) to the current API
+      # endpoint class. If this method is called multiple times on the same class,
+      # the scopes are all aggregated.
+      def allow_access_with_scope(scopes, options = {})
+        Array(scopes).each do |scope|
+          allowed_scopes << Scope.new(scope, options)
+        end
+      end
+
+      def allowed_scopes
+        @scopes ||= []
+      end
+    end
+
     # Helper Methods for Grape Endpoint
     module HelperMethods
       # Invokes the doorkeeper guard.
@@ -47,7 +64,7 @@ module API
         access_token = find_access_token
         return nil unless access_token
 
-        case AccessTokenValidationService.new(access_token).validate(scopes: scopes)
+        case AccessTokenValidationService.new(access_token, request: request).validate(scopes: scopes)
         when AccessTokenValidationService::INSUFFICIENT_SCOPE
           raise InsufficientScopeError.new(scopes)
 
@@ -74,18 +91,6 @@ module API
         @current_user
       end
 
-      # Set the authorization scope(s) allowed for the current request.
-      #
-      # Note: A call to this method adds to any previous scopes in place. This is done because
-      # `Grape` callbacks run from the outside-in: the top-level callback (API::API) runs first, then
-      # the next-level callback (API::API::Users, for example) runs. All these scopes are valid for the
-      # given endpoint (GET `/api/users` is accessible by the `api` and `read_user` scopes), and so they
-      # need to be stored.
-      def allow_access_with_scope(*scopes)
-        @scopes ||= []
-        @scopes.concat(scopes.map(&:to_s))
-      end
-
       private
 
       def find_user_by_authentication_token(token_string)
@@ -96,7 +101,7 @@ module API
         access_token = PersonalAccessToken.active.find_by_token(token_string)
         return unless access_token
 
-        if AccessTokenValidationService.new(access_token).include_any_scope?(scopes)
+        if AccessTokenValidationService.new(access_token, request: request).include_any_scope?(scopes)
           User.find(access_token.user_id)
         end
       end

lib/api/helpers.rb

@@ -360,8 +360,8 @@ module API
     def initial_current_user
       return @initial_current_user if defined?(@initial_current_user)
       Gitlab::Auth::UniqueIpsLimiter.limit_user! do
-        @initial_current_user ||= find_user_by_private_token(scopes: @scopes)
-        @initial_current_user ||= doorkeeper_guard(scopes: @scopes)
+        @initial_current_user ||= find_user_by_private_token(scopes: scopes_registered_for_endpoint)
+        @initial_current_user ||= doorkeeper_guard(scopes: scopes_registered_for_endpoint)
         @initial_current_user ||= find_user_from_warden
 
         unless @initial_current_user && Gitlab::UserAccess.new(@initial_current_user).allowed?
@@ -429,5 +429,22 @@ module API
 
       exception.status == 500
     end
+
+    # An array of scopes that were registered (using `allow_access_with_scope`)
+    # for the current endpoint class. It also returns scopes registered on
+    # `API::API`, since these are meant to apply to all API routes.
+    def scopes_registered_for_endpoint
+      @scopes_registered_for_endpoint ||=
+        begin
+          endpoint_classes = [options[:for].presence, ::API::API].compact
+          endpoint_classes.reduce([]) do |memo, endpoint|
+            if endpoint.respond_to?(:allowed_scopes)
+              memo.concat(endpoint.allowed_scopes)
+            else
+              memo
+            end
+          end
+        end
+    end
   end
 end

lib/api/scope.rb

+# Encapsulate a scope used for authorization, such as `api`, or `read_user`
+module API
+  class Scope
+    attr_reader :name, :if
+
+    def initialize(name, options = {})
+      @name = name.to_sym
+      @if = options[:if]
+    end
+
+    # Are the `scopes` passed in sufficient to adequately authorize the passed
+    # request for the scope represented by the current instance of this class?
+    def sufficient?(scopes, request)
+      scopes.include?(self.name) && verify_if_condition(request)
+    end
+
+    private
+
+    def verify_if_condition(request)
+      self.if.nil? || self.if.call(request)
+    end
+  end
+end

lib/api/users.rb

 module API
   class Users < Grape::API
     include PaginationParams
+    include APIGuard
 
-    before do
-      allow_access_with_scope :read_user if request.get?
-    end
+    allow_access_with_scope :read_user, if: -> (request) { request.get? }
 
     resource :users, requirements: { uid: /[0-9]*/, id: /[0-9]*/ } do
       before do

lib/api/v3/users.rb

@@ -2,9 +2,11 @@ module API
   module V3
     class Users < Grape::API
       include PaginationParams
+      include APIGuard
+
+      allow_access_with_scope :read_user, if: -> (request) { request.get? }
 
       before do
-        allow_access_with_scope :read_user if request.get?
         authenticate!
       end
 

lib/gitlab/auth.rb

@@ -132,13 +132,13 @@ module Gitlab
 
         token = PersonalAccessTokensFinder.new(state: 'active').find_by(token: password)
 
-        if token && valid_scoped_token?(token, AVAILABLE_SCOPES.map(&:to_s))
+        if token && valid_scoped_token?(token, AVAILABLE_SCOPES)
           Gitlab::Auth::Result.new(token.user, nil, :personal_token, abilities_for_scope(token.scopes))
         end
       end
 
       def valid_oauth_token?(token)
-        token && token.accessible? && valid_scoped_token?(token, ["api"])
+        token && token.accessible? && valid_scoped_token?(token, [:api])
       end
 
       def valid_scoped_token?(token, scopes)

spec/requests/api/helpers_spec.rb

@@ -14,6 +14,10 @@ describe API::Helpers do
   let(:request) { Rack::Request.new(env) }
   let(:header) { }
 
+  before do
+    allow_any_instance_of(self.class).to receive(:options).and_return({})
+  end
+
   def set_env(user_or_token, identifier)
     clear_env
     clear_param
@@ -167,7 +171,6 @@ describe API::Helpers do
       it "returns nil for a token without the appropriate scope" do
         personal_access_token = create(:personal_access_token, user: user, scopes: ['read_user'])
         env[API::APIGuard::PRIVATE_TOKEN_HEADER] = personal_access_token.token
-        allow_access_with_scope('write_user')
 
         expect(current_user).to be_nil
       end

spec/requests/api/users_spec.rb

@@ -402,6 +402,14 @@ describe API::Users do
         expect(json_response['identities'].first['provider']).to eq('github')
       end
     end
+
+    context "scopes" do
+      let(:user) { admin }
+      let(:path) { '/users' }
+      let(:api_call) { method(:api) }
+
+      include_examples 'does not allow the "read_user" scope'
+    end
   end
 
   describe "GET /users/sign_up" do
@@ -918,6 +926,13 @@ describe API::Users do
         expect(response).to match_response_schema('public_api/v4/user/public')
         expect(json_response['id']).to eq(user.id)
       end
+
+      context "scopes" do
+        let(:path) { "/user" }
+        let(:api_call) { method(:api) }
+
+        include_examples 'allows the "read_user" scope'
+      end
     end
 
     context 'with admin' do
@@ -987,6 +1002,13 @@ describe API::Users do
         expect(json_response).to be_an Array
         expect(json_response.first["title"]).to eq(key.title)
       end
+
+      context "scopes" do
+        let(:path) { "/user/keys" }
+        let(:api_call) { method(:api) }
+
+        include_examples 'allows the "read_user" scope'
+      end
     end
   end
 
@@ -1020,6 +1042,13 @@ describe API::Users do
 
       expect(response).to have_http_status(404)
     end
+
+    context "scopes" do
+      let(:path) { "/user/keys/#{key.id}" }
+      let(:api_call) { method(:api) }
+
+      include_examples 'allows the "read_user" scope'
+    end
   end
 
   describe "POST /user/keys" do
@@ -1109,6 +1138,13 @@ describe API::Users do
         expect(json_response).to be_an Array
         expect(json_response.first["email"]).to eq(email.email)
       end
+
+      context "scopes" do
+        let(:path) { "/user/emails" }
+        let(:api_call) { method(:api) }
+
+        include_examples 'allows the "read_user" scope'
+      end
     end
   end
 
@@ -1141,6 +1177,13 @@ describe API::Users do
 
       expect(response).to have_http_status(404)
     end
+
+    context "scopes" do
+      let(:path) { "/user/emails/#{email.id}" }
+      let(:api_call) { method(:api) }
+
+      include_examples 'allows the "read_user" scope'
+    end
   end
 
   describe "POST /user/emails" do

spec/requests/api/v3/users_spec.rb

@@ -67,6 +67,19 @@ describe API::V3::Users do
         expect(json_response.first['title']).to eq(key.title)
       end
     end
+
+    context "scopes" do
+      let(:user) { admin }
+      let(:path) { "/users/#{user.id}/keys" }
+      let(:api_call) { method(:v3_api) }
+
+      before do
+        user.keys << key
+        user.save
+      end
+
+      include_examples 'allows the "read_user" scope'
+    end
   end
 
   describe 'GET /user/:id/emails' do
@@ -287,7 +300,7 @@ describe API::V3::Users do
     end
 
     it 'returns a 404 error if not found' do
-      get v3_api('/users/42/events', user)
+      get v3_api('/users/420/events', user)
 
       expect(response).to have_http_status(404)
       expect(json_response['message']).to eq('404 User Not Found')
@@ -312,5 +325,13 @@ describe API::V3::Users do
 
       expect(json_response['is_admin']).to be_nil
     end
+
+    context "scopes" do
+      let(:user) { admin }
+      let(:path) { '/users' }
+      let(:api_call) { method(:v3_api) }
+
+      include_examples 'does not allow the "read_user" scope'
+    end
   end
 end

spec/services/access_token_validation_service_spec.rb

@@ -2,40 +2,71 @@ require 'spec_helper'
 
 describe AccessTokenValidationService, services: true do
   describe ".include_any_scope?" do
+    let(:request) { double("request") }
+
     it "returns true if the required scope is present in the token's scopes" do
       token = double("token", scopes: [:api, :read_user])
+      scopes = [:api]
 
-      expect(described_class.new(token).include_any_scope?([:api])).to be(true)
+      expect(described_class.new(token, request: request).include_any_scope?(scopes)).to be(true)
     end
 
     it "returns true if more than one of the required scopes is present in the token's scopes" do
       token = double("token", scopes: [:api, :read_user, :other_scope])
+      scopes = [:api, :other_scope]
 
-      expect(described_class.new(token).include_any_scope?([:api, :other_scope])).to be(true)
+      expect(described_class.new(token, request: request).include_any_scope?(scopes)).to be(true)
     end
 
     it "returns true if the list of required scopes is an exact match for the token's scopes" do
       token = double("token", scopes: [:api, :read_user, :other_scope])
+      scopes = [:api, :read_user, :other_scope]
 
-      expect(described_class.new(token).include_any_scope?([:api, :read_user, :other_scope])).to be(true)
+      expect(described_class.new(token, request: request).include_any_scope?(scopes)).to be(true)
     end
 
     it "returns true if the list of required scopes contains all of the token's scopes, in addition to others" do
       token = double("token", scopes: [:api, :read_user])
+      scopes = [:api, :read_user, :other_scope]
 
-      expect(described_class.new(token).include_any_scope?([:api, :read_user, :other_scope])).to be(true)
+      expect(described_class.new(token, request: request).include_any_scope?(scopes)).to be(true)
     end
 
     it 'returns true if the list of required scopes is blank' do
       token = double("token", scopes: [])
+      scopes = []
 
-      expect(described_class.new(token).include_any_scope?([])).to be(true)
+      expect(described_class.new(token, request: request).include_any_scope?(scopes)).to be(true)
     end
 
     it "returns false if there are no scopes in common between the required scopes and the token scopes" do
       token = double("token", scopes: [:api, :read_user])
+      scopes = [:other_scope]
+
+      expect(described_class.new(token, request: request).include_any_scope?(scopes)).to be(false)
+    end
+
+    context "conditions" do
+      it "ignores any scopes whose `if` condition returns false" do
+        token = double("token", scopes: [:api, :read_user])
+        scopes = [API::Scope.new(:api, if: ->(_) { false })]
+
+        expect(described_class.new(token, request: request).include_any_scope?(scopes)).to be(false)
+      end
+
+      it "does not ignore scopes whose `if` condition is not set" do
+        token = double("token", scopes: [:api, :read_user])
+        scopes = [API::Scope.new(:api, if: ->(_) { false }), :read_user]
+
+        expect(described_class.new(token, request: request).include_any_scope?(scopes)).to be(true)
+      end
 
-      expect(described_class.new(token).include_any_scope?([:other_scope])).to be(false)
+      it "does not ignore scopes whose `if` condition returns true" do
+        token = double("token", scopes: [:api, :read_user])
+        scopes = [API::Scope.new(:api, if: ->(_) { true }), API::Scope.new(:read_user, if: ->(_) { false })]
+
+        expect(described_class.new(token, request: request).include_any_scope?(scopes)).to be(true)
+      end
     end
   end
 end

spec/support/api/scopes/read_user_shared_examples.rb

+shared_examples_for 'allows the "read_user" scope' do
+  context 'for personal access tokens' do
+    context 'when the requesting token has the "api" scope' do
+      let(:token) { create(:personal_access_token, scopes: ['api'], user: user) }
+
+      it 'returns a "200" response' do
+        get api_call.call(path, user, personal_access_token: token)
+
+        expect(response).to have_http_status(200)
+      end
+    end
+
+    context 'when the requesting token has the "read_user" scope' do
+      let(:token) { create(:personal_access_token, scopes: ['read_user'], user: user) }
+
+      it 'returns a "200" response' do
+        get api_call.call(path, user, personal_access_token: token)
+
+        expect(response).to have_http_status(200)
+      end
+    end
+
+    context 'when the requesting token does not have any required scope' do
+      let(:token) { create(:personal_access_token, scopes: ['read_registry'], user: user) }
+
+      it 'returns a "401" response' do
+        get api_call.call(path, user, personal_access_token: token)
+
+        expect(response).to have_http_status(401)
+      end
+    end
+  end
+
+  context 'for doorkeeper (OAuth) tokens' do
+    let!(:application) { Doorkeeper::Application.create!(name: "MyApp", redirect_uri: "https://app.com", owner: user) }
+
+    context 'when the requesting token has the "api" scope' do
+      let!(:token) { Doorkeeper::AccessToken.create! application_id: application.id, resource_owner_id: user.id, scopes: "api" }
+
+      it 'returns a "200" response' do
+        get api_call.call(path, user, oauth_access_token: token)
+
+        expect(response).to have_http_status(200)
+      end
+    end
+
+    context 'when the requesting token has the "read_user" scope' do
+      let!(:token) { Doorkeeper::AccessToken.create! application_id: application.id, resource_owner_id: user.id, scopes: "read_user" }
+
+      it 'returns a "200" response' do
+        get api_call.call(path, user, oauth_access_token: token)
+
+        expect(response).to have_http_status(200)
+      end
+    end
+
+    context 'when the requesting token does not have any required scope' do
+      let!(:token) { Doorkeeper::AccessToken.create! application_id: application.id, resource_owner_id: user.id, scopes: "invalid" }
+
+      it 'returns a "403" response' do
+        get api_call.call(path, user, oauth_access_token: token)
+
+        expect(response).to have_http_status(403)
+      end
+    end
+  end
+end
+
+shared_examples_for 'does not allow the "read_user" scope' do
+  context 'when the requesting token has the "read_user" scope' do
+    let(:token) { create(:personal_access_token, scopes: ['read_user'], user: user) }
+
+    it 'returns a "401" response' do
+      post api_call.call(path, user, personal_access_token: token), attributes_for(:user, projects_limit: 3)
+
+      expect(response).to have_http_status(401)
+    end
+  end
+end

spec/support/api_helpers.rb

@@ -17,14 +17,18 @@ module ApiHelpers
   #   => "/api/v2/issues?foo=bar&private_token=..."
   #
   # Returns the relative path to the requested API resource
-  def api(path, user = nil, version: API::API.version)
+  def api(path, user = nil, version: API::API.version, personal_access_token: nil, oauth_access_token: nil)
     "/api/#{version}#{path}" +
 
       # Normalize query string
       (path.index('?') ? '' : '?') +
 
+      if personal_access_token.present?
+        "&private_token=#{personal_access_token.token}"
+      elsif oauth_access_token.present?
+        "&access_token=#{oauth_access_token.token}"
       # Append private_token if given a User object
-      if user.respond_to?(:private_token)
+      elsif user.respond_to?(:private_token)
         "&private_token=#{user.private_token}"
       else
         ''
@@ -32,8 +36,14 @@ module ApiHelpers
   end
 
   # Temporary helper method for simplifying V3 exclusive API specs
-  def v3_api(path, user = nil)
-    api(path, user, version: 'v3')
+  def v3_api(path, user = nil, personal_access_token: nil, oauth_access_token: nil)
+    api(
+      path,
+      user,
+      version: 'v3',
+      personal_access_token: personal_access_token,
+      oauth_access_token: oauth_access_token
+    )
   end
 
   def ci_api(path, user = nil)