This allows users to be removed from issuables such as confidential
ones which they might otherwise have access to despite being removed
from the project.