This is to prevent inconsistent state that prevents a resync
from happening as seen in https://gitlab.com/gitlab-com/migration/issues/408.