Removing this dependency also allows us to remove a transitive
dependency on gitlab_grit - which is the whole point of this exercise.

I don't think we can EOL gitlab_grit until it's removed as a dependency
from gitaly-ruby, but this at least gets it out of gitlab-ce.