This can be slow because it must potentially remove a lot of remote
refs (if the remote being removed has a lot of refs)