Improve performance of toggling diff view type

Closes #48666 and #48733

See merge request gitlab-org/gitlab-ce!20278