Make sure we don't set grouping reference on lines that have a valid
group only for source (or destination) but not for the other side.