This reverts commit b2d24404.

It's true that creating rxe on top of 802.1q interfaces doesn't work.
Thus, commit fd49ddaf ("RDMA/rxe: prevent rxe creation on top of vlan
interface") was absolutely correct.

But b2d24404 was incorrect assuming that with this change, RDMA and
VLAN don't work togehter at all. It just has to be set up
differently. Rather than creating rxe on top of the VLAN interface, rxe
must be created on top of the physical interface.  RDMA then works just
fine through VLAN interfaces on top of that physical interface, via the
"upper device" logic.

This is hard to see in the rxe logic because it never talks about vlan,
but instead rxe carefully selects upper vlan netdevices when working with
packets which in turn imply certain vlan tagging. This is all done
correctly and interacts with the gid table with VLAN support the same as
real HW does.

b2d24404 broke this setup deliberately and should thus be
reverted. Also, b2d24404 removed rxe_dma_device(), so adapt the revert
to discard that hunk.

Fixes: b2d24404 ("RDMA/rxe: Remove VLAN code leftovers from RXE")
Link: https://lore.kernel.org/r/20210120161913.7347-1-mwilck@suse.comSigned-off-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>