commit 35b1de55 ("rdma/cxgb4: Fixes cxgb4 probe failure in VM when PF is
exposed through PCI Passthrough") moved the code to check for SR-IOV PF[0..3]
much further down in init_one() past the point where we allocate a (struct
adapter) for PF[0..3]. As a result, we left four of these on ever module remove.

Fix: Allocate adapter structure only for PF4
Signed-off-by: Hariprasad Shenai <hariprasad@chelsio.com>
Signed-off-by: David S. Miller <davem@davemloft.net>