Do not remove the device from the IOMMU while the driver is
still attached.
Signed-off-by: Joerg Roedel <jroedel@suse.de>