When the card has been added to the device model, it might be bound
to a card driver. Therefore, we have to release the host lock when
trying to remove it as we otherwise might deadlock with the driver.
Signed-off-by: Pierre Ossman <drzeus@drzeus.cx>