Don't leave any bridge or panel attached to a stale driver instance
when unbinding, to allow reattachment on a rebind.
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>