Right now the host is only unregistered when the driver is used via the
bridge api and not via the component api, leading to the host staying
registered in cases like probe deferral.

So move the host unregister to the general remove function, so that it
gets cleaned up in all cases.
Signed-off-by: Heiko Stuebner <heiko@sntech.de>
Reviewed-by: Andrzej Hajda <a.hajda@samsung.com>
Reviewed-by: Philippe Cornu <philippe.cornu@st.com>
Tested-by: Philippe Cornu <philippe.cornu@st.com>
Signed-off-by: Andrzej Hajda <a.hajda@samsung.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20181001123845.11818-2-heiko@sntech.de