Keep the runtime active during the remove operation, and disable
related clocks.
Signed-off-by: Peter Chen <peter.chen@nxp.com>