The delayed worker isn't needed and is racey. Remove it and do
the state change in line.
Reviewed-by: Yakir Yang <ykk@rock-chips.com>
Tested-by: Yakir Yang <ykk@rock-chips.com>
Signed-off-by: Sean Paul <seanpaul@chromium.org>