commit 81a60b7f upstream.

we don't to gate clocks until our children are
done with their remove path.

Fixes: af310e96 (usb: dwc3: omap: use runtime API's to enable clocks)
Signed-off-by: Felipe Balbi <balbi@ti.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>