When device is stopped, there is no need to handle ISR.
Especially when otg switch to HOST mode.
Signed-off-by: Neil Zhang <zhangwm@marvell.com>
Signed-off-by: Felipe Balbi <balbi@ti.com>