That trick is only needed if we end up with an error, so
there's no point in messing that outside of an error path.

In fact doing so causes problems when removing dwc3.ko,
problems which commit c5a1fbca (usb: dwc3: dwc3-omap: Fix
the crash on module removal) mistakenly tried to fix.
Signed-off-by: Felipe Balbi <balbi@ti.com>