A number of things can currently go wrong after the XUSB controller has
been enabled, which means that it might need to be disabled again before
it has ever been used.

Avoid this by delaying runtime PM enablement until it's really required
right before registers are accessed for the first time.
Signed-off-by: Thierry Reding <treding@nvidia.com>
Link: https://lore.kernel.org/r/20191206140653.2085561-8-thierry.reding@gmail.comSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>