Otherwise, the system will hang if USB driver try to access
portsc register.

Cc: André Draszik <git@andred.net>
Signed-off-by: Peter Chen <peter.chen@nxp.com>
Signed-off-by: Shawn Guo <shawnguo@kernel.org>