It doesn't need to enable/disable L1 on the fly for EP0 transfer,
we only need to enable L1 after SET_CONFIGURATION.

This code may be introduced by careless.

Cc: Pawel Laszczak <pawell@cadence.com>
Signed-off-by: Peter Chen <peter.chen@nxp.com>
Signed-off-by: Felipe Balbi <balbi@kernel.org>