This reverts commit 7c86d44c.
Stephen says:

IIRC, I proposed it before solely to solve some suspend/resume ordering
issues, and Colin Cross NAKd it. These days, deferred probe should make
this change unnecessary.
Reported-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Wolfram Sang <w.sang@pengutronix.de>