This reverts commit 8f86a5b4.

It has been established that this causes a boot regression on
both Baytrail and Cherrytrail SoCs, and we can't have that in
the final kernel release, so we need to revert it.
Reported-by: Hans de Goede <hdegoede@redhat.com>
Acked-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>