This reverts commit 4c875409.

This revert is a prerequisite for the later revert of commit
8f86a5b4.
Reported-by: Hans de Goede <hdegoede@redhat.com>
Acked-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>