This reverts commit 6658f87f.

This revert is a prerequisite for the later revert of commit
8f86a5b4.
Reported-by: Hans de Goede <hdegoede@redhat.com>
Acked-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>