This reverts commit 9dbaab56.

Turns out it was a bad idea and was fixed up "properly" in 4.7 but those
patches are too big to put into 4.6, so let's just revert it for now.
Reported-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Peter Frühberger <peter.fruehberger@gmail.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Jani Nikula <jani.nikula@intel.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>