This reverts commit d45bb9c5.

Just got a report that this causes some suspend/resume issues,
so back it out and I'll investigate it later.
Reported-by: Mike Galbraith <efault@gmx.de>
Signed-off-by: Dave Airlie <airlied@redhat.com>