We can not allow the worker to run after its fbdev, or even the module,
has been removed.

Fixes: cfe63423 ("drm/fb-helper: Add drm_fb_helper_set_suspend_unlocked()")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Noralf Trønnes <noralf@tronnes.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Jani Nikula <jani.nikula@linux.intel.com>
Cc: Sean Paul <seanpaul@chromium.org>
Cc: dri-devel@lists.freedesktop.org
Cc: <stable@vger.kernel.org> # v4.9+
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/20170207124956.14954-2-chris@chris-wilson.co.uk