And initialize workers and notifiers as soon as possible.

This avoids a potential race if irqs are enabled and triggered too early
before the worker is properly set up.
Suggested-by: Grygorii Strashko <grygorii.strashko@ti.com>
Signed-off-by: H. Nikolaus Schaller <hns@goldelico.com>
Signed-off-by: Sebastian Reichel <sebastian.reichel@collabora.co.uk>