If the FLL is being shut down we will exit early so there is no need to
check here and in fact we're checking the wrong thing anyway.
Reported-by: Graeme Gregory <gg@slimlogic.co.uk>
Signed-off-by: Mark Brown <broonie@opensource.wolfsonmicro.com>