We can have valid but very low clocks in accessory detection modes.
Signed-off-by: Mark Brown <broonie@opensource.wolfsonmicro.com>