Avoid ending up with a higher frequency than requested
Signed-off-by: Matthias Seidel <kernel@mseidel.net>
Signed-off-by: Mark Brown <broonie@kernel.org>