Avoid config.speed_hz is 0 when it is a divisor.

Signed-off-by: Han Xu <han.xu@nxp.com
Signed-off-by: Clark Wang <xiaoning.wang@nxp.com
Link: https://lore.kernel.org/r/20230505064159.3964473-1-xiaoning.wang@nxp.com
Signed-off-by: Mark Brown <broonie@kernel.org