It is waiting udelay(50) x 1024 (= 50ms) for status check
in worst case, but it is overkill.
And we shouldn't use udelay() for 50us
(linux/Documentation/timers/timers-howto.txt)

Waiting maximum udelay(5) x 1024 (= 5ms) is very enough
for status check.
This patch fixes these issue.
Reported-by: Hiromitsu Yamasaki <hiromitsu.yamasaki.ym@renesas.com>
Signed-off-by: Kuninori Morimoto <kuninori.morimoto.gx@renesas.com>
Signed-off-by: Mark Brown <broonie@kernel.org>