Avoids wasting cycles at boot specially on slower simulators

Signed-off-by: Vineet Gupta <vgupta@synopsys.com>
Cc: Michal Marek <mmarek@suse.cz>
Cc: Francois Bedard <fbedard@synopsys.com>
Cc: linux-kernel@vger.kernel.org
Acked-by: David Daney <david.daney@cavium.com>