We want commands to finish ASAP, so let's remove
that udelay() call.
Signed-off-by: Felipe Balbi <felipe.balbi@linux.intel.com>