Patch from Tom Rini.

If we don't change the divisor, we don't want to change what we claim
as the uart clock either.  Without this I don't get a usable serial
console on my Motorola Sandpoint.