This reverts commit 01338bb8.

With clock improvements in place, this seems to be no longer
necessary. Set the CLRSIPO to default setting recommended by
manufacturer.
Reviewed-by: Alexander Stein <alexander.stein@ew.tq-group.com>
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Robert Foss <rfoss@kernel.org>
Link: https://patchwork.freedesktop.org/patch/msgid/20240625120334.145320-5-marex@denx.de