Ultimately we want everything to be going through the clock framework and
runtime pm, so kill off the per-port callbacks that enabled ports to
bypass the common infrastructure.
Signed-off-by: Paul Mundt <lethal@linux-sh.org>