After configuration accesses have been completed deassert the
configuration access enable cleanly.
Signed-off-by: Paul Mundt <lethal@linux-sh.org>