Since IMX8MM_CLK_USB_CORE_REF is not used at all, so remove the setting
for it.
Signed-off-by: Li Jun <jun.li@nxp.com>
Signed-off-by: Shawn Guo <shawnguo@kernel.org>