We should free any hwspinlocks when we destroy the regmap, do so.
Signed-off-by: Mark Brown <broonie@kernel.org>