Store the source/destination slave address separately into the channel
structure.  This moves us towards being able to avoid a configuration
call each time we use the channel.
Acked-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>