On 1 and 2 bytes per word, the transfer of the 3 last bytes will access
memory outside tx_ptr.

Although this has not trigger any error on real hardware, we should
better fix this.

Fixes: 24ba5e59 (Remove rx_fn and tx_fn pointer)
Signed-off-by: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>
Signed-off-by: Mark Brown <broonie@kernel.org>