If this code isn't triggering this assert by now, it never will, so just
remove it, it's pointless.

Cc: Duncan Sands <duncan.sands@free.fr>
Acked-by: Felipe Balbi <balbi@ti.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>