Drop the paranoid port structure sanity checks which are confusing at
best.

The driver data port pointer is set at port probe and never cleared,
while USB serial core sets the tty driver data at install and won't call
any driver functions with a NULL port pointer.
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Johan Hovold <johan@kernel.org>