The connector drivers need a handle to the source they are connected to
in order to control the source.

All drivers get that handle at probe time, resulting in probe deferral
when the source hasn't been probed yet. However they don't need the
handle until their connect handler is called.

Move retrieval of the source handle to the connect handler to avoid
probe deferrals.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Sebastian Reichel <sebastian.reichel@collabora.co.uk>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ti.com>