This just takes a reference on the connector when we set a mode
in the non-atomic paths.

v2: Follow Daniel Stone's suggestions on when to take/drop
references.
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>