Because of workqueue delay, the put_device could be called before
device_del, so move it to del_conn.

Signed-off-by: Dave Young <hidave.darkstar@gmail.com> 
Signed-off-by: David S. Miller <davem@davemloft.net>