The current sub-module unload exit path is a mess, it tries
to abuse the idr. Just keep a list of devices per driver struct
and free them in-order on rmmod.
Signed-off-by: Dave Airlie <airlied@redhat.com>