It is unsafe thing to do (no locking, no reference counting etc).
Just remove module_exit() as it was done for IDE PCI drivers.