Remove the work queue for serial_remove; events always happen
in process context now so the work queue gains us nothing.