There's an underlying race condition with the unjoin_work() call that is
sometimes triggered depending on scheduling order and the phase of the
moon.  This doesn't fix the race condition, but it does remove the
ill-advised BUG_ON() call in an easily-recoverable situation.
Signed-off-by: Solomon Peachy <pizza@shaftnet.org>
Signed-off-by: John W. Linville <linville@tuxdriver.com>