Otherwise we try to sleep with preemption disabled, etc.

Noticed by Thomas Gleixner.
Signed-off-by: David S. Miller <davem@davemloft.net>