• Eric Dumazet's avatar
    net: ipv4: fix RCU races on dst refcounts · 404e0a8b
    Eric Dumazet authored
    commit c6cffba4 (ipv4: Fix input route performance regression.)
    added various fatal races with dst refcounts.
    
    crashes happen on tcp workloads if routes are added/deleted at the same
    time.
    
    The dst_free() calls from free_fib_info_rcu() are clearly racy.
    
    We need instead regular dst refcounting (dst_release()) and make
    sure dst_release() is aware of RCU grace periods :
    
    Add DST_RCU_FREE flag so that dst_release() respects an RCU grace period
    before dst destruction for cached dst
    
    Introduce a new inet_sk_rx_dst_set() helper, using atomic_inc_not_zero()
    to make sure we dont increase a zero refcount (On a dst currently
    waiting an rcu grace period before destruction)
    
    rt_cache_route() must take a reference on the new cached route, and
    release it if was not able to install it.
    
    With this patch, my machines survive various benchmarks.
    Signed-off-by: default avatarEric Dumazet <edumazet@google.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    404e0a8b
tcp_input.c 174 KB