• Eric Dumazet's avatar
    net: reorganize sk_buff for faster __copy_skb_header() · b1937227
    Eric Dumazet authored
    With proliferation of bit fields in sk_buff, __copy_skb_header() became
    quite expensive, showing as the most expensive function in a GSO
    workload.
    
    __copy_skb_header() performance is also critical for non GSO TCP
    operations, as it is used from skb_clone()
    
    This patch carefully moves all the fields that were not copied in a
    separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
    
    Then I moved all other fields and all other copied fields in a section
    delimited by headers_start[0]/headers_end[0] section so that we
    can use a single memcpy() call, inlined by compiler using long
    word load/stores.
    
    I also tried to make all copies in the natural orders of sk_buff,
    to help hardware prefetching.
    
    I made sure sk_buff size did not change.
    Signed-off-by: default avatarEric Dumazet <edumazet@google.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    b1937227
skbuff.c 103 KB