• Alexei Starovoitov's avatar
    bpf: make programs see skb->data == L2 for ingress and egress · 3431205e
    Alexei Starovoitov authored
    eBPF programs attached to ingress and egress qdiscs see inconsistent skb->data.
    For ingress L2 header is already pulled, whereas for egress it's present.
    This is known to program writers which are currently forced to use
    BPF_LL_OFF workaround.
    Since programs don't change skb internal pointers it is safe to do
    pull/push right around invocation of the program and earlier taps and
    later pt->func() will not be affected.
    Multiple taps via packet_rcv(), tpacket_rcv() are doing the same trick
    around run_filter/BPF_PROG_RUN even if skb_shared.
    
    This fix finally allows programs to use optimized LD_ABS/IND instructions
    without BPF_LL_OFF for higher performance.
    tc ingress + cls_bpf + samples/bpf/tcbpf1_kern.o
           w/o JIT   w/JIT
    before  20.5     23.6 Mpps
    after   21.8     26.6 Mpps
    
    Old programs with BPF_LL_OFF will still work as-is.
    
    We can now undo most of the earlier workaround commit:
    a166151c ("bpf: fix bpf helpers to use skb->mac_header relative offsets")
    Signed-off-by: default avatarAlexei Starovoitov <ast@plumgrid.com>
    Acked-by: default avatarJamal Hadi Salim <jhs@mojatatu.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    3431205e
cls_bpf.c 10.8 KB