1. 08 Aug, 2023 38 commits
  2. 07 Aug, 2023 2 commits
    • Jakub Kicinski's avatar
      Merge branch 'page_pool-a-couple-of-assorted-optimizations' · 66244337
      Jakub Kicinski authored
      Alexander Lobakin says:
      
      ====================
      page_pool: a couple of assorted optimizations
      
      That initially was a spin-off of the IAVF PP series[0], but has grown
      (and shrunk) since then a bunch. In fact, it consists of three
      semi-independent blocks:
      
      * #1-2: Compile-time optimization. Split page_pool.h into 2 headers to
        not overbloat the consumers not needing complex inline helpers and
        then stop including it in skbuff.h at all. The first patch is also
        prereq for the whole series.
      * #3: Improve cacheline locality for users of the Page Pool frag API.
      * #4-6: Use direct cache recycling more aggressively, when it is safe
        obviously. In addition, make sure nobody wants to use Page Pool API
        with disabled interrupts.
      
      Patches #1 and #5 are authored by Yunsheng and Jakub respectively, with
      small modifications from my side as per ML discussions.
      For the perf numbers for #3-6, please see individual commit messages.
      
      Also available on my GH with many more Page Pool goodies[1].
      
      [0] https://lore.kernel.org/netdev/20230530150035.1943669-1-aleksander.lobakin@intel.com
      [1] https://github.com/alobakin/linux/commits/iavf-pp-frag
      ====================
      
      Link: https://lore.kernel.org/r/20230804180529.2483231-1-aleksander.lobakin@intel.comSigned-off-by: default avatarJakub Kicinski <kuba@kernel.org>
      66244337
    • Alexander Lobakin's avatar
      net: skbuff: always try to recycle PP pages directly when in softirq · 4a36d018
      Alexander Lobakin authored
      Commit 8c48eea3 ("page_pool: allow caching from safely localized
      NAPI") allowed direct recycling of skb pages to their PP for some cases,
      but unfortunately missed a couple of other majors.
      For example, %XDP_DROP in skb mode. The netstack just calls kfree_skb(),
      which unconditionally passes `false` as @napi_safe. Thus, all pages go
      through ptr_ring and locks, although most of time we're actually inside
      the NAPI polling this PP is linked with, so that it would be perfectly
      safe to recycle pages directly.
      Let's address such. If @napi_safe is true, we're fine, don't change
      anything for this path. But if it's false, check whether we are in the
      softirq context. It will most likely be so and then if ->list_owner
      is our current CPU, we're good to use direct recycling, even though
      @napi_safe is false -- concurrent access is excluded. in_softirq()
      protection is needed mostly due to we can hit this place in the
      process context (not the hardirq though).
      For the mentioned xdp-drop-skb-mode case, the improvement I got is
      3-4% in Mpps. As for page_pool stats, recycle_ring is now 0 and
      alloc_slow counter doesn't change most of time, which means the
      MM layer is not even called to allocate any new pages.
      
      Suggested-by: Jakub Kicinski <kuba@kernel.org> # in_softirq()
      Signed-off-by: default avatarAlexander Lobakin <aleksander.lobakin@intel.com>
      Reviewed-by: default avatarAlexander Duyck <alexanderduyck@fb.com>
      Link: https://lore.kernel.org/r/20230804180529.2483231-7-aleksander.lobakin@intel.comSigned-off-by: default avatarJakub Kicinski <kuba@kernel.org>
      4a36d018