Skip to content

L
linux
Commit 59d0f481 authored by Paolo Abeni's avatar Paolo Abeni
Browse files
Options

Merge branch 'improve-gbeth-performance-on-renesas-rz-g2l-and-related-socs' 

Paul Barker says:

====================
Improve GbEth performance on Renesas RZ/G2L and related SoCs

This series aims to improve performance of the GbEth IP in the Renesas
RZ/G2L SoC family and the RZ/G3S SoC, which use the ravb driver. Along
the way, we do some refactoring and ensure that napi_complete_done() is
used in accordance with the NAPI documentation for both GbEth and R-Car
code paths.

Much of the performance improvement comes from enabling SW IRQ
Coalescing for all SoCs using the GbEth IP, and NAPI Threaded mode for
single core SoCs using the GbEth IP. These can be enabled/disabled at
runtime via sysfs, but our goal is to set sensible defaults which get
good performance on the affected SoCs.

The rest of the performance improvement comes from using a page pool to
allocate RX buffers, and reducing the allocation size from >8kB to 2kB.

The overall performance impact of this patch series seen in testing with
iperf3 is as follows (see patches 5-7 for more detailed results):
  * RZ/G2L:
    * TCP TX: +1.8% bandwidth
    * TCP RX: +1% bandwidth at 47% less CPU load
    * UDP RX: +1% bandwidth at 26% less CPU load

  * RZ/G2UL:
    * TCP TX: +37% bandwidth
    * TCP RX: +43% bandwidth
    * UDP TX: -8% bandwidth
    * UDP RX: +32500% bandwidth (!)

  * RZ/G3S:
    * TCP TX: +25% bandwidth
    * TCP RX: +76% bandwidth
    * UDP TX: -9% bandwidth
    * UDP RX: +37900% bandwidth (!)

  * RZ/Five:
    * TCP TX: +18% bandwidth
    * TCP RX: +212% bandwidth
    * UDP TX: +2% bandwidth
    * UDP RX: +inf bandwidth (test no longer crashes)

There is no significant impact on bandwidth or CPU load in testing on
RZ/G2H or R-Car M3N.

Fixing the crash in UDP RX testing for RZ/Five is a cumulative effect of
patches 1, 2, 5 & 6 so this is very difficult to break out as a bugfix
for backporting.

Changes v4->v5:
  * Added Sergey's Reviewed-by tags.
  * Improved the commit message for patch 2/7.
  * Re-wrapped to 80 cols, except where this would significantly impact
    readability.
  * Use lower case `skb` consistently in comments.
  * Included <net/page_pool/types.h> in ravb.h.
  * Moved rx_buffer_size so it is in the same place in ravb_hw_info as
    rx_max_desc_use was previously.
  * Used reverse xmas tree ordering in variable declarations.
  * Split lines after binary operators, instead of before.
  * Factor subtraction of sizeof(__sum16) out of the if condition in
    ravb_rx_csum_gbeth().
  * Add blank lines after variable declarations where needed.
  * Used goto instead of break to handle napi_build_skb() failure in
    ravb_rx_gbeth(). Break was incorrectly scoped to the surrounding
    switch statement, when it's the outer loop we really want to break
    out of.
  * Used continue instead of break to handle NULL priv->rx_1st_skb in
    ravb_rx_gbeth() as we may still be able to process further
    descriptors.
  * Unconditionally set priv->rx_1st_skb = NULL after processing a
    packet in ravb_rx_gbeth(). We don't need to check die_dt as this
    will be a no-op for single descriptor packets.
  * Moved napi_build_skb() call after dma_sync_single_for_cpu() in
    ravb_rx_rcar() to align the order of operations with ravb_rx_gbeth()
    and ensure the data is sync'd before it is accessed.
  * Moved zeroing of rx_buff->page to the end of packet processing in
    ravb_rx_rcar() to align the order of operations with
    ravb_rx_gbeth().

Changes v3->v4:
  * Dependency patches have merged so this is no longer an RFC.
  * Fixed update of stats->rx_packets.
  * Simplified refactoring following feedback from Niklas and Sergey.
  * Renamed needs_irq_coalesce -> coalesce_irqs.
  * Used a separate page pool for each RX queue.
  * Passed struct ravb_rx_desc to ravb_alloc_rx_buffer() so that we can
    simplify the calling function.
  * Explained the calculation of rx_desc->ds_cc.
  * Added handling of nonlinear SKBs in ravb_rx_csum_gbeth().
  * Used Niklas' suggested commit message for patch 2/7.
  * Added Sergey's Reviewed-by tags to patches 5/7 and 6/7.

Changes v2->v3:
  * Incorporated feedback on RFC v2 from Sergey.
  * Split out bugfixes and rebased. This changed the order of what was
    the first 5 patches of v2 and things look a little different so I've
    not picked up Reviewed-by tags from v2.
  * Further refactoring and tidy up of RX ring refill and
    ravb_rx_gbeth().
  * Switched to using a page pool to allocate RX buffers.
  * Re-tested and provided updated performance figures.

Changes v1->v2:
  * Marked as RFC as the series depends on unmerged patches.
  * Refactored R-Car code paths as well as GbEth code paths.
  * Updated references to the patches this series depends on.
====================

Link: https://lore.kernel.org/r/20240604072825.7490-1-paul.barker.ct@bp.renesas.comSigned-off-by: default avatarPaolo Abeni <pabeni@redhat.com>
parents 7da375e2 96672632
Hide whitespace changes
Inline Side-by-side
Showing with 270 additions and 221 deletions
drivers/net/ethernet/renesas/ravb.h
View file @ 59d0f481
......@@ -19,6 +19,7 @@
#include <linux/phy.h>
#include <linux/platform_device.h>
#include <linux/ptp_clock_kernel.h>
#include <net/page_pool/types.h>
#define BE_TX_RING_SIZE 64 /* TX ring size for Best Effort */
#define BE_RX_RING_SIZE 1024 /* RX ring size for Best Effort */
......@@ -1039,7 +1040,7 @@ struct ravb_ptp {
};
struct ravb_hw_info {
bool (*receive)(struct net_device *ndev, int *quota, int q);
int (*receive)(struct net_device *ndev, int budget, int q);
void (*set_rate)(struct net_device *ndev);
int (*set_feature)(struct net_device *ndev, netdev_features_t features);
int (*dmac_init)(struct net_device *ndev);
......@@ -1051,9 +1052,10 @@ struct ravb_hw_info {
int stats_len;
u32 tccr_mask;
u32 rx_max_frame_size;
u32 rx_max_desc_use;
u32 rx_buffer_size;
u32 rx_desc_size;
unsigned aligned_tx: 1;
unsigned coalesce_irqs:1; /* Needs software IRQ coalescing */
/* hardware features */
unsigned internal_delay:1; /* AVB-DMAC has internal delays */
......@@ -1070,6 +1072,11 @@ struct ravb_hw_info {
unsigned half_duplex:1; /* E-MAC supports half duplex mode */
};
struct ravb_rx_buffer {
struct page *page;
unsigned int offset;
};
struct ravb_private {
struct net_device *ndev;
struct platform_device *pdev;
......@@ -1093,7 +1100,8 @@ struct ravb_private {
struct ravb_tx_desc *tx_ring[NUM_TX_QUEUE];
void *tx_align[NUM_TX_QUEUE];
struct sk_buff *rx_1st_skb;
struct sk_buff **rx_skb[NUM_RX_QUEUE];
struct page_pool *rx_pool[NUM_RX_QUEUE];
struct ravb_rx_buffer *rx_buffers[NUM_RX_QUEUE];
struct sk_buff **tx_skb[NUM_TX_QUEUE];
u32 rx_over_errors;
u32 rx_fifo_errors;
......
drivers/net/ethernet/renesas/ravb_main.c
View file @ 59d0f481
......@@ -30,6 +30,7 @@
#include <linux/reset.h>
#include <linux/math64.h>
#include <net/ip.h>
#include <net/page_pool/helpers.h>
#include "ravb.h"
......@@ -113,25 +114,6 @@ static void ravb_set_rate_rcar(struct net_device *ndev)
}
}
static struct sk_buff *
ravb_alloc_skb(struct net_device *ndev, const struct ravb_hw_info *info,
gfp_t gfp_mask)
{
struct sk_buff *skb;
u32 reserve;
skb = __netdev_alloc_skb(ndev, info->rx_max_frame_size + RAVB_ALIGN - 1,
gfp_mask);
if (!skb)
return NULL;
reserve = (unsigned long)skb->data & (RAVB_ALIGN - 1);
if (reserve)
skb_reserve(skb, RAVB_ALIGN - reserve);
return skb;
}
/* Get MAC address from the MAC address registers
*
* Ethernet AVB device doesn't have ROM for MAC address.
......@@ -257,21 +239,10 @@ static void ravb_rx_ring_free(struct net_device *ndev, int q)
{
struct ravb_private *priv = netdev_priv(ndev);
unsigned int ring_size;
unsigned int i;
if (!priv->rx_ring[q].raw)
return;
for (i = 0; i < priv->num_rx_ring[q]; i++) {
struct ravb_rx_desc *desc = ravb_rx_get_desc(priv, q, i);
if (!dma_mapping_error(ndev->dev.parent,
le32_to_cpu(desc->dptr)))
dma_unmap_single(ndev->dev.parent,
le32_to_cpu(desc->dptr),
priv->info->rx_max_frame_size,
DMA_FROM_DEVICE);
}
ring_size = priv->info->rx_desc_size * (priv->num_rx_ring[q] + 1);
dma_free_coherent(ndev->dev.parent, ring_size, priv->rx_ring[q].raw,
priv->rx_desc_dma[q]);
......@@ -298,13 +269,16 @@ static void ravb_ring_free(struct net_device *ndev, int q)
priv->tx_ring[q] = NULL;
}
/* Free RX skb ringbuffer */
if (priv->rx_skb[q]) {
for (i = 0; i < priv->num_rx_ring[q]; i++)
dev_kfree_skb(priv->rx_skb[q][i]);
/* Free RX buffers */
for (i = 0; i < priv->num_rx_ring[q]; i++) {
if (priv->rx_buffers[q][i].page)
page_pool_put_page(priv->rx_pool[q],
priv->rx_buffers[q][i].page,
0, true);
}
kfree(priv->rx_skb[q]);
priv->rx_skb[q] = NULL;
kfree(priv->rx_buffers[q]);
priv->rx_buffers[q] = NULL;
page_pool_destroy(priv->rx_pool[q]);
/* Free aligned TX buffers */
kfree(priv->tx_align[q]);
......@@ -317,35 +291,64 @@ static void ravb_ring_free(struct net_device *ndev, int q)
priv->tx_skb[q] = NULL;
}
static void ravb_rx_ring_format(struct net_device *ndev, int q)
static int
ravb_alloc_rx_buffer(struct net_device *ndev, int q, u32 entry, gfp_t gfp_mask,
struct ravb_rx_desc *rx_desc)
{
struct ravb_private *priv = netdev_priv(ndev);
struct ravb_rx_desc *rx_desc;
unsigned int rx_ring_size;
const struct ravb_hw_info *info = priv->info;
struct ravb_rx_buffer *rx_buff;
dma_addr_t dma_addr;
unsigned int i;
unsigned int size;
rx_ring_size = priv->info->rx_desc_size * priv->num_rx_ring[q];
memset(priv->rx_ring[q].raw, 0, rx_ring_size);
/* Build RX ring buffer */
for (i = 0; i < priv->num_rx_ring[q]; i++) {
/* RX descriptor */
rx_desc = ravb_rx_get_desc(priv, q, i);
rx_desc->ds_cc = cpu_to_le16(priv->info->rx_max_desc_use);
dma_addr = dma_map_single(ndev->dev.parent, priv->rx_skb[q][i]->data,
priv->info->rx_max_frame_size,
DMA_FROM_DEVICE);
rx_buff = &priv->rx_buffers[q][entry];
size = info->rx_buffer_size;
rx_buff->page = page_pool_alloc(priv->rx_pool[q], &rx_buff->offset,
&size, gfp_mask);
if (unlikely(!rx_buff->page)) {
/* We just set the data size to 0 for a failed mapping which
* should prevent DMA from happening...
*/
if (dma_mapping_error(ndev->dev.parent, dma_addr))
rx_desc->ds_cc = cpu_to_le16(0);
rx_desc->dptr = cpu_to_le32(dma_addr);
rx_desc->ds_cc = cpu_to_le16(0);
return -ENOMEM;
}
dma_addr = page_pool_get_dma_addr(rx_buff->page) + rx_buff->offset;
dma_sync_single_for_device(ndev->dev.parent, dma_addr,
info->rx_buffer_size, DMA_FROM_DEVICE);
rx_desc->dptr = cpu_to_le32(dma_addr);
/* The end of the RX buffer is used to store skb shared data, so we need
* to ensure that the hardware leaves enough space for this.
*/
rx_desc->ds_cc = cpu_to_le16(info->rx_buffer_size -
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) -
ETH_FCS_LEN + sizeof(__sum16));
return 0;
}
static u32
ravb_rx_ring_refill(struct net_device *ndev, int q, u32 count, gfp_t gfp_mask)
{
struct ravb_private *priv = netdev_priv(ndev);
struct ravb_rx_desc *rx_desc;
u32 i, entry;
for (i = 0; i < count; i++) {
entry = (priv->dirty_rx[q] + i) % priv->num_rx_ring[q];
rx_desc = ravb_rx_get_desc(priv, q, entry);
if (!priv->rx_buffers[q][entry].page) {
if (unlikely(ravb_alloc_rx_buffer(ndev, q, entry,
gfp_mask, rx_desc)))
break;
}
/* Descriptor type must be set after all the above writes */
dma_wmb();
rx_desc->die_dt = DT_FEMPTY;
}
rx_desc = ravb_rx_get_desc(priv, q, i);
rx_desc->dptr = cpu_to_le32((u32)priv->rx_desc_dma[q]);
rx_desc->die_dt = DT_LINKFIX; /* type */
return i;
}
/* Format skb and descriptor buffer for Ethernet AVB */
......@@ -353,6 +356,7 @@ static void ravb_ring_format(struct net_device *ndev, int q)
{
struct ravb_private *priv = netdev_priv(ndev);
unsigned int num_tx_desc = priv->num_tx_desc;
struct ravb_rx_desc *rx_desc;
struct ravb_tx_desc *tx_desc;
struct ravb_desc *desc;
unsigned int tx_ring_size = sizeof(*tx_desc) * priv->num_tx_ring[q] *
......@@ -364,7 +368,13 @@ static void ravb_ring_format(struct net_device *ndev, int q)
priv->dirty_rx[q] = 0;
priv->dirty_tx[q] = 0;
ravb_rx_ring_format(ndev, q);
/* Regular RX descriptors have already been initialized by
* ravb_rx_ring_refill(), we just need to initialize the final link
* descriptor.
*/
rx_desc = ravb_rx_get_desc(priv, q, priv->num_rx_ring[q]);
rx_desc->dptr = cpu_to_le32((u32)priv->rx_desc_dma[q]);
rx_desc->die_dt = DT_LINKFIX; /* type */
memset(priv->tx_ring[q], 0, tx_ring_size);
/* Build TX ring buffer */
......@@ -408,26 +418,47 @@ static void *ravb_alloc_rx_desc(struct net_device *ndev, int q)
static int ravb_ring_init(struct net_device *ndev, int q)
{
struct ravb_private *priv = netdev_priv(ndev);
const struct ravb_hw_info *info = priv->info;
unsigned int num_tx_desc = priv->num_tx_desc;
struct page_pool_params params = {
.order = 0,
.flags = PP_FLAG_DMA_MAP,
.pool_size = priv->num_rx_ring[q],
.nid = NUMA_NO_NODE,
.dev = ndev->dev.parent,
.dma_dir = DMA_FROM_DEVICE,
};
unsigned int ring_size;
struct sk_buff *skb;
unsigned int i;
u32 num_filled;
/* Allocate RX page pool and buffers */
priv->rx_pool[q] = page_pool_create(&params);
if (IS_ERR(priv->rx_pool[q]))
goto error;
/* Allocate RX buffers */
priv->rx_buffers[q] = kcalloc(priv->num_rx_ring[q],
sizeof(*priv->rx_buffers[q]), GFP_KERNEL);
if (!priv->rx_buffers[q])
goto error;
/* Allocate RX and TX skb rings */
priv->rx_skb[q] = kcalloc(priv->num_rx_ring[q],
sizeof(*priv->rx_skb[q]), GFP_KERNEL);
/* Allocate TX skb rings */
priv->tx_skb[q] = kcalloc(priv->num_tx_ring[q],
sizeof(*priv->tx_skb[q]), GFP_KERNEL);
if (!priv->rx_skb[q] || !priv->tx_skb[q])
if (!priv->tx_skb[q])
goto error;
for (i = 0; i < priv->num_rx_ring[q]; i++) {
skb = ravb_alloc_skb(ndev, info, GFP_KERNEL);
if (!skb)
goto error;
priv->rx_skb[q][i] = skb;
}
/* Allocate all RX descriptors. */
if (!ravb_alloc_rx_desc(ndev, q))
goto error;
/* Populate RX ring buffer. */
priv->dirty_rx[q] = 0;
ring_size = priv->info->rx_desc_size * priv->num_rx_ring[q];
memset(priv->rx_ring[q].raw, 0, ring_size);
num_filled = ravb_rx_ring_refill(ndev, q, priv->num_rx_ring[q],
GFP_KERNEL);
if (num_filled != priv->num_rx_ring[q])
goto error;
if (num_tx_desc > 1) {
/* Allocate rings for the aligned buffers */
......@@ -437,12 +468,6 @@ static int ravb_ring_init(struct net_device *ndev, int q)
goto error;
}
/* Allocate all RX descriptors. */
if (!ravb_alloc_rx_desc(ndev, q))
goto error;
priv->dirty_rx[q] = 0;
/* Allocate all TX descriptors. */
ring_size = sizeof(struct ravb_tx_desc) *
(priv->num_tx_ring[q] * num_tx_desc + 1);
......@@ -706,7 +731,9 @@ static void ravb_get_tx_tstamp(struct net_device *ndev)
static void ravb_rx_csum_gbeth(struct sk_buff *skb)
{
struct skb_shared_info *shinfo = skb_shinfo(skb);
__wsum csum_ip_hdr, csum_proto;
skb_frag_t *last_frag;
u8 *hw_csum;
/* The hardware checksum status is contained in sizeof(__sum16) * 2 = 4
......@@ -716,12 +743,24 @@ static void ravb_rx_csum_gbeth(struct sk_buff *skb)
if (unlikely(skb->len < sizeof(__sum16) * 2))
return;
hw_csum = skb_tail_pointer(skb) - sizeof(__sum16);
if (skb_is_nonlinear(skb)) {
last_frag = &shinfo->frags[shinfo->nr_frags - 1];
hw_csum = skb_frag_address(last_frag) +
skb_frag_size(last_frag);
} else {
hw_csum = skb_tail_pointer(skb);
}
hw_csum -= sizeof(__sum16);
csum_proto = csum_unfold((__force __sum16)get_unaligned_le16(hw_csum));
hw_csum -= sizeof(__sum16);
csum_ip_hdr = csum_unfold((__force __sum16)get_unaligned_le16(hw_csum));
skb_trim(skb, skb->len - 2 * sizeof(__sum16));
if (skb_is_nonlinear(skb))
skb_frag_size_sub(last_frag, 2 * sizeof(__sum16));
else
skb_trim(skb, skb->len - 2 * sizeof(__sum16));
/* TODO: IPV6 Rx checksum */
if (skb->protocol == htons(ETH_P_IP) && !csum_ip_hdr && !csum_proto)
......@@ -743,30 +782,14 @@ static void ravb_rx_csum(struct sk_buff *skb)
skb_trim(skb, skb->len - sizeof(__sum16));
}
static struct sk_buff *ravb_get_skb_gbeth(struct net_device *ndev, int entry,
struct ravb_rx_desc *desc)
{
struct ravb_private *priv = netdev_priv(ndev);
struct sk_buff *skb;
skb = priv->rx_skb[RAVB_BE][entry];
priv->rx_skb[RAVB_BE][entry] = NULL;
dma_unmap_single(ndev->dev.parent, le32_to_cpu(desc->dptr),
ALIGN(priv->info->rx_max_frame_size, 16),
DMA_FROM_DEVICE);
return skb;
}
/* Packet receive function for Gigabit Ethernet */
static bool ravb_rx_gbeth(struct net_device *ndev, int *quota, int q)
static int ravb_rx_gbeth(struct net_device *ndev, int budget, int q)
{
struct ravb_private *priv = netdev_priv(ndev);
const struct ravb_hw_info *info = priv->info;
struct net_device_stats *stats;
struct ravb_rx_desc *desc;
struct sk_buff *skb;
dma_addr_t dma_addr;
int rx_packets = 0;
u8 desc_status;
u16 desc_len;
......@@ -781,7 +804,7 @@ static bool ravb_rx_gbeth(struct net_device *ndev, int *quota, int q)
for (i = 0; i < limit; i++, priv->cur_rx[q]++) {
entry = priv->cur_rx[q] % priv->num_rx_ring[q];
desc = &priv->rx_ring[q].desc[entry];
if (rx_packets == *quota || desc->die_dt == DT_FEMPTY)
if (rx_packets == budget || desc->die_dt == DT_FEMPTY)
break;
/* Descriptor type must be checked before all other reads */
......@@ -807,87 +830,110 @@ static bool ravb_rx_gbeth(struct net_device *ndev, int *quota, int q)
if (desc_status & MSC_CEEF)
stats->rx_missed_errors++;
} else {
struct ravb_rx_buffer *rx_buff;
void *rx_addr;
rx_buff = &priv->rx_buffers[q][entry];
rx_addr = page_address(rx_buff->page) + rx_buff->offset;
die_dt = desc->die_dt & 0xF0;
dma_sync_single_for_cpu(ndev->dev.parent,
le32_to_cpu(desc->dptr),
desc_len, DMA_FROM_DEVICE);
switch (die_dt) {
case DT_FSINGLE:
skb = ravb_get_skb_gbeth(ndev, entry, desc);
skb_put(skb, desc_len);
skb->protocol = eth_type_trans(skb, ndev);
if (ndev->features & NETIF_F_RXCSUM)
ravb_rx_csum_gbeth(skb);
napi_gro_receive(&priv->napi[q], skb);
rx_packets++;
stats->rx_bytes += desc_len;
break;
case DT_FSTART:
priv->rx_1st_skb = ravb_get_skb_gbeth(ndev, entry, desc);
skb_put(priv->rx_1st_skb, desc_len);
/* Start of packet: Set initial data length. */
skb = napi_build_skb(rx_addr,
info->rx_buffer_size);
if (unlikely(!skb)) {
stats->rx_errors++;
page_pool_put_page(priv->rx_pool[q],
rx_buff->page, 0,
true);
goto refill;
}
skb_mark_for_recycle(skb);
skb_put(skb, desc_len);
/* Save this skb if the packet spans multiple
* descriptors.
*/
if (die_dt == DT_FSTART)
priv->rx_1st_skb = skb;
break;
case DT_FMID:
skb = ravb_get_skb_gbeth(ndev, entry, desc);
skb_copy_to_linear_data_offset(priv->rx_1st_skb,
priv->rx_1st_skb->len,
skb->data,
desc_len);
skb_put(priv->rx_1st_skb, desc_len);
dev_kfree_skb(skb);
break;
case DT_FEND:
skb = ravb_get_skb_gbeth(ndev, entry, desc);
skb_copy_to_linear_data_offset(priv->rx_1st_skb,
priv->rx_1st_skb->len,
skb->data,
desc_len);
skb_put(priv->rx_1st_skb, desc_len);
dev_kfree_skb(skb);
priv->rx_1st_skb->protocol =
eth_type_trans(priv->rx_1st_skb, ndev);
/* Continuing a packet: Add this buffer as an RX
* frag.
*/
/* rx_1st_skb will be NULL if napi_build_skb()
* failed for the first descriptor of a
* multi-descriptor packet.
*/
if (unlikely(!priv->rx_1st_skb)) {
stats->rx_errors++;
page_pool_put_page(priv->rx_pool[q],
rx_buff->page, 0,
true);
/* We may find a DT_FSINGLE or DT_FSTART
* descriptor in the queue which we can
* process, so don't give up yet.
*/
continue;
}
skb_add_rx_frag(priv->rx_1st_skb,
skb_shinfo(priv->rx_1st_skb)->nr_frags,
rx_buff->page, rx_buff->offset,
desc_len, info->rx_buffer_size);
/* Set skb to point at the whole packet so that
* we only need one code path for finishing a
* packet.
*/
skb = priv->rx_1st_skb;
}
switch (die_dt) {
case DT_FSINGLE:
case DT_FEND:
/* Finishing a packet: Determine protocol &
* checksum, hand off to NAPI and update our
* stats.
*/
skb->protocol = eth_type_trans(skb, ndev);
if (ndev->features & NETIF_F_RXCSUM)
ravb_rx_csum_gbeth(priv->rx_1st_skb);
stats->rx_bytes += priv->rx_1st_skb->len;
napi_gro_receive(&priv->napi[q],
priv->rx_1st_skb);
ravb_rx_csum_gbeth(skb);
stats->rx_bytes += skb->len;
napi_gro_receive(&priv->napi[q], skb);
rx_packets++;
break;
/* Clear rx_1st_skb so that it will only be
* non-NULL when valid.
*/
priv->rx_1st_skb = NULL;
}
/* Mark this RX buffer as consumed. */
rx_buff->page = NULL;
}
}
refill:
/* Refill the RX ring buffers. */
for (; priv->cur_rx[q] - priv->dirty_rx[q] > 0; priv->dirty_rx[q]++) {
entry = priv->dirty_rx[q] % priv->num_rx_ring[q];
desc = &priv->rx_ring[q].desc[entry];
desc->ds_cc = cpu_to_le16(priv->info->rx_max_desc_use);
if (!priv->rx_skb[q][entry]) {
skb = ravb_alloc_skb(ndev, info, GFP_ATOMIC);
if (!skb)
break;
dma_addr = dma_map_single(ndev->dev.parent,
skb->data,
priv->info->rx_max_frame_size,
DMA_FROM_DEVICE);
skb_checksum_none_assert(skb);
/* We just set the data size to 0 for a failed mapping
* which should prevent DMA from happening...
*/
if (dma_mapping_error(ndev->dev.parent, dma_addr))
desc->ds_cc = cpu_to_le16(0);
desc->dptr = cpu_to_le32(dma_addr);
priv->rx_skb[q][entry] = skb;
}
/* Descriptor type must be set after all the above writes */
dma_wmb();
desc->die_dt = DT_FEMPTY;
}
priv->dirty_rx[q] += ravb_rx_ring_refill(ndev, q,
priv->cur_rx[q] - priv->dirty_rx[q],
GFP_ATOMIC);
stats->rx_packets += rx_packets;
*quota -= rx_packets;
return *quota == 0;
return rx_packets;
}
/* Packet receive function for Ethernet AVB */
static bool ravb_rx_rcar(struct net_device *ndev, int *quota, int q)
static int ravb_rx_rcar(struct net_device *ndev, int budget, int q)
{
struct ravb_private *priv = netdev_priv(ndev);
const struct ravb_hw_info *info = priv->info;
......@@ -895,7 +941,6 @@ static bool ravb_rx_rcar(struct net_device *ndev, int *quota, int q)
struct ravb_ex_rx_desc *desc;
unsigned int limit, i;
struct sk_buff *skb;
dma_addr_t dma_addr;
struct timespec64 ts;
int rx_packets = 0;
u8 desc_status;
......@@ -906,7 +951,7 @@ static bool ravb_rx_rcar(struct net_device *ndev, int *quota, int q)
for (i = 0; i < limit; i++, priv->cur_rx[q]++) {
entry = priv->cur_rx[q] % priv->num_rx_ring[q];
desc = &priv->rx_ring[q].ex_desc[entry];
if (rx_packets == *quota || desc->die_dt == DT_FEMPTY)
if (rx_packets == budget || desc->die_dt == DT_FEMPTY)
break;
/* Descriptor type must be checked before all other reads */
......@@ -934,12 +979,23 @@ static bool ravb_rx_rcar(struct net_device *ndev, int *quota, int q)
stats->rx_missed_errors++;
} else {
u32 get_ts = priv->tstamp_rx_ctrl & RAVB_RXTSTAMP_TYPE;
skb = priv->rx_skb[q][entry];
priv->rx_skb[q][entry] = NULL;
dma_unmap_single(ndev->dev.parent, le32_to_cpu(desc->dptr),
priv->info->rx_max_frame_size,
DMA_FROM_DEVICE);
struct ravb_rx_buffer *rx_buff;
void *rx_addr;
rx_buff = &priv->rx_buffers[q][entry];
rx_addr = page_address(rx_buff->page) + rx_buff->offset;
dma_sync_single_for_cpu(ndev->dev.parent,
le32_to_cpu(desc->dptr),
pkt_len, DMA_FROM_DEVICE);
skb = napi_build_skb(rx_addr, info->rx_buffer_size);
if (unlikely(!skb)) {
stats->rx_errors++;
page_pool_put_page(priv->rx_pool[q],
rx_buff->page, 0, true);
break;
}
skb_mark_for_recycle(skb);
get_ts &= (q == RAVB_NC) ?
RAVB_RXTSTAMP_TYPE_V2_L2_EVENT :
~RAVB_RXTSTAMP_TYPE_V2_L2_EVENT;
......@@ -961,48 +1017,28 @@ static bool ravb_rx_rcar(struct net_device *ndev, int *quota, int q)
napi_gro_receive(&priv->napi[q], skb);
rx_packets++;
stats->rx_bytes += pkt_len;
/* Mark this RX buffer as consumed. */
rx_buff->page = NULL;
}
}
/* Refill the RX ring buffers. */
for (; priv->cur_rx[q] - priv->dirty_rx[q] > 0; priv->dirty_rx[q]++) {
entry = priv->dirty_rx[q] % priv->num_rx_ring[q];
desc = &priv->rx_ring[q].ex_desc[entry];
desc->ds_cc = cpu_to_le16(priv->info->rx_max_desc_use);
if (!priv->rx_skb[q][entry]) {
skb = ravb_alloc_skb(ndev, info, GFP_ATOMIC);
if (!skb)
break; /* Better luck next round. */
dma_addr = dma_map_single(ndev->dev.parent, skb->data,
priv->info->rx_max_frame_size,
DMA_FROM_DEVICE);
skb_checksum_none_assert(skb);
/* We just set the data size to 0 for a failed mapping
* which should prevent DMA from happening...
*/
if (dma_mapping_error(ndev->dev.parent, dma_addr))
desc->ds_cc = cpu_to_le16(0);
desc->dptr = cpu_to_le32(dma_addr);
priv->rx_skb[q][entry] = skb;
}
/* Descriptor type must be set after all the above writes */
dma_wmb();
desc->die_dt = DT_FEMPTY;
}
priv->dirty_rx[q] += ravb_rx_ring_refill(ndev, q,
priv->cur_rx[q] - priv->dirty_rx[q],
GFP_ATOMIC);
stats->rx_packets += rx_packets;
*quota -= rx_packets;
return *quota == 0;
return rx_packets;
}
/* Packet receive function for Ethernet AVB */
static bool ravb_rx(struct net_device *ndev, int *quota, int q)
static int ravb_rx(struct net_device *ndev, int budget, int q)
{
struct ravb_private *priv = netdev_priv(ndev);
const struct ravb_hw_info *info = priv->info;
return info->receive(ndev, quota, q);
return info->receive(ndev, budget, q);
}
static void ravb_rcv_snd_disable(struct net_device *ndev)
......@@ -1319,13 +1355,12 @@ static int ravb_poll(struct napi_struct *napi, int budget)
unsigned long flags;
int q = napi - priv->napi;
int mask = BIT(q);
int quota = budget;
bool unmask;
int work_done;
/* Processing RX Descriptor Ring */
/* Clear RX interrupt */
ravb_write(ndev, ~(mask | RIS0_RESERVED), RIS0);
unmask = !ravb_rx(ndev, &quota, q);
work_done = ravb_rx(ndev, budget, q);
/* Processing TX Descriptor Ring */
spin_lock_irqsave(&priv->lock, flags);
......@@ -1344,24 +1379,20 @@ static int ravb_poll(struct napi_struct *napi, int budget)
if (priv->rx_fifo_errors != ndev->stats.rx_fifo_errors)
ndev->stats.rx_fifo_errors = priv->rx_fifo_errors;
if (!unmask)
goto out;
napi_complete(napi);
/* Re-enable RX/TX interrupts */
spin_lock_irqsave(&priv->lock, flags);
if (!info->irq_en_dis) {
ravb_modify(ndev, RIC0, mask, mask);
ravb_modify(ndev, TIC, mask, mask);
} else {
ravb_write(ndev, mask, RIE0);
ravb_write(ndev, mask, TIE);
if (work_done < budget && napi_complete_done(napi, work_done)) {
/* Re-enable RX/TX interrupts */
spin_lock_irqsave(&priv->lock, flags);
if (!info->irq_en_dis) {
ravb_modify(ndev, RIC0, mask, mask);
ravb_modify(ndev, TIC, mask, mask);
} else {
ravb_write(ndev, mask, RIE0);
ravb_write(ndev, mask, TIE);
}
spin_unlock_irqrestore(&priv->lock, flags);
}
spin_unlock_irqrestore(&priv->lock, flags);
out:
return budget - quota;
return work_done;
}
static void ravb_set_duplex_gbeth(struct net_device *ndev)
......@@ -2634,7 +2665,8 @@ static const struct ravb_hw_info ravb_gen3_hw_info = {
.stats_len = ARRAY_SIZE(ravb_gstrings_stats),
.tccr_mask = TCCR_TSRQ0 | TCCR_TSRQ1 | TCCR_TSRQ2 | TCCR_TSRQ3,
.rx_max_frame_size = SZ_2K,
.rx_max_desc_use = SZ_2K - ETH_FCS_LEN + sizeof(__sum16),
.rx_buffer_size = SZ_2K +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
.rx_desc_size = sizeof(struct ravb_ex_rx_desc),
.internal_delay = 1,
.tx_counters = 1,
......@@ -2658,7 +2690,8 @@ static const struct ravb_hw_info ravb_gen2_hw_info = {
.stats_len = ARRAY_SIZE(ravb_gstrings_stats),
.tccr_mask = TCCR_TSRQ0 | TCCR_TSRQ1 | TCCR_TSRQ2 | TCCR_TSRQ3,
.rx_max_frame_size = SZ_2K,
.rx_max_desc_use = SZ_2K - ETH_FCS_LEN + sizeof(__sum16),
.rx_buffer_size = SZ_2K +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
.rx_desc_size = sizeof(struct ravb_ex_rx_desc),
.aligned_tx = 1,
.gptp = 1,
......@@ -2679,7 +2712,8 @@ static const struct ravb_hw_info ravb_rzv2m_hw_info = {
.stats_len = ARRAY_SIZE(ravb_gstrings_stats),
.tccr_mask = TCCR_TSRQ0 | TCCR_TSRQ1 | TCCR_TSRQ2 | TCCR_TSRQ3,
.rx_max_frame_size = SZ_2K,
.rx_max_desc_use = SZ_2K - ETH_FCS_LEN + sizeof(__sum16),
.rx_buffer_size = SZ_2K +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
.rx_desc_size = sizeof(struct ravb_ex_rx_desc),
.multi_irqs = 1,
.err_mgmt_irqs = 1,
......@@ -2702,9 +2736,10 @@ static const struct ravb_hw_info gbeth_hw_info = {
.stats_len = ARRAY_SIZE(ravb_gstrings_stats_gbeth),
.tccr_mask = TCCR_TSRQ0,
.rx_max_frame_size = SZ_8K,
.rx_max_desc_use = 4080,
.rx_buffer_size = SZ_2K,
.rx_desc_size = sizeof(struct ravb_rx_desc),
.aligned_tx = 1,
.coalesce_irqs = 1,
.tx_counters = 1,
.carrier_counters = 1,
.half_duplex = 1,
......@@ -2981,6 +3016,12 @@ static int ravb_probe(struct platform_device *pdev)
if (info->nc_queues)
netif_napi_add(ndev, &priv->napi[RAVB_NC], ravb_poll);
if (info->coalesce_irqs) {
netdev_sw_irq_coalesce_default_on(ndev);
if (num_present_cpus() == 1)
dev_set_threaded(ndev, true);
}
/* Network device register */
error = register_netdev(ndev);
if (error)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7