rtrs_sess is in fact a path. This makes it confusing and difficult to get
into the code. So let's rename the structure and related use cases of it.

Coccinelle was used to do the transformation for most of the occurrences
and remaining ones were handled manually.

Link: https://lore.kernel.org/r/20220105180708.7774-2-jinpu.wang@ionos.comSigned-off-by: Vaishali Thakkar <vaishali.thakkar@ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@ionos.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

HIP09 EQ does not support level 2 addressing.

Link: https://lore.kernel.org/r/20211231101341.45759-3-liangwenpeng@huawei.comSigned-off-by: Wenpeng Liang <liangwenpeng@huawei.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

The following changes were made:
1. Align function signatures to 80 characters per line.
2. Remove tabs for variable assignment and use 1 space instead.
3. Don't compare to NULL in "if" clause.
4. Remove strange indentations.

This will ease on the maintenance of the driver for the future.

Link: https://lore.kernel.org/r/20211215135721.3662-7-mgurtovoy@nvidia.comSigned-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

The void pointer can be typecasted to/from any type.

Link: https://lore.kernel.org/r/20211215135721.3662-6-mgurtovoy@nvidia.comSigned-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

In order to complete a scsi command and guarantee that the HCA will never
perform an access violation when retrying a send operation we must
complete a scsi request only when both send and receive completions has
arrived. This is a preparation commit that remove the send completions
suppression. Next step will be taking care of the local invalidation
mechanism and adding a reference counter for commands.  Currently, we
don't do anything upon getting the send completion and just "consume" it.

Link: https://lore.kernel.org/r/20211215135721.3662-5-mgurtovoy@nvidia.comSigned-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Reviewed-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Use more common name for return values ("ret"). This commit doesn't change
any logic.

Link: https://lore.kernel.org/r/20211215135721.3662-4-mgurtovoy@nvidia.comSigned-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Reviewed-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Some users complain about RNR errors on the target, when heavy
high-priority tasks run on the initiator. After the investigation, we
found out that the receive WRs were exhausted, because the initiator could
not post them on time.

Receive work reqeusts are posted in chunks to reduce the number of hits to
the HCA. The WRs are posted in the receive completion handler when the
number of free receive buffers reaches the threshold. But on a high-loaded
host, receive CQEs processing can be delayed and all receive WRs will be
exhausted. In this case, the target will get an RNR error.

To avoid this, we post receive WR, as soon as possible and not in a
batch. This increases the number of hits to the HCA, but also the common
implementation in most of Linux ULPs (e.g. NVMe-oF/RDMA). As a rule of
thumb, performance improvements and heuristics are being added to the RDMA
core layer or vendors low level drivers and it's about time to align iSER
as well.

Link: https://lore.kernel.org/r/20211215135721.3662-3-mgurtovoy@nvidia.comSigned-off-by: Sergey Gorenko <sergeygo@nvidia.com>
Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Reviewed-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

No need for this dead code. This commit doesn't change any functionality
since one can still run "modprobe ib_iser pi_guard=<type>".

Link: https://lore.kernel.org/r/20211215135721.3662-2-mgurtovoy@nvidia.comSigned-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Under MAD query port, Report NDR speed when NDR is supported in the port
capability mask.

Link: https://lore.kernel.org/r/a2ab630d2a634547db9b581faa9d65da2edb9d05.1639554831.git.leonro@nvidia.comSigned-off-by: Maher Sanalla <msanalla@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

The API for ib_query_qp requires the driver to set cur_qp_state on return,
add the missing set.

Fixes: 67bbc055 ("RDMA/cxgb4: Add query_qp support")
Link: https://lore.kernel.org/r/20211220152530.60399-1-kamalheib1@gmail.comSigned-off-by: Kamal Heib <kamalheib1@gmail.com>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

HIP06 is no longer supported. In order to reduce unnecessary maintenance,
the code of HIP06 is removed.

Link: https://lore.kernel.org/r/20211220130558.61585-1-liangwenpeng@huawei.comSigned-off-by: Chengchang Tang <tangchengchang@huawei.com>
Signed-off-by: Wenpeng Liang <liangwenpeng@huawei.com>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Starting from the commit 66920e1b ("rdma_rxe: Use netlink messages
to add/delete links") from the 2019, the RXE modules parameters are marked
as deprecated in favour of rdmatool. So remove the kernel code too.

Link: https://lore.kernel.org/r/c8376d7517aebe7cc851f0baaeef7b13707cf767.1641372460.git.leonro@nvidia.comSigned-off-by: Leon Romanovsky <leonro@nvidia.com>
Reviewed-by: Zhu Yanjun <zyjzyj2000@gmail.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Several drivers used same function to initialize query MAD,
so move that function to global header file.

Link: https://lore.kernel.org/r/af6f35c590ff5ef56d0137351b8b295af0f7c13c.1641369858.git.leonro@nvidia.comSigned-off-by: Leon Romanovsky <leonro@nvidia.com>
Reviewed-by: Håkon Bugge <haakon.bugge@oracle.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

* Fix these up to always have the '+', and '|' on the continuing line
  which is the normal kernel style.
* Fix indentations correspondingly

NOTE: this patch also remove the 2 redundant plus in
IB_OPCODE_RD_FETCH_ADD and IB_OPCODE_RD_COMPARE_SWAP

Link: https://lore.kernel.org/r/20220105042605.14343-1-lizhijian@fujitsu.comSigned-off-by: Li Zhijian <lizhijian@fujitsu.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

There are currently 2 ways to create a set of sysfs files for a kobj_type,
through the default_attrs field, and the default_groups field.  Move the
IB code to use default_groups field which has been the preferred way since
commit aa30f47c ("kobject: Add support for default attribute groups to
kobj_type") so that we can soon get rid of the obsolete default_attrs
field.

Link: https://lore.kernel.org/r/20220103152259.531034-1-gregkh@linuxfoundation.orgSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

mlx5_handle_error_cqe() only dump the content of the CQE which is raw hex
data, and not straighforward for debug.  Print WC status message when we
got CQE error and dump is need.

Here is an example of how the dmesg log looks like with this:

 infiniband mlx5_0: mlx5_handle_error_cqe:333:(pid 0): WC error: 10, message: remote access error
 infiniband mlx5_0: dump_cqe:272:(pid 0): dump error cqe
 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00000030: 00 00 00 00 00 00 88 13 08 03 61 b3 1e a1 42 d3

Link: https://lore.kernel.org/r/20211227123806.47530-1-dust.li@linux.alibaba.comSigned-off-by: Dust Li <dust.li@linux.alibaba.com>
Acked-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

There is a redundant ']' in the name of opcode IB_OPCODE_RC_SEND_MIDDLE,
so just fix it.

Fixes: 8700e3e7 ("Soft RoCE driver")
Link: https://lore.kernel.org/r/20211218112320.3558770-1-cgxu519@mykernel.netSigned-off-by: Chengguang Xu <cgxu519@mykernel.net>
Acked-by: Zhu Yanjun <zyjzyj2000@gmail.com>
Reviewed-by: Bob Pearson <rpearsonhpe@gmail.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

The member variable xmit_errors can be replaced with

 rxe_counter_inc(rxe, RXE_CNT_SEND_ERR)

Link: https://lore.kernel.org/r/20211216054842.1099428-1-yanjun.zhu@linux.devSigned-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Return value directly instead of taking this in another redundant
variable.

Link: https://lore.kernel.org/20211215075258.442930-1-chi.minghao@zte.com.cnReported-by: Zeal Robot <zealci@zte.com.cn>
Signed-off-by: Minghao Chi <chi.minghao@zte.com.cn>
Acked-by: Zhu Yanjun <zyjzyj2000@gmail.com>
Reviewed-by: Devesh Sharma <Devesh.s.sharma@oracle.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Return status directly from function called.

Link: https://lore.kernel.org/r/20211215055421.441375-1-chi.minghao@zte.com.cnReported-by: Zeal Robot <zealci@zte.com.cn>
Signed-off-by: Minghao Chi <chi.minghao@zte.com.cn>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Saeed Mahameed says:
====================
Currently, the driver ignores the user's priority for flow steering rules
in FDB namespace. Change it and create the rule in the right priority.

It will allow to create FDB steering rules in up to 16 different
priorities.
====================
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

* mellanox/mlx5-next:
  RDMA/mlx5: Add support to multiple priorities for FDB rules
  net/mlx5: Create more priorities for FDB bypass namespace
  net/mlx5: Refactor mlx5_get_flow_namespace
  net/mlx5: Separate FDB namespace

In preparation for FORTIFY_SOURCE performing compile-time and run-time
field bounds checking for memset(), avoid intentionally writing across
neighboring fields.

Use memset_startat() so memset() doesn't get confused about writing beyond
the destination member that is intended to be the starting point of
zeroing through the end of the struct.

Link: https://lore.kernel.org/r/20211213223331.135412-15-keescook@chromium.orgSigned-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

In preparation for FORTIFY_SOURCE performing compile-time and run-time
field bounds checking for memset(), avoid intentionally writing across
neighboring fields.

Use memset_startat() so memset() doesn't get confused about writing beyond
the destination member that is intended to be the starting point of
zeroing through the end of the struct. Additionally, since everything
appears to perform a roundup (including allocation), just change the size
of the struct itself and add a build-time check to validate the expected
size.

Link: https://lore.kernel.org/r/20211213223331.135412-13-keescook@chromium.orgSigned-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

In preparation for FORTIFY_SOURCE performing compile-time and run-time
field bounds checking for memset(), avoid intentionally writing across
neighboring fields.

Use memset_after() to zero the end of struct mlx5_ib_mr that should be
initialized.

Link: https://lore.kernel.org/r/20211213223331.135412-10-keescook@chromium.orgAcked-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Required due to dependencies in following patches.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Currently, when cma_resolve_ib_dev() searches for a matching GID it will
stop searching after encountering the first empty GID table entry. This
behavior is wrong since neither IB nor RoCE spec enforce tightly packed
GID tables.

For example, when the matching valid GID entry exists at index N, and if a
GID entry is empty at index N-1, cma_resolve_ib_dev() will fail to find
the matching valid entry.

Fix it by making cma_resolve_ib_dev() continue searching even after
encountering missing entries.

Fixes: f17df3b0 ("RDMA/cma: Add support for AF_IB to rdma_resolve_addr()")
Link: https://lore.kernel.org/r/b7346307e3bb396c43d67d924348c6c496493991.1639055490.git.leonro@nvidia.comSigned-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Mark Zhang <markzhang@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Currently, ib_find_gid() will stop searching after encountering the first
empty GID table entry. This behavior is wrong since neither IB nor RoCE
spec enforce tightly packed GID tables.

For example, when a valid GID entry exists at index N, and if a GID entry
is empty at index N-1, ib_find_gid() will fail to find the valid entry.

Fix it by making ib_find_gid() continue searching even after encountering
missing entries.

Fixes: 5eb620c8 ("IB/core: Add helpers for uncached GID and P_Key searches")
Link: https://lore.kernel.org/r/e55d331b96cecfc2cf19803d16e7109ea966882d.1639055490.git.leonro@nvidia.comSigned-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Mark Zhang <markzhang@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Modify rdma_query_gid() to return -ENOENT for empty entries. This will
make error reporting more accurate and will be used in next patches.

Link: https://lore.kernel.org/r/1f2b65dfb4d995e74b621e3e21e7c7445d187956.1639055490.git.leonro@nvidia.comSigned-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Mark Zhang <markzhang@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

The current write wqe mechanism is to write to DDR first, and then notify
the hardware through doorbell to read the data. Direct wqe is a mechanism
to fill wqe directly into the hardware. In the case of light load, the wqe
will be filled into pcie bar space of the hardware, this will reduce one
memory access operation and therefore reduce the latency. SIMD
instructions allows cpu to write the 512 bits at one time to device
memory, thus it can be used for posting direct wqe.

Add direct wqe enable switch and address mapping.

Link: https://lore.kernel.org/r/20211207124901.42123-2-liangwenpeng@huawei.comSigned-off-by: Yixing Liu <liuyixing1@huawei.com>
Signed-off-by: Wenpeng Liang <liangwenpeng@huawei.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Currently, the driver ignores the user's priority for flow steering
rules in FDB namespace. Change it and create the rule in the right
priority.
It will allow to create FDB steering rules in up to 16 different
priorities.
Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Acked-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Create 16 flow steering priorities for FDB bypass users.
Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Have all the namespace type check in the same switch case.
Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

This patch doesn't add an additional namespaces, but just separates the
naming to be used by each FDB user, bypass and kernel.
Downstream patches will actually split this up and allow to have more
than single priority for the bypass users.
Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Acked-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Pull USB fixes from Greg KH:
 "Here are some small USB fixes for 5.16-rc5.  They include:

   - gadget driver fixes for reported issues

   - xhci fixes for reported problems.

   - config endpoint parsing fixes for where we got bitfields wrong

  Most of these have been in linux-next, the remaining few were not, but
  got lots of local testing in my systems and in some cloud testing
  infrastructures"

* tag 'usb-5.16-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb:
  usb: core: config: using bit mask instead of individual bits
  usb: core: config: fix validation of wMaxPacketValue entries
  USB: gadget: zero allocate endpoint 0 buffers
  USB: gadget: detect too-big endpoint 0 requests
  xhci: avoid race between disable slot command and host runtime suspend
  xhci: Remove CONFIG_USB_DEFAULT_PERSIST to prevent xHCI from runtime suspending
  Revert "usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default"

Pull char/misc driver fixes from Greg KH:
 "Here are a bunch of small char/misc and other driver subsystem fixes.

  Included in here are:

   - iio driver fixes for reported problems

   - phy driver fixes for a number of reported problems

   - mhi resume bugfix for broken hardware

   - nvmem driver fix

   - rtsx driver fix for irq issues

   - fastrpc packet parsing fix

  All of these have been in linux-next for a while with no reported
  issues"

* tag 'char-misc-5.16-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc: (33 commits)
  bus: mhi: core: Add support for forced PM resume
  iio: trigger: stm32-timer: fix MODULE_ALIAS
  misc: rtsx: Avoid mangling IRQ during runtime PM
  nvmem: eeprom: at25: fix FRAM byte_len
  misc: fastrpc: fix improper packet size calculation
  MAINTAINERS: add maintainer for Qualcomm FastRPC driver
  bus: mhi: pci_generic: Fix device recovery failed issue
  iio: adc: stm32: fix null pointer on defer_probe error
  phy: HiSilicon: Fix copy and paste bug in error handling
  dt-bindings: phy: zynqmp-psgtr: fix USB phy name
  phy: ti: omap-usb2: Fix the kernel-doc style
  phy: qualcomm: ipq806x-usb: Fix kernel-doc style
  iio: at91-sama5d2: Fix incorrect sign extension
  iio: adc: axp20x_adc: fix charging current reporting on AXP22x
  iio: gyro: adxrs290: fix data signedness
  phy: ti: tusb1210: Fix the kernel-doc warn
  phy: qualcomm: usb-hsic: Fix the kernel-doc warn
  phy: qualcomm: qmp: Add missing struct documentation
  phy: mvebu-cp110-utmi: Fix kernel-doc warns
  iio: ad7768-1: Call iio_trigger_notify_done() on error
  ...

Pull timer fixes from Thomas Gleixner:
 "Two fixes for clock chip drivers:

   - A regression fix for the Designware APB timer. A recent change to
     the error checking code transformed the error condition wrongly so
     it turned into a fail if good condition.

   - Fix a clang build fail of the ARM architected timer driver"

* tag 'timers-urgent-2021-12-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  clocksource/drivers/arm_arch_timer: Force inlining of erratum_set_next_event_generic()
  clocksource/drivers/dw_apb_timer_of: Fix probe failure

Pull irq fixes from Thomas Gleixner:
 "A set of interrupt chip driver fixes:

   - Fix the multi vector MSI allocation on Armada 370XP

   - Do interrupt acknowledgement correctly in the aspeed-scu driver

   - Make the IPR register offset correct in the NVIC driver

   - Make redistribution table flushing correct by issueing a SYNC
     command to ensure that the invalidation command has been executed

   - Plug a device tree node reference leak in the bcm7210-l2 driver

   - Trivial fixes in the MIPS GIC and the Apple AIC drivers"

* tag 'irq-urgent-2021-12-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  irqchip/irq-bcm7120-l2: Add put_device() after of_find_device_by_node()
  irqchip/irq-gic-v3-its.c: Force synchronisation when issuing INVALL
  irqchip/apple-aic: Mark aic_init_smp() as __init
  irqchip: nvic: Fix offset for Interrupt Priority Offsets
  irqchip/mips-gic: Use bitfield helpers
  irqchip/aspeed-scu: Replace update_bits with write_bits.
  irqchip/armada-370-xp: Fix support for Multi-MSI interrupts
  irqchip/armada-370-xp: Fix return value of armada_370_xp_msi_alloc()

Pull scheduler fix from Thomas Gleixner:
 "A single fix for the x86 scheduler topology:

  Using cluster topology on hybrid CPUs, e.g. Alder Lake, biases the
  scheduler towards the ATOM cluster as that has more total capacity.
  Use selection based on CPU priority instead"

* tag 'sched-urgent-2021-12-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  sched,x86: Don't use cluster topology for x86 hybrid CPUs

Pull csky from Guo Ren:
 "Only one fix for csky: fix fpu config macro"

* tag 'csky-for-linus-5.16-rc5' of git://github.com/c-sky/csky-linux:
  csky: fix typo of fpu config macro