• Vlad Buslov's avatar
    net/mlx5e: Fix race condition on nhe->n pointer in neigh update · 1253935a
    Vlad Buslov authored
    Current neigh update event handler implementation takes reference to
    neighbour structure, assigns it to nhe->n, tries to schedule workqueue task
    and releases the reference if task was already enqueued. This results
    potentially overwriting existing nhe->n pointer with another neighbour
    instance, which causes double release of the instance (once in neigh update
    handler that failed to enqueue to workqueue and another one in neigh update
    workqueue task that processes updated nhe->n pointer instead of original
    one):
    
    [ 3376.512806] ------------[ cut here ]------------
    [ 3376.513534] refcount_t: underflow; use-after-free.
    [ 3376.521213] Modules linked in: act_skbedit act_mirred act_tunnel_key vxlan ip6_udp_tunnel udp_tunnel nfnetlink act_gact cls_flower sch_ingress openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 mlx5_ib mlx5_core mlxfw pci_hyperv_intf ptp pps_core nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd
     grace fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp rpcrdma rdma_ucm ib_umad ib_ipoib ib_iser rdma_cm ib_cm iw_cm rfkill ib_uverbs ib_core sunrpc kvm_intel kvm iTCO_wdt iTCO_vendor_support virtio_net irqbypass net_failover crc32_pclmul lpc_ich i2c_i801 failover pcspkr i2c_smbus mfd_core ghash_clmulni_intel sch_fq_codel drm i2c
    _core ip_tables crc32c_intel serio_raw [last unloaded: mlxfw]
    [ 3376.529468] CPU: 8 PID: 22756 Comm: kworker/u20:5 Not tainted 5.9.0-rc5+ #6
    [ 3376.530399] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
    [ 3376.531975] Workqueue: mlx5e mlx5e_rep_neigh_update [mlx5_core]
    [ 3376.532820] RIP: 0010:refcount_warn_saturate+0xd8/0xe0
    [ 3376.533589] Code: ff 48 c7 c7 e0 b8 27 82 c6 05 0b b6 09 01 01 e8 94 93 c1 ff 0f 0b c3 48 c7 c7 88 b8 27 82 c6 05 f7 b5 09 01 01 e8 7e 93 c1 ff <0f> 0b c3 0f 1f 44 00 00 8b 07 3d 00 00 00 c0 74 12 83 f8 01 74 13
    [ 3376.536017] RSP: 0018:ffffc90002a97e30 EFLAGS: 00010286
    [ 3376.536793] RAX: 0000000000000000 RBX: ffff8882de30d648 RCX: 0000000000000000
    [ 3376.537718] RDX: ffff8882f5c28f20 RSI: ffff8882f5c18e40 RDI: ffff8882f5c18e40
    [ 3376.538654] RBP: ffff8882cdf56c00 R08: 000000000000c580 R09: 0000000000001a4d
    [ 3376.539582] R10: 0000000000000731 R11: ffffc90002a97ccd R12: 0000000000000000
    [ 3376.540519] R13: ffff8882de30d600 R14: ffff8882de30d640 R15: ffff88821e000900
    [ 3376.541444] FS:  0000000000000000(0000) GS:ffff8882f5c00000(0000) knlGS:0000000000000000
    [ 3376.542732] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [ 3376.543545] CR2: 0000556e5504b248 CR3: 00000002c6f10005 CR4: 0000000000770ee0
    [ 3376.544483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [ 3376.545419] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    [ 3376.546344] PKRU: 55555554
    [ 3376.546911] Call Trace:
    [ 3376.547479]  mlx5e_rep_neigh_update.cold+0x33/0xe2 [mlx5_core]
    [ 3376.548299]  process_one_work+0x1d8/0x390
    [ 3376.548977]  worker_thread+0x4d/0x3e0
    [ 3376.549631]  ? rescuer_thread+0x3e0/0x3e0
    [ 3376.550295]  kthread+0x118/0x130
    [ 3376.550914]  ? kthread_create_worker_on_cpu+0x70/0x70
    [ 3376.551675]  ret_from_fork+0x1f/0x30
    [ 3376.552312] ---[ end trace d84e8f46d2a77eec ]---
    
    Fix the bug by moving work_struct to dedicated dynamically-allocated
    structure. This enabled every event handler to work on its own private
    neighbour pointer and removes the need for handling the case when task is
    already enqueued.
    
    Fixes: 232c0013 ("net/mlx5e: Add support to neighbour update flow")
    Signed-off-by: default avatarVlad Buslov <vladbu@nvidia.com>
    Reviewed-by: default avatarRoi Dayan <roid@nvidia.com>
    Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
    1253935a
neigh.c 10.3 KB