1. 15 Mar, 2021 1 commit
  2. 14 Mar, 2021 1 commit
    • Jens Axboe's avatar
      io_uring: convert io_buffer_idr to XArray · 9e15c3a0
      Jens Axboe authored
      Like we did for the personality idr, convert the IO buffer idr to use
      XArray. This avoids a use-after-free on removal of entries, since idr
      doesn't like doing so from inside an iterator, and it nicely reduces
      the amount of code we need to support this feature.
      
      Fixes: 5a2e745d ("io_uring: buffer registration infrastructure")
      Cc: stable@vger.kernel.org
      Cc: Matthew Wilcox <willy@infradead.org>
      Cc: yangerkun <yangerkun@huawei.com>
      Reported-by: default avatarHulk Robot <hulkci@huawei.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      9e15c3a0
  3. 13 Mar, 2021 2 commits
  4. 12 Mar, 2021 5 commits
    • Pavel Begunkov's avatar
      io_uring: fix OP_ASYNC_CANCEL across tasks · 58f99373
      Pavel Begunkov authored
      IORING_OP_ASYNC_CANCEL tries io-wq cancellation only for current task.
      If it fails go over tctx_list and try it out for every single tctx.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      58f99373
    • Pavel Begunkov's avatar
      io_uring: cancel sqpoll via task_work · 521d6a73
      Pavel Begunkov authored
      1) The first problem is io_uring_cancel_sqpoll() ->
      io_uring_cancel_task_requests() basically doing park(); park(); and so
      hanging.
      
      2) Another one is more subtle, when the master task is doing cancellations,
      but SQPOLL task submits in-between the end of the cancellation but
      before finish() requests taking a ref to the ctx, and so eternally
      locking it up.
      
      3) Yet another is a dying SQPOLL task doing io_uring_cancel_sqpoll() and
      same io_uring_cancel_sqpoll() from the owner task, they race for
      tctx->wait events. And there probably more of them.
      
      Instead do SQPOLL cancellations from within SQPOLL task context via
      task_work, see io_sqpoll_cancel_sync(). With that we don't need temporal
      park()/unpark() during cancellation, which is ugly, subtle and anyway
      doesn't allow to do io_run_task_work() properly.
      
      io_uring_cancel_sqpoll() is called only from SQPOLL task context and
      under sqd locking, so all parking is removed from there. And so,
      io_sq_thread_[un]park() and io_sq_thread_stop() are not used now by
      SQPOLL task, and that spare us from some headache.
      
      Also remove ctx->sqd_list early to avoid 2). And kill tctx->sqpoll,
      which is not used anymore.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      521d6a73
    • Pavel Begunkov's avatar
      io_uring: prevent racy sqd->thread checks · 26984fbf
      Pavel Begunkov authored
      SQPOLL thread to which we're trying to attach may be going away, it's
      not nice but a more serious problem is if io_sq_offload_create() sees
      sqd->thread==NULL, and tries to init it with a new thread. There are
      tons of ways it can be exploited or fail.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      26984fbf
    • Pavel Begunkov's avatar
      io_uring: remove useless ->startup completion · 0df8ea60
      Pavel Begunkov authored
      We always do complete(&sqd->startup) almost right after sqd->thread
      creation, either in the success path or in io_sq_thread_finish(). It's
      specifically created not started for us to be able to set some stuff
      like sqd->thread and io_uring_alloc_task_context() before following
      right after wake_up_new_task().
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      0df8ea60
    • Pavel Begunkov's avatar
      io_uring: cancel deferred requests in try_cancel · e1915f76
      Pavel Begunkov authored
      As io_uring_cancel_files() and others let SQO to run between
      io_uring_try_cancel_requests(), SQO may generate new deferred requests,
      so it's safer to try to cancel them in it.
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      e1915f76
  5. 11 Mar, 2021 2 commits
    • Jens Axboe's avatar
      io_uring: perform IOPOLL reaping if canceler is thread itself · d052d1d6
      Jens Axboe authored
      We bypass IOPOLL completion polling (and reaping) for the SQPOLL thread,
      but if it's the thread itself invoking cancelations, then we still need
      to perform it or no one will.
      
      Fixes: 9936c7c2 ("io_uring: deduplicate core cancellations sequence")
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      d052d1d6
    • Jens Axboe's avatar
      io_uring: force creation of separate context for ATTACH_WQ and non-threads · 5c2469e0
      Jens Axboe authored
      Earlier kernels had SQPOLL threads that could share across anything, as
      we grabbed the context we needed on a per-ring basis. This is no longer
      the case, so only allow attaching directly if we're in the same thread
      group. That is the common use case. For non-group tasks, just setup a
      new context and thread as we would've done if sharing wasn't set. This
      isn't 100% ideal in terms of CPU utilization for the forked and share
      case, but hopefully that isn't much of a concern. If it is, there are
      plans in motion for how to improve that. Most importantly, we want to
      avoid app side regressions where sharing worked before and now doesn't.
      With this patch, functionality is equivalent to previous kernels that
      supported IORING_SETUP_ATTACH_WQ with SQPOLL.
      Reported-by: default avatarStefan Metzmacher <metze@samba.org>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      5c2469e0
  6. 10 Mar, 2021 15 commits
  7. 07 Mar, 2021 10 commits
  8. 06 Mar, 2021 4 commits
    • Jens Axboe's avatar
      io-wq: always track creds for async issue · 003e8dcc
      Jens Axboe authored
      If we go async with a request, grab the creds that the task currently has
      assigned and make sure that the async side switches to them. This is
      handled in the same way that we do for registered personalities.
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      003e8dcc
    • Jens Axboe's avatar
      io-wq: fix race in freeing 'wq' and worker access · 886d0137
      Jens Axboe authored
      Ran into a use-after-free on the main io-wq struct, wq. It has a worker
      ref and completion event, but the manager itself isn't holding a
      reference. This can lead to a race where the manager thinks there are
      no workers and exits, but a worker is being added. That leads to the
      following trace:
      
      BUG: KASAN: use-after-free in io_wqe_worker+0x4c0/0x5e0
      Read of size 8 at addr ffff888108baa8a0 by task iou-wrk-3080422/3080425
      
      CPU: 5 PID: 3080425 Comm: iou-wrk-3080422 Not tainted 5.12.0-rc1+ #110
      Hardware name: Micro-Star International Co., Ltd. MS-7C60/TRX40 PRO 10G (MS-7C60), BIOS 1.60 05/13/2020
      Call Trace:
       dump_stack+0x90/0xbe
       print_address_description.constprop.0+0x67/0x28d
       ? io_wqe_worker+0x4c0/0x5e0
       kasan_report.cold+0x7b/0xd4
       ? io_wqe_worker+0x4c0/0x5e0
       __asan_load8+0x6d/0xa0
       io_wqe_worker+0x4c0/0x5e0
       ? io_worker_handle_work+0xc00/0xc00
       ? recalc_sigpending+0xe5/0x120
       ? io_worker_handle_work+0xc00/0xc00
       ? io_worker_handle_work+0xc00/0xc00
       ret_from_fork+0x1f/0x30
      
      Allocated by task 3080422:
       kasan_save_stack+0x23/0x60
       __kasan_kmalloc+0x80/0xa0
       kmem_cache_alloc_node_trace+0xa0/0x480
       io_wq_create+0x3b5/0x600
       io_uring_alloc_task_context+0x13c/0x380
       io_uring_add_task_file+0x109/0x140
       __x64_sys_io_uring_enter+0x45f/0x660
       do_syscall_64+0x32/0x80
       entry_SYSCALL_64_after_hwframe+0x44/0xae
      
      Freed by task 3080422:
       kasan_save_stack+0x23/0x60
       kasan_set_track+0x20/0x40
       kasan_set_free_info+0x24/0x40
       __kasan_slab_free+0xe8/0x120
       kfree+0xa8/0x400
       io_wq_put+0x14a/0x220
       io_wq_put_and_exit+0x9a/0xc0
       io_uring_clean_tctx+0x101/0x140
       __io_uring_files_cancel+0x36e/0x3c0
       do_exit+0x169/0x1340
       __x64_sys_exit+0x34/0x40
       do_syscall_64+0x32/0x80
       entry_SYSCALL_64_after_hwframe+0x44/0xae
      
      Have the manager itself hold a reference, and now both drop points drop
      and complete if we hit zero, and the manager can unconditionally do a
      wait_for_completion() instead of having a race between reading the ref
      count and waiting if it was non-zero.
      
      Fixes: fb3a1f6c ("io-wq: have manager wait for all workers to exit")
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      886d0137
    • Linus Torvalds's avatar
      Linux 5.12-rc2 · a38fd874
      Linus Torvalds authored
      a38fd874
    • Linus Torvalds's avatar
      Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma · f3ed4de6
      Linus Torvalds authored
      Pull rdma fixes from Jason Gunthorpe:
       "Nothing special here, though Bob's regression fixes for rxe would have
        made it before the rc cycle had there not been such strong winter
        weather!
      
         - Fix corner cases in the rxe reference counting cleanup that are
           causing regressions in blktests for SRP
      
         - Two kdoc fixes so W=1 is clean
      
         - Missing error return in error unwind for mlx5
      
         - Wrong lock type nesting in IB CM"
      
      * tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma:
        RDMA/rxe: Fix errant WARN_ONCE in rxe_completer()
        RDMA/rxe: Fix extra deref in rxe_rcv_mcast_pkt()
        RDMA/rxe: Fix missed IB reference counting in loopback
        RDMA/uverbs: Fix kernel-doc warning of _uverbs_alloc
        RDMA/mlx5: Set correct kernel-doc identifier
        IB/mlx5: Add missing error code
        RDMA/rxe: Fix missing kconfig dependency on CRYPTO
        RDMA/cm: Fix IRQ restore in ib_send_cm_sidr_rep
      f3ed4de6