- 09 Jan, 2022 1 commit
-
-
Pavel Begunkov authored
tctx_task_work() may get run after io_uring cancellation and so there will be no one to put cached in tctx task refs that may have been added back by tw handlers using inline completion infra, Call io_uring_drop_tctx_refs() at the end of the main tw handler to release them. Cc: stable@vger.kernel.org # 5.15+ Reported-by:
Lukas Bulwahn <lukas.bulwahn@gmail.com> Fixes: e98e49b2 ("io_uring: extend task put optimisations") Signed-off-by:
Pavel Begunkov <asml.silence@gmail.com> Link: https://lore.kernel.org/r/69f226b35fbdb996ab799a8bbc1c06bf634ccec1.1641688805.git.asml.silence@gmail.comSigned-off-by:
Jens Axboe <axboe@kernel.dk>
-
- 05 Jan, 2022 2 commits
-
-
GuoYong Zheng authored
When show fdinfo, SqMask follow two tab space, which is inconsistent with other parameters. Remove one, so it lines up nicely. Signed-off-by:
GuoYong Zheng <zhenggy@chinatelecom.cn> Link: https://lore.kernel.org/r/1641377585-1891-1-git-send-email-zhenggy@chinatelecom.cnSigned-off-by:
-
GuoYong Zheng authored
Parameter res2 is not used in __io_complete_rw, remove it. Fixes: 6b19b766 ("fs: get rid of the res2 iocb->ki_complete argument") Signed-off-by:
-
- 28 Dec, 2021 7 commits
-
-
Pavel Begunkov authored
Use __io_req_complete() in io_poll_update(), so we can utilise completion batching for both update/remove request and the poll we're killing (if any). Signed-off-by:
-
Pavel Begunkov authored
We don't need to poll oneshot request if we've got a desired mask in io_poll_wake(), task_work will clean it up correctly, but as we already hold a wq spinlock, we can remove ourselves and save on additional spinlocking in io_poll_remove_entries(). Signed-off-by:
-
Pavel Begunkov authored
It's not possible to go forward with the current state of io_uring polling, we need a more straightforward and easier synchronisation. There are a lot of problems with how it is at the moment, including missing events on rewait. The main idea here is to introduce a notion of request ownership while polling, no one but the owner can modify any part but ->poll_refs of struct io_kiocb, that grants us protection against all sorts of races. Main users of such exclusivity are poll task_work handler, so before queueing a tw one should have/acquire ownership, which will be handed off to the tw handler. The other user is __io_arm_poll_handler() do initial poll arming. It starts taking the ownership, so tw handlers won't be run until it's released later in the function after vfs_poll. note: also prevents races in __io_queue_proc(). Poll wake/etc. may not be able to get ownership, then they need to increase the poll refcount and the task_work should notice it and retry if necessary, see io_poll_check_events(). There is also IO_POLL_CANCEL_FLAG flag to notify that we want to kill request. It makes cancellations more reliable, enables double multishot polling, fixes double poll rewait, fixes missing poll events and fixes another bunch of races. Even though it adds some overhead for new refcounting, and there are a couple of nice performance wins: - no req->refs refcounting for poll requests anymore - if the data is already there (once measured for some test to be 1-2% of all apoll requests), it removes it doesn't add atomics and removes spin_lock/unlock pair. - works well with multishots, we don't do remove from queue / add to queue for each new poll event. Signed-off-by:
-
Pavel Begunkov authored
With IORING_FEAT_FAST_POLL in place, io_put_req_find_next() for poll requests doesn't make much sense, and in any case re-adding it shouldn't be a problem considering batching in tctx_task_work(). We can remove it. Signed-off-by:
-
Pavel Begunkov authored
Move some poll helpers/etc up, we'll need them there shortly Signed-off-by:
-
Pavel Begunkov authored
Clean up io_poll_update() and unify cancellation paths for remove and update. Signed-off-by:
-
Pavel Begunkov authored
Before updating a poll request we should remove it from poll queues, including the double poll entry. Fixes: b69de288 ("io_uring: allow events and user_data update of running poll requests") Signed-off-by:
-
- 14 Dec, 2021 1 commit
-
-
Hao Xu authored
There are some functions doing ctx = req->ctx while still using req->ctx, update those places. Signed-off-by:
Hao Xu <haoxu@linux.alibaba.com> Link: https://lore.kernel.org/r/20211214055904.61772-1-haoxu@linux.alibaba.comSigned-off-by:
-
- 08 Dec, 2021 1 commit
-
-
Hao Xu authored
In previous patches, we have already gathered some tw with io_req_task_complete() as callback in prior_task_list, let's complete them in batch while we cannot grab uring lock. In this way, we batch the req_complete_post path. Signed-off-by:
-
- 07 Dec, 2021 4 commits
-
-
Hao Xu authored
Split io_req_complete_post(), this is a prep for the next patch. Signed-off-by:
-
Hao Xu authored
Add a helper for task work execution code. We will use it later. Reviewed-by:
-
Hao Xu authored
Now we have a lot of task_work users, some are just to complete a req and generate a cqe. Let's put the work to a new tw list which has a higher priority, so that it can be handled quickly and thus to reduce avg req latency and users can issue next round of sqes earlier. An explanatory case: origin timeline: submit_sqe-->irq-->add completion task_work -->run heavy work0~n-->run completion task_work now timeline: submit_sqe-->irq-->add completion task_work -->run completion task_work-->run heavy work0~n Limitation: this optimization is only for those that submission and reaping process are in different threads. Otherwise anyhow we have to submit new sqes after returning to userspace, then the order of TWs doesn't matter. Tested this patch(and the following ones) by manually replace __io_queue_sqe() in io_queue_sqe() by io_req_task_queue() to construct 'heavy' task works. Then test with fio: ioengine=io_uring sqpoll=1 thread=1 bs=4k direct=1 rw=randread time_based=1 runtime=600 randrepeat=0 group_reporting=1 filename=/dev/nvme0n1 Tried various iodepth. The peak IOPS for this patch is 710K, while the old one is 665K. For avg latency, difference shows when iodepth grow: depth and avg latency(usec): depth new old 1 7.05 7.10 2 8.47 8.60 4 10.42 10.42 8 13.78 13.22 16 27.41 24.33 32 49.40 53.08 64 102.53 103.36 128 196.98 205.61 256 372.99 414.88 512 747.23 791.30 1024 1472.59 1538.72 2048 3153.49 3329.01 4096 6387.86 6682.54 8192 12150.25 12774.14 16384 23085.58 26044.71 Signed-off-by: -
Hao Xu authored
add a helper to merge two wq_lists, it will be useful in the next patches. Reviewed-by:
-
- 05 Dec, 2021 4 commits
-
-
Pavel Begunkov authored
With kbuf unification io_req_task_complete() is now a generic function, use it for timeout's tw completions. Signed-off-by:
-
Pavel Begunkov authored
When iopolling the userspace specifies the minimum number of "events" it expects. Previously, we had one CQE per request, so the definition of an "event" was unequivocal, but that's not more the case anymore with REQ_F_CQE_SKIP. Currently it counts the number of completed requests, replace it with the number of posted CQEs. This allows users of the "one CQE per link" scheme to wait for all N links in a single syscall, which is not possible without the patch and requires extra context switches. Signed-off-by:
-
Pavel Begunkov authored
As selected buffers are now stored in a separate field in a request, get rid of rw/recv specific helpers and simplify the code. Signed-off-by:
-
Hao Xu authored
Move them up to avoid explicit declaration. We will use them in later patches. Reviewed-by:
-
- 29 Nov, 2021 1 commit
-
-
Ye Bin authored
Like commit f6223ff7, timeout removal should also validate the timespec that is being passed in. Signed-off-by:
Ye Bin <yebin10@huawei.com> Link: https://lore.kernel.org/r/20211129041537.1936270-1-yebin10@huawei.comSigned-off-by:
-
- 25 Nov, 2021 2 commits
-
-
Hao Xu authored
It's better to use REQ_F_IO_DRAIN for req->flags rather than IOSQE_IO_DRAIN though they have same value. Signed-off-by:
-
Hao Xu authored
ctx->cq_extra should be protected by completion lock so that the req_need_defer() does the right check. Cc: stable@vger.kernel.org Signed-off-by:
-
- 24 Nov, 2021 4 commits
-
-
Pavel Begunkov authored
Current IOSQE_IO_DRAIN implementation doesn't work well with CQE skipping and it's not allowed, otherwise some requests might be not executed until the ring is destroyed and the userspace would hang. Let's fail all drain requests after seeing IOSQE_CQE_SKIP_SUCCESS at least once. All drained requests prior to that will get run normally, so there should be no stalls. However, even though such mixing wouldn't lead to issues at the moment, it's still not allowed as the behaviour may change. Signed-off-by:
-
Pavel Begunkov authored
When no of queued for the batch completion requests need to post an CQE, see IOSQE_CQE_SKIP_SUCCESS, avoid grabbing ->completion_lock and other commit/post. Signed-off-by:
-
Pavel Begunkov authored
Emitting a CQE is expensive from the kernel perspective. Often, it's also not convenient for the userspace, spends some cycles on processing and just complicates the logic. A similar problems goes for linked requests, where we post an CQE for each request in the link. Introduce a new flags, IOSQE_CQE_SKIP_SUCCESS, trying to help with it. When set and a request completed successfully, it won't generate a CQE. When fails, it produces an CQE, but all following linked requests will be CQE-less, regardless whether they have IOSQE_CQE_SKIP_SUCCESS or not. The notion of "fail" is the same as for link failing-cancellation, where it's opcode dependent, and _usually_ result >= 0 is a success, but not always. Linked timeouts are a bit special. When the requests it's linked to was not attempted to be executed, e.g. failing linked requests, it follows the description above. Otherwise, whether a linked timeout will post a completion or not solely depends on IOSQE_CQE_SKIP_SUCCESS of that linked timeout request. Linked timeout never "fail" during execution, so for them it's unconditional. It's expected for users to not really care about the result of it but rely solely on the result of the master request. Another reason for such a treatment is that it's racy, and the timeout callback may be running awhile the master request posts its completion. use case 1: If one doesn't care about results of some requests, e.g. normal timeouts, just set IOSQE_CQE_SKIP_SUCCESS. Error result will still be posted and need to be handled. use case 2: Set IOSQE_CQE_SKIP_SUCCESS for all requests of a link but the last, and it'll post a completion only for the last one if everything goes right, otherwise there will be one only one CQE for the first failed request. Signed-off-by:
-
Pavel Begunkov authored
Split io_cqring_fill_event() into a couple of more targeted functions. The first on is io_fill_cqe_aux() for completions that are not associated with request completions and doing the ->cq_extra accounting. Examples are additional CQEs from multishot poll and rsrc notifications. The second is io_fill_cqe_req(), should be called when it's a normal request completion. Nothing more to it at the moment, will be used in later patches. The last one is inlined __io_fill_cqe() for a finer grained control, should be used with caution and in hottest places. Signed-off-by:
-
- 23 Nov, 2021 4 commits
-
-
Pavel Begunkov authored
kiocb_done() accepts a pointer to struct kiocb, pass struct io_kiocb (i.e. io_uring's request) instead so we can get rid of useless container_of(). Signed-off-by:
-
Pavel Begunkov authored
Apparently, implicit 0 to NULL conversion with ERR_PTR is not recommended and makes some tooling like Smatch to complain. Handle it explicitly, compilers are perfectly capable to optimise it out. Link: https://lore.kernel.org/all/20211108134937.GA2863@kili/Reported-by:
Dan Carpenter <dan.carpenter@oracle.com> Signed-off-by:
-
Pavel Begunkov authored
Hide all error handling under common if block, removes two extra ifs on the success path and keeps the handling more condensed. Signed-off-by:
-
Pavel Begunkov authored
Simplify failed resubmission prep in kiocb_done(), it's a bit ugly with conditional logic and hand handling cflags / select buffers. Instead, punt to tw and use io_req_task_complete() already handling all the cases. Signed-off-by:
-
- 21 Nov, 2021 5 commits
-
-
Linus Torvalds authored
-
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tipLinus Torvalds authored
Pull x86 fixes from Thomas Gleixner: - Move the command line preparation and the early command line parsing earlier so that the command line parameters which affect early_reserve_memory(), e.g. efi=nosftreserve, are taken into account. This was broken when the invocation of early_reserve_memory() was moved recently. - Use an atomic type for the SGX page accounting, which is read and written locklessly, to plug various race conditions related to it. * tag 'x86-urgent-2021-11-21' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: x86/sgx: Fix free page accounting x86/boot: Pull up cmdline preparation and early param parsing
-
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tipLinus Torvalds authored
Pull x86 perf fixes from Thomas Gleixner: - Remove unneded PEBS disabling when taking LBR snapshots to prevent an unchecked MSR access error. - Fix IIO event constraints for Snowridge and Skylake server chips. * tag 'perf-urgent-2021-11-21' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: x86/perf: Fix snapshot_branch_stack warning in VM perf/x86/intel/uncore: Fix IIO event constraints for Snowridge perf/x86/intel/uncore: Fix IIO event constraints for Skylake Server perf/x86/intel/uncore: Fix filter_tid mask for CHA events on Skylake Server
-
git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linuxLinus Torvalds authored
Pull more powerpc fixes from Michael Ellerman: - Fix a bug in copying of sigset_t for 32-bit systems, which caused X to not start. - Fix handling of shared LSIs (rare) with the xive interrupt controller (Power9/10). - Fix missing TOC setup in some KVM code, which could result in oopses depending on kernel data layout. - Fix DMA mapping when we have persistent memory and only one DMA window available. - Fix further problems with STRICT_KERNEL_RWX on 8xx, exposed by a recent fix. - A couple of other minor fixes. Thanks to Alexey Kardashevskiy, Aneesh Kumar K.V, Cédric Le Goater, Christian Zigotzky, Christophe Leroy, Daniel Axtens, Finn Thain, Greg Kurz, Masahiro Yamada, Nicholas Piggin, and Uwe Kleine-König. * tag 'powerpc-5.16-2' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux: powerpc/xive: Change IRQ domain to a tree domain powerpc/8xx: Fix pinned TLBs with CONFIG_STRICT_KERNEL_RWX powerpc/signal32: Fix sigset_t copy powerpc/book3e: Fix TLBCAM preset at boot powerpc/pseries/ddw: Do not try direct mapping with persistent memory and one window powerpc/pseries/ddw: simplify enable_ddw() powerpc/pseries/ddw: Revert "Extend upper limit for huge DMA window for persistent memory" powerpc/pseries: Fix numa FORM2 parsing fallback code powerpc/pseries: rename numa_dist_table to form2_distances powerpc: clean vdso32 and vdso64 directories powerpc/83xx/mpc8349emitx: Drop unused variable KVM: PPC: Book3S HV: Use GLOBAL_TOC for kvmppc_h_set_dabr/xdabr()
-
Geert Uytterhoeven authored
On 32-bit: fs/pstore/blk.c: In function ‘__best_effort_init’: include/linux/kern_levels.h:5:18: warning: format ‘%zu’ expects argument of type ‘size_t’, but argument 3 has type ‘long unsigned int’ [-Wformat=] 5 | #define KERN_SOH "\001" /* ASCII Start Of Header */ | ^~~~~~ include/linux/kern_levels.h:14:19: note: in expansion of macro ‘KERN_SOH’ 14 | #define KERN_INFO KERN_SOH "6" /* informational */ | ^~~~~~~~ include/linux/printk.h:373:9: note: in expansion of macro ‘KERN_INFO’ 373 | printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__) | ^~~~~~~~~ fs/pstore/blk.c:314:3: note: in expansion of macro ‘pr_info’ 314 | pr_info("attached %s (%zu) (no dedicated panic_write!)\n", | ^~~~~~~ Cc: stable@vger.kernel.org Fixes: 7bb9557b ("pstore/blk: Use the normal block device I/O path") Signed-off-by:Geert Uytterhoeven <geert@linux-m68k.org> Signed-off-by:
Kees Cook <keescook@chromium.org> Link: https://lore.kernel.org/r/20210629103700.1935012-1-geert@linux-m68k.org Cc: Jens Axboe <axboe@kernel.dk> Reviewed-by:
Christoph Hellwig <hch@lst.de> Signed-off-by:
Linus Torvalds <torvalds@linux-foundation.org>
-
- 20 Nov, 2021 4 commits
-
-
Linus Torvalds authored
Merge misc fixes from Andrew Morton: "15 patches. Subsystems affected by this patch series: ipc, hexagon, mm (swap, slab-generic, kmemleak, hugetlb, kasan, damon, and highmem), and proc" * emailed patches from Andrew Morton <akpm@linux-foundation.org>: proc/vmcore: fix clearing user buffer by properly using clear_user() kmap_local: don't assume kmap PTEs are linear arrays in memory mm/damon/dbgfs: fix missed use of damon_dbgfs_lock mm/damon/dbgfs: use '__GFP_NOWARN' for user-specified size buffer allocation kasan: test: silence intentional read overflow warnings hugetlb, userfaultfd: fix reservation restore on userfaultfd error hugetlb: fix hugetlb cgroup refcounting during mremap mm: kmemleak: slob: respect SLAB_NOLEAKTRACE flag hexagon: ignore vmlinux.lds hexagon: clean up timer-regs.h hexagon: export raw I/O routines for modules mm: emit the "free" trace report before freeing memory in kmem_cache_free() shm: extend forced shm destroy to support objects from several IPC nses ipc: WARN if trying to remove ipc object which is absent mm/swap.c:put_pages_list(): reinitialise the page list
-
git://git.kernel.dk/linux-blockLinus Torvalds authored
Pull block fixes from Jens Axboe: - Flip a cap check to avoid a selinux error (Alistair) - Fix for a regression this merge window where we can miss a queue ref put (me) - Un-mark pstore-blk as broken, as the condition that triggered that change has been rectified (Kees) - Queue quiesce and sync fixes (Ming) - FUA insertion fix (Ming) - blk-cgroup error path put fix (Yu) * tag 'block-5.16-2021-11-19' of git://git.kernel.dk/linux-block: blk-mq: don't insert FUA request with data into scheduler queue blk-cgroup: fix missing put device in error path from blkg_conf_pref() block: avoid to quiesce queue in elevator_init_mq Revert "mark pstore-blk as broken" blk-mq: cancel blk-mq dispatch work in both blk_cleanup_queue and disk_release() block: fix missing queue put in error path block: Check ADMIN before NICE for IOPRIO_CLASS_RT
-
git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrlLinus Torvalds authored
Pull pin control fixes from Linus Walleij: "There is an ACPI stubs fix which is ACKed by the ACPI maintainer for merging through my tree. One item stand out and that is that I delete the <linux/sdb.h> header that is used by nothing. I deleted this subsystem (through the GPIO tree) a while back so I feel responsible for tidying up the floor. Other than that it is the usual mistakes, a bit noisy around build issue and Kconfig then driver fixes. Specifics: - Fix some stubs causing compile issues for ACPI. - Fix some wakeups on AMD IRQs shared between GPIO and SCI. - Fix a build warning in the Tegra driver. - Fix a Kconfig issue in the Qualcomm driver. - Add a missing include the RALink driver. - Return a valid type for the Apple pinctrl IRQs. - Implement some Qualcomm SDM845 dual-edge errata. - Remove the unused <linux/sdb.h> header. (The subsystem was once deleted by the pinctrl maintainer...) - Fix a duplicate initialized in the Tegra driver. - Fix register offsets for UFS and SDC in the Qualcomm SM8350 driver" * tag 'pinctrl-v5.16-2' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl: pinctrl: qcom: sm8350: Correct UFS and SDC offsets pinctrl: tegra194: remove duplicate initializer again Remove unused header <linux/sdb.h> pinctrl: qcom: sdm845: Enable dual edge errata pinctrl: apple: Always return valid type in apple_gpio_irq_type pinctrl: ralink: include 'ralink_regs.h' in 'pinctrl-mt7620.c' pinctrl: qcom: fix unmet dependencies on GPIOLIB for GPIOLIB_IRQCHIP pinctrl: tegra: Return const pointer from tegra_pinctrl_get_group() pinctrl: amd: Fix wakeups when IRQ is shared with SCI ACPI: Add stubs for wakeup handler functions -
git://git.kernel.org/pub/scm/linux/kernel/git/s390/linuxLinus Torvalds authored
Pull s390 updates from Heiko Carstens: - Add missing Kconfig option for ftrace direct multi sample, so it can be compiled again, and also add s390 support for this sample. - Update Christian Borntraeger's email address. - Various fixes for memory layout setup. Besides other this makes it possible to load shared DCSS segments again. - Fix copy to user space of swapped kdump oldmem. - Remove -mstack-guard and -mstack-size compile options when building vdso binaries. This can happen when CONFIG_VMAP_STACK is disabled and results in broken vdso code which causes more or less random exceptions. Also remove the not needed -nostdlib option. - Fix memory leak on cpu hotplug and return code handling in kexec code. - Wire up futex_waitv system call. - Replace snprintf with sysfs_emit where appropriate. * tag 's390-5.16-3' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux: ftrace/samples: add s390 support for ftrace direct multi sample ftrace/samples: add missing Kconfig option for ftrace direct multi sample MAINTAINERS: update email address of Christian Borntraeger s390/kexec: fix memory leak of ipl report buffer s390/kexec: fix return code handling s390/dump: fix copying to user-space of swapped kdump oldmem s390: wire up sys_futex_waitv system call s390/vdso: filter out -mstack-guard and -mstack-size s390/vdso: remove -nostdlib compiler flag s390: replace snprintf in show functions with sysfs_emit s390/boot: simplify and fix kernel memory layout setup s390/setup: re-arrange memblock setup s390/setup: avoid using memblock_enforce_memory_limit s390/setup: avoid reserving memory above identity mapping
-
