Skip to content

L
linux
Switch branch/tag
  1. 03 Sep, 2021 21 commits
    • Vlastimil Babka's avatar
      mm, slub: move irq control into unfreeze_partials() · f3ab8b6b
      Vlastimil Babka authored 
      unfreeze_partials() can be optimized so that it doesn't need irqs disabled for
the whole time. As the first step, move irq control into the function and
remove it from the put_cpu_partial() caller.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      f3ab8b6b
    • Vlastimil Babka's avatar
      mm, slub: call deactivate_slab() without disabling irqs · cfdf836e
      Vlastimil Babka authored 
      The function is now safe to be called with irqs enabled, so move the calls
outside of irq disabled sections.

When called from ___slab_alloc() -> flush_slab() we have irqs disabled, so to
reenable them before deactivate_slab() we need to open-code flush_slab() in
___slab_alloc() and reenable irqs after modifying the kmem_cache_cpu fields.
But that means a IRQ handler meanwhile might have assigned a new page to
kmem_cache_cpu.page so we have to retry the whole check.

The remaining callers of flush_slab() are the IPI handler which has disabled
irqs anyway, and slub_cpu_dead() which will be dealt with in the following
patch.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      cfdf836e
    • Vlastimil Babka's avatar
      mm, slub: make locking in deactivate_slab() irq-safe · 3406e91b
      Vlastimil Babka authored 
      dectivate_slab() now no longer touches the kmem_cache_cpu structure, so it will
be possible to call it with irqs enabled. Just convert the spin_lock calls to
their irq saving/restoring variants to make it irq-safe.

Note we now have to use cmpxchg_double_slab() for irq-safe slab_lock(), because
in some situations we don't take the list_lock, which would disable irqs.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      3406e91b
    • Vlastimil Babka's avatar
      mm, slub: move reset of c->page and freelist out of deactivate_slab() · a019d201
      Vlastimil Babka authored 
      deactivate_slab() removes the cpu slab by merging the cpu freelist with slab's
freelist and putting the slab on the proper node's list. It also sets the
respective kmem_cache_cpu pointers to NULL.

By extracting the kmem_cache_cpu operations from the function, we can make it
not dependent on disabled irqs.

Also if we return a single free pointer from ___slab_alloc, we no longer have
to assign kmem_cache_cpu.page before deactivation or care if somebody preempted
us and assigned a different page to our kmem_cache_cpu in the process.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      a019d201
    • Vlastimil Babka's avatar
      mm, slub: stop disabling irqs around get_partial() · 4b1f449d
      Vlastimil Babka authored 
      The function get_partial() does not need to have irqs disabled as a whole. It's
sufficient to convert spin_lock operations to their irq saving/restoring
versions.

As a result, it's now possible to reach the page allocator from the slab
allocator without disabling and re-enabling interrupts on the way.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      4b1f449d
    • Vlastimil Babka's avatar
      mm, slub: check new pages with restored irqs · 9f101ee8
      Vlastimil Babka authored 
      Building on top of the previous patch, re-enable irqs before checking new
pages. alloc_debug_processing() is now called with enabled irqs so we need to
remove VM_BUG_ON(!irqs_disabled()); in check_slab() - there doesn't seem to be
a need for it anyway.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      9f101ee8
    • Vlastimil Babka's avatar
      mm, slub: validate slab from partial list or page allocator before making it cpu slab · 3f2b77e3
      Vlastimil Babka authored 
      When we obtain a new slab page from node partial list or page allocator, we
assign it to kmem_cache_cpu, perform some checks, and if they fail, we undo
the assignment.

In order to allow doing the checks without irq disabled, restructure the code
so that the checks are done first, and kmem_cache_cpu.page assignment only
after they pass.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      3f2b77e3
    • Vlastimil Babka's avatar
      mm, slub: restore irqs around calling new_slab() · 6c1dbb67
      Vlastimil Babka authored 
      allocate_slab() currently re-enables irqs before calling to the page allocator.
It depends on gfpflags_allow_blocking() to determine if it's safe to do so.
Now we can instead simply restore irq before calling it through new_slab().
The other caller early_kmem_cache_node_alloc() is unaffected by this.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      6c1dbb67
    • Vlastimil Babka's avatar
      mm, slub: move disabling irqs closer to get_partial() in ___slab_alloc() · fa417ab7
      Vlastimil Babka authored 
      Continue reducing the irq disabled scope. Check for per-cpu partial slabs with
first with irqs enabled and then recheck with irqs disabled before grabbing
the slab page. Mostly preparatory for the following patches.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      fa417ab7
    • Vlastimil Babka's avatar
      mm, slub: do initial checks in ___slab_alloc() with irqs enabled · 0b303fb4
      Vlastimil Babka authored 
      As another step of shortening irq disabled sections in ___slab_alloc(), delay
disabling irqs until we pass the initial checks if there is a cached percpu
slab and it's suitable for our allocation.

Now we have to recheck c->page after actually disabling irqs as an allocation
in irq handler might have replaced it.

Because we call pfmemalloc_match() as one of the checks, we might hit
VM_BUG_ON_PAGE(!PageSlab(page)) in PageSlabPfmemalloc in case we get
interrupted and the page is freed. Thus introduce a pfmemalloc_match_unsafe()
variant that lacks the PageSlab check.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarMel Gorman <mgorman@techsingularity.net>
      0b303fb4
    • Vlastimil Babka's avatar
      mm, slub: move disabling/enabling irqs to ___slab_alloc() · e500059b
      Vlastimil Babka authored 
      Currently __slab_alloc() disables irqs around the whole ___slab_alloc().  This
includes cases where this is not needed, such as when the allocation ends up in
the page allocator and has to awkwardly enable irqs back based on gfp flags.
Also the whole kmem_cache_alloc_bulk() is executed with irqs disabled even when
it hits the __slab_alloc() slow path, and long periods with disabled interrupts
are undesirable.

As a first step towards reducing irq disabled periods, move irq handling into
___slab_alloc(). Callers will instead prevent the s->cpu_slab percpu pointer
from becoming invalid via get_cpu_ptr(), thus preempt_disable(). This does not
protect against modification by an irq handler, which is still done by disabled
irq for most of ___slab_alloc(). As a small immediate benefit,
slab_out_of_memory() from ___slab_alloc() is now called with irqs enabled.

kmem_cache_alloc_bulk() disables irqs for its fastpath and then re-enables them
before calling ___slab_alloc(), which then disables them at its discretion. The
whole kmem_cache_alloc_bulk() operation also disables preemption.

When  ___slab_alloc() calls new_slab() to allocate a new page, re-enable
preemption, because new_slab() will re-enable interrupts in contexts that allow
blocking (this will be improved by later patches).

The patch itself will thus increase overhead a bit due to disabled preemption
(on configs where it matters) and increased disabling/enabling irqs in
kmem_cache_alloc_bulk(), but that will be gradually improved in the following
patches.

Note in __slab_alloc() we need to change the #ifdef CONFIG_PREEMPT guard to
CONFIG_PREEMPT_COUNT to make sure preempt disable/enable is properly paired in
all configurations. On configs without involuntary preemption and debugging
the re-read of kmem_cache_cpu pointer is still compiled out as it was before.

[ Mike Galbraith <efault@gmx.de>: Fix kmem_cache_alloc_bulk() error path ]
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      e500059b
    • Vlastimil Babka's avatar
      mm, slub: simplify kmem_cache_cpu and tid setup · 9b4bc85a
      Vlastimil Babka authored 
      In slab_alloc_node() and do_slab_free() fastpaths we need to guarantee that
our kmem_cache_cpu pointer is from the same cpu as the tid value. Currently
that's done by reading the tid first using this_cpu_read(), then the
kmem_cache_cpu pointer and verifying we read the same tid using the pointer and
plain READ_ONCE().

This can be simplified to just fetching kmem_cache_cpu pointer and then reading
tid using the pointer. That guarantees they are from the same cpu. We don't
need to read the tid using this_cpu_read() because the value will be validated
by this_cpu_cmpxchg_double(), making sure we are on the correct cpu and the
freelist didn't change by anyone preempting us since reading the tid.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarMel Gorman <mgorman@techsingularity.net>
      9b4bc85a
    • Vlastimil Babka's avatar
      mm, slub: restructure new page checks in ___slab_alloc() · 1572df7c
      Vlastimil Babka authored 
      When we allocate slab object from a newly acquired page (from node's partial
list or page allocator), we usually also retain the page as a new percpu slab.
There are two exceptions - when pfmemalloc status of the page doesn't match our
gfp flags, or when the cache has debugging enabled.

The current code for these decisions is not easy to follow, so restructure it
and add comments. The new structure will also help with the following changes.
No functional change.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarMel Gorman <mgorman@techsingularity.net>
      1572df7c
    • Vlastimil Babka's avatar
      mm, slub: return slab page from get_partial() and set c->page afterwards · 75c8ff28
      Vlastimil Babka authored 
      The function get_partial() finds a suitable page on a partial list, acquires
and returns its freelist and assigns the page pointer to kmem_cache_cpu.
In later patch we will need more control over the kmem_cache_cpu.page
assignment, so instead of passing a kmem_cache_cpu pointer, pass a pointer to a
pointer to a page that get_partial() can fill and the caller can assign the
kmem_cache_cpu.page pointer. No functional change as all of this still happens
with disabled IRQs.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      75c8ff28
    • Vlastimil Babka's avatar
      mm, slub: dissolve new_slab_objects() into ___slab_alloc() · 53a0de06
      Vlastimil Babka authored 
      The later patches will need more fine grained control over individual actions
in ___slab_alloc(), the only caller of new_slab_objects(), so dissolve it
there. This is a preparatory step with no functional change.

The only minor change is moving WARN_ON_ONCE() for using a constructor together
with __GFP_ZERO to new_slab(), which makes it somewhat less frequent, but still
able to catch a development change introducing a systematic misuse.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarChristoph Lameter <cl@linux.com>
Acked-by: default avatarMel Gorman <mgorman@techsingularity.net>
      53a0de06
    • Vlastimil Babka's avatar
      mm, slub: extract get_partial() from new_slab_objects() · 2a904905
      Vlastimil Babka authored 
      The later patches will need more fine grained control over individual actions
in ___slab_alloc(), the only caller of new_slab_objects(), so this is a first
preparatory step with no functional change.

This adds a goto label that appears unnecessary at this point, but will be
useful for later changes.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarChristoph Lameter <cl@linux.com>
      2a904905
    • Vlastimil Babka's avatar
      mm, slub: remove redundant unfreeze_partials() from put_cpu_partial() · 976b805c
      Vlastimil Babka authored 
      Commit d6e0b7fa ("slub: make dead caches discard free slabs immediately")
introduced cpu partial flushing for kmemcg caches, based on setting the target
cpu_partial to 0 and adding a flushing check in put_cpu_partial().
This code that sets cpu_partial to 0 was later moved by c9fc5864 ("slab:
introduce __kmemcg_cache_deactivate()") and ultimately removed by 9855609b
("mm: memcg/slab: use a single set of kmem_caches for all accounted
allocations"). However the check and flush in put_cpu_partial() was never
removed, although it's effectively a dead code. So this patch removes it.

Note that d6e0b7fa also added preempt_disable()/enable() to
unfreeze_partials() which could be thus also considered unnecessary. But
further patches will rely on it, so keep it.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      976b805c
    • Vlastimil Babka's avatar
      mm, slub: don't disable irq for debug_check_no_locks_freed() · 84048039
      Vlastimil Babka authored 
      In slab_free_hook() we disable irqs around the debug_check_no_locks_freed()
call, which is unnecessary, as irqs are already being disabled inside the call.
This seems to be leftover from the past where there were more calls inside the
irq disabled sections. Remove the irq disable/enable operations.

Mel noted:
> Looks like it was needed for kmemcheck which went away back in 4.15
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarMel Gorman <mgorman@techsingularity.net>
      84048039
    • Vlastimil Babka's avatar
      mm, slub: allocate private object map for validate_slab_cache() · 0a19e7dd
      Vlastimil Babka authored 
      validate_slab_cache() is called either to handle a sysfs write, or from a
self-test context. In both situations it's straightforward to preallocate a
private object bitmap instead of grabbing the shared static one meant for
critical sections, so let's do that.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarChristoph Lameter <cl@linux.com>
Acked-by: default avatarMel Gorman <mgorman@techsingularity.net>
      0a19e7dd
    • Vlastimil Babka's avatar
      mm, slub: allocate private object map for debugfs listings · b3fd64e1
      Vlastimil Babka authored 
      Slub has a static spinlock protected bitmap for marking which objects are on
freelist when it wants to list them, for situations where dynamically
allocating such map can lead to recursion or locking issues, and on-stack
bitmap would be too large.

The handlers of debugfs files alloc_traces and free_traces also currently use this
shared bitmap, but their syscall context makes it straightforward to allocate a
private map before entering locked sections, so switch these processing paths
to use a private bitmap.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarChristoph Lameter <cl@linux.com>
Acked-by: default avatarMel Gorman <mgorman@techsingularity.net>
      b3fd64e1
    • Vlastimil Babka's avatar
      mm, slub: don't call flush_all() from slab_debug_trace_open() · eafb1d64
      Vlastimil Babka authored 
      slab_debug_trace_open() can only be called on caches with SLAB_STORE_USER flag
and as with all slub debugging flags, such caches avoid cpu or percpu partial
slabs altogether, so there's nothing to flush.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarChristoph Lameter <cl@linux.com>
      eafb1d64
  3. 29 Aug, 2021 8 commits
  5. 28 Aug, 2021 3 commits
  7. 27 Aug, 2021 8 commits
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7