• Kirill A. Shutemov's avatar
    x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds · 09ef4939
    Kirill A. Shutemov authored
    In split page table lock case, we embed spinlock_t into struct page.
    For obvious reason, we don't want to increase size of struct page if
    spinlock_t is too big, like with DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC or
    on -rt kernel.  So we disable split page table lock, if spinlock_t is
    too big.
    
    This patchset allows to allocate the lock dynamically if spinlock_t is
    big.  In this page->ptl is used to store pointer to spinlock instead of
    spinlock itself.  It costs additional cache line for indirect access,
    but fix page fault scalability for multi-threaded applications.
    
    LOCK_STAT depends on DEBUG_SPINLOCK, so on current kernel enabling
    LOCK_STAT to analyse scalability issues breaks scalability.  ;)
    
    The patchset mostly fixes this.  Results for ./thp_memscale -c 80 -b 512M
    on 4-socket machine:
    
    baseline, no CONFIG_LOCK_STAT:	9.115460703 seconds time elapsed
    baseline, CONFIG_LOCK_STAT=y:	53.890567123 seconds time elapsed
    patched, no CONFIG_LOCK_STAT:	8.852250368 seconds time elapsed
    patched, CONFIG_LOCK_STAT=y:	11.069770759 seconds time elapsed
    
    Patch count is scary, but most of them trivial. Overview:
    
     Patches 1-4	Few bug fixes. No dependencies to other patches.
    		Probably should applied as soon as possible.
    
     Patch 5	Changes signature of pgtable_page_ctor(). We will use it
    		for dynamic lock allocation, so it can fail.
    
     Patches 6-8	Add missing constructor/destructor calls on few archs.
    		It's fixes NR_PAGETABLE accounting and prepare to use
    		split ptl.
    
     Patches 9-33	Add pgtable_page_ctor() fail handling to all archs.
    
     Patches 34	Finally adds support of dynamically-allocated page->pte.
    		Also contains documentation for split page table lock.
    
    This patch (of 34):
    
    I've missed that we preallocate few pmds on pgd_alloc() if X86_PAE
    enabled.  Let's add missed constructor/destructor calls.
    
    I haven't noticed it during testing since prep_new_page() clears
    page->mapping and therefore page->ptl.  It's effectively equal to
    spin_lock_init(&page->ptl).
    Signed-off-by: default avatarKirill A. Shutemov <kirill.shutemov@linux.intel.com>
    Acked-by: default avatarIngo Molnar <mingo@kernel.org>
    Cc: "H. Peter Anvin" <hpa@zytor.com>
    Cc: "James E.J. Bottomley" <jejb@parisc-linux.org>
    Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
    Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
    Cc: Catalin Marinas <catalin.marinas@arm.com>
    Cc: Chen Liqin <liqin.chen@sunplusct.com>
    Cc: Chris Metcalf <cmetcalf@tilera.com>
    Cc: Chris Zankel <chris@zankel.net>
    Cc: Christoph Lameter <cl@linux.com>
    Cc: David Howells <dhowells@redhat.com>
    Cc: David S. Miller <davem@davemloft.net>
    Cc: Fenghua Yu <fenghua.yu@intel.com>
    Cc: Geert Uytterhoeven <geert@linux-m68k.org>
    Cc: Grant Likely <grant.likely@linaro.org>
    Cc: Guan Xuetao <gxt@mprc.pku.edu.cn>
    Cc: Haavard Skinnemoen <hskinnemoen@gmail.com>
    Cc: Hans-Christian Egtvedt <egtvedt@samfundet.no>
    Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
    Cc: Helge Deller <deller@gmx.de>
    Cc: Hirokazu Takata <takata@linux-m32r.org>
    Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
    Cc: James Hogan <james.hogan@imgtec.com>
    Cc: Jeff Dike <jdike@addtoit.com>
    Cc: Jesper Nilsson <jesper.nilsson@axis.com>
    Cc: Jonas Bonn <jonas@southpole.se>
    Cc: Koichi Yasutake <yasutake.koichi@jp.panasonic.com>
    Cc: Lennox Wu <lennox.wu@gmail.com>
    Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
    Cc: Matt Turner <mattst88@gmail.com>
    Cc: Max Filippov <jcmvbkbc@gmail.com>
    Cc: Michal Simek <monstr@monstr.eu>
    Cc: Mikael Starvik <starvik@axis.com>
    Cc: Paul Mackerras <paulus@samba.org>
    Cc: Paul Mundt <lethal@linux-sh.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Ralf Baechle <ralf@linux-mips.org>
    Cc: Richard Henderson <rth@twiddle.net>
    Cc: Richard Kuo <rkuo@codeaurora.org>
    Cc: Richard Weinberger <richard@nod.at>
    Cc: Rob Herring <rob.herring@calxeda.com>
    Cc: Russell King <linux@arm.linux.org.uk>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Tony Luck <tony.luck@intel.com>
    Cc: Vineet Gupta <vgupta@synopsys.com>
    Cc: Will Deacon <will.deacon@arm.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    09ef4939
pgtable.c 10.7 KB