• Pavel Tatashin's avatar
    sparc64: optimize struct page zeroing · 78c94366
    Pavel Tatashin authored
    Add an optimized mm_zero_struct_page(), so struct page's are zeroed
    without calling memset().  We do eight to ten regular stores based on
    the size of struct page.  Compiler optimizes out the conditions of
    switch() statement.
    
    SPARC-M6 with 15T of memory, single thread performance:
    
                                   BASE            FIX  OPTIMIZED_FIX
            bootmem_init   28.440467985s   2.305674818s   2.305161615s
    free_area_init_nodes  202.845901673s 225.343084508s 172.556506560s
                          --------------------------------------------
    Total                 231.286369658s 227.648759326s 174.861668175s
    
    BASE:  current linux
    FIX:   This patch series without "optimized struct page zeroing"
    OPTIMIZED_FIX: This patch series including the current patch.
    
    bootmem_init() is where memory for struct pages is zeroed during
    allocation.  Note, about two seconds in this function is a fixed time:
    it does not increase as memory is increased.
    
    Link: http://lkml.kernel.org/r/20171013173214.27300-11-pasha.tatashin@oracle.comSigned-off-by: default avatarPavel Tatashin <pasha.tatashin@oracle.com>
    Reviewed-by: default avatarSteven Sistare <steven.sistare@oracle.com>
    Reviewed-by: default avatarDaniel Jordan <daniel.m.jordan@oracle.com>
    Reviewed-by: default avatarBob Picco <bob.picco@oracle.com>
    Acked-by: default avatarDavid S. Miller <davem@davemloft.net>
    Cc: Alexander Potapenko <glider@google.com>
    Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
    Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
    Cc: Catalin Marinas <catalin.marinas@arm.com>
    Cc: Christian Borntraeger <borntraeger@de.ibm.com>
    Cc: Dmitry Vyukov <dvyukov@google.com>
    Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
    Cc: "H. Peter Anvin" <hpa@zytor.com>
    Cc: Ingo Molnar <mingo@redhat.com>
    Cc: Mark Rutland <mark.rutland@arm.com>
    Cc: Matthew Wilcox <willy@infradead.org>
    Cc: Mel Gorman <mgorman@techsingularity.net>
    Cc: Michal Hocko <mhocko@kernel.org>
    Cc: Michal Hocko <mhocko@suse.com>
    Cc: Sam Ravnborg <sam@ravnborg.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Will Deacon <will.deacon@arm.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    78c94366
pgtable_64.h 30.7 KB