• Minchan Kim's avatar
    mm: use put_page() to free page instead of putback_lru_page() · c6c919eb
    Minchan Kim authored
    Recently, I got many reports about perfermance degradation in embedded
    system(Android mobile phone, webOS TV and so on) and easy fork fail.
    
    The problem was fragmentation caused by zram and GPU driver mainly.
    With memory pressure, their pages were spread out all of pageblock and
    it cannot be migrated with current compaction algorithm which supports
    only LRU pages.  In the end, compaction cannot work well so reclaimer
    shrinks all of working set pages.  It made system very slow and even to
    fail to fork easily which requires order-[2 or 3] allocations.
    
    Other pain point is that they cannot use CMA memory space so when OOM
    kill happens, I can see many free pages in CMA area, which is not memory
    efficient.  In our product which has big CMA memory, it reclaims zones
    too exccessively to allocate GPU and zram page although there are lots
    of free space in CMA so system becomes very slow easily.
    
    To solve these problem, this patch tries to add facility to migrate
    non-lru pages via introducing new functions and page flags to help
    migration.
    
    struct address_space_operations {
    	..
    	..
    	bool (*isolate_page)(struct page *, isolate_mode_t);
    	void (*putback_page)(struct page *);
    	..
    }
    
    new page flags
    
    	PG_movable
    	PG_isolated
    
    For details, please read description in "mm: migrate: support non-lru
    movable page migration".
    
    Originally, Gioh Kim had tried to support this feature but he moved so I
    took over the work.  I took many code from his work and changed a little
    bit and Konstantin Khlebnikov helped Gioh a lot so he should deserve to
    have many credit, too.
    
    And I should mention Chulmin who have tested this patchset heavily so I
    can find many bugs from him.  :)
    
    Thanks, Gioh, Konstantin and Chulmin!
    
    This patchset consists of five parts.
    
    1. clean up migration
      mm: use put_page to free page instead of putback_lru_page
    
    2. add non-lru page migration feature
      mm: migrate: support non-lru movable page migration
    
    3. rework KVM memory-ballooning
      mm: balloon: use general non-lru movable page feature
    
    4. zsmalloc refactoring for preparing page migration
      zsmalloc: keep max_object in size_class
      zsmalloc: use bit_spin_lock
      zsmalloc: use accessor
      zsmalloc: factor page chain functionality out
      zsmalloc: introduce zspage structure
      zsmalloc: separate free_zspage from putback_zspage
      zsmalloc: use freeobj for index
    
    5. zsmalloc page migration
      zsmalloc: page migration support
      zram: use __GFP_MOVABLE for memory allocation
    
    This patch (of 12):
    
    Procedure of page migration is as follows:
    
    First of all, it should isolate a page from LRU and try to migrate the
    page.  If it is successful, it releases the page for freeing.
    Otherwise, it should put the page back to LRU list.
    
    For LRU pages, we have used putback_lru_page for both freeing and
    putback to LRU list.  It's okay because put_page is aware of LRU list so
    if it releases last refcount of the page, it removes the page from LRU
    list.  However, It makes unnecessary operations (e.g., lru_cache_add,
    pagevec and flags operations.  It would be not significant but no worth
    to do) and harder to support new non-lru page migration because put_page
    isn't aware of non-lru page's data structure.
    
    To solve the problem, we can add new hook in put_page with PageMovable
    flags check but it can increase overhead in hot path and needs new
    locking scheme to stabilize the flag check with put_page.
    
    So, this patch cleans it up to divide two semantic(ie, put and putback).
    If migration is successful, use put_page instead of putback_lru_page and
    use putback_lru_page only on failure.  That makes code more readable and
    doesn't add overhead in put_page.
    
    Comment from Vlastimil
     "Yeah, and compaction (perhaps also other migration users) has to drain
      the lru pvec...  Getting rid of this stuff is worth even by itself."
    
    Link: http://lkml.kernel.org/r/1464736881-24886-2-git-send-email-minchan@kernel.orgSigned-off-by: default avatarMinchan Kim <minchan@kernel.org>
    Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: Mel Gorman <mgorman@suse.de>
    Cc: Hugh Dickins <hughd@google.com>
    Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
    Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    c6c919eb
migrate.c 48.5 KB