• Li Rongqing's avatar
    ipc: prevent lockup on alloc_msg and free_msg · d6a2946a
    Li Rongqing authored
    msgctl10 of ltp triggers the following lockup When CONFIG_KASAN is
    enabled on large memory SMP systems, the pages initialization can take a
    long time, if msgctl10 requests a huge block memory, and it will block
    rcu scheduler, so release cpu actively.
    
    After adding schedule() in free_msg, free_msg can not be called when
    holding spinlock, so adding msg to a tmp list, and free it out of
    spinlock
    
      rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
      rcu:     Tasks blocked on level-1 rcu_node (CPUs 16-31): P32505
      rcu:     Tasks blocked on level-1 rcu_node (CPUs 48-63): P34978
      rcu:     (detected by 11, t=35024 jiffies, g=44237529, q=16542267)
      msgctl10        R  running task    21608 32505   2794 0x00000082
      Call Trace:
       preempt_schedule_irq+0x4c/0xb0
       retint_kernel+0x1b/0x2d
      RIP: 0010:__is_insn_slot_addr+0xfb/0x250
      Code: 82 1d 00 48 8b 9b 90 00 00 00 4c 89 f7 49 c1 ee 03 e8 59 83 1d 00 48 b8 00 00 00 00 00 fc ff df 4c 39 eb 48 89 9d 58 ff ff ff <41> c6 04 06 f8 74 66 4c 8d 75 98 4c 89 f1 48 c1 e9 03 48 01 c8 48
      RSP: 0018:ffff88bce041f758 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
      RAX: dffffc0000000000 RBX: ffffffff8471bc50 RCX: ffffffff828a2a57
      RDX: dffffc0000000000 RSI: dffffc0000000000 RDI: ffff88bce041f780
      RBP: ffff88bce041f828 R08: ffffed15f3f4c5b3 R09: ffffed15f3f4c5b3
      R10: 0000000000000001 R11: ffffed15f3f4c5b2 R12: 000000318aee9b73
      R13: ffffffff8471bc50 R14: 1ffff1179c083ef0 R15: 1ffff1179c083eec
       kernel_text_address+0xc1/0x100
       __kernel_text_address+0xe/0x30
       unwind_get_return_address+0x2f/0x50
       __save_stack_trace+0x92/0x100
       create_object+0x380/0x650
       __kmalloc+0x14c/0x2b0
       load_msg+0x38/0x1a0
       do_msgsnd+0x19e/0xcf0
       do_syscall_64+0x117/0x400
       entry_SYSCALL_64_after_hwframe+0x49/0xbe
    
      rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
      rcu:     Tasks blocked on level-1 rcu_node (CPUs 0-15): P32170
      rcu:     (detected by 14, t=35016 jiffies, g=44237525, q=12423063)
      msgctl10        R  running task    21608 32170  32155 0x00000082
      Call Trace:
       preempt_schedule_irq+0x4c/0xb0
       retint_kernel+0x1b/0x2d
      RIP: 0010:lock_acquire+0x4d/0x340
      Code: 48 81 ec c0 00 00 00 45 89 c6 4d 89 cf 48 8d 6c 24 20 48 89 3c 24 48 8d bb e4 0c 00 00 89 74 24 0c 48 c7 44 24 20 b3 8a b5 41 <48> c1 ed 03 48 c7 44 24 28 b4 25 18 84 48 c7 44 24 30 d0 54 7a 82
      RSP: 0018:ffff88af83417738 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
      RAX: dffffc0000000000 RBX: ffff88bd335f3080 RCX: 0000000000000002
      RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88bd335f3d64
      RBP: ffff88af83417758 R08: 0000000000000000 R09: 0000000000000000
      R10: 0000000000000001 R11: ffffed13f3f745b2 R12: 0000000000000000
      R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000
       is_bpf_text_address+0x32/0xe0
       kernel_text_address+0xec/0x100
       __kernel_text_address+0xe/0x30
       unwind_get_return_address+0x2f/0x50
       __save_stack_trace+0x92/0x100
       save_stack+0x32/0xb0
       __kasan_slab_free+0x130/0x180
       kfree+0xfa/0x2d0
       free_msg+0x24/0x50
       do_msgrcv+0x508/0xe60
       do_syscall_64+0x117/0x400
       entry_SYSCALL_64_after_hwframe+0x49/0xbe
    
    Davidlohr said:
     "So after releasing the lock, the msg rbtree/list is empty and new
      calls will not see those in the newly populated tmp_msg list, and
      therefore they cannot access the delayed msg freeing pointers, which
      is good. Also the fact that the node_cache is now freed before the
      actual messages seems to be harmless as this is wanted for
      msg_insert() avoiding GFP_ATOMIC allocations, and after releasing the
      info->lock the thing is freed anyway so it should not change things"
    
    Link: http://lkml.kernel.org/r/1552029161-4957-1-git-send-email-lirongqing@baidu.comSigned-off-by: default avatarLi RongQing <lirongqing@baidu.com>
    Signed-off-by: default avatarZhang Yu <zhangyu31@baidu.com>
    Reviewed-by: default avatarDavidlohr Bueso <dbueso@suse.de>
    Cc: Manfred Spraul <manfred@colorfullife.com>
    Cc: Arnd Bergmann <arnd@arndb.de>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    d6a2946a
msgutil.c 3.67 KB