• Steven Rostedt (VMware)'s avatar
    printk/tracing: Do not trace printk_nmi_enter() · d1c392c9
    Steven Rostedt (VMware) authored
    I hit the following splat in my tests:
    
    ------------[ cut here ]------------
    IRQs not enabled as expected
    WARNING: CPU: 3 PID: 0 at kernel/time/tick-sched.c:982 tick_nohz_idle_enter+0x44/0x8c
    Modules linked in: ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables ipv6
    CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.0-rc2-test+ #2
    Hardware name: MSI MS-7823/CSM-H87M-G43 (MS-7823), BIOS V1.6 02/22/2014
    EIP: tick_nohz_idle_enter+0x44/0x8c
    Code: ec 05 00 00 00 75 26 83 b8 c0 05 00 00 00 75 1d 80 3d d0 36 3e c1 00
    75 14 68 94 63 12 c1 c6 05 d0 36 3e c1 01 e8 04 ee f8 ff <0f> 0b 58 fa bb a0
    e5 66 c1 e8 25 0f 04 00 64 03 1d 28 31 52 c1 8b
    EAX: 0000001c EBX: f26e7f8c ECX: 00000006 EDX: 00000007
    ESI: f26dd1c0 EDI: 00000000 EBP: f26e7f40 ESP: f26e7f38
    DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010296
    CR0: 80050033 CR2: 0813c6b0 CR3: 2f342000 CR4: 001406f0
    Call Trace:
     do_idle+0x33/0x202
     cpu_startup_entry+0x61/0x63
     start_secondary+0x18e/0x1ed
     startup_32_smp+0x164/0x168
    irq event stamp: 18773830
    hardirqs last  enabled at (18773829): [<c040150c>] trace_hardirqs_on_thunk+0xc/0x10
    hardirqs last disabled at (18773830): [<c040151c>] trace_hardirqs_off_thunk+0xc/0x10
    softirqs last  enabled at (18773824): [<c0ddaa6f>] __do_softirq+0x25f/0x2bf
    softirqs last disabled at (18773767): [<c0416bbe>] call_on_stack+0x45/0x4b
    ---[ end trace b7c64aa79e17954a ]---
    
    After a bit of debugging, I found what was happening. This would trigger
    when performing "perf" with a high NMI interrupt rate, while enabling and
    disabling function tracer. Ftrace uses breakpoints to convert the nops at
    the start of functions to calls to the function trampolines. The breakpoint
    traps disable interrupts and this makes calls into lockdep via the
    trace_hardirqs_off_thunk in the entry.S code. What happens is the following:
    
      do_idle {
    
        [interrupts enabled]
    
        <interrupt> [interrupts disabled]
    	TRACE_IRQS_OFF [lockdep says irqs off]
    	[...]
    	TRACE_IRQS_IRET
    	    test if pt_regs say return to interrupts enabled [yes]
    	    TRACE_IRQS_ON [lockdep says irqs are on]
    
    	    <nmi>
    		nmi_enter() {
    		    printk_nmi_enter() [traced by ftrace]
    		    [ hit ftrace breakpoint ]
    		    <breakpoint exception>
    			TRACE_IRQS_OFF [lockdep says irqs off]
    			[...]
    			TRACE_IRQS_IRET [return from breakpoint]
    			   test if pt_regs say interrupts enabled [no]
    			   [iret back to interrupt]
    	   [iret back to code]
    
        tick_nohz_idle_enter() {
    
    	lockdep_assert_irqs_enabled() [lockdep say no!]
    
    Although interrupts are indeed enabled, lockdep thinks it is not, and since
    we now do asserts via lockdep, it gives a false warning. The issue here is
    that printk_nmi_enter() is called before lockdep_off(), which disables
    lockdep (for this reason) in NMIs. By simply not allowing ftrace to see
    printk_nmi_enter() (via notrace annotation) we keep lockdep from getting
    confused.
    
    Cc: stable@vger.kernel.org
    Fixes: 42a0bb3f ("printk/nmi: generic solution for safe printk in NMI")
    Acked-by: default avatarSergey Senozhatsky <sergey.senozhatsky@gmail.com>
    Acked-by: default avatarPetr Mladek <pmladek@suse.com>
    Signed-off-by: default avatarSteven Rostedt (VMware) <rostedt@goodmis.org>
    d1c392c9
printk_safe.c 11.2 KB