• Mark Rutland's avatar
    arm64: entry: fix non-NMI kernel<->kernel transitions · 7cd1ea10
    Mark Rutland authored
    There are periods in kernel mode when RCU is not watching and/or the
    scheduler tick is disabled, but we can still take exceptions such as
    interrupts. The arm64 exception handlers do not account for this, and
    it's possible that RCU is not watching while an exception handler runs.
    
    The x86/generic entry code handles this by ensuring that all (non-NMI)
    kernel exception handlers call irqentry_enter() and irqentry_exit(),
    which handle RCU, lockdep, and IRQ flag tracing. We can't yet move to
    the generic entry code, and already hadnle the user<->kernel transitions
    elsewhere, so we add new kernel<->kernel transition helpers alog the
    lines of the generic entry code.
    
    Since we now track interrupts becoming masked when an exception is
    taken, local_daif_inherit() is modified to track interrupts becoming
    re-enabled when the original context is inherited. To balance the
    entry/exit paths, each handler masks all DAIF exceptions before
    exit_to_kernel_mode().
    Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
    Cc: Catalin Marinas <catalin.marinas@arm.com>
    Cc: James Morse <james.morse@arm.com>
    Cc: Will Deacon <will@kernel.org>
    Link: https://lore.kernel.org/r/20201130115950.22492-10-mark.rutland@arm.comSigned-off-by: default avatarWill Deacon <will@kernel.org>
    7cd1ea10
daifflags.h 3.29 KB