1. 10 Aug, 2017 4 commits
  2. 09 Aug, 2017 13 commits
    • Ard Biesheuvel's avatar
      md/raid6: implement recovery using ARM NEON intrinsics · 6ec4e251
      Ard Biesheuvel authored
      Provide a NEON accelerated implementation of the recovery algorithm,
      which supersedes the default byte-by-byte one.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      6ec4e251
    • Ard Biesheuvel's avatar
      md/raid6: use faster multiplication for ARM NEON delta syndrome · 35129dde
      Ard Biesheuvel authored
      The P/Q left side optimization in the delta syndrome simply involves
      repeatedly multiplying a value by polynomial 'x' in GF(2^8). Given
      that 'x * x * x * x' equals 'x^4' even in the polynomial world, we
      can accelerate this substantially by performing up to 4 such operations
      at once, using the NEON instructions for polynomial multiplication.
      
      Results on a Cortex-A57 running in 64-bit mode:
      
        Before:
        -------
        raid6: neonx1   xor()  1680 MB/s
        raid6: neonx2   xor()  2286 MB/s
        raid6: neonx4   xor()  3162 MB/s
        raid6: neonx8   xor()  3389 MB/s
      
        After:
        ------
        raid6: neonx1   xor()  2281 MB/s
        raid6: neonx2   xor()  3362 MB/s
        raid6: neonx4   xor()  3787 MB/s
        raid6: neonx8   xor()  4239 MB/s
      
      While we're at it, simplify MASK() by using a signed shift rather than
      a vector compare involving a temp register.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      35129dde
    • Catalin Marinas's avatar
      Merge tag 'arm64-iort-for-v4.14' of... · f39c3f9b
      Catalin Marinas authored
      Merge tag 'arm64-iort-for-v4.14' of git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux into for-next/core
      
      Two patches for the 4.14 release merge window:
      
      - IORT PCI aliases detection improvement to cater for systems with
        complex PCI topologies that current code mishandles (R.Murphy)
      - IORT SMMUv3 proximity domain (ie NUMA) handling (respective ACPICA
        changes will be merged separately) (G. Kulkarni)
      
      * tag 'arm64-iort-for-v4.14' of git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux:
        ACPI/IORT: numa: Add numa node mapping for smmuv3 devices
        ACPI/IORT: Handle PCI aliases properly for IOMMUs
      f39c3f9b
    • Catalin Marinas's avatar
      Merge branch 'arm64/exception-stack' of... · 05538967
      Catalin Marinas authored
      Merge branch 'arm64/exception-stack' of git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux into for-next/core
      
      * 'arm64/exception-stack' of git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux:
        arm64: unwind: remove sp from struct stackframe
        arm64: unwind: reference pt_regs via embedded stack frame
        arm64: unwind: disregard frame.sp when validating frame pointer
        arm64: unwind: avoid percpu indirection for irq stack
        arm64: move non-entry code out of .entry.text
        arm64: consistently use bl for C exception entry
        arm64: Add ASM_BUG()
      05538967
    • Ard Biesheuvel's avatar
      arm64: unwind: remove sp from struct stackframe · 31e43ad3
      Ard Biesheuvel authored
      The unwind code sets the sp member of struct stackframe to
      'frame pointer + 0x10' unconditionally, without regard for whether
      doing so produces a legal value. So let's simply remove it now that
      we have stopped using it anyway.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      31e43ad3
    • Ard Biesheuvel's avatar
      arm64: unwind: reference pt_regs via embedded stack frame · 73267498
      Ard Biesheuvel authored
      As it turns out, the unwind code is slightly broken, and probably has
      been for a while. The problem is in the dumping of the exception stack,
      which is intended to dump the contents of the pt_regs struct at each
      level in the call stack where an exception was taken and routed to a
      routine marked as __exception (which means its stack frame is right
      below the pt_regs struct on the stack).
      
      'Right below the pt_regs struct' is ill defined, though: the unwind
      code assigns 'frame pointer + 0x10' to the .sp member of the stackframe
      struct at each level, and dump_backtrace() happily dereferences that as
      the pt_regs pointer when encountering an __exception routine. However,
      the actual size of the stack frame created by this routine (which could
      be one of many __exception routines we have in the kernel) is not known,
      and so frame.sp is pretty useless to figure out where struct pt_regs
      really is.
      
      So it seems the only way to ensure that we can find our struct pt_regs
      when walking the stack frames is to put it at a known fixed offset of
      the stack frame pointer that is passed to such __exception routines.
      The simplest way to do that is to put it inside pt_regs itself, which is
      the main change implemented by this patch. As a bonus, doing this allows
      us to get rid of a fair amount of cruft related to walking from one stack
      to the other, which is especially nice since we intend to introduce yet
      another stack for overflow handling once we add support for vmapped
      stacks. It also fixes an inconsistency where we only add a stack frame
      pointing to ELR_EL1 if we are executing from the IRQ stack but not when
      we are executing from the task stack.
      
      To consistly identify exceptions regs even in the presence of exceptions
      taken from entry code, we must check whether the next frame was created
      by entry text, rather than whether the current frame was crated by
      exception text.
      
      To avoid backtracing using PCs that fall in the idmap, or are controlled
      by userspace, we must explcitly zero the FP and LR in startup paths, and
      must ensure that the frame embedded in pt_regs is zeroed upon entry from
      EL0. To avoid these NULL entries showin in the backtrace, unwind_frame()
      is updated to avoid them.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      [Mark: compare current frame against .entry.text, avoid bogus PCs]
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      73267498
    • Dmitry Safonov's avatar
      arm64/vdso: Support mremap() for vDSO · 73958695
      Dmitry Safonov authored
      vDSO VMA address is saved in mm_context for the purpose of using
      restorer from vDSO page to return to userspace after signal handling.
      
      In Checkpoint Restore in Userspace (CRIU) project we place vDSO VMA
      on restore back to the place where it was on the dump.
      With the exception for x86 (where there is API to map vDSO with
      arch_prctl()), we move vDSO inherited from CRIU task to restoree
      position by mremap().
      
      CRIU does support arm64 architecture, but kernel doesn't update
      context.vdso pointer after mremap(). Which results in translation
      fault after signal handling on restored application:
      https://github.com/xemul/criu/issues/288
      
      Make vDSO code track the VMA address by supplying .mremap() fops
      the same way it's done for x86 and arm32 by:
      commit b059a453 ("x86/vdso: Add mremap hook to vm_special_mapping")
      commit 280e87e9 ("ARM: 8683/1: ARM32: Support mremap() for sigpage/vDSO").
      
      Cc: Russell King <rmk+kernel@armlinux.org.uk>
      Cc: linux-arm-kernel@lists.infradead.org
      Cc: Cyrill Gorcunov <gorcunov@openvz.org>
      Cc: Pavel Emelyanov <xemul@virtuozzo.com>
      Cc: Christopher Covington <cov@codeaurora.org>
      Reviewed-by: default avatarWill Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarDmitry Safonov <dsafonov@virtuozzo.com>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      73958695
    • Robin Murphy's avatar
      arm64: uaccess: Implement *_flushcache variants · 5d7bdeb1
      Robin Murphy authored
      Implement the set of copy functions with guarantees of a clean cache
      upon completion necessary to support the pmem driver.
      Reviewed-by: default avatarWill Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarRobin Murphy <robin.murphy@arm.com>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      5d7bdeb1
    • Robin Murphy's avatar
      arm64: Implement pmem API support · d50e071f
      Robin Murphy authored
      Add a clean-to-point-of-persistence cache maintenance helper, and wire
      up the basic architectural support for the pmem driver based on it.
      Reviewed-by: default avatarWill Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarRobin Murphy <robin.murphy@arm.com>
      [catalin.marinas@arm.com: move arch_*_pmem() functions to arch/arm64/mm/flush.c]
      [catalin.marinas@arm.com: change dmb(sy) to dmb(osh)]
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      d50e071f
    • Robin Murphy's avatar
      arm64: Handle trapped DC CVAP · e1bc5d1b
      Robin Murphy authored
      Cache clean to PoP is subject to the same access controls as to PoC, so
      if we are trapping userspace cache maintenance with SCTLR_EL1.UCI, we
      need to be prepared to handle it. To avoid getting into complicated
      fights with binutils about ARMv8.2 options, we'll just cheat and use the
      raw SYS instruction rather than the 'proper' DC alias.
      Reviewed-by: default avatarWill Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarRobin Murphy <robin.murphy@arm.com>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      e1bc5d1b
    • Robin Murphy's avatar
      arm64: Expose DC CVAP to userspace · 7aac405e
      Robin Murphy authored
      The ARMv8.2-DCPoP feature introduces persistent memory support to the
      architecture, by defining a point of persistence in the memory
      hierarchy, and a corresponding cache maintenance operation, DC CVAP.
      Expose the support via HWCAP and MRS emulation.
      Reviewed-by: default avatarWill Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarRobin Murphy <robin.murphy@arm.com>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      7aac405e
    • Robin Murphy's avatar
      arm64: Convert __inval_cache_range() to area-based · d46befef
      Robin Murphy authored
      __inval_cache_range() is already the odd one out among our data cache
      maintenance routines as the only remaining range-based one; as we're
      going to want an invalidation routine to call from C code for the pmem
      API, let's tweak the prototype and name to bring it in line with the
      clean operations, and to make its relationship with __dma_inv_area()
      neatly mirror that of __clean_dcache_area_poc() and __dma_clean_area().
      The loop clearing the early page tables gets mildly massaged in the
      process for the sake of consistency.
      Reviewed-by: default avatarWill Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarRobin Murphy <robin.murphy@arm.com>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      d46befef
    • Robin Murphy's avatar
      arm64: mm: Fix set_memory_valid() declaration · 09c2a7dc
      Robin Murphy authored
      Clearly, set_memory_valid() has never been seen in the same room as its
      declaration... Whilst the type mismatch is such that kexec probably
      wasn't broken in practice, fix it to match the definition as it should.
      
      Fixes: 9b0aa14e ("arm64: mm: add set_memory_valid()")
      Reviewed-by: default avatarWill Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarRobin Murphy <robin.murphy@arm.com>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      09c2a7dc
  3. 08 Aug, 2017 7 commits
    • Ganapatrao Kulkarni's avatar
      ACPI/IORT: numa: Add numa node mapping for smmuv3 devices · 5fe0ce3b
      Ganapatrao Kulkarni authored
      ARM IORT specification(rev. C) has added  provision to define proximity
      domain in SMMUv3 IORT table. Adding required code to parse Proximity
      domain and set numa_node of smmv3 platform devices.
      Signed-off-by: default avatarGanapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
      [lorenzo.pieralisi@arm.com: update pr_info()/commit log]
      Signed-off-by: default avatarLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      5fe0ce3b
    • Ard Biesheuvel's avatar
      arm64: unwind: disregard frame.sp when validating frame pointer · c7365330
      Ard Biesheuvel authored
      Currently, when unwinding the call stack, we validate the frame pointer
      of each frame against frame.sp, whose value is not clearly defined, and
      which makes it more difficult to link stack frames together across
      different stacks. It is far better to simply check whether the frame
      pointer itself points into a valid stack.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      c7365330
    • Mark Rutland's avatar
      arm64: unwind: avoid percpu indirection for irq stack · 09668372
      Mark Rutland authored
      Our IRQ_STACK_PTR() and on_irq_stack() helpers both take a cpu argument,
      used to generate a percpu address. In all cases, they are passed
      {raw_,}smp_processor_id(), so this parameter is redundant.
      
      Since {raw_,}smp_processor_id() use a percpu variable internally, this
      approach means we generate a percpu offset to find the current cpu, then
      use this to index an array of percpu offsets, which we then use to find
      the current CPU's IRQ stack pointer. Thus, most of the work is
      redundant.
      
      Instead, we can consistently use raw_cpu_ptr() to generate the CPU's
      irq_stack pointer by simply adding the percpu offset to the irq_stack
      address, which is simpler in both respects.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      09668372
    • Mark Rutland's avatar
      arm64: move non-entry code out of .entry.text · ed84b4e9
      Mark Rutland authored
      Currently, cpu_switch_to and ret_from_fork both live in .entry.text,
      though neither form the critical path for an exception entry.
      
      In subsequent patches, we will require that code in .entry.text is part
      of the critical path for exception entry, for which we can assume
      certain properties (e.g. the presence of exception regs on the stack).
      
      Neither cpu_switch_to nor ret_from_fork will meet these requirements, so
      we must move them out of .entry.text. To ensure that neither are kprobed
      after being moved out of .entry.text, we must explicitly blacklist them,
      requiring a new NOKPROBE() asm helper.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      ed84b4e9
    • Mark Rutland's avatar
      arm64: consistently use bl for C exception entry · 2d0e751a
      Mark Rutland authored
      In most cases, our exception entry assembly branches to C handlers with
      a BL instruction, but in cases where we do not expect to return, we use
      B instead.
      
      While this is correct today, it means that backtraces for fatal
      exceptions miss the entry assembly (as the LR is stale at the point we
      call C code), while non-fatal exceptions have the entry assembly in the
      LR. In subsequent patches, we will need the LR to be set in these cases
      in order to backtrace reliably.
      
      This patch updates these sites to use a BL, ensuring consistency, and
      preparing for backtrace rework. An ASM_BUG() is added after each of
      these new BLs, which both catches unexpected returns, and ensures that
      the LR value doesn't point to another function label.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      2d0e751a
    • Mark Rutland's avatar
      arm64: Add ASM_BUG() · db44e9c5
      Mark Rutland authored
      Currently. we can only use BUG() from C code, though there are
      situations where we would like an equivalent mechanism in assembly code.
      
      This patch refactors our BUG() definition such that it can be used in
      either C or assembly, in the form of a new ASM_BUG().
      
      The refactoring requires the removal of escape sequences, such as '\n'
      and '\t', but these aren't strictly necessary as we can use ';' to
      terminate assembler statements.
      
      The low-level assembly is factored out into <asm/asm-bug.h>, with
      <asm/bug.h> retained as the C wrapper.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Dave Martin <dave.martin@arm.com>
      Cc: James Morse <james.morse@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      db44e9c5
    • Robin Murphy's avatar
      ACPI/IORT: Handle PCI aliases properly for IOMMUs · bc8648d4
      Robin Murphy authored
      When a PCI device has DMA quirks, we need to ensure that an upstream
      IOMMU knows about all possible aliases, since the presence of a DMA
      quirk does not preclude the device still also emitting transactions
      (e.g. MSIs) on its 'real' RID. Similarly, the rules for bridge aliasing
      are relatively complex, and some bridges may only take ownership of
      transactions under particular transient circumstances, leading again to
      multiple RIDs potentially being seen at the IOMMU for the given device.
      
      Take all this into account in iort_iommu_configure() by mapping every
      RID produced by the alias walk, not just whichever one comes out last.
      Since adding any more internal PTR_ERR() juggling would have confused me
      no end, a bit of refactoring happens in the process - we know where to
      find the ops if everything succeeded, so we're free to just pass regular
      error codes around up until then.
      
      CC: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      CC: Hanjun Guo <hanjun.guo@linaro.org>
      CC: Sudeep Holla <sudeep.holla@arm.com>
      Signed-off-by: default avatarRobin Murphy <robin.murphy@arm.com>
      [lorenzo.pieralisi@arm.com: tagged __get_pci_rid __maybe_unused]
      Signed-off-by: default avatarLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      bc8648d4
  4. 07 Aug, 2017 4 commits
    • Julien Thierry's avatar
      arm64: Decode information from ESR upon mem faults · 1f9b8936
      Julien Thierry authored
      When receiving unhandled faults from the CPU, description is very sparse.
      Adding information about faults decoded from ESR.
      
      Added defines to esr.h corresponding ESR fields. Values are based on ARM
      Archtecture Reference Manual (DDI 0487B.a), section D7.2.28 ESR_ELx, Exception
      Syndrome Register (ELx) (pages D7-2275 to D7-2280).
      
      New output is of the form:
      [   77.818059] Mem abort info:
      [   77.820826]   Exception class = DABT (current EL), IL = 32 bits
      [   77.826706]   SET = 0, FnV = 0
      [   77.829742]   EA = 0, S1PTW = 0
      [   77.832849] Data abort info:
      [   77.835713]   ISV = 0, ISS = 0x00000070
      [   77.839522]   CM = 0, WnR = 1
      Signed-off-by: default avatarJulien Thierry <julien.thierry@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      [catalin.marinas@arm.com: fix "%lu" in a pr_alert() call]
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      1f9b8936
    • Dave Martin's avatar
      arm64: Abstract syscallno manipulation · 17c28958
      Dave Martin authored
      The -1 "no syscall" value is written in various ways, shared with
      the user ABI in some places, and generally obscure.
      
      This patch attempts to make things a little more consistent and
      readable by replacing all these uses with a single #define.  A
      couple of symbolic helpers are provided to clarify the intent
      further.
      
      Because the in-syscall check in do_signal() is changed from >= 0 to
      != NO_SYSCALL by this patch, different behaviour may be observable
      if syscallno is set to values less than -1 by a tracer.  However,
      this is not different from the behaviour that is already observable
      if a tracer sets syscallno to a value >= __NR_(compat_)syscalls.
      
      It appears that this can cause spurious syscall restarting, but
      that is not a new behaviour either, and does not appear harmful.
      Signed-off-by: default avatarDave Martin <Dave.Martin@arm.com>
      Acked-by: default avatarWill Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      17c28958
    • Dave Martin's avatar
      arm64: syscallno is secretly an int, make it official · 35d0e6fb
      Dave Martin authored
      The upper 32 bits of the syscallno field in thread_struct are
      handled inconsistently, being sometimes zero extended and sometimes
      sign-extended.  In fact, only the lower 32 bits seem to have any
      real significance for the behaviour of the code: it's been OK to
      handle the upper bits inconsistently because they don't matter.
      
      Currently, the only place I can find where those bits are
      significant is in calling trace_sys_enter(), which may be
      unintentional: for example, if a compat tracer attempts to cancel a
      syscall by passing -1 to (COMPAT_)PTRACE_SET_SYSCALL at the
      syscall-enter-stop, it will be traced as syscall 4294967295
      rather than -1 as might be expected (and as occurs for a native
      tracer doing the same thing).  Elsewhere, reads of syscallno cast
      it to an int or truncate it.
      
      There's also a conspicuous amount of code and casting to bodge
      around the fact that although semantically an int, syscallno is
      stored as a u64.
      
      Let's not pretend any more.
      
      In order to preserve the stp x instruction that stores the syscall
      number in entry.S, this patch special-cases the layout of struct
      pt_regs for big endian so that the newly 32-bit syscallno field
      maps onto the low bits of the stored value.  This is not beautiful,
      but benchmarking of the getpid syscall on Juno suggests indicates a
      minor slowdown if the stp is split into an stp x and stp w.
      Signed-off-by: default avatarDave Martin <Dave.Martin@arm.com>
      Acked-by: default avatarWill Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      35d0e6fb
    • Linus Torvalds's avatar
      Linux 4.13-rc4 · aae4e7a8
      Linus Torvalds authored
      aae4e7a8
  5. 06 Aug, 2017 12 commits