1. 30 Nov, 2018 3 commits
    • Mark Rutland's avatar
      arm64: ftrace: use GLOBAL() · e4fe1966
      Mark Rutland authored
      The global exports of ftrace_call and ftrace_graph_call are somewhat
      painful to read. Let's use the generic GLOBAL() macro to ameliorate
      matters.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: AKASHI Takahiro <takahiro.akashi@linaro.org>
      Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Torsten Duwe <duwe@suse.de>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      e4fe1966
    • Mark Rutland's avatar
      linkage: add generic GLOBAL() macro · ad697a1a
      Mark Rutland authored
      Declaring a global symbol in assembly is tedious, error-prone, and
      painful to read. While ENTRY() exists, this is supposed to be used for
      function entry points, and this affects alignment in a potentially
      undesireable manner.
      
      Instead, let's add a generic GLOBAL() macro for this, as x86 added
      locally in commit:
      
        95695547 ("x86: asm linkage - introduce GLOBAL macro")
      
      ... thus allowing us to use this more freely in the kernel.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: AKASHI Takahiro <takahiro.akashi@linaro.org>
      Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Torsten Duwe <duwe@suse.de>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      ad697a1a
    • Ard Biesheuvel's avatar
      arm64: drop linker script hack to hide __efistub_ symbols · dd6846d7
      Ard Biesheuvel authored
      Commit 1212f7a1 ("scripts/kallsyms: filter arm64's __efistub_
      symbols") updated the kallsyms code to filter out symbols with
      the __efistub_ prefix explicitly, so we no longer require the
      hack in our linker script to emit them as absolute symbols.
      
      Cc: Nick Desaulniers <ndesaulniers@google.com>
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      dd6846d7
  2. 29 Nov, 2018 1 commit
    • Will Deacon's avatar
      arm64: io: Ensure value passed to __iormb() is held in a 64-bit register · 1b57ec8c
      Will Deacon authored
      As of commit 6460d320 ("arm64: io: Ensure calls to delay routines
      are ordered against prior readX()"), MMIO reads smaller than 64 bits
      fail to compile under clang because we end up mixing 32-bit and 64-bit
      register operands for the same data processing instruction:
      
      ./include/asm-generic/io.h:695:9: warning: value size does not match register size specified by the constraint and modifier [-Wasm-operand-widths]
              return readb(addr);
                     ^
      ./arch/arm64/include/asm/io.h:147:58: note: expanded from macro 'readb'
                                                                             ^
      ./include/asm-generic/io.h:695:9: note: use constraint modifier "w"
      ./arch/arm64/include/asm/io.h:147:50: note: expanded from macro 'readb'
                                                                     ^
      ./arch/arm64/include/asm/io.h:118:24: note: expanded from macro '__iormb'
              asm volatile("eor       %0, %1, %1\n"                           \
                                          ^
      
      Fix the build by casting the macro argument to 'unsigned long' when used
      as an input to the inline asm.
      Reported-by: default avatarNick Desaulniers <nick.desaulniers@gmail.com>
      Reported-by: default avatarNathan Chancellor <natechancellor@gmail.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      1b57ec8c
  3. 27 Nov, 2018 6 commits
    • Will Deacon's avatar
      arm64: tlbi: Set MAX_TLBI_OPS to PTRS_PER_PTE · 3d65b6bb
      Will Deacon authored
      In order to reduce the possibility of soft lock-ups, we bound the
      maximum number of TLBI operations performed by a single call to
      flush_tlb_range() to an arbitrary constant of 1024.
      
      Whilst this does the job of avoiding lock-ups, we can actually be a bit
      smarter by defining this as PTRS_PER_PTE. Due to the structure of our
      page tables, using PTRS_PER_PTE means that an outer loop calling
      flush_tlb_range() for entire table entries will end up performing just a
      single TLBI operation for each entry. As an example, mremap()ing a 1GB
      range mapped using 4k pages now requires only 512 TLBI operations when
      moving the page tables as opposed to 262144 operations (512*512) when
      using the current threshold of 1024.
      
      Cc: Joel Fernandes <joel@joelfernandes.org>
      Acked-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      3d65b6bb
    • Ard Biesheuvel's avatar
      arm64/module: switch to ADRP/ADD sequences for PLT entries · bdb85cd1
      Ard Biesheuvel authored
      Now that we have switched to the small code model entirely, and
      reduced the extended KASLR range to 4 GB, we can be sure that the
      targets of relative branches that are out of range are in range
      for a ADRP/ADD pair, which is one instruction shorter than our
      current MOVN/MOVK/MOVK sequence, and is more idiomatic and so it
      is more likely to be implemented efficiently by micro-architectures.
      
      So switch over the ordinary PLT code and the special handling of
      the Cortex-A53 ADRP errata, as well as the ftrace trampline
      handling.
      Reviewed-by: default avatarTorsten Duwe <duwe@lst.de>
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      [will: Added a couple of comments in the plt equality check]
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      bdb85cd1
    • Ard Biesheuvel's avatar
      arm64/insn: add support for emitting ADR/ADRP instructions · 7aaf7b2f
      Ard Biesheuvel authored
      Add support for emitting ADR and ADRP instructions so we can switch
      over our PLT generation code in a subsequent patch.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      7aaf7b2f
    • James Morse's avatar
      arm64: Use a raw spinlock in __install_bp_hardening_cb() · d8797b12
      James Morse authored
      __install_bp_hardening_cb() is called via stop_machine() as part
      of the cpu_enable callback. To force each CPU to take its turn
      when allocating slots, they take a spinlock.
      
      With the RT patches applied, the spinlock becomes a mutex,
      and we get warnings about sleeping while in stop_machine():
      | [    0.319176] CPU features: detected: RAS Extension Support
      | [    0.319950] BUG: scheduling while atomic: migration/3/36/0x00000002
      | [    0.319955] Modules linked in:
      | [    0.319958] Preemption disabled at:
      | [    0.319969] [<ffff000008181ae4>] cpu_stopper_thread+0x7c/0x108
      | [    0.319973] CPU: 3 PID: 36 Comm: migration/3 Not tainted 4.19.1-rt3-00250-g330fc2c2a880 #2
      | [    0.319975] Hardware name: linux,dummy-virt (DT)
      | [    0.319976] Call trace:
      | [    0.319981]  dump_backtrace+0x0/0x148
      | [    0.319983]  show_stack+0x14/0x20
      | [    0.319987]  dump_stack+0x80/0xa4
      | [    0.319989]  __schedule_bug+0x94/0xb0
      | [    0.319991]  __schedule+0x510/0x560
      | [    0.319992]  schedule+0x38/0xe8
      | [    0.319994]  rt_spin_lock_slowlock_locked+0xf0/0x278
      | [    0.319996]  rt_spin_lock_slowlock+0x5c/0x90
      | [    0.319998]  rt_spin_lock+0x54/0x58
      | [    0.320000]  enable_smccc_arch_workaround_1+0xdc/0x260
      | [    0.320001]  __enable_cpu_capability+0x10/0x20
      | [    0.320003]  multi_cpu_stop+0x84/0x108
      | [    0.320004]  cpu_stopper_thread+0x84/0x108
      | [    0.320008]  smpboot_thread_fn+0x1e8/0x2b0
      | [    0.320009]  kthread+0x124/0x128
      | [    0.320010]  ret_from_fork+0x10/0x18
      
      Switch this to a raw spinlock, as we know this is only called with
      IRQs masked.
      Signed-off-by: default avatarJames Morse <james.morse@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      d8797b12
    • Jeremy Linton's avatar
      arm64: acpi: Prepare for longer MADTs · 9eb1c92b
      Jeremy Linton authored
      The BAD_MADT_GICC_ENTRY check is a little too strict because
      it rejects MADT entries that don't match the currently known
      lengths. We should remove this restriction to avoid problems
      if the table length changes. Future code which might depend on
      additional fields should be written to validate those fields
      before using them, rather than trying to globally check
      known MADT version lengths.
      
      Link: https://lkml.kernel.org/r/20181012192937.3819951-1-jeremy.linton@arm.comSigned-off-by: default avatarJeremy Linton <jeremy.linton@arm.com>
      [lorenzo.pieralisi@arm.com: added MADT macro comments]
      Signed-off-by: default avatarLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Acked-by: default avatarSudeep Holla <sudeep.holla@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Al Stone <ahs3@redhat.com>
      Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      9eb1c92b
    • Will Deacon's avatar
      arm64: io: Ensure calls to delay routines are ordered against prior readX() · 6460d320
      Will Deacon authored
      A relatively standard idiom for ensuring that a pair of MMIO writes to a
      device arrive at that device with a specified minimum delay between them
      is as follows:
      
      	writel_relaxed(42, dev_base + CTL1);
      	readl(dev_base + CTL1);
      	udelay(10);
      	writel_relaxed(42, dev_base + CTL2);
      
      the intention being that the read-back from the device will push the
      prior write to CTL1, and the udelay will hold up the write to CTL1 until
      at least 10us have elapsed.
      
      Unfortunately, on arm64 where the underlying delay loop is implemented
      as a read of the architected counter, the CPU does not guarantee
      ordering from the readl() to the delay loop and therefore the delay loop
      could in theory be speculated and not provide the desired interval
      between the two writes.
      
      Fix this in a similar manner to PowerPC by introducing a dummy control
      dependency on the output of readX() which, combined with the ISB in the
      read of the architected counter, guarantees that a subsequent delay loop
      can not be executed until the readX() has returned its result.
      
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Arnd Bergmann <arnd@arndb.de>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      6460d320
  4. 26 Nov, 2018 1 commit
    • Alex Van Brunt's avatar
      arm64: mm: Don't wait for completion of TLB invalidation when page aging · 3403e56b
      Alex Van Brunt authored
      When transitioning a PTE from young to old as part of page aging, we
      can avoid waiting for the TLB invalidation to complete and therefore
      drop the subsequent DSB instruction. Whilst this opens up a race with
      page reclaim, where a PTE in active use via a stale, young TLB entry
      does not update the underlying descriptor, the worst thing that happens
      is that the page is reclaimed and then immediately faulted back in.
      
      Given that we have a DSB in our context-switch path, the window for a
      spurious reclaim is fairly limited and eliding the barrier claims to
      boost NVMe/SSD accesses by over 10% on some platforms.
      
      A similar optimisation was made for x86 in commit b13b1d2d ("x86/mm:
      In the PTE swapout page reclaim case clear the accessed bit instead of
      flushing the TLB").
      Signed-off-by: default avatarAlex Van Brunt <avanbrunt@nvidia.com>
      Signed-off-by: default avatarAshish Mhetre <amhetre@nvidia.com>
      [will: rewrote patch]
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      3403e56b
  5. 20 Nov, 2018 3 commits
    • Jessica Yu's avatar
      arm64/module: use plt section indices for relocations · c8ebf64e
      Jessica Yu authored
      Instead of saving a pointer to the .plt and .init.plt sections to apply
      plt-based relocations, save and use their section indices instead.
      
      The mod->arch.{core,init}.plt pointers were problematic for livepatch
      because they pointed within temporary section headers (provided by the
      module loader via info->sechdrs) that would be freed after module load.
      Since livepatch modules may need to apply relocations post-module-load
      (for example, to patch a module that is loaded later), using section
      indices to offset into the section headers (instead of accessing them
      through a saved pointer) allows livepatch modules on arm64 to pass in
      their own copy of the section headers to apply_relocate_add() to apply
      delayed relocations.
      Reviewed-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Reviewed-by: default avatarMiroslav Benes <mbenes@suse.cz>
      Signed-off-by: default avatarJessica Yu <jeyu@kernel.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      c8ebf64e
    • Ard Biesheuvel's avatar
      arm64: mm: apply r/o permissions of VM areas to its linear alias as well · c55191e9
      Ard Biesheuvel authored
      On arm64, we use block mappings and contiguous hints to map the linear
      region, to minimize the TLB footprint. However, this means that the
      entire region is mapped using read/write permissions, which we cannot
      modify at page granularity without having to take intrusive measures to
      prevent TLB conflicts.
      
      This means the linear aliases of pages belonging to read-only mappings
      (executable or otherwise) in the vmalloc region are also mapped read/write,
      and could potentially be abused to modify things like module code, bpf JIT
      code or other read-only data.
      
      So let's fix this, by extending the set_memory_ro/rw routines to take
      the linear alias into account. The consequence of enabling this is
      that we can no longer use block mappings or contiguous hints, so in
      cases where the TLB footprint of the linear region is a bottleneck,
      performance may be affected.
      
      Therefore, allow this feature to be runtime en/disabled, by setting
      rodata=full (or 'on' to disable just this enhancement, or 'off' to
      disable read-only mappings for code and r/o data entirely) on the
      kernel command line. Also, allow the default value to be set via a
      Kconfig option.
      Tested-by: default avatarLaura Abbott <labbott@redhat.com>
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      c55191e9
    • Ard Biesheuvel's avatar
      arm64: mm: purge lazily unmapped vm regions before changing permissions · b34d2ef0
      Ard Biesheuvel authored
      Call vm_unmap_aliases() every time we apply any changes to permission
      attributes of mappings in the vmalloc region. This avoids any potential
      issues resulting from lingering writable or executable aliases of
      mappings that should be read-only or non-executable, respectively.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      b34d2ef0
  6. 18 Nov, 2018 23 commits
  7. 16 Nov, 2018 3 commits