1. 20 Apr, 2016 7 commits
  2. 19 Apr, 2016 5 commits
  3. 15 Apr, 2016 9 commits
  4. 14 Apr, 2016 18 commits
    • James Morse's avatar
      arm64: mm: Add trace_irqflags annotations to do_debug_exception() · 6afedcd2
      James Morse authored
      With CONFIG_PROVE_LOCKING, CONFIG_DEBUG_LOCKDEP and CONFIG_TRACE_IRQFLAGS
      enabled, lockdep will compare current->hardirqs_enabled with the flags from
      local_irq_save().
      
      When a debug exception occurs, interrupts are disabled in entry.S, but
      lockdep isn't told, resulting in:
      DEBUG_LOCKS_WARN_ON(current->hardirqs_enabled)
      ------------[ cut here ]------------
      WARNING: at ../kernel/locking/lockdep.c:3523
      Modules linked in:
      CPU: 3 PID: 1752 Comm: perf Not tainted 4.5.0-rc4+ #2204
      Hardware name: ARM Juno development board (r1) (DT)
      task: ffffffc974868000 ti: ffffffc975f40000 task.ti: ffffffc975f40000
      PC is at check_flags.part.35+0x17c/0x184
      LR is at check_flags.part.35+0x17c/0x184
      pc : [<ffffff80080fc93c>] lr : [<ffffff80080fc93c>] pstate: 600003c5
      [...]
      ---[ end trace 74631f9305ef5020 ]---
      Call trace:
      [<ffffff80080fc93c>] check_flags.part.35+0x17c/0x184
      [<ffffff80080ffe30>] lock_acquire+0xa8/0xc4
      [<ffffff8008093038>] breakpoint_handler+0x118/0x288
      [<ffffff8008082434>] do_debug_exception+0x3c/0xa8
      [<ffffff80080854b4>] el1_dbg+0x18/0x6c
      [<ffffff80081e82f4>] do_filp_open+0x64/0xdc
      [<ffffff80081d6e60>] do_sys_open+0x140/0x204
      [<ffffff80081d6f58>] SyS_openat+0x10/0x18
      [<ffffff8008085d30>] el0_svc_naked+0x24/0x28
      possible reason: unannotated irqs-off.
      irq event stamp: 65857
      hardirqs last  enabled at (65857): [<ffffff80081fb1c0>] lookup_mnt+0xf4/0x1b4
      hardirqs last disabled at (65856): [<ffffff80081fb188>] lookup_mnt+0xbc/0x1b4
      softirqs last  enabled at (65790): [<ffffff80080bdca4>] __do_softirq+0x1f8/0x290
      softirqs last disabled at (65757): [<ffffff80080be038>] irq_exit+0x9c/0xd0
      
      This patch adds the annotations to do_debug_exception(), while trying not
      to call trace_hardirqs_off() if el1_dbg() interrupted a task that already
      had irqs disabled.
      Signed-off-by: default avatarJames Morse <james.morse@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      6afedcd2
    • Anna-Maria Gleixner's avatar
      arm64: hw-breakpoint: Remove superfluous SMP function call · 4bc49274
      Anna-Maria Gleixner authored
      Since commit 1cf4f629 ("cpu/hotplug: Move online calls to
      hotplugged cpu") it is ensured that callbacks of CPU_ONLINE and
      CPU_DOWN_PREPARE are processed on the hotplugged CPU. Due to this SMP
      function calls are no longer required.
      
      Replace smp_call_function_single() with a direct call of
      hw_breakpoint_reset(). To keep the calling convention, interrupts are
      explicitly disabled around the call.
      
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: linux-arm-kernel@lists.infradead.org
      Signed-off-by: default avatarAnna-Maria Gleixner <anna-maria@linutronix.de>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      4bc49274
    • Anna-Maria Gleixner's avatar
      arm64/debug: Remove superfluous SMP function call · 499c8150
      Anna-Maria Gleixner authored
      Since commit 1cf4f629 ("cpu/hotplug: Move online calls to
      hotplugged cpu") it is ensured that callbacks of CPU_ONLINE and
      CPU_DOWN_PREPARE are processed on the hotplugged CPU. Due to this SMP
      function calls are no longer required.
      
      Replace smp_call_function_single() with a direct call to
      clear_os_lock(). The function writes the OSLAR register to clear OS
      locking. This does not require to be called with interrupts disabled,
      therefore the smp_call_function_single() calling convention is not
      preserved.
      
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: linux-arm-kernel@lists.infradead.org
      Signed-off-by: default avatarAnna-Maria Gleixner <anna-maria@linutronix.de>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      499c8150
    • Ard Biesheuvel's avatar
      arm64: simplify kernel segment mapping granularity · 97740051
      Ard Biesheuvel authored
      The mapping of the kernel consist of four segments, each of which is mapped
      with different permission attributes and/or lifetimes. To optimize the TLB
      and translation table footprint, we define various opaque constants in the
      linker script that resolve to different aligment values depending on the
      page size and whether CONFIG_DEBUG_ALIGN_RODATA is set.
      
      Considering that
      - a 4 KB granule kernel benefits from a 64 KB segment alignment (due to
        the fact that it allows the use of the contiguous bit),
      - the minimum alignment of the .data segment is THREAD_SIZE already, not
        PAGE_SIZE (i.e., we already have padding between _data and the start of
        the .data payload in many cases),
      - 2 MB is a suitable alignment value on all granule sizes, either for
        mapping directly (level 2 on 4 KB), or via the contiguous bit (level 3 on
        16 KB and 64 KB),
      - anything beyond 2 MB exceeds the minimum alignment mandated by the boot
        protocol, and can only be mapped efficiently if the physical alignment
        happens to be the same,
      
      we can simplify this by standardizing on 64 KB (or 2 MB) explicitly, i.e.,
      regardless of granule size, all segments are aligned either to 64 KB, or to
      2 MB if CONFIG_DEBUG_ALIGN_RODATA=y. This also means we can drop the Kconfig
      dependency of CONFIG_DEBUG_ALIGN_RODATA on CONFIG_ARM64_4K_PAGES.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      97740051
    • Ard Biesheuvel's avatar
      arm64: cover the .head.text section in the .text segment mapping · 7eb90f2f
      Ard Biesheuvel authored
      Keeping .head.text out of the .text mapping buys us very little: its actual
      payload is only 4 KB, most of which is padding, but the page alignment may
      add up to 2 MB (in case of CONFIG_DEBUG_ALIGN_RODATA=y) of additional
      padding to the uncompressed kernel Image.
      
      Also, on 4 KB granule kernels, the 4 KB misalignment of .text forces us to
      map the adjacent 56 KB of code without the PTE_CONT attribute, and since
      this region contains things like the vector table and the GIC interrupt
      handling entry point, this region is likely to benefit from the reduced TLB
      pressure that results from PTE_CONT mappings.
      
      So remove the alignment between the .head.text and .text sections, and use
      the [_text, _etext) rather than the [_stext, _etext) interval for mapping
      the .text segment.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      7eb90f2f
    • Ard Biesheuvel's avatar
      arm64: move early boot code to the .init segment · 546c8c44
      Ard Biesheuvel authored
      Apart from the arm64/linux and EFI header data structures, there is nothing
      in the .head.text section that must reside at the beginning of the Image.
      So let's move it to the .init section where it belongs.
      
      Note that this involves some minor tweaking of the EFI header, primarily
      because the address of 'stext' no longer coincides with the start of the
      .text section. It also requires a couple of relocated symbol references
      to be slightly rewritten or their definition moved to the linker script.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      546c8c44
    • Ard Biesheuvel's avatar
      arm64: use 'segment' rather than 'chunk' to describe mapped kernel regions · 2c09ec06
      Ard Biesheuvel authored
      Replace the poorly defined term chunk with segment, which is a term that is
      already used by the ELF spec to describe contiguous mappings with the same
      permission attributes of statically allocated ranges of an executable.
      Acked-by: default avatarMark Rutland <mark.rutland@arm.com>
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      2c09ec06
    • Ard Biesheuvel's avatar
      arm64: mm: restrict virt_to_page() to the linear mapping · 9f287591
      Ard Biesheuvel authored
      Now that the vmemmap region has been redefined to cover the linear region
      rather than the entire physical address space, we no longer need to
      perform a virtual-to-physical translation in the implementaion of
      virt_to_page(). This restricts virt_to_page() translations to the linear
      region, so redefine virt_addr_valid() as well.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      9f287591
    • Ard Biesheuvel's avatar
      arm64: mm: move vmemmap region right below the linear region · 3e1907d5
      Ard Biesheuvel authored
      This moves the vmemmap region right below PAGE_OFFSET, aka the start
      of the linear region, and redefines its size to be a power of two.
      Due to the placement of PAGE_OFFSET in the middle of the address space,
      whose size is a power of two as well, this guarantees that virt to
      page conversions and vice versa can be implemented efficiently, by
      masking and shifting rather than ordinary arithmetic.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      3e1907d5
    • Ard Biesheuvel's avatar
      arm64: insn: avoid virt_to_page() translations on core kernel symbols · e44308e6
      Ard Biesheuvel authored
      Before restricting virt_to_page() to the linear mapping, ensure that
      the text patching code does not use it to resolve references into the
      core kernel text, which is mapped in the vmalloc area.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      e44308e6
    • Ard Biesheuvel's avatar
      arm64: mm: avoid virt_to_page() translation for the zero page · 22b6f3b0
      Ard Biesheuvel authored
      The zero page is statically allocated, so grab its struct page pointer
      without using virt_to_page(), which will be restricted to the linear
      mapping later.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      22b6f3b0
    • Ard Biesheuvel's avatar
      arm64: mm: free __init memory via the linear mapping · d386825c
      Ard Biesheuvel authored
      The implementation of free_initmem_default() expects __init_begin
      and __init_end to be covered by the linear mapping, which is no
      longer the case. So open code it instead, using addresses that are
      explicitly translated from kernel virtual to linear virtual.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      d386825c
    • Ard Biesheuvel's avatar
      arm64: vdso: avoid virt_to_page() translations on kernel symbols · 97bbb54e
      Ard Biesheuvel authored
      The translation performed by virt_to_page() is only valid for linear
      addresses, and kernel symbols are no longer in the linear mapping.
      So perform the __pa() translation explicitly, which does the right
      thing in either case, and only then translate to a struct page offset.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      97bbb54e
    • Ard Biesheuvel's avatar
      arm64: remove the now unneeded relocate_initrd() · 8923a166
      Ard Biesheuvel authored
      This removes the relocate_initrd() implementation and invocation, which are
      no longer needed now that the placement of the initrd is guaranteed to be
      covered by the linear mapping.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      8923a166
    • Ard Biesheuvel's avatar
      arm64: add the initrd region to the linear mapping explicitly · 177e15f0
      Ard Biesheuvel authored
      Instead of going out of our way to relocate the initrd if it turns out
      to occupy memory that is not covered by the linear mapping, just add the
      initrd to the linear mapping. This puts the burden on the bootloader to
      pass initrd= and mem= options that are mutually consistent.
      
      Note that, since the placement of the linear region in the PA space is
      also dependent on the placement of the kernel Image, which may reside
      anywhere in memory, we may still end up with a situation where the initrd
      and the kernel Image are simply too far apart to be covered by the linear
      region.
      
      Since we now leave it up to the bootloader to pass the initrd in memory
      that is guaranteed to be accessible by the kernel, add a mention of this to
      the arm64 boot protocol specification as well.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      177e15f0
    • Ard Biesheuvel's avatar
      Revert "arm64: account for sparsemem section alignment when choosing vmemmap offset" · 3bab79ed
      Ard Biesheuvel authored
      This reverts commit 36e5cd6b, since the
      section alignment is now guaranteed by construction when choosing the
      value of memstart_addr.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      3bab79ed
    • Ard Biesheuvel's avatar
      arm64: choose memstart_addr based on minimum sparsemem section alignment · 06e9bf2f
      Ard Biesheuvel authored
      This redefines ARM64_MEMSTART_ALIGN in terms of the minimal alignment
      required by sparsemem vmemmap. This comes down to using 1 GB for all
      translation granules if CONFIG_SPARSEMEM_VMEMMAP is enabled.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      06e9bf2f
    • Ard Biesheuvel's avatar
      arm64/mm: ensure memstart_addr remains sufficiently aligned · 2958987f
      Ard Biesheuvel authored
      After choosing memstart_addr to be the highest multiple of
      ARM64_MEMSTART_ALIGN less than or equal to the first usable physical memory
      address, we clip the memblocks to the maximum size of the linear region.
      Since the kernel may be high up in memory, we take care not to clip the
      kernel itself, which means we have to clip some memory from the bottom if
      this occurs, to ensure that the distance between the first and the last
      usable physical memory address can be covered by the linear region.
      
      However, we fail to update memstart_addr if this clipping from the bottom
      occurs, which means that we may still end up with virtual addresses that
      wrap into the userland range. So increment memstart_addr as appropriate to
      prevent this from happening.
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      2958987f
  5. 13 Apr, 2016 1 commit
    • Jisheng Zhang's avatar
      arm64: cpuidle: make arm_cpuidle_suspend() a bit more efficient · b5fda7ed
      Jisheng Zhang authored
      Currently, we check two pointers: cpu_ops and cpu_suspend on every idle
      state entry. These pointers check can be avoided:
      
      If cpu_ops has not been registered, arm_cpuidle_init() will return
      -EOPNOTSUPP, so arm_cpuidle_suspend() will never have chance to
      run. In other word, the cpu_ops check can be avoid.
      
      Similarly, the cpu_suspend check could be avoided in this hot path by
      moving it into arm_cpuidle_init().
      
      I measured the 4096 * time from arm_cpuidle_suspend entry point to the
      cpu_psci_cpu_suspend entry point. HW platform is Marvell BG4CT STB
      board.
      
      1. only one shell, no other process, hot-unplug secondary cpus, execute
      the following cmd
      
      while true
      do
      	sleep 0.2
      done
      
      before the patch: 1581220ns
      
      after the patch: 1579630ns
      
      reduced by 0.1%
      
      2. only one shell, no other process, hot-unplug secondary cpus, execute
      the following cmd
      
      while true
      do
      	md5sum /tmp/testfile
      	sleep 0.2
      done
      
      NOTE: the testfile size should be larger than L1+L2 cache size
      
      before the patch: 1961960ns
      after the patch: 1912500ns
      
      reduced by 2.5%
      
      So the more complex the system load, the bigger the improvement.
      Signed-off-by: default avatarJisheng Zhang <jszhang@marvell.com>
      Acked-by: default avatarLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      b5fda7ed