1. 10 Dec, 2018 17 commits
    • Steve Capper's avatar
      arm64: mm: Prevent mismatched 52-bit VA support · a96a33b1
      Steve Capper authored
      For cases where there is a mismatch in ARMv8.2-LVA support between CPUs
      we have to be careful in allowing secondary CPUs to boot if 52-bit
      virtual addresses have already been enabled on the boot CPU.
      
      This patch adds code to the secondary startup path. If the boot CPU has
      enabled 52-bit VAs then ID_AA64MMFR2_EL1 is checked to see if the
      secondary can also enable 52-bit support. If not, the secondary is
      prevented from booting and an error message is displayed indicating why.
      
      Technically this patch could be implemented using the cpufeature code
      when considering 52-bit userspace support. However, we employ low level
      checks here as the cpufeature code won't be able to run if we have
      mismatched 52-bit kernel va support.
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      a96a33b1
    • Steve Capper's avatar
      arm64: mm: Offset TTBR1 to allow 52-bit PTRS_PER_PGD · e842dfb5
      Steve Capper authored
      Enabling 52-bit VAs on arm64 requires that the PGD table expands from 64
      entries (for the 48-bit case) to 1024 entries. This quantity,
      PTRS_PER_PGD is used as follows to compute which PGD entry corresponds
      to a given virtual address, addr:
      
      pgd_index(addr) -> (addr >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1)
      
      Userspace addresses are prefixed by 0's, so for a 48-bit userspace
      address, uva, the following is true:
      (uva >> PGDIR_SHIFT) & (1024 - 1) == (uva >> PGDIR_SHIFT) & (64 - 1)
      
      In other words, a 48-bit userspace address will have the same pgd_index
      when using PTRS_PER_PGD = 64 and 1024.
      
      Kernel addresses are prefixed by 1's so, given a 48-bit kernel address,
      kva, we have the following inequality:
      (kva >> PGDIR_SHIFT) & (1024 - 1) != (kva >> PGDIR_SHIFT) & (64 - 1)
      
      In other words a 48-bit kernel virtual address will have a different
      pgd_index when using PTRS_PER_PGD = 64 and 1024.
      
      If, however, we note that:
      kva = 0xFFFF << 48 + lower (where lower[63:48] == 0b)
      and, PGDIR_SHIFT = 42 (as we are dealing with 64KB PAGE_SIZE)
      
      We can consider:
      (kva >> PGDIR_SHIFT) & (1024 - 1) - (kva >> PGDIR_SHIFT) & (64 - 1)
       = (0xFFFF << 6) & 0x3FF - (0xFFFF << 6) & 0x3F	// "lower" cancels out
       = 0x3C0
      
      In other words, one can switch PTRS_PER_PGD to the 52-bit value globally
      provided that they increment ttbr1_el1 by 0x3C0 * 8 = 0x1E00 bytes when
      running with 48-bit kernel VAs (TCR_EL1.T1SZ = 16).
      
      For kernel configuration where 52-bit userspace VAs are possible, this
      patch offsets ttbr1_el1 and sets PTRS_PER_PGD corresponding to the
      52-bit value.
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Reviewed-by: default avatarSuzuki K Poulose <suzuki.poulose@arm.com>
      Suggested-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      [will: added comment to TTBR1_BADDR_4852_OFFSET calculation]
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      e842dfb5
    • Steve Capper's avatar
      arm64: mm: Define arch_get_mmap_end, arch_get_mmap_base · e5d99157
      Steve Capper authored
      Now that we have DEFAULT_MAP_WINDOW defined, we can arch_get_mmap_end
      and arch_get_mmap_base helpers to allow for high addresses in mmap.
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      e5d99157
    • Steve Capper's avatar
      arm64: mm: Introduce DEFAULT_MAP_WINDOW · 363524d2
      Steve Capper authored
      We wish to introduce a 52-bit virtual address space for userspace but
      maintain compatibility with software that assumes the maximum VA space
      size is 48 bit.
      
      In order to achieve this, on 52-bit VA systems, we make mmap behave as
      if it were running on a 48-bit VA system (unless userspace explicitly
      requests a VA where addr[51:48] != 0).
      
      On a system running a 52-bit userspace we need TASK_SIZE to represent
      the 52-bit limit as it is used in various places to distinguish between
      kernelspace and userspace addresses.
      
      Thus we need a new limit for mmap, stack, ELF loader and EFI (which uses
      TTBR0) to represent the non-extended VA space.
      
      This patch introduces DEFAULT_MAP_WINDOW and DEFAULT_MAP_WINDOW_64 and
      switches the appropriate logic to use that instead of TASK_SIZE.
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      363524d2
    • Steve Capper's avatar
      mm: mmap: Allow for "high" userspace addresses · f6795053
      Steve Capper authored
      This patch adds support for "high" userspace addresses that are
      optionally supported on the system and have to be requested via a hint
      mechanism ("high" addr parameter to mmap).
      
      Architectures such as powerpc and x86 achieve this by making changes to
      their architectural versions of arch_get_unmapped_* functions. However,
      on arm64 we use the generic versions of these functions.
      
      Rather than duplicate the generic arch_get_unmapped_* implementations
      for arm64, this patch instead introduces two architectural helper macros
      and applies them to arch_get_unmapped_*:
       arch_get_mmap_end(addr) - get mmap upper limit depending on addr hint
       arch_get_mmap_base(addr, base) - get mmap_base depending on addr hint
      
      If these macros are not defined in architectural code then they default
      to (TASK_SIZE) and (base) so should not introduce any behavioural
      changes to architectures that do not define them.
      Signed-off-by: default avatarSteve Capper <steve.capper@arm.com>
      Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Acked-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      f6795053
    • Qian Cai's avatar
      arm64: kasan: Increase stack size for KASAN_EXTRA · 6e883067
      Qian Cai authored
      If the kernel is configured with KASAN_EXTRA, the stack size is
      increased significantly due to setting the GCC -fstack-reuse option to
      "none" [1]. As a result, it can trigger a stack overrun quite often with
      32k stack size compiled using GCC 8. For example, this reproducer
      
        https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/syscalls/madvise/madvise06.c
      
      can trigger a "corrupted stack end detected inside scheduler" very
      reliably with CONFIG_SCHED_STACK_END_CHECK enabled. There are other
      reports at:
      
        https://lore.kernel.org/lkml/1542144497.12945.29.camel@gmx.us/
        https://lore.kernel.org/lkml/721E7B42-2D55-4866-9C1A-3E8D64F33F9C@gmx.us/
      
      There are just too many functions that could have a large stack with
      KASAN_EXTRA due to large local variables that have been called over and
      over again without being able to reuse the stacks. Some noticiable ones
      are,
      
      size
      7536 shrink_inactive_list
      7440 shrink_page_list
      6560 fscache_stats_show
      3920 jbd2_journal_commit_transaction
      3216 try_to_unmap_one
      3072 migrate_page_move_mapping
      3584 migrate_misplaced_transhuge_page
      3920 ip_vs_lblcr_schedule
      4304 lpfc_nvme_info_show
      3888 lpfc_debugfs_nvmestat_data.constprop
      
      There are other 49 functions over 2k in size while compiling kernel with
      "-Wframe-larger-than=" on this machine. Hence, it is too much work to
      change Makefiles for each object to compile without
      -fsanitize-address-use-after-scope individually.
      
      [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23Signed-off-by: default avatarQian Cai <cai@lca.pw>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      6e883067
    • Will Deacon's avatar
      arm64: Fix minor issues with the dcache_by_line_op macro · 33309ecd
      Will Deacon authored
      The dcache_by_line_op macro suffers from a couple of small problems:
      
      First, the GAS directives that are currently being used rely on
      assembler behavior that is not documented, and probably not guaranteed
      to produce the correct behavior going forward. As a result, we end up
      with some undefined symbols in cache.o:
      
      $ nm arch/arm64/mm/cache.o
               ...
               U civac
               ...
               U cvac
               U cvap
               U cvau
      
      This is due to the fact that the comparisons used to select the
      operation type in the dcache_by_line_op macro are comparing symbols
      not strings, and even though it seems that GAS is doing the right
      thing here (undefined symbols by the same name are equal to each
      other), it seems unwise to rely on this.
      
      Second, when patching in a DC CVAP instruction on CPUs that support it,
      the fallback path consists of a DC CVAU instruction which may be
      affected by CPU errata that require ARM64_WORKAROUND_CLEAN_CACHE.
      
      Solve these issues by unrolling the various maintenance routines and
      using the conditional directives that are documented as operating on
      strings. To avoid the complexity of nested alternatives, we move the
      DC CVAP patching to __clean_dcache_area_pop, falling back to a branch
      to __clean_dcache_area_poc if DCPOP is not supported by the CPU.
      Reported-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Suggested-by: default avatarRobin Murphy <robin.murphy@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      33309ecd
    • Mark Rutland's avatar
      arm64: remove arm64ksyms.c · 2a9cee5b
      Mark Rutland authored
      Now that arm64ksyms.c has been reduced to a stub, let's remove it
      entirely. New exports should be associated with their function
      definition.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      2a9cee5b
    • Mark Rutland's avatar
      arm64: frace: use asm EXPORT_SYMBOL() · dbd31962
      Mark Rutland authored
      For a while now it's been possible to use EXPORT_SYMBOL() in assembly
      files, which allows us to place exports immediately after assembly
      functions, as we do for C functions.
      
      As a step towards removing arm64ksyms.c, let's move the ftrace exports
      to the assembly files the functions are defined in.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      dbd31962
    • Mark Rutland's avatar
      arm64: string: use asm EXPORT_SYMBOL() · ac0e8c72
      Mark Rutland authored
      For a while now it's been possible to use EXPORT_SYMBOL() in assembly
      files, which allows us to place exports immediately after assembly
      functions, as we do for C functions.
      
      As a step towards removing arm64ksyms.c, let's move the string routine
      exports to the assembly files the functions are defined in. Routines
      which should only be exported for !KASAN builds are exported using the
      EXPORT_SYMBOL_NOKASAN() helper.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      ac0e8c72
    • Mark Rutland's avatar
      arm64: uaccess: use asm EXPORT_SYMBOL() · 56c08ec5
      Mark Rutland authored
      For a while now it's been possible to use EXPORT_SYMBOL() in assembly
      files, which allows us to place exports immediately after assembly
      functions, as we do for C functions.
      
      As a step towards removing arm64ksyms.c, let's move the uaccess exports
      to the assembly files the functions are defined in.  As we have to
      include <asm/assembler.h>, the existing includes are fixed to follow the
      usual ordering conventions.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      56c08ec5
    • Mark Rutland's avatar
      arm64: page: use asm EXPORT_SYMBOL() · 50fdecb2
      Mark Rutland authored
      For a while now it's been possible to use EXPORT_SYMBOL() in assembly
      files, which allows us to place exports immediately after assembly
      functions, as we do for C functions.
      
      As a step towards removing arm64ksyms.c, let's move the copy_page and
      clear_page exports to the assembly files the functions are defined in.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      50fdecb2
    • Mark Rutland's avatar
      arm64: smccc: use asm EXPORT_SYMBOL() · 23fe04c0
      Mark Rutland authored
      For a while now it's been possible to use EXPORT_SYMBOL() in assembly
      files, which allows us to place exports immediately after assembly
      functions, as we do for C functions.
      
      As a step towards removing arm64ksyms.c, let's move the SMCCC exports to
      the assembly file the functions are defined in.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      23fe04c0
    • Mark Rutland's avatar
      arm64: tishift: use asm EXPORT_SYMBOL() · abb77f3d
      Mark Rutland authored
      For a while now it's been possible to use EXPORT_SYMBOL() in assembly
      files, which allows us to place exports immediately after assembly
      functions, as we do for C functions.
      
      As a step towards removing arm64ksyms.c, let's move the tishift exports
      to the assembly file the functions are defined in.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      abb77f3d
    • Mark Rutland's avatar
      arm64: add EXPORT_SYMBOL_NOKASAN() · 386b3c7b
      Mark Rutland authored
      So that we can export symbols directly from assembly files, let's make
      use of the generic <asm/export.h>. We have a few symbols that we'll want
      to conditionally export for !KASAN kernel builds, so we add a helper for
      that in <asm/assembler.h>.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      386b3c7b
    • Mark Rutland's avatar
      arm64: move memstart_addr export inline · 03ef055f
      Mark Rutland authored
      Since we define memstart_addr in a C file, we can have the export
      immediately after the definition of the symbol, as we do elsewhere.
      
      As a step towards removing arm64ksyms.c, move the export of
      memstart_addr to init.c, where the symbol is defined.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      03ef055f
    • Mark Rutland's avatar
      arm64: remove bitop exports · 2d7c89b0
      Mark Rutland authored
      Now that the arm64 bitops are inlines built atop of the regular atomics,
      we don't need to export anything.
      
      Remove the redundant exports.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      2d7c89b0
  2. 07 Dec, 2018 6 commits
  3. 06 Dec, 2018 13 commits
  4. 04 Dec, 2018 1 commit
    • Ard Biesheuvel's avatar
      arm64: relocatable: fix inconsistencies in linker script and options · 3bbd3db8
      Ard Biesheuvel authored
      readelf complains about the section layout of vmlinux when building
      with CONFIG_RELOCATABLE=y (for KASLR):
      
        readelf: Warning: [21]: Link field (0) should index a symtab section.
        readelf: Warning: [21]: Info field (0) should index a relocatable section.
      
      Also, it seems that our use of '-pie -shared' is contradictory, and
      thus ambiguous. In general, the way KASLR is wired up at the moment
      is highly tailored to how ld.bfd happens to implement (and conflate)
      PIE executables and shared libraries, so given the current effort to
      support other toolchains, let's fix some of these issues as well.
      
      - Drop the -pie linker argument and just leave -shared. In ld.bfd,
        the differences between them are unclear (except for the ELF type
        of the produced image [0]) but lld chokes on seeing both at the
        same time.
      
      - Rename the .rela output section to .rela.dyn, as is customary for
        shared libraries and PIE executables, so that it is not misidentified
        by readelf as a static relocation section (producing the warnings
        above).
      
      - Pass the -z notext and -z norelro options to explicitly instruct the
        linker to permit text relocations, and to omit the RELRO program
        header (which requires a certain section layout that we don't adhere
        to in the kernel). These are the defaults for current versions of
        ld.bfd.
      
      - Discard .eh_frame and .gnu.hash sections to avoid them from being
        emitted between .head.text and .text, screwing up the section layout.
      
      These changes only affect the ELF image, and produce the same binary
      image.
      
      [0] b9dce7f1 ("arm64: kernel: force ET_DYN ELF type for ...")
      
      Cc: Nick Desaulniers <ndesaulniers@google.com>
      Cc: Peter Smith <peter.smith@linaro.org>
      Tested-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      3bbd3db8
  5. 30 Nov, 2018 3 commits
    • Ard Biesheuvel's avatar
      arm64/lib: improve CRC32 performance for deep pipelines · efdb25ef
      Ard Biesheuvel authored
      Improve the performance of the crc32() asm routines by getting rid of
      most of the branches and small sized loads on the common path.
      
      Instead, use a branchless code path involving overlapping 16 byte
      loads to process the first (length % 32) bytes, and process the
      remainder using a loop that processes 32 bytes at a time.
      
      Tested using the following test program:
      
        #include <stdlib.h>
      
        extern void crc32_le(unsigned short, char const*, int);
      
        int main(void)
        {
          static const char buf[4096];
      
          srand(20181126);
      
          for (int i = 0; i < 100 * 1000 * 1000; i++)
            crc32_le(0, buf, rand() % 1024);
      
          return 0;
        }
      
      On Cortex-A53 and Cortex-A57, the performance regresses but only very
      slightly. On Cortex-A72 however, the performance improves from
      
        $ time ./crc32
      
        real  0m10.149s
        user  0m10.149s
        sys   0m0.000s
      
      to
      
        $ time ./crc32
      
        real  0m7.915s
        user  0m7.915s
        sys   0m0.000s
      
      Cc: Rui Sun <sunrui26@huawei.com>
      Signed-off-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      efdb25ef
    • Mark Rutland's avatar
      arm64: ftrace: always pass instrumented pc in x0 · 7dc48bf9
      Mark Rutland authored
      The core ftrace hooks take the instrumented PC in x0, but for some
      reason arm64's prepare_ftrace_return() takes this in x1.
      
      For consistency, let's flip the argument order and always pass the
      instrumented PC in x0.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: AKASHI Takahiro <takahiro.akashi@linaro.org>
      Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Torsten Duwe <duwe@suse.de>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      7dc48bf9
    • Mark Rutland's avatar
      arm64: ftrace: remove return_regs macros · 49e258e0
      Mark Rutland authored
      The save_return_regs and restore_return_regs macros are only used by
      return_to_handler, and having them defined out-of-line only serves to
      obscure the logic.
      
      Before we complicate, let's clean this up and fold the logic directly
      into return_to_handler, saving a few lines of macro boilerplate in the
      process. At the same time, a missing trailing space is added to the
      comments, fixing a code style violation.
      
      There should be no functional change as a result of this patch.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Cc: AKASHI Takahiro <takahiro.akashi@linaro.org>
      Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Torsten Duwe <duwe@suse.de>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      49e258e0