- 24 Mar, 2016 2 commits
-
-
Mark Rutland authored
Commit 324420bf ("arm64: add support for ioremap() block mappings") added new p?d_set_huge functions which do the hard work to generate and set a correct block entry. These differ from open-coded huge page creation in the early page table code by explicitly setting the P?D_TYPE_SECT bits (which are implicitly retained by mk_sect_prot() for any valid prot), but are otherwise identical (and cannot fail on arm64). For simplicity and consistency, make use of these in the initial page table creation code. Signed-off-by:
Mark Rutland <mark.rutland@arm.com> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> Cc: Will Deacon <will.deacon@arm.com> Signed-off-by:
Catalin Marinas <catalin.marinas@arm.com>
-
Ard Biesheuvel authored
The KASLR code incorrectly expects the contents of x18 to be preserved across a call into C code, and uses it to stash the contents of SCTLR_EL1 before enabling the MMU. If the MMU needs to be disabled again to create the randomized kernel mapping, x18 is written back to SCTLR_EL1, which is likely to crash the system if x18 has been clobbered by kasan_early_init() or kaslr_early_init(). So use x22 instead, which is not in use so far in head.S Signed-off-by:
Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by:
-
- 21 Mar, 2016 4 commits
-
-
Catalin Marinas authored
The printk() implementation has a limit of LOG_LINE_MAX (== 1024 - 32) buffer per call which the arm64 mem_init() breaches when printing the virtual memory layout with CONFIG_KASAN enabled. The result is that the last line is no longer printed. This patch splits the call into a pr_notice() + additional pr_cont() calls. Signed-off-by:
-
Kefeng Wang authored
After commit 65da0a8e ("arm64: use non-global mappings for UEFI runtime regions"), nobody use __local_flush_icache_all() anymore, so drop it. Signed-off-by:
Kefeng Wang <wangkefeng.wang@huawei.com> Acked-by:
-
Mark Rutland authored
Commit f80fb3a3 ("arm64: add support for kernel ASLR") missed a DSB necessary to complete I-cache maintenance in the primary boot path, and hence stale instructions may still be present in the I-cache and may be executed until the I-cache maintenance naturally completes. Since commit 8ec41987 ("arm64: mm: ensure patched kernel text is fetched from PoU"), all CPUs invalidate their I-caches after their MMU is enabled. Prior a CPU's MMU having been enabled, arbitrary lines may have been fetched from the PoC into I-caches. We never patch text expected to be executed with the MMU off. Thus, it is unnecessary to perform broadcast I-cache maintenance in the primary boot path. This patch reduces the scope of the I-cache maintenance to the local CPU, and adds the missing DSB with similar scope, matching prior maintenance in the primary boot path. Signed-off-by:
-
Ard Biesheuvel authored
The implementation of macro inv_entry refers to its 'el' argument without the required leading backslash, which results in an undefined symbol 'el' to be passed into the kernel_entry macro rather than the index of the exception level as intended. This undefined symbol strangely enough does not result in build failures, although it is visible in vmlinux: $ nm -n vmlinux |head U el 0000000000000000 A _kernel_flags_le_hi32 0000000000000000 A _kernel_offset_le_hi32 0000000000000000 A _kernel_size_le_hi32 000000000000000a A _kernel_flags_le_lo32 ..... However, it does result in incorrect code being generated for invalid exceptions taken from EL0, since the argument check in kernel_entry assumes EL1 if its argument does not equal '0'. Signed-off-by:
-
- 11 Mar, 2016 3 commits
-
-
Catalin Marinas authored
With the 16KB and 64KB page size configurations, SWAPPER_BLOCK_SIZE is PAGE_SIZE and ARM64_SWAPPER_USES_SECTION_MAPS is 0. Since kimg_shadow_end is not page aligned (_end shifted by KASAN_SHADOW_SCALE_SHIFT), the edges of previously mapped kernel image shadow via vmemmap_populate() may be overridden by subsequent calls to kasan_populate_zero_shadow(), leading to kernel panics like below: ------------------------------------------------------------------------------ Unable to handle kernel paging request at virtual address fffffc100135068c pgd = fffffc8009ac0000 [fffffc100135068c] *pgd=00000009ffee0003, *pud=00000009ffee0003, *pmd=00000009ffee0003, *pte=00e0000081a00793 Internal error: Oops: 9600004f [#1] PREEMPT SMP Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.5.0-rc4+ #1984 Hardware name: Juno (DT) task: fffffe09001a0000 ti: fffffe0900200000 task.ti: fffffe0900200000 PC is at __memset+0x4c/0x200 LR is at kasan_unpoison_shadow+0x34/0x50 pc : [<fffffc800846f1cc>] lr : [<fffffc800821ff54>] pstate: 00000245 sp : fffffe0900203db0 x29: fffffe0900203db0 x28: 0000000000000000 x27: 0000000000000000 x26: 0000000000000000 x25: fffffc80099b69d0 x24: 0000000000000001 x23: 0000000000000000 x22: 0000000000002000 x21: dffffc8000000000 x20: 1fffff9001350a8c x19: 0000000000002000 x18: 0000000000000008 x17: 0000000000000147 x16: ffffffffffffffff x15: 79746972100e041d x14: ffffff0000000000 x13: ffff000000000000 x12: 0000000000000000 x11: 0101010101010101 x10: 1fffffc11c000000 x9 : 0000000000000000 x8 : fffffc100135068c x7 : 0000000000000000 x6 : 000000000000003f x5 : 0000000000000040 x4 : 0000000000000004 x3 : fffffc100134f651 x2 : 0000000000000400 x1 : 0000000000000000 x0 : fffffc100135068c Process swapper/0 (pid: 1, stack limit = 0xfffffe0900200020) Call trace: [<fffffc800846f1cc>] __memset+0x4c/0x200 [<fffffc8008220044>] __asan_register_globals+0x5c/0xb0 [<fffffc8008a09d34>] _GLOBAL__sub_I_65535_1_sunrpc_cache_lookup+0x1c/0x28 [<fffffc8008f20d28>] kernel_init_freeable+0x104/0x274 [<fffffc80089e1948>] kernel_init+0x10/0xf8 [<fffffc8008093a00>] ret_from_fork+0x10/0x50 ------------------------------------------------------------------------------ This patch aligns kimg_shadow_start and kimg_shadow_end to SWAPPER_BLOCK_SIZE in all configurations. Fixes: f9040773 ("arm64: move kernel image to base of vmalloc area") Signed-off-by:
-
Catalin Marinas authored
With the 16KB or 64KB page configurations, the generic vmemmap_populate() implementation warns on potential offnode page_structs via vmemmap_verify() because the arm64 kasan_init() passes NUMA_NO_NODE instead of the actual node for the kernel image memory. Fixes: f9040773 ("arm64: move kernel image to base of vmalloc area") Signed-off-by:
James Morse <james.morse@arm.com> Acked-by:
-
Catalin Marinas authored
The set_pte_at() function must update the hardware PTE_RDONLY bit depending on the state of the PTE_WRITE and PTE_DIRTY bits of the given entry value. However, it currently only performs this for pte_valid() entries, ignoring PTE_PROT_NONE. The side-effect is that PROT_NONE mappings would not have the PTE_RDONLY bit set. Without CONFIG_ARM64_HW_AFDBM, this is not an issue since such PROT_NONE pages are not accessible anyway. With commit 2f4b829c ("arm64: Add support for hardware updates of the access and dirty pte bits"), the ptep_set_wrprotect() function was re-written to cope with automatic hardware updates of the dirty state. As an optimisation, only PTE_RDONLY is checked to assess the "dirty" status. Since set_pte_at() does not set this bit for PROT_NONE mappings, such pages may be considered "dirty" as a result of ptep_set_wrprotect(). This patch updates the pte_valid() check to pte_present() in set_pte_at(). It also adds PTE_PROT_NONE to the swap entry bits comment. Fixes: 2f4b829c ("arm64: Add support for hardware updates of the access and dirty pte bits") Signed-off-by:
Ganapatrao Kulkarni <gkulkarni@caviumnetworks.com> Tested-by:
Ganapatrao Kulkarni <gkulkarni@cavium.com> Cc: <stable@vger.kernel.org>
-
- 04 Mar, 2016 3 commits
-
-
Adam Buchbinder authored
Signed-off-by:
Adam Buchbinder <adam.buchbinder@gmail.com> Signed-off-by:
-
Ard Biesheuvel authored
The prologue of the EFI entry point pushes x29 and x30 onto the stack but fails to create the stack frame correctly by omitting the assignment of x29 to the new value of the stack pointer. So fix that. Signed-off-by:
-
Mark Rutland authored
Commit 0f54b14e ("arm64: cpufeature: Change read_cpuid() to use sysreg's mrs_s macro") changed read_cpuid to require a SYS_ prefix on register names, to allow manual assembly of registers unknown by the toolchain, using tables in sysreg.h. This interacts poorly with commit 42b55734 ("efi/arm64: Check for h/w support before booting a >4 KB granular kernel"), which is curretly queued via the tip tree, and uses read_cpuid without a SYS_ prefix. Due to this, a build of next-20160304 fails if EFI and 64K pages are selected. To avoid this issue when trees are merged, move the required SYS_ prefixing into read_cpuid, and revert all of the updated callsites to pass plain register names. This effectively reverts the bulk of commit 0f54b14e. Signed-off-by:
-
- 03 Mar, 2016 1 commit
-
-
Ard Biesheuvel authored
In spite of its name, CONFIG_DEBUG_RODATA is an important hardening feature for production kernels, and distros all enable it by default in their kernel configs. However, since enabling it used to result in more granular, and thus less efficient kernel mappings, it is not enabled by default for performance reasons. However, since commit 2f39b5f9 ("arm64: mm: Mark .rodata as RO"), the various kernel segments (.text, .rodata, .init and .data) are already mapped individually, and the only effect of setting CONFIG_DEBUG_RODATA is that the existing .text and .rodata mappings are updated late in the boot sequence to have their read-only attributes set, which means that any performance concerns related to enabling CONFIG_DEBUG_RODATA are no longer valid. So from now on, make CONFIG_DEBUG_RODATA default to 'y' Signed-off-by:
Kees Cook <keescook@chromium.org> Signed-off-by:
-
- 02 Mar, 2016 2 commits
-
-
Mark Rutland authored
We validate pstate using PSR_MODE32_BIT, which is part of the user-provided pstate (and cannot be trusted). Also, we conflate validation of AArch32 and AArch64 pstate values, making the code difficult to reason about. Instead, validate the pstate value based on the associated task. The task may or may not be current (e.g. when using ptrace), so this must be passed explicitly by callers. To avoid circular header dependencies via sched.h, is_compat_task is pulled out of asm/ptrace.h. To make the code possible to reason about, the AArch64 and AArch32 validation is split into separate functions. Software must respect the RES0 policy for SPSR bits, and thus the kernel mirrors the hardware policy (RAZ/WI) for bits as-yet unallocated. When these acquire an architected meaning writes may be permitted (potentially with additional validation). Signed-off-by:
Will Deacon <will.deacon@arm.com> Cc: Dave Martin <dave.martin@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Peter Maydell <peter.maydell@linaro.org> Signed-off-by:
-
Ard Biesheuvel authored
Commit 8439e62a ("arm64: mm: use bit ops rather than arithmetic in pa/va translations") changed the boundary check against PAGE_OFFSET from an arithmetic comparison to a bit test. This means we now silently assume that PAGE_OFFSET is a power of 2 that divides the kernel virtual address space into two equal halves. So make that assumption explicit. Signed-off-by:
-
- 01 Mar, 2016 1 commit
-
-
Marc Zyngier authored
In order to reduce the risk of a bad merge, let's move the new kvm_call_hyp back to its original location in the file. This has zero impact from a code point of view. Signed-off-by:
Marc Zyngier <marc.zyngier@arm.com> Acked-by:
-
- 29 Feb, 2016 2 commits
-
-
Ard Biesheuvel authored
Commit c031a421 ("arm64: kaslr: randomize the linear region") implements randomization of the linear region, by subtracting a random multiple of PUD_SIZE from memstart_addr. This causes the virtual mapping of system RAM to move upwards in the linear region, and at the same time causes memstart_addr to assume a value which may be negative if the offset of system RAM in the physical space is smaller than its offset relative to PAGE_OFFSET in the virtual space. Since memstart_addr is effectively an offset now, redefine its type as s64 so that expressions involving shifting or division preserve its sign. Signed-off-by:
-
Ard Biesheuvel authored
In the boot log, instead of listing .init first, list .text, .rodata, .init and .data in the same order they appear in memory Signed-off-by:
-
- 26 Feb, 2016 9 commits
-
-
Ard Biesheuvel authored
The LSE atomics implementation uses runtime patching to patch in calls to out of line non-LSE atomics implementations on cores that lack hardware support for LSE. To avoid paying the overhead cost of a function call even if no call ends up being made, the bl instruction is kept invisible to the compiler, and the out of line implementations preserve all registers, not just the ones that they are required to preserve as per the AAPCS64. However, commit fd045f6c ("arm64: add support for module PLTs") added support for routing branch instructions via veneers if the branch target offset exceeds the range of the ordinary relative branch instructions. Since this deals with jump and call instructions that are exposed to ELF relocations, the PLT code uses x16 to hold the address of the branch target when it performs an indirect branch-to-register, something which is explicitly allowed by the AAPCS64 (and ordinary compiler generated code does not expect register x16 or x17 to retain their values across a bl instruction). Since the lse runtime patched bl instructions don't adhere to the AAPCS64, they don't deal with this clobbering of registers x16 and x17. So add them to the clobber list of the asm() statements that perform the call instructions, and drop x16 and x17 from the list of registers that are callee saved in the out of line non-LSE implementations. In addition, since we have given these functions two scratch registers, they no longer need to stack/unstack temp registers. Signed-off-by:
-
Kefeng Wang authored
Use VA_START macro in asm/memory.h instead of private LOWEST_ADDR definition in dump.c. Signed-off-by:
-
Will Deacon authored
UAO is a feature of ARMv8.2, so add a submenu like we have for 8.1. Signed-off-by:
-
Lorenzo Pieralisi authored
When secondary cpus are booted through the ACPI parking protocol, the booted cpu should check that FW has correctly cleared its mailbox entry point value to make sure the boot process was correctly executed. The entry point check is carried in the cpu_ops->cpu_postboot method, that is executed by secondary cpus when entering the kernel with irqs disabled. The ACPI parking protocol cpu_ops maps/unmaps the mailboxes on the primary CPU to trigger secondary boot in the cpu_ops->cpu_boot method and on secondary processors to carry out FW checks on the booted CPU to verify the boot protocol was successfully executed in the cpu_ops->cpu_postboot method. Therefore, the cpu_ops->cpu_postboot method is forced to ioremap/unmap the mailboxes, which is wrong in that ioremap cannot be safely be carried out with irqs disabled. To fix this issue, this patch reshuffles the code so that the mailboxes are still mapped after the boot processor executes the cpu_ops->cpu_boot method for a given cpu, and the VA at which a mailbox is mapped for a given cpu is stashed in the per-cpu data struct so that secondary cpus can retrieve them in the cpu_ops->cpu_postboot and complete the required FW checks. Signed-off-by:
Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> Reported-by:
Itaru Kitayama <itaru.kitayama@riken.jp> Tested-by:
Loc Ho <lho@apm.com> Tested-by:
-
Suzuki K Poulose authored
ARMv8.2 extensions [1] include an optional feature, which supports half precision(16bit) floating point/asimd data processing instructions. This patch adds support for detecting and exposing the same to the userspace via HWCAPs [1] https://community.arm.com/groups/processors/blog/2016/01/05/armv8-a-architecture-evolutionSigned-off-by:
Suzuki K. Poulose <suzuki.poulose@arm.com> Signed-off-by:
-
Mark Rutland authored
The asm-generic fixmap.h depends on each architecture's fixmap.h to pull in the definition of PAGE_KERNEL_RO, if this exists. In the absence of this, FIXMAP_PAGE_RO will not be defined. In mm/early_ioremap.c the definition of early_memremap_ro is predicated on FIXMAP_PAGE_RO being defined. Currently, the arm64 fixmap.h doesn't include pgtable.h for the definition of PAGE_KERNEL_RO, and as a knock-on effect early_memremap_ro is not always defined, leading to link-time failures when it is used. This has been observed with defconfig on next-20160226. Unfortunately, as pgtable.h includes fixmap.h, adding the include introduces a circular dependency, which is just as fragile. Instead, this patch factors out PAGE_KERNEL_RO and other prot definitions into a new pgtable-prot header which can be included by poth pgtable.h and fixmap.h, avoiding the circular dependency, and ensuring that early_memremap_ro is alwyas defined where it is used. Signed-off-by:
-
Andrew Pinski authored
On ThunderX T88 pass 1.x through 2.1 parts, broadcast TLBI instructions may cause the icache to become corrupted if it contains data for a non-current ASID. This patch implements the workaround (which invalidates the local icache when switching the mm) by using code patching. Signed-off-by:
Andrew Pinski <apinski@cavium.com> Signed-off-by:
David Daney <david.daney@cavium.com> Reviewed-by:
-
Jeremy Linton authored
Currently the .rodata section is actually still executable when DEBUG_RODATA is enabled. This changes that so the .rodata is actually read only, no execute. It also adds the .rodata section to the mem_init banner. Signed-off-by:
Jeremy Linton <jeremy.linton@arm.com> Reviewed-by:
-
Miles Chen authored
Remove the unnecessary boundary check since there is a huge gap between user and kernel address that they would never overlap. (arm64 does not have enough levels of page tables to cover 64-bit virtual address) See Documentation/arm64/memory.txt Signed-off-by:
Miles Chen <miles.chen@mediatek.com> Acked-by:
-
- 25 Feb, 2016 9 commits
-
-
Catalin Marinas authored
In such configuration, Linux uses only two pages of page tables and __pud_populate() should not be used. However, the BUILD_BUG() triggers since pud_sect() is still defined and the compiler cannot eliminate such code, even though at run-time it should not be triggered. This patch extends the #ifdef ARM64_64K_PAGES condition for pud_sect to include PGTABLE_LEVELS < 3. Signed-off-by: -
Suzuki K Poulose authored
Now that we have a clear understanding of the sign of a feature, rename the routines to reflect the sign, so that it is not misused. The cpuid_feature_extract_field() now accepts a 'sign' parameter. Signed-off-by:
-
Suzuki K Poulose authored
Use the appropriate accessor for the feature bit by keeping track of the sign of the feature Signed-off-by:
-
Suzuki K Poulose authored
There is a confusion on whether the values of a feature are signed or not in ARM. This is not clearly mentioned in the ARM ARM either. We have dealt most of the bits as signed so far, and marked the rest as unsigned explicitly. This fixed in ARM ARM and will be rolled out soon. Here is the criteria in a nutshell: 1) The fields, which are either signed or unsigned, use increasing numerical values to indicate an increase in functionality. Thus, if a value of 0x1 indicates the presence of some instructions, then the 0x2 value will indicate the presence of those instructions plus some additional instructions or functionality. 2) For ID field values where the value 0x0 defines that a feature is not present, the number is an unsigned value. 3) For some features where the feature was made optional or removed after the start of the definition of the architecture, the value 0x0 is used to indicate the presence of a feature, and 0xF indicates the absence of the feature. In these cases, the fields are, in effect, holding signed values. So with these rules applied, we have only the following fields which are signed and the rest are unsigned. a) ID_AA64PFR0_EL1: {FP, ASIMD} b) ID_AA64MMFR0_EL1: {TGran4K, TGran64K} c) ID_AA64DFR0_EL1: PMUVer (0xf - PMUv3 not implemented) d) ID_DFR0_EL1: PerfMon e) ID_MMFR0_EL1: {InnerShr, OuterShr} Signed-off-by: -
Suzuki K Poulose authored
Correct the feature bit entries for : ID_DFR0 ID_MMFR0 to fix the default safe value for some of the bits. Signed-off-by:
-
Suzuki K Poulose authored
Adds a hook for checking whether a secondary CPU has the features used already by the kernel during early boot, based on the boot CPU and plugs in the check for ASID size. The ID_AA64MMFR0_EL1:ASIDBits determines the size of the mm context id and is used in the early boot to make decisions. The value is picked up from the Boot CPU and cannot be delayed until other CPUs are up. If a secondary CPU has a smaller size than that of the Boot CPU, things will break horribly and the usual SANITY check is not good enough to prevent the system from crashing. So, crash the system with enough information. Cc: Mark Rutland <mark.rutland@arm.com> Acked-by:
-
Suzuki K Poulose authored
Add a helper to extract ASIDBits on the current cpu Cc: Mark Rutland <mark.rutland@arm.com> Acked-by:
-
Suzuki K Poulose authored
We verify the capabilities of the secondary CPUs only when hotplug is enabled. The boot time activated CPUs do not go through the verification by checking whether the system wide capabilities were initialised or not. This patch removes the capability check dependency on CONFIG_HOTPLUG_CPU, to make sure that all the secondary CPUs go through the check. The boot time activated CPUs will still skip the system wide capability check. The plan is to hook in a check for CPU features used by the kernel at early boot up, based on the Boot CPU values. Cc: Mark Rutland <mark.rutland@arm.com> Acked-by:
-
Suzuki K Poulose authored
A secondary CPU could fail to come online due to insufficient capabilities and could simply die or loop in the kernel. e.g, a CPU with no support for the selected kernel PAGE_SIZE loops in kernel with MMU turned off. or a hotplugged CPU which doesn't have one of the advertised system capability will die during the activation. There is no way to synchronise the status of the failing CPU back to the master. This patch solves the issue by adding a field to the secondary_data which can be updated by the failing CPU. If the secondary CPU fails even before turning the MMU on, it updates the status in a special variable reserved in the head.txt section to make sure that the update can be cache invalidated safely without possible sharing of cache write back granule. Here are the possible states : -1. CPU_MMU_OFF - Initial value set by the master CPU, this value indicates that the CPU could not turn the MMU on, hence the status could not be reliably updated in the secondary_data. Instead, the CPU has updated the status @ __early_cpu_boot_status. 0. CPU_BOOT_SUCCESS - CPU has booted successfully. 1. CPU_KILL_ME - CPU has invoked cpu_ops->die, indicating the master CPU to synchronise by issuing a cpu_ops->cpu_kill. 2. CPU_STUCK_IN_KERNEL - CPU couldn't invoke die(), instead is looping in the kernel. This information could be used by say, kexec to check if it is really safe to do a kexec reboot. 3. CPU_PANIC_KERNEL - CPU detected some serious issues which requires kernel to crash immediately. The secondary CPU cannot call panic() until it has initialised the GIC. This flag can be used to instruct the master to do so. Cc: Mark Rutland <mark.rutland@arm.com> Acked-by:
-
- 24 Feb, 2016 4 commits
-
-
Suzuki K Poulose authored
This patch moves cpu_die_early to smp.c, where it fits better. No functional changes, except for adding the necessary checks for CONFIG_HOTPLUG_CPU. Cc: Mark Rutland <mark.rutland@arm.com> Acked-by:
-
Suzuki K Poulose authored
Or in other words, make fail_incapable_cpu() reusable. We use fail_incapable_cpu() to kill a secondary CPU early during the bringup, which doesn't have the system advertised capabilities. This patch makes the routine more generic, to kill a secondary booting CPU, getting rid of the dependency on capability struct. This can be used by checks which are not necessarily attached to a capability struct (e.g, cpu ASIDBits). In that process, renames the function to cpu_die_early() to better match its functionality. This will be moved to arch/arm64/kernel/smp.c later. Cc: Mark Rutland <mark.rutland@arm.com> Acked-by:
-
Suzuki K Poulose authored
Adds a routine which can be used to park CPUs (spinning in kernel) when they can't be killed. Cc: Mark Rutland <mark.rutland@arm.com> Acked-by:
-
Ard Biesheuvel authored
Since arm64 does not use a decompressor that supplies an execution environment where it is feasible to some extent to provide a source of randomness, the arm64 KASLR kernel depends on the bootloader to supply some random bits in the /chosen/kaslr-seed DT property upon kernel entry. On UEFI systems, we can use the EFI_RNG_PROTOCOL, if supplied, to obtain some random bits. At the same time, use it to randomize the offset of the kernel Image in physical memory. Reviewed-by:
Matt Fleming <matt@codeblueprint.co.uk> Signed-off-by:
-
