- 09 Oct, 2018 40 commits
-
-
Vasily Gorbik authored
By default 3-level paging is used when the kernel is compiled with kasan support. Add 4-level paging option to support systems with more then 3TB of physical memory and to cover 4-level paging specific code with kasan as well. Reviewed-by:
Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by:
Vasily Gorbik <gor@linux.ibm.com> Signed-off-by:
-
Vasily Gorbik authored
Kasan initialization code is changed to populate persistent shadow first, save allocator position into pgalloc_freeable and proceed with early identity mapping creation. This way early identity mapping paging structures could be freed at once after switching to swapper_pg_dir when early identity mapping is not needed anymore. Reviewed-by:
-
Vasily Gorbik authored
By defining KASAN_SHADOW_OFFSET in Kconfig stack and global variables memory access check instrumentation is enabled. gcc version 4.9.2 or newer is also required. Reviewed-by:
-
Vasily Gorbik authored
Walking async_stack produces false positives. Disable __dump_trace function instrumentation for now. Signed-off-by:
-
Vasily Gorbik authored
Some functions from both arch/s390/kernel/ipl.c and arch/s390/kernel/machine_kexec.c are called without DAT enabled (or with and without DAT enabled code paths). There is no easy way to partially disable kasan for those files without a substantial rework. Disable kasan for both files for now. To avoid disabling kasan for arch/s390/kernel/diag.c DAT flag is enabled in diag308 call. pcpu_delegate which disables DAT is marked with __no_sanitize_address to disable instrumentation for that one function. Signed-off-by:
-
Vasily Gorbik authored
smp_start_secondary function is called without DAT enabled. To avoid disabling kasan instrumentation for entire arch/s390/kernel/smp.c smp_start_secondary has been split in 2 parts. smp_start_secondary has instrumentation disabled, it does minimal setup and enables DAT. Then instrumentated __smp_start_secondary is called to do the rest. __load_psw_mask function instrumentation has been disabled as well to be able to call it from smp_start_secondary. Reviewed-by:
-
Vasily Gorbik authored
Due to conflict between kasan instrumentation and inlining https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67368 functions which are defined as inline could not be called from functions defined with __no_sanitize_address. Introduce __no_sanitize_address_or_inline which would expand to __no_sanitize_address when the kernel is built with kasan support and to inline otherwise. This helps to avoid disabling kasan instrumentation for entire files. Reviewed-by:
-
Vasily Gorbik authored
To lower memory footprint and speed up kasan initialisation detect EDAT availability and use large pages if possible. As we know how much memory is needed for initialisation, another simplistic large page allocator is introduced to avoid memory fragmentation. Since facilities list is retrieved anyhow, detect noexec support and adjust pages attributes. Handle noexec kernel option to avoid inconsistent kasan shadow memory pages flags. Signed-off-by:
-
Vasily Gorbik authored
Move from modules area entire shadow memory preallocation to dynamic allocation per module load. This behaivior has been introduced for x86 with bebf56a1: "This patch also forces module_alloc() to return 8*PAGE_SIZE aligned address making shadow memory handling ( kasan_module_alloc()/kasan_module_free() ) more simple. Such alignment guarantees that each shadow page backing modules address space correspond to only one module_alloc() allocation" Signed-off-by:
-
Vasily Gorbik authored
This change adds address space markers for kasan shadow memory. Reviewed-by:
-
Vasily Gorbik authored
Kasan instrumentation adds "store" check for variables marked as modified by inline assembly. With user pointers containing addresses from another address space this produces false positives. static inline unsigned long clear_user_xc(void __user *to, ...) { asm volatile( ... : "+a" (to) ... User space access functions are wrapped by manually instrumented functions in kasan common code, which should be sufficient to catch errors. So, we just disable uaccess.o instrumentation altogether. Reviewed-by: -
Vasily Gorbik authored
Kasan stack instrumentation pads stack variables with redzones, which increases stack frames size significantly. Stack sizes are increased from 16k to 32k in the code, as well as for the kernel stack overflow detection option (CHECK_STACK). Reviewed-by:
-
Vasily Gorbik authored
Kasan needs 1/8 of kernel virtual address space to be reserved as the shadow area. And eventually it requires the shadow memory offset to be known at compile time (passed to the compiler when full instrumentation is enabled). Any value picked as the shadow area offset for 3-level paging would eat up identity mapping on 4-level paging (with 1PB shadow area size). So, the kernel sticks to 3-level paging when kasan is enabled. 3TB border is picked as the shadow offset. The memory layout is adjusted so, that physical memory border does not exceed KASAN_SHADOW_START and vmemmap does not go below KASAN_SHADOW_END. Due to the fact that on s390 paging is set up very late and to cover more code with kasan instrumentation, temporary identity mapping and final shadow memory are set up early. The shadow memory mapping is later carried over to init_mm.pgd during paging_init. For the needs of paging structures allocation and shadow memory population a primitive allocator is used, which simply chops off memory blocks from the end of the physical memory. Kasan currenty doesn't track vmemmap and vmalloc areas. Current memory layout (for 3-level paging, 2GB physical memory). ---[ Identity Mapping ]--- 0x0000000000000000-0x0000000000100000 ---[ Kernel Image Start ]--- 0x0000000000100000-0x0000000002b00000 ---[ Kernel Image End ]--- 0x0000000002b00000-0x0000000080000000 2G <- physical memory border 0x0000000080000000-0x0000030000000000 3070G PUD I ---[ Kasan Shadow Start ]--- 0x0000030000000000-0x0000030010000000 256M PMD RW X <- shadow for 2G memory 0x0000030010000000-0x0000037ff0000000 523776M PTE RO NX <- kasan zero ro page 0x0000037ff0000000-0x0000038000000000 256M PMD RW X <- shadow for 2G modules ---[ Kasan Shadow End ]--- 0x0000038000000000-0x000003d100000000 324G PUD I ---[ vmemmap Area ]--- 0x000003d100000000-0x000003e080000000 ---[ vmalloc Area ]--- 0x000003e080000000-0x000003ff80000000 ---[ Modules Area ]--- 0x000003ff80000000-0x0000040000000000 2G Acked-by:
-
Vasily Gorbik authored
Add pgd_page primitive which is required by kasan common code. Also fixes typo in p4d_page definition. Reviewed-by:
-
Vasily Gorbik authored
Kasan common code requires MAX_PTRS_PER_P4D definition, which in case of s390 is always PTRS_PER_P4D. Reviewed-by:
-
Vasily Gorbik authored
Follow the common kasan approach: "KASan replaces memory functions with manually instrumented variants. Original functions declared as weak symbols so strong definitions in mm/kasan/kasan.c could replace them. Original functions have aliases with '__' prefix in name, so we could call non-instrumented variant if needed." Signed-off-by: -
Vasily Gorbik authored
Instrumented C code cannot run without the kasan shadow area. Exempt source code files from kasan which are running before / used during kasan initialization. Reviewed-by:
-
Vasily Gorbik authored
vdso is mapped into user space processes, which won't have kasan shodow mapped. Reviewed-by:
-
Vasily Gorbik authored
kasan common code uses pfn_to_kaddr, which is defined by many other architectures. Adding it as well to avoid a build error. Reviewed-by:
-
Vasily Gorbik authored
To distinguish zfcpdump case and to be able to parse some of the kernel command line arguments early (e.g. mem=) ipl block retrieval and command line construction code is moved to the early boot phase. "memory_end" is set up correctly respecting "mem=" and hsa_size in case of the zfcpdump. arch/s390/boot/string.c is introduced to provide string handling and command line parsing functions to early boot phase code for the compressed kernel image case. Signed-off-by:
-
Vasily Gorbik authored
Introduce sclp_early_get_hsa_size function to be used during early memory detection. This function allows to find a memory limit imposed during zfcpdump. Reviewed-by:
Heiko Carstens <heiko.carstens@de.ibm.com> Reviewed-by:
-
Vasily Gorbik authored
Print mem_detect info source when memblock=debug is specified. Reviewed-by:
-
Vasily Gorbik authored
In a situation when other memory detection methods are not available (no SCLP and no z/VM diag260), continuous online memory is assumed. Replacing tprot loop with faster binary search, as only online memory end has to be found. Reviewed-by:
-
Vasily Gorbik authored
When neither SCLP storage info, nor z/VM diag260 "storage configuration" are available assume a continuous online memory of size specified by SCLP info. Reviewed-by:
-
Vasily Gorbik authored
In the case when z/VM memory is defined with "define storage config" command, SCLP storage info is not available. Utilize diag260 "storage configuration" call, to get information about z/VM specific guest memory definitions with potential memory holes. Reviewed-by:
-
Vasily Gorbik authored
SCLP storage info allows to detect continuous and non-continuous online memory under LPAR, z/VM and KVM, when standby memory is defined. Reviewed-by:
-
Vasily Gorbik authored
Make sure that .boot.data sections of vmlinux and arch/s390/compressed/vmlinux match before producing the compressed kernel image. Symbols presence, order and sizes are cross-checked. Reviewed-by:
-
Vasily Gorbik authored
Move memory detection to early boot phase. To store online memory regions "struct mem_detect_info" has been introduced together with for_each_mem_detect_block iterator. mem_detect_info is later converted to memblock. Also introduces sclp_early_get_meminfo function to get maximum physical memory and maximum increment number. Reviewed-by:
-
Vasily Gorbik authored
To enable early online memory detection sclp_early_read_info has been moved to sclp_early_core.c. sclp_info_sccb has been made a part of .boot.data, which allows to reuse it later during early kernel startup and make sclp_early_read_info call just once. Reviewed-by:
-
Vasily Gorbik authored
Introduce .boot.data section which is "shared" between the decompressor code and the decompressed kernel. The decompressor will store values in it, and copy over to the decompressed image before starting it. This method allows to avoid using pre-defined addresses and other hacks to pass values between those boot phases. .boot.data section is a part of init data, and will be freed after kernel initialization is complete. For uncompressed kernel image, .boot.data section is basically the same as .init.data Reviewed-by:
-
Vasily Gorbik authored
Since compressed/misc.c is conditionally compiled move error reporting code to boot/main.c. With that being done compressed/misc.c has no "miscellaneous" functions left and is all about plain decompression now. Rename it accordingly. Reviewed-by:
-
Vasily Gorbik authored
To avoid multi-stage initrd rescue operation and to simplify assumptions during early memory allocations move initrd at some final safe destination as early as possible. This would also allow us to drop .bss usage restrictions for some files. Reviewed-by:
-
Vasily Gorbik authored
Architecture documentation suggests that hsa_size has been available in the read info since the list-directed ipl dump has been introduced. By using this value few early sclp calls could be avoided. Reviewed-by:
-
Vasily Gorbik authored
Using .bss in early code should be avoided. It might overlay initrd image or not yet be initialized. Clean up the last couple of places in the decompressor's code where .bss is used and enfore no .bss usage check on boot/compressed/misc.c. In particular: - initializing free_mem_ptr and free_mem_end_ptr with values guarantee that these variables won't end up in the .bss section. - define STATIC_RW_DATA to go into .data section. Reviewed-by:
-
Vasily Gorbik authored
The kernel decompressor has to know several bits of information about uncompressed image. Currently this info is collected by running "nm" on uncompressed vmlinux + "sed" and producing sizes.h file. This method worked well, but it has several disadvantages. Obscure symbols name pattern matching is fragile. Adding new values makes pattern even longer. Logic is spread across code and make file. Limited ability to adjust symbols values (currently magic lma value of 0x100000 is always subtracted). Apart from that same pieces of information (and more) would be needed for early memory detection and features like KASLR outside of boot/compressed/ folder where sizes.h is generated. To overcome limitations new "struct vmlinux_info" has been introduced to include values needed for the decompressor and the rest of the boot code. The only static instance of vmlinux_info is produced during vmlinux link step by filling in struct fields by the linker (like it is done with input_data in boot/compressed/vmlinux.scr.lds.S). This way individual values could be adjusted with all the knowledge linker has and arithmetic it supports. Later .vmlinux.info section (which contains struct vmlinux_info) is transplanted into the decompressor image and dropped from uncompressed image altogether. While doing that replace "compressed/vmlinux.scr.lds.S" linker script (whose purpose is to rename .data section in piggy.o to .rodata.compressed) with plain objcopy command. And simplify decompressor's linker script. Reviewed-by:
-
Vasily Gorbik authored
Decompressor's head.S provided "data mover" sole purpose of which has been to safely move uncompressed kernel at 0x100000 and jump to it. With current bzImage layout entire decompressor's code guaranteed to be in a safe location under 0x100000, and hence could not be overwritten during kernel move. For that reason head.S could be replaced with simple memmove function. To do so introduce early boot code phase which is executed from arch/s390/boot/head.S after "verify_facilities" and takes care of optional kernel image decompression and transition to it. Reviewed-by:
-
Vasily Gorbik authored
Remove STACK_ORDER and STACK_SIZE in favour of identical THREAD_SIZE_ORDER and THREAD_SIZE definitions. THREAD_SIZE and THREAD_SIZE_ORDER naming is misleading since it is used as general kernel stack size information. But both those definitions are used in the common code and throughout architectures specific code, so changing the naming is problematic. Reviewed-by:
-
Martin Schwidefsky authored
With virtually mapped kernel stacks the kernel stack overflow detection is now fault based, every stack has a guard page in the vmalloc space. The panic_stack is renamed to nodat_stack and is used for all function that need to run without DAT, e.g. memcpy_real or do_start_kdump. The main effect is a reduction in the kernel image size as with vmap stacks the old style overflow checking that adds two instructions per function is not needed anymore. Result from bloat-o-meter: add/remove: 20/1 grow/shrink: 13/26854 up/down: 2198/-216240 (-214042) In regard to performance the micro-benchmark for fork has a hit of a few microseconds, allocating 4 pages in vmalloc space is more expensive compare to an order-2 page allocation. But with real workload I could not find a noticeable difference. Acked-by:
-
Martin Schwidefsky authored
Reviewed-by:
-
Martin Schwidefsky authored
With CONFIG_VMAP_STACK=y the kernel stack of all tasks should be allocated in the vmalloc space. The initial stack used for all the early init code is in the init_thread_union. To be able to switch from this early stack to a properly allocated stack from vmalloc the architecture needs a switch-over point. Introduce the arch_call_rest_init() function with a weak definition in init/main.c with the only purpose to call rest_init() from the end of start_kernel(). The architecture override can then do the necessary magic to switch to the new vmalloc'ed stack. Signed-off-by:
-
