system/linux
system/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux synced 2024-10-03 09:48:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
linux/arch/arm/kernel/entry-armv.S

1339 lines
33 KiB
ArmAsm
Raw Normal View History

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500 Based on 2 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation # extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 4122 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Enrico Weigelt <info@metux.net> Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org> Reviewed-by: Allison Randal <allison@lohutok.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-06-04 08:11:33 +00:00
/* SPDX-License-Identifier: GPL-2.0-only */
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*
* linux/arch/arm/kernel/entry-armv.S
*
* Copyright (C) 1996,1997,1998 Russell King.
* ARM700 fix by Matthew Godbolt (linux-user@willothewisp.demon.co.uk)
[ARM] Support register switch in nommu mode This patch adds register switch support in nommu mode. Signed-off-by: Hyok S. Choi <hyok.choi@samsung.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-01-13 21:05:25 +00:00
* nommu support by Hyok S. Choi (hyok.choi@samsung.com)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
*
* Low-level vector interface routines
*
[ARM] 4689/1: small comment wrap fix Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-12-04 13:33:33 +00:00
* Note: there is a StrongARM bug in the STMIA rn, {regs}^ instruction
* that causes it to save wrong values... Be aware!
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
*/
arm: fix implicit #include <linux/init.h> in entry asm. They use the "_INIT" macro and friends, and hence need to source this header file, vs. relying on getting it implicitly. Cc: Russell King <linux@arm.linux.org.uk> Cc: linux-arm-kernel@lists.infradead.org Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
2015-05-02 00:13:42 +00:00
#include <linux/init.h>
ARM: create a common IOMEM definition Several platforms create IOMEM defines for casting to 'void __iomem *', and other platforms are incorrectly using __io() macro for the same purpose. This creates a common definition and removes all the platform specific versions. Rather than try to make linux/io.h and asm/io.h assembly safe, the assembly version of IOMEM is moved into asm/assembler.h. Signed-off-by: Rob Herring <rob.herring@calxeda.com> Cc: Russell King <linux@arm.linux.org.uk> Cc: Sekhar Nori <nsekhar@ti.com> Cc: Kevin Hilman <khilman@ti.com> Acked-by: H Hartley Sweeten <hsweeten@visionengravers.com> Cc: Ryan Mallon <rmallon@gmail.com> Cc: Eric Miao <eric.y.miao@gmail.com> Cc: Haojian Zhuang <haojian.zhuang@marvell.com> Acked-by: David Brown <davidb@codeaurora.org> Cc: Daniel Walker <dwalker@fifo99.com> Cc: Bryan Huntsman <bryanh@codeaurora.org> Cc: Sascha Hauer <kernel@pengutronix.de> Cc: Shawn Guo <shawn.guo@linaro.org> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Paul Walmsley <paul@pwsan.com> Acked-by: Viresh Kumar <viresh.kumar@st.com> Cc: Rajeev Kumar <rajeev-dlh.kumar@st.com> Cc: Colin Cross <ccross@android.com> Cc: Olof Johansson <olof@lixom.net> Cc: Stephen Warren <swarren@nvidia.com> Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-10 16:30:31 +00:00
#include <asm/assembler.h>
[ARM] 3060/1: allow constants found in asm/memory.h to be used in asm code Patch from Nicolas Pitre This patch allows for assorted type of cleanups by letting assembly code use the same set of defines for constant values and avoid duplicated definitions that might not always be in sync, or that might simply be confusing due to the different names for the same thing. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-10-29 20:44:55 +00:00
#include <asm/memory.h>
ARM: move cache/processor/fault glue to separate include files This allows the cache/processor/fault glue to be more easily used from assembler code. Tested on Assabet and Tegra 2. Tested-by: Colin Cross <ccross@android.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-02-06 15:32:24 +00:00
#include <asm/glue-df.h>
#include <asm/glue-pf.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
#include <asm/vfpmacros.h>
[ARM] Add thread_notify infrastructure Some machine classes need to allow VFP support to be built into the kernel, but still allow the kernel to run even though VFP isn't present. Unfortunately, the kernel hard-codes VFP instructions into the thread switch, which prevents this being run-time selectable. Solve this by introducing a notifier which things such as VFP can hook into to be informed of events which affect the VFP subsystem (eg, creation and destruction of threads, switches between threads.) Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-06-21 12:31:52 +00:00
#include <asm/thread_notify.h>
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
#include <asm/unwind.h>
ARM: Use a definition for the userspace cmpxchg emulation syscall Use a definition for the cmpxchg SWI instead of hard-coding the number. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk> Acked-by: Nicolas Pitre <nico@fluxnic.net>
2009-11-09 23:53:29 +00:00
#include <asm/unistd.h>
ARM: 6207/1: Replace CONFIG_HAS_TLS_REG with HWCAP_TLS and check for it on V6 The TLS register is only available on ARM1136 r1p0 and later. Set HWCAP_TLS flags if hardware TLS is available and test for it if CONFIG_CPU_32v6K is not set for V6. Note that we set the TLS instruction in __kuser_get_tls dynamically as suggested by Jamie Lokier <jamie@shareable.org>. Also the __switch_to code is optimized out in most cases as suggested by Nicolas Pitre <nico@fluxnic.net>. Reviewed-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Tony Lindgren <tony@atomide.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-07-05 13:53:10 +00:00
#include <asm/tls.h>
Disintegrate asm/system.h for ARM Disintegrate asm/system.h for ARM. Signed-off-by: David Howells <dhowells@redhat.com> cc: Russell King <linux@arm.linux.org.uk> cc: linux-arm-kernel@lists.infradead.org
2012-03-28 17:30:01 +00:00
#include <asm/system_info.h>
ARM: uaccess: consolidate uaccess asm to asm/uaccess-asm.h Consolidate the user access assembly code to asm/uaccess-asm.h. This moves the csdb, check_uaccess, uaccess_mask_range_ptr, uaccess_enable, uaccess_disable, uaccess_save, uaccess_restore macros, and creates two new ones for exception entry and exit - uaccess_entry and uaccess_exit. This makes the uaccess_save and uaccess_restore macros private to asm/uaccess-asm.h. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2020-05-03 12:03:54 +00:00
#include <asm/uaccess-asm.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
#include "entry-header.S"
ARM: kprobes: disallow probing stack consuming instructions This patch prohibits probing instructions for which the stack requirements are unable to be determined statically. Some test cases are found not work again after the modification, this patch also removes them. Signed-off-by: Wang Nan <wangnan0@huawei.com> Reviewed-by: Jon Medhurst <tixy@linaro.org> Signed-off-by: Jon Medhurst <tixy@linaro.org>
2015-01-05 11:29:25 +00:00
#include <asm/probes.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
[PATCH] ARM SMP: consolidate main IRQ handler code Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-21 17:14:44 +00:00
/*
ARM: entry: re-allocate registers in irq entry assembly macros This avoids the irq entry assembly corrupting r5, thereby allowing it to be preserved through to the svc exit code. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 09:34:02 +00:00
* Interrupt handling.
[PATCH] ARM SMP: consolidate main IRQ handler code Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-21 17:14:44 +00:00
*/
ARM: implement IRQ stacks Now that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well. Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Acked-by: Linus Walleij <linus.walleij@linaro.org> Tested-by: Keith Packard <keithpac@amazon.com> Acked-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-05 07:15:40 +00:00
.macro irq_handler, from_user:req
ARM: 6532/1: Allow machine to specify it's own IRQ handlers at run-time Normally different ARM platform has different way to decode the IRQ hardware status and demultiplex to the corresponding IRQ handler. This is highly optimized by macro irq_handler in entry-armv.S, and each machine defines their own macro to decode the IRQ number. However, this prevents multiple machine classes to be built into a single kernel. By allowing each machine to specify thier own handler, and making function pointer 'handle_arch_irq' to point to it at run time, this can be solved. And introduce CONFIG_MULTI_IRQ_HANDLER to allow both solutions to work. Comparing with the highly optimized macro of irq_handler, the new function must be written with care not to lose too much performance. And the IPI stuff on SMP is expected to move to the provided arch IRQ handler as well. The assembly code to invoke handle_arch_irq is optimized by Russell King. Signed-off-by: Eric Miao <eric.miao@canonical.com> Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-12-13 08:42:34 +00:00
mov r0, sp
ARM: implement IRQ stacks Now that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well. Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Acked-by: Linus Walleij <linus.walleij@linaro.org> Tested-by: Keith Packard <keithpac@amazon.com> Acked-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-05 07:15:40 +00:00
#ifdef CONFIG_IRQSTACKS
mov_l r2, irq_stack_ptr @ Take base address
mrc p15, 0, r3, c13, c0, 4 @ Get CPU offset
#ifdef CONFIG_UNWINDER_ARM
mov fpreg, sp @ Preserve original SP
#else
mov r8, fp @ Preserve original FP
mov r9, sp @ Preserve original SP
#endif
ldr sp, [r2, r3] @ Load SP from per-CPU var
.if \from_user == 0
UNWIND( .setfp fpreg, sp )
@
@ If we took the interrupt while running in the kernel, we may already
@ be using the IRQ stack, so revert to the original value in that case.
@
subs r2, sp, r0 @ SP above bottom of IRQ stack?
rsbscs r2, r2, #THREAD_SIZE @ ... and below the top?
ARM: implement support for vmap'ed stacks Wire up the generic support for managing task stack allocations via vmalloc, and implement the entry code that detects whether we faulted because of a stack overrun (or future stack overrun caused by pushing the pt_regs array) While this adds a fair amount of tricky entry asm code, it should be noted that it only adds a TST + branch to the svc_entry path. The code implementing the non-trivial handling of the overflow stack is emitted out-of-line into the .text section. Since on ARM, we rely on do_translation_fault() to keep PMD level page table entries that cover the vmalloc region up to date, we need to ensure that we don't hit such a stale PMD entry when accessing the stack. So we do a dummy read from the new stack while still running from the old one on the context switch path, and bump the vmalloc_seq counter when PMD level entries in the vmalloc range are modified, so that the MM switch fetches the latest version of the entries. Note that we need to increase the per-mode stack by 1 word, to gain some space to stash a GPR until we know it is safe to touch the stack. However, due to the cacheline alignment of the struct, this does not actually increase the memory footprint of the struct stack array at all. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Keith Packard <keithpac@amazon.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-09-23 07:15:53 +00:00
#ifdef CONFIG_VMAP_STACK
ARM: assembler: add optimized ldr/str macros to load variables from memory We will be adding variable loads to various hot paths, so it makes sense to add a helper macro that can load variables from asm code without the use of literal pool entries. On v7 or later, we can simply use MOVW/MOVT pairs, but on earlier cores, this requires a bit of hackery to emit a instruction sequence that implements this using a sequence of ADD/LDR instructions. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-11-26 18:37:27 +00:00
ldr_va r2, high_memory, cc @ End of the linear region
ARM: implement support for vmap'ed stacks Wire up the generic support for managing task stack allocations via vmalloc, and implement the entry code that detects whether we faulted because of a stack overrun (or future stack overrun caused by pushing the pt_regs array) While this adds a fair amount of tricky entry asm code, it should be noted that it only adds a TST + branch to the svc_entry path. The code implementing the non-trivial handling of the overflow stack is emitted out-of-line into the .text section. Since on ARM, we rely on do_translation_fault() to keep PMD level page table entries that cover the vmalloc region up to date, we need to ensure that we don't hit such a stale PMD entry when accessing the stack. So we do a dummy read from the new stack while still running from the old one on the context switch path, and bump the vmalloc_seq counter when PMD level entries in the vmalloc range are modified, so that the MM switch fetches the latest version of the entries. Note that we need to increase the per-mode stack by 1 word, to gain some space to stash a GPR until we know it is safe to touch the stack. However, due to the cacheline alignment of the struct, this does not actually increase the memory footprint of the struct stack array at all. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Keith Packard <keithpac@amazon.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-09-23 07:15:53 +00:00
cmpcc r2, r0 @ Stack pointer was below it?
#endif
ARM: implement IRQ stacks Now that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well. Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Acked-by: Linus Walleij <linus.walleij@linaro.org> Tested-by: Keith Packard <keithpac@amazon.com> Acked-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-05 07:15:40 +00:00
movcs sp, r0 @ If so, revert to incoming SP
#ifndef CONFIG_UNWINDER_ARM
@
@ Inform the frame pointer unwinder where the next frame lives
@
movcc lr, pc @ Make LR point into .entry.text so
@ that we will get a dump of the
@ exception stack for this frame.
#ifdef CONFIG_CC_IS_GCC
movcc ip, r0 @ Store the old SP in the frame record.
stmdbcc sp!, {fp, ip, lr, pc} @ Push frame record
addcc fp, sp, #12
#else
stmdbcc sp!, {fp, lr} @ Push frame record
movcc fp, sp
#endif // CONFIG_CC_IS_GCC
#endif // CONFIG_UNWINDER_ARM
.endif
#endif // CONFIG_IRQSTACKS
irq: arm: perform irqentry in entry code In preparation for removing HANDLE_DOMAIN_IRQ_IRQENTRY, have arch/arm perform all the irqentry accounting in its entry code. For configurations with CONFIG_GENERIC_IRQ_MULTI_HANDLER, we can use generic_handle_arch_irq(). Other than asm_do_IRQ(), all C calls to handle_IRQ() are from irqchip handlers which will be called from generic_handle_arch_irq(), so to avoid double accounting IRQ entry, the entry logic is moved from handle_IRQ() into asm_do_IRQ(). For ARMv7M the entry assembly is tightly coupled with the NVIC irqchip, and while the entry code should logically live under arch/arm/, moving the entry logic there makes things more convoluted. So for now, place the entry logic in the NVIC irqchip, but separated into a separate function to make the split of responsibility clear. For all other configurations without CONFIG_GENERIC_IRQ_MULTI_HANDLER, IRQ entry is already handled in arch code, and requires no changes. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M Cc: Russell King <linux@armlinux.org.uk> Cc: Thomas Gleixner <tglx@linutronix.de>
2021-10-19 17:17:17 +00:00
bl generic_handle_arch_irq
ARM: implement IRQ stacks Now that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well. Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Acked-by: Linus Walleij <linus.walleij@linaro.org> Tested-by: Keith Packard <keithpac@amazon.com> Acked-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-05 07:15:40 +00:00
#ifdef CONFIG_IRQSTACKS
#ifdef CONFIG_UNWINDER_ARM
mov sp, fpreg @ Restore original SP
#else
mov fp, r8 @ Restore original FP
mov sp, r9 @ Restore original SP
#endif // CONFIG_UNWINDER_ARM
#endif // CONFIG_IRQSTACKS
[PATCH] ARM SMP: consolidate main IRQ handler code Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-21 17:14:44 +00:00
.endm
ARM: entry: prefetch/data abort helpers: convert to macros Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 09:22:08 +00:00
.macro pabt_helper
ARM: entry: prefetch abort: tail-call the main prefetch abort handler Tail-call the main C prefetch abort handler code from the per-CPU helper code. Also note that the helper function becomes ABI compliant in terms of the registers preserved. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 11:37:35 +00:00
@ PABORT handler takes pt_regs in r2, fault address in r4 and psr in r5
ARM: entry: prefetch/data abort helpers: convert to macros Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 09:22:08 +00:00
#ifdef MULTI_PABORT
ARM: entry: prefetch/data abort helpers: avoid corrupting r4 Replace r4 with ip for calling abort helpers - ip is allowed to be corrupted by called functions in the ABI, so it makes more sense to use such a register. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:46:08 +00:00
ldr ip, .LCprocfns
ARM: entry: prefetch/data abort helpers: convert to macros Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 09:22:08 +00:00
mov lr, pc
ARM: entry: prefetch/data abort helpers: avoid corrupting r4 Replace r4 with ip for calling abort helpers - ip is allowed to be corrupted by called functions in the ABI, so it makes more sense to use such a register. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:46:08 +00:00
ldr pc, [ip, #PROCESSOR_PABT_FUNC]
ARM: entry: prefetch/data abort helpers: convert to macros Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 09:22:08 +00:00
#else
bl CPU_PABORT_HANDLER
#endif
.endm
.macro dabt_helper
@
@ Call the processor-specific abort handler:
@
ARM: entry: data abort: tail-call the main data abort handler Tail-call the main C data abort handler code from the per-CPU helper code. Update the comments in the code wrt the new calling and return register state. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 15:01:26 +00:00
@ r2 - pt_regs
ARM: entry: data abort: arrange for CPU abort helpers to take pc/psr in r4/r5 Re-jig the CPU abort helpers to take the PC/PSR in r4/r5 rather than r2/r3. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 13:35:07 +00:00
@ r4 - aborted context pc
@ r5 - aborted context psr
ARM: entry: prefetch/data abort helpers: convert to macros Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 09:22:08 +00:00
@
@ The abort handler must return the aborted address in r0, and
@ the fault status register in r1. r9 must be preserved.
@
#ifdef MULTI_DABORT
ARM: entry: prefetch/data abort helpers: avoid corrupting r4 Replace r4 with ip for calling abort helpers - ip is allowed to be corrupted by called functions in the ABI, so it makes more sense to use such a register. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:46:08 +00:00
ldr ip, .LCprocfns
ARM: entry: prefetch/data abort helpers: convert to macros Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 09:22:08 +00:00
mov lr, pc
ARM: entry: prefetch/data abort helpers: avoid corrupting r4 Replace r4 with ip for calling abort helpers - ip is allowed to be corrupted by called functions in the ABI, so it makes more sense to use such a register. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:46:08 +00:00
ldr pc, [ip, #PROCESSOR_DABT_FUNC]
ARM: entry: prefetch/data abort helpers: convert to macros Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 09:22:08 +00:00
#else
bl CPU_DABORT_HANDLER
#endif
.endm
ARM: probes: avoid adding kprobes to sensitive kernel-entry/exit code Avoid adding kprobes to any of the kernel entry/exit or startup assembly code, or code in the identity-mapped region. This code does not conform to the standard C conventions, which means that the expectations of the kprobes code is not forfilled. Placing kprobes at some of these locations results in the kernel trying to return to userspace addresses while retaining the CPU in kernel mode. Tested-by: Naresh Kamboju <naresh.kamboju@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-11-24 23:54:22 +00:00
.section .entry.text,"ax",%progbits
ARM kprobes: prevent some functions involved with kprobes from being probed Signed-off-by: Nicolas Pitre <nico@marvell.com>
2007-12-03 20:27:56 +00:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*
* Invalid mode handlers
*/
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
.macro inv_entry, reason
ARM: rename S_FRAME_SIZE to PT_REGS_SIZE S_FRAME_SIZE is no longer the size of the kernel stack frame, so this name is misleading. It is the size of the kernel pt_regs structure. Name it so. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2016-05-10 15:34:27 +00:00
sub sp, sp, #PT_REGS_SIZE
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
ARM( stmib sp, {r1 - lr} )
THUMB( stmia sp, {r0 - r12} )
THUMB( str sp, [sp, #S_SP] )
THUMB( str lr, [sp, #S_LR] )
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
mov r1, #\reason
.endm
__pabt_invalid:
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
inv_entry BAD_PREFETCH
b common_invalid
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__pabt_invalid)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
__dabt_invalid:
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
inv_entry BAD_DATA
b common_invalid
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__dabt_invalid)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
__irq_invalid:
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
inv_entry BAD_IRQ
b common_invalid
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__irq_invalid)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
__und_invalid:
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
inv_entry BAD_UNDEFINSTR
@
@ XXX fall through to common_invalid
@
@
@ common_invalid - generic code for failed exception (re-entrant version of handlers)
@
common_invalid:
zero_fp
ldmia r0, {r4 - r6}
add r0, sp, #S_PC @ here for interlock avoidance
mov r7, #-1 @ "" "" "" ""
str r4, [sp] @ save preserved r0
stmia r0, {r5 - r7} @ lr_<exception>,
@ cpsr_<exception>, "old_r0"
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
mov r0, sp
b bad_mode
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__und_invalid)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*
* SVC mode handlers
*/
[ARM] 3102/1: ARM EABI: stack pointer must be 64-bit aligned after a CPU exception Patch from Nicolas Pitre The ARM EABI says that the stack pointer has to be 64-bit aligned for reasons already mentioned in patch #3101 when calling C functions. We therefore must verify and adjust sp accordingly when taking an exception from kernel mode since sp might not necessarily be 64-bit aligned if the exception occurs in the middle of a kernel function. If the exception occurs while in user mode then no sp fixup is needed as long as sizeof(struct pt_regs) as well as any additional syscall data stack space remain multiples of 8. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-01-14 16:18:08 +00:00
#if defined(CONFIG_AEABI) && (__LINUX_ARM_ARCH__ >= 5)
#define SPFIX(code...) code
#else
#define SPFIX(code...)
#endif
ARM: implement support for vmap'ed stacks Wire up the generic support for managing task stack allocations via vmalloc, and implement the entry code that detects whether we faulted because of a stack overrun (or future stack overrun caused by pushing the pt_regs array) While this adds a fair amount of tricky entry asm code, it should be noted that it only adds a TST + branch to the svc_entry path. The code implementing the non-trivial handling of the overflow stack is emitted out-of-line into the .text section. Since on ARM, we rely on do_translation_fault() to keep PMD level page table entries that cover the vmalloc region up to date, we need to ensure that we don't hit such a stale PMD entry when accessing the stack. So we do a dummy read from the new stack while still running from the old one on the context switch path, and bump the vmalloc_seq counter when PMD level entries in the vmalloc range are modified, so that the MM switch fetches the latest version of the entries. Note that we need to increase the per-mode stack by 1 word, to gain some space to stash a GPR until we know it is safe to touch the stack. However, due to the cacheline alignment of the struct, this does not actually increase the memory footprint of the struct stack array at all. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Keith Packard <keithpac@amazon.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-09-23 07:15:53 +00:00
.macro svc_entry, stack_hole=0, trace=1, uaccess=1, overflow_check=1
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnstart )
ARM: entry: rework stack realignment code in svc_entry The original Thumb-2 enablement patches updated the stack realignment code in svc_entry to work around the lack of a STMIB instruction in Thumb-2, by subtracting 4 from the frame size, inverting the sense of the misaligment check, and changing to a STMIA instruction and a final stack push of a 4 byte quantity that results in the stack becoming aligned at the end of the sequence. It also pushes and pops R0 to the stack in order to have a temp register that Thumb-2 allows in general purpose ALU instructions, as TST using SP is not permitted. Both are a bit problematic for vmap'ed stacks, as using the stack is only permitted after we decide that we did not overflow the stack, or have already switched to the overflow stack. As for the alignment check: the current approach creates a corner case where, if the initial SUB of SP ends up right at the start of the stack, we will end up subtracting another 8 bytes and overflowing it. This means we would need to add the overflow check *after* the SUB that deliberately misaligns the stack. However, this would require us to keep local state (i.e., whether we performed the subtract or not) across the overflow check, but without any GPRs or stack available. So let's switch to an approach where we don't use the stack, and where the alignment check of the stack pointer occurs in the usual way, as this is guaranteed not to result in overflow. This means we will be able to do the overflow check first. While at it, switch to R1 so the mode stack pointer in R0 remains accessible. Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-17 15:23:47 +00:00
sub sp, sp, #(SVC_REGS_SIZE + \stack_hole)
ARM: implement support for vmap'ed stacks Wire up the generic support for managing task stack allocations via vmalloc, and implement the entry code that detects whether we faulted because of a stack overrun (or future stack overrun caused by pushing the pt_regs array) While this adds a fair amount of tricky entry asm code, it should be noted that it only adds a TST + branch to the svc_entry path. The code implementing the non-trivial handling of the overflow stack is emitted out-of-line into the .text section. Since on ARM, we rely on do_translation_fault() to keep PMD level page table entries that cover the vmalloc region up to date, we need to ensure that we don't hit such a stale PMD entry when accessing the stack. So we do a dummy read from the new stack while still running from the old one on the context switch path, and bump the vmalloc_seq counter when PMD level entries in the vmalloc range are modified, so that the MM switch fetches the latest version of the entries. Note that we need to increase the per-mode stack by 1 word, to gain some space to stash a GPR until we know it is safe to touch the stack. However, due to the cacheline alignment of the struct, this does not actually increase the memory footprint of the struct stack array at all. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Keith Packard <keithpac@amazon.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-09-23 07:15:53 +00:00
THUMB( add sp, r1 ) @ get SP in a GPR without
THUMB( sub r1, sp, r1 ) @ using a temp register
.if \overflow_check
UNWIND(.save {r0 - pc} )
do_overflow_check (SVC_REGS_SIZE + \stack_hole)
.endif
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
#ifdef CONFIG_THUMB2_KERNEL
ARM: entry: rework stack realignment code in svc_entry The original Thumb-2 enablement patches updated the stack realignment code in svc_entry to work around the lack of a STMIB instruction in Thumb-2, by subtracting 4 from the frame size, inverting the sense of the misaligment check, and changing to a STMIA instruction and a final stack push of a 4 byte quantity that results in the stack becoming aligned at the end of the sequence. It also pushes and pops R0 to the stack in order to have a temp register that Thumb-2 allows in general purpose ALU instructions, as TST using SP is not permitted. Both are a bit problematic for vmap'ed stacks, as using the stack is only permitted after we decide that we did not overflow the stack, or have already switched to the overflow stack. As for the alignment check: the current approach creates a corner case where, if the initial SUB of SP ends up right at the start of the stack, we will end up subtracting another 8 bytes and overflowing it. This means we would need to add the overflow check *after* the SUB that deliberately misaligns the stack. However, this would require us to keep local state (i.e., whether we performed the subtract or not) across the overflow check, but without any GPRs or stack available. So let's switch to an approach where we don't use the stack, and where the alignment check of the stack pointer occurs in the usual way, as this is guaranteed not to result in overflow. This means we will be able to do the overflow check first. While at it, switch to R1 so the mode stack pointer in R0 remains accessible. Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-17 15:23:47 +00:00
tst r1, #4 @ test stack pointer alignment
sub r1, sp, r1 @ restore original R1
sub sp, r1 @ restore original SP
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
#else
[ARM] 3102/1: ARM EABI: stack pointer must be 64-bit aligned after a CPU exception Patch from Nicolas Pitre The ARM EABI says that the stack pointer has to be 64-bit aligned for reasons already mentioned in patch #3101 when calling C functions. We therefore must verify and adjust sp accordingly when taking an exception from kernel mode since sp might not necessarily be 64-bit aligned if the exception occurs in the middle of a kernel function. If the exception occurs while in user mode then no sp fixup is needed as long as sizeof(struct pt_regs) as well as any additional syscall data stack space remain multiples of 8. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-01-14 16:18:08 +00:00
SPFIX( tst sp, #4 )
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
#endif
ARM: entry: rework stack realignment code in svc_entry The original Thumb-2 enablement patches updated the stack realignment code in svc_entry to work around the lack of a STMIB instruction in Thumb-2, by subtracting 4 from the frame size, inverting the sense of the misaligment check, and changing to a STMIA instruction and a final stack push of a 4 byte quantity that results in the stack becoming aligned at the end of the sequence. It also pushes and pops R0 to the stack in order to have a temp register that Thumb-2 allows in general purpose ALU instructions, as TST using SP is not permitted. Both are a bit problematic for vmap'ed stacks, as using the stack is only permitted after we decide that we did not overflow the stack, or have already switched to the overflow stack. As for the alignment check: the current approach creates a corner case where, if the initial SUB of SP ends up right at the start of the stack, we will end up subtracting another 8 bytes and overflowing it. This means we would need to add the overflow check *after* the SUB that deliberately misaligns the stack. However, this would require us to keep local state (i.e., whether we performed the subtract or not) across the overflow check, but without any GPRs or stack available. So let's switch to an approach where we don't use the stack, and where the alignment check of the stack pointer occurs in the usual way, as this is guaranteed not to result in overflow. This means we will be able to do the overflow check first. While at it, switch to R1 so the mode stack pointer in R0 remains accessible. Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-17 15:23:47 +00:00
SPFIX( subne sp, sp, #4 )
ARM( stmib sp, {r1 - r12} )
THUMB( stmia sp, {r0 - r12} ) @ No STMIB in Thumb-2
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
ldmia r0, {r3 - r5}
ARM: entry: rework stack realignment code in svc_entry The original Thumb-2 enablement patches updated the stack realignment code in svc_entry to work around the lack of a STMIB instruction in Thumb-2, by subtracting 4 from the frame size, inverting the sense of the misaligment check, and changing to a STMIA instruction and a final stack push of a 4 byte quantity that results in the stack becoming aligned at the end of the sequence. It also pushes and pops R0 to the stack in order to have a temp register that Thumb-2 allows in general purpose ALU instructions, as TST using SP is not permitted. Both are a bit problematic for vmap'ed stacks, as using the stack is only permitted after we decide that we did not overflow the stack, or have already switched to the overflow stack. As for the alignment check: the current approach creates a corner case where, if the initial SUB of SP ends up right at the start of the stack, we will end up subtracting another 8 bytes and overflowing it. This means we would need to add the overflow check *after* the SUB that deliberately misaligns the stack. However, this would require us to keep local state (i.e., whether we performed the subtract or not) across the overflow check, but without any GPRs or stack available. So let's switch to an approach where we don't use the stack, and where the alignment check of the stack pointer occurs in the usual way, as this is guaranteed not to result in overflow. This means we will be able to do the overflow check first. While at it, switch to R1 so the mode stack pointer in R0 remains accessible. Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-17 15:23:47 +00:00
add r7, sp, #S_SP @ here for interlock avoidance
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
mov r6, #-1 @ "" "" "" ""
ARM: entry: rework stack realignment code in svc_entry The original Thumb-2 enablement patches updated the stack realignment code in svc_entry to work around the lack of a STMIB instruction in Thumb-2, by subtracting 4 from the frame size, inverting the sense of the misaligment check, and changing to a STMIA instruction and a final stack push of a 4 byte quantity that results in the stack becoming aligned at the end of the sequence. It also pushes and pops R0 to the stack in order to have a temp register that Thumb-2 allows in general purpose ALU instructions, as TST using SP is not permitted. Both are a bit problematic for vmap'ed stacks, as using the stack is only permitted after we decide that we did not overflow the stack, or have already switched to the overflow stack. As for the alignment check: the current approach creates a corner case where, if the initial SUB of SP ends up right at the start of the stack, we will end up subtracting another 8 bytes and overflowing it. This means we would need to add the overflow check *after* the SUB that deliberately misaligns the stack. However, this would require us to keep local state (i.e., whether we performed the subtract or not) across the overflow check, but without any GPRs or stack available. So let's switch to an approach where we don't use the stack, and where the alignment check of the stack pointer occurs in the usual way, as this is guaranteed not to result in overflow. This means we will be able to do the overflow check first. While at it, switch to R1 so the mode stack pointer in R0 remains accessible. Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-17 15:23:47 +00:00
add r2, sp, #(SVC_REGS_SIZE + \stack_hole)
SPFIX( addne r2, r2, #4 )
str r3, [sp] @ save the "real" r0 copied
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
@ from the exception stack
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
mov r3, lr
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@
@ We are now ready to fill in the remaining blanks on the stack:
@
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
@ r2 - sp_svc
@ r3 - lr_svc
@ r4 - lr_<exception>, already fixed up for correct return/restart
@ r5 - spsr_<exception>
@ r6 - orig_r0 (see pt_regs definition in ptrace.h)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
stmia r7, {r2 - r6}
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: save and reset the address limit when entering an exception When we enter an exception, the current address limit should not apply to the exception context: if the exception context wishes to access kernel space via the user accessors (eg, perf code), it must explicitly request such access. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2016-05-13 10:40:20 +00:00
get_thread_info tsk
ARM: uaccess: consolidate uaccess asm to asm/uaccess-asm.h Consolidate the user access assembly code to asm/uaccess-asm.h. This moves the csdb, check_uaccess, uaccess_mask_range_ptr, uaccess_enable, uaccess_disable, uaccess_save, uaccess_restore macros, and creates two new ones for exception entry and exit - uaccess_entry and uaccess_exit. This makes the uaccess_save and uaccess_restore macros private to asm/uaccess-asm.h. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2020-05-03 12:03:54 +00:00
uaccess_entry tsk, r0, r1, r2, \uaccess
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
.if \trace
ARM: entry: avoid enabling interrupts in prefetch/data abort handlers Avoid enabling interrupts if the parent context had interrupts enabled in the abort handler assembly code, and move this into the breakpoint/ page/alignment fault handlers instead. This gets rid of some special-casing for the breakpoint fault handlers from the low level abort handler path. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 10:44:06 +00:00
#ifdef CONFIG_TRACE_IRQFLAGS
bl trace_hardirqs_off
#endif
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
.endif
ARM: entry: consolidate trace_hardirqs_off into (svc|usr)_entry macros All handlers now call trace_hardirqs_off, so move this common code into the (svc|usr)_entry assembler macros. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 16:35:19 +00:00
.endm
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: entry: consolidate trace_hardirqs_off into (svc|usr)_entry macros All handlers now call trace_hardirqs_off, so move this common code into the (svc|usr)_entry assembler macros. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 16:35:19 +00:00
.align 5
__dabt_svc:
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
svc_entry uaccess=0
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
mov r2, sp
ARM: entry: data abort: tail-call the main data abort handler Tail-call the main C data abort handler code from the per-CPU helper code. Update the comments in the code wrt the new calling and return register state. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 15:01:26 +00:00
dabt_helper
ARM: 7876/1: clear Thumb-2 IT state on exception handling The exception handling code fails to clear the IT state, potentially leading to incorrect execution of the fixup if the size of the IT block is more than one. Let fixup_exception do the IT sanitizing if a fixup has been found, and restore CPSR from the stack when returning from a data abort. Cc: Will Deacon <will.deacon@arm.com> Cc: stable@vger.kernel.org Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-11-04 10:42:29 +00:00
THUMB( ldr r5, [sp, #S_PSR] ) @ potentially updated CPSR
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
svc_exit r5 @ return from exception
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnend )
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__dabt_svc)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.align 5
__irq_svc:
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
svc_entry
ARM: implement IRQ stacks Now that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well. Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Acked-by: Linus Walleij <linus.walleij@linaro.org> Tested-by: Keith Packard <keithpac@amazon.com> Acked-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-05 07:15:40 +00:00
irq_handler from_user=0
ARM: entry: no need to increase preempt count for IRQ handlers irq_enter() and irq_exit() already take care of the preempt_count handling for interrupts, which increment and decrement the hardirq bits of the preempt count. So we can remove the preempt count handing in our IRQ entry/exit assembly, like x86 did some 9 years ago. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 09:57:57 +00:00
sched/rt, ARM: Use CONFIG_PREEMPTION CONFIG_PREEMPTION is selected by CONFIG_PREEMPT and by CONFIG_PREEMPT_RT. Both PREEMPT and PREEMPT_RT require the same functionality which today depends on CONFIG_PREEMPT. Switch the entry code, cache over to use CONFIG_PREEMPTION and add output in show_stack() for PREEMPT_RT. [bigeasy: +traps.c] Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Russell King <linux@armlinux.org.uk> Cc: linux-arm-kernel@lists.infradead.org Link: https://lore.kernel.org/r/20191015191821.11479-2-bigeasy@linutronix.de Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-10-15 19:17:48 +00:00
#ifdef CONFIG_PREEMPTION
ARM: entry: no need to increase preempt count for IRQ handlers irq_enter() and irq_exit() already take care of the preempt_count handling for interrupts, which increment and decrement the hardirq bits of the preempt count. So we can remove the preempt count handing in our IRQ entry/exit assembly, like x86 did some 9 years ago. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 09:57:57 +00:00
ldr r8, [tsk, #TI_PREEMPT] @ get preempt count
[PATCH] ARM SMP: reallocate main IRQ handler code registers By changing r9 -> r8 and r8 to 'tsk' (r9) we are able to remove one instruction from the preempt path. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-21 17:15:45 +00:00
ldr r0, [tsk, #TI_FLAGS] @ get flags
[ARM] Fix kernel mode preemption Luc Van Oostenryck reported: The code removed by this patch tested the irq_cpustat_t members __local_irq_count and __local_bh_count but these fields have been removed some time ago: http://git.kernel.org/?p=linux/kernel/git/tglx/history.git;a=commitdiff;h=3ab146c93e039dec99fec8d441a8dd046fe510cc Fix this oversight. Acked-by: Bill Gatliff <bgat@billgatliff.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-04-13 16:47:35 +00:00
teq r8, #0 @ if preempt count != 0
movne r0, #0 @ force flags to 0
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
tst r0, #_TIF_NEED_RESCHED
blne svc_preempt
#endif
ARM: entry: no need to reload the SPSR value from struct pt_regs The SVC IRQ, prefetch and data abort handlers preserve the SPSR value via r5 across the exception. Rather than re-loading it from pt_regs, use the preserved value instead. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 11:47:08 +00:00
ARM: entry: move IRQ tracing exit into svc_exit The IRQ tracing exit path is much the same between all SVC mode exits, so move this into the svc_exit macro. Use a macro parameter to identify the IRQ case, which is the only different case there is. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-03-28 12:57:40 +00:00
svc_exit r5, irq = 1 @ return from exception
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnend )
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__irq_svc)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.ltorg
sched/rt, ARM: Use CONFIG_PREEMPTION CONFIG_PREEMPTION is selected by CONFIG_PREEMPT and by CONFIG_PREEMPT_RT. Both PREEMPT and PREEMPT_RT require the same functionality which today depends on CONFIG_PREEMPT. Switch the entry code, cache over to use CONFIG_PREEMPTION and add output in show_stack() for PREEMPT_RT. [bigeasy: +traps.c] Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Russell King <linux@armlinux.org.uk> Cc: linux-arm-kernel@lists.infradead.org Link: https://lore.kernel.org/r/20191015191821.11479-2-bigeasy@linutronix.de Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-10-15 19:17:48 +00:00
#ifdef CONFIG_PREEMPTION
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
svc_preempt:
[ARM] Fix kernel mode preemption Luc Van Oostenryck reported: The code removed by this patch tested the irq_cpustat_t members __local_irq_count and __local_bh_count but these fields have been removed some time ago: http://git.kernel.org/?p=linux/kernel/git/tglx/history.git;a=commitdiff;h=3ab146c93e039dec99fec8d441a8dd046fe510cc Fix this oversight. Acked-by: Bill Gatliff <bgat@billgatliff.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-04-13 16:47:35 +00:00
mov r8, lr
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
1: bl preempt_schedule_irq @ irq en/disable is done inside
[PATCH] ARM SMP: reallocate main IRQ handler code registers By changing r9 -> r8 and r8 to 'tsk' (r9) we are able to remove one instruction from the preempt path. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-21 17:15:45 +00:00
ldr r0, [tsk, #TI_FLAGS] @ get new tasks TI_FLAGS
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
tst r0, #_TIF_NEED_RESCHED
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
reteq r8 @ go again
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
b 1b
#endif
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
__und_fault:
@ Correct the PC such that it is pointing at the instruction
@ which caused the fault. If the faulting instruction was ARM
@ the PC will be pointing at the next instruction, and have to
@ subtract 4. Otherwise, it is Thumb, and the PC will be
@ pointing at the second half of the Thumb instruction. We
@ have to subtract 2.
ldr r2, [r0, #S_PC]
sub r2, r2, r1
str r2, [r0, #S_PC]
b do_undefinstr
ENDPROC(__und_fault)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.align 5
__und_svc:
ARM kprobes: don't let a single-stepped stmdb corrupt the exception stack If kprobes installs a breakpoint on a "stmdb sp!, {...}" instruction, and then single-step it by simulation from the exception context, it will corrupt the saved regs on the stack from the previous context. To avoid this, let's add an optional parameter to the svc_entry macro allowing for a hole to be created on the stack before saving the interrupted context, and use it in the undef_svc handler when kprobes is enabled. Signed-off-by: Nicolas Pitre <nico@marvell.com>
2007-12-14 20:56:01 +00:00
#ifdef CONFIG_KPROBES
@ If a kprobe is about to simulate a "stmdb sp..." instruction,
@ it obviously needs free stack space which then will belong to
@ the saved context.
ARM: kprobes: disallow probing stack consuming instructions This patch prohibits probing instructions for which the stack requirements are unable to be determined statically. Some test cases are found not work again after the modification, this patch also removes them. Signed-off-by: Wang Nan <wangnan0@huawei.com> Reviewed-by: Jon Medhurst <tixy@linaro.org> Signed-off-by: Jon Medhurst <tixy@linaro.org>
2015-01-05 11:29:25 +00:00
svc_entry MAX_STACK_SIZE
ARM kprobes: don't let a single-stepped stmdb corrupt the exception stack If kprobes installs a breakpoint on a "stmdb sp!, {...}" instruction, and then single-step it by simulation from the exception context, it will corrupt the saved regs on the stack from the previous context. To avoid this, let's add an optional parameter to the svc_entry macro allowing for a hole to be created on the stack before saving the interrupted context, and use it in the undef_svc handler when kprobes is enabled. Signed-off-by: Nicolas Pitre <nico@marvell.com>
2007-12-14 20:56:01 +00:00
#else
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
svc_entry
ARM kprobes: don't let a single-stepped stmdb corrupt the exception stack If kprobes installs a breakpoint on a "stmdb sp!, {...}" instruction, and then single-step it by simulation from the exception context, it will corrupt the saved regs on the stack from the previous context. To avoid this, let's add an optional parameter to the svc_entry macro allowing for a hole to be created on the stack before saving the interrupted context, and use it in the undef_svc handler when kprobes is enabled. Signed-off-by: Nicolas Pitre <nico@marvell.com>
2007-12-14 20:56:01 +00:00
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
mov r1, #4 @ PC correction to apply
ARM: 9030/1: entry: omit FP emulation for UND exceptions taken in kernel mode There are a couple of problems with the exception entry code that deals with FP exceptions (which are reported as UND exceptions) when building the kernel in Thumb2 mode: - the conditional branch to vfp_kmode_exception in vfp_support_entry() may be out of range for its target, depending on how the linker decides to arrange the sections; - when the UND exception is taken in kernel mode, the emulation handling logic is entered via the 'call_fpe' label, which means we end up using the wrong value/mask pairs to match and detect the NEON opcodes. Since UND exceptions in kernel mode are unlikely to occur on a hot path (as opposed to the user mode version which is invoked for VFP support code and lazy restore), we can use the existing undef hook machinery for any kernel mode instruction emulation that is needed, including calling the existing vfp_kmode_exception() routine for unexpected cases. So drop the call to call_fpe, and instead, install an undef hook that will get called for NEON and VFP instructions that trigger an UND exception in kernel mode. While at it, make sure that the PC correction is accurate for the execution mode where the exception was taken, by checking the PSR Thumb bit. Cc: Dmitry Osipenko <digetx@gmail.com> Cc: Kees Cook <keescook@chromium.org> Fixes: eff8728fe698 ("vmlinux.lds.h: Add PGO and AutoFDO input sections") Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Reviewed-by: Linus Walleij <linus.walleij@linaro.org> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2020-11-19 17:09:16 +00:00
THUMB( tst r5, #PSR_T_BIT ) @ exception taken in Thumb mode?
THUMB( movne r1, #2 ) @ if so, fix up PC correction
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
mov r0, sp @ struct pt_regs *regs
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
bl __und_fault
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
__und_svc_finish:
ARM: fix address limit restoration for undefined instructions During boot, sometimes the kernel will test to see if an instruction causes an undefined instruction exception. Unfortunately, the exit path for these exceptions did not restore the address limit, which causes the rootfs mount code to fail. Fix the missing address limit restoration. Tested-by: Guenter Roeck <linux@roeck-us.net> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2016-08-03 09:33:35 +00:00
get_thread_info tsk
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
ldr r5, [sp, #S_PSR] @ Get SVC cpsr
svc_exit r5 @ return from exception
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnend )
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__und_svc)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.align 5
__pabt_svc:
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
svc_entry
ARM: 5727/1: Pass IFSR register to do_PrefetchAbort() Instruction fault status register, IFSR, was introduced on ARMv6 to provide status information about the last insturction fault. It needed for proper prefetch abort handling. Now we have three prefetch abort model: * legacy - for CPUs before ARMv6. They doesn't provide neither IFSR nor IFAR. We simulate IFSR with section translation fault status for them to generalize code; * ARMv6 - provides IFSR, but not IFAR; * ARMv7 - provides both IFSR and IFAR. Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-09-25 12:39:47 +00:00
mov r2, sp @ regs
ARM: entry: prefetch abort: tail-call the main prefetch abort handler Tail-call the main C prefetch abort handler code from the per-CPU helper code. Also note that the helper function becomes ABI compliant in terms of the registers preserved. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 11:37:35 +00:00
pabt_helper
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
svc_exit r5 @ return from exception
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnend )
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__pabt_svc)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
.align 5
__fiq_svc:
svc_entry trace=0
mov r0, sp @ struct pt_regs *regs
bl handle_fiq_as_nmi
svc_exit_via_fiq
UNWIND(.fnend )
ENDPROC(__fiq_svc)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.align 5
[PATCH] ARM SMP: convert alignment enable The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch makes the alignment trap enable code independent of the way we handle the save areas. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 17:02:00 +00:00
.LCcralign:
.word cr_alignment
Add a prefetch abort handler This patch adds a prefetch abort handler similar to the data abort one and renames the latter for consistency. Initial implementation by Paul Brook with some renaming by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:07 +00:00
#ifdef MULTI_DABORT
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.LCprocfns:
.word processor
#endif
.LCfp:
.word fp_enter
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
/*
* Abort mode handlers
*/
@
@ Taking a FIQ in abort mode is similar to taking a FIQ in SVC mode
@ and reuses the same macros. However in abort mode we must also
@ save/restore lr_abt and spsr_abt to make nested aborts safe.
@
.align 5
__fiq_abt:
svc_entry trace=0
ARM( msr cpsr_c, #ABT_MODE | PSR_I_BIT | PSR_F_BIT )
THUMB( mov r0, #ABT_MODE | PSR_I_BIT | PSR_F_BIT )
THUMB( msr cpsr_c, r0 )
mov r1, lr @ Save lr_abt
mrs r2, spsr @ Save spsr_abt, abort is now safe
ARM( msr cpsr_c, #SVC_MODE | PSR_I_BIT | PSR_F_BIT )
THUMB( mov r0, #SVC_MODE | PSR_I_BIT | PSR_F_BIT )
THUMB( msr cpsr_c, r0 )
stmfd sp!, {r1 - r2}
add r0, sp, #8 @ struct pt_regs *regs
bl handle_fiq_as_nmi
ldmfd sp!, {r1 - r2}
ARM( msr cpsr_c, #ABT_MODE | PSR_I_BIT | PSR_F_BIT )
THUMB( mov r0, #ABT_MODE | PSR_I_BIT | PSR_F_BIT )
THUMB( msr cpsr_c, r0 )
mov lr, r1 @ Restore lr_abt, abort is unsafe
msr spsr_cxsf, r2 @ Restore spsr_abt
ARM( msr cpsr_c, #SVC_MODE | PSR_I_BIT | PSR_F_BIT )
THUMB( mov r0, #SVC_MODE | PSR_I_BIT | PSR_F_BIT )
THUMB( msr cpsr_c, r0 )
svc_exit_via_fiq
UNWIND(.fnend )
ENDPROC(__fiq_abt)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*
* User mode handlers
[ARM] 3102/1: ARM EABI: stack pointer must be 64-bit aligned after a CPU exception Patch from Nicolas Pitre The ARM EABI says that the stack pointer has to be 64-bit aligned for reasons already mentioned in patch #3101 when calling C functions. We therefore must verify and adjust sp accordingly when taking an exception from kernel mode since sp might not necessarily be 64-bit aligned if the exception occurs in the middle of a kernel function. If the exception occurs while in user mode then no sp fixup is needed as long as sizeof(struct pt_regs) as well as any additional syscall data stack space remain multiples of 8. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-01-14 16:18:08 +00:00
*
ARM: rename S_FRAME_SIZE to PT_REGS_SIZE S_FRAME_SIZE is no longer the size of the kernel stack frame, so this name is misleading. It is the size of the kernel pt_regs structure. Name it so. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2016-05-10 15:34:27 +00:00
* EABI note: sp_svc is always 64-bit aligned here, so should PT_REGS_SIZE
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
*/
[ARM] 3102/1: ARM EABI: stack pointer must be 64-bit aligned after a CPU exception Patch from Nicolas Pitre The ARM EABI says that the stack pointer has to be 64-bit aligned for reasons already mentioned in patch #3101 when calling C functions. We therefore must verify and adjust sp accordingly when taking an exception from kernel mode since sp might not necessarily be 64-bit aligned if the exception occurs in the middle of a kernel function. If the exception occurs while in user mode then no sp fixup is needed as long as sizeof(struct pt_regs) as well as any additional syscall data stack space remain multiples of 8. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-01-14 16:18:08 +00:00
ARM: rename S_FRAME_SIZE to PT_REGS_SIZE S_FRAME_SIZE is no longer the size of the kernel stack frame, so this name is misleading. It is the size of the kernel pt_regs structure. Name it so. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2016-05-10 15:34:27 +00:00
#if defined(CONFIG_AEABI) && (__LINUX_ARM_ARCH__ >= 5) && (PT_REGS_SIZE & 7)
[ARM] 3102/1: ARM EABI: stack pointer must be 64-bit aligned after a CPU exception Patch from Nicolas Pitre The ARM EABI says that the stack pointer has to be 64-bit aligned for reasons already mentioned in patch #3101 when calling C functions. We therefore must verify and adjust sp accordingly when taking an exception from kernel mode since sp might not necessarily be 64-bit aligned if the exception occurs in the middle of a kernel function. If the exception occurs while in user mode then no sp fixup is needed as long as sizeof(struct pt_regs) as well as any additional syscall data stack space remain multiples of 8. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-01-14 16:18:08 +00:00
#error "sizeof(struct pt_regs) must be a multiple of 8"
#endif
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
.macro usr_entry, trace=1, uaccess=1
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnstart )
UNWIND(.cantunwind ) @ don't unwind the user space
ARM: rename S_FRAME_SIZE to PT_REGS_SIZE S_FRAME_SIZE is no longer the size of the kernel stack frame, so this name is misleading. It is the size of the kernel pt_regs structure. Name it so. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2016-05-10 15:34:27 +00:00
sub sp, sp, #PT_REGS_SIZE
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
ARM( stmib sp, {r1 - r12} )
THUMB( stmia sp, {r0 - r12} )
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
ARM: Avoid writing to control register on every exception If we are not changing the control register value, avoid writing to it. Writes to the control register can be very expensive, taking around a hundred cycles or so. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-08-28 12:08:14 +00:00
ATRAP( mrc p15, 0, r7, c1, c0, 0)
ATRAP( ldr r8, .LCcralign)
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
ldmia r0, {r3 - r5}
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
add r0, sp, #S_PC @ here for interlock avoidance
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
mov r6, #-1 @ "" "" "" ""
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
str r3, [sp] @ save the "real" r0 copied
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
@ from the exception stack
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: Avoid writing to control register on every exception If we are not changing the control register value, avoid writing to it. Writes to the control register can be very expensive, taking around a hundred cycles or so. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-08-28 12:08:14 +00:00
ATRAP( ldr r8, [r8, #0])
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@
@ We are now ready to fill in the remaining blanks on the stack:
@
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
@ r4 - lr_<exception>, already fixed up for correct return/restart
@ r5 - spsr_<exception>
@ r6 - orig_r0 (see pt_regs definition in ptrace.h)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@
@ Also, separately save sp_usr and lr_usr
@
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
stmia r0, {r4 - r6}
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
ARM( stmdb r0, {sp, lr}^ )
THUMB( store_user_sp_lr r0, r1, S_SP - S_PC )
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
.if \uaccess
uaccess_disable ip
.endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@ Enable the alignment trap while in kernel mode
ARM: Avoid writing to control register on every exception If we are not changing the control register value, avoid writing to it. Writes to the control register can be very expensive, taking around a hundred cycles or so. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-08-28 12:08:14 +00:00
ATRAP( teq r8, r7)
ATRAP( mcrne p15, 0, r8, c1, c0, 0)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: smp: Store current pointer in TPIDRURO register if available Now that the user space TLS register is assigned on every return to user space, we can use it to keep the 'current' pointer while running in the kernel. This removes the need to access it via thread_info, which is located at the base of the stack, but will be moved out of there in a subsequent patch. Use the __builtin_thread_pointer() helper when available - this will help GCC understand that reloading the value within the same function is not necessary, even when using the per-task stack protector (which also generates accesses via the TLS register). For example, the generated code below loads TPIDRURO only once, and uses it to access both the stack canary and the preempt_count fields. <do_one_initcall>: e92d 41f0 stmdb sp!, {r4, r5, r6, r7, r8, lr} ee1d 4f70 mrc 15, 0, r4, cr13, cr0, {3} 4606 mov r6, r0 b094 sub sp, #80 ; 0x50 f8d4 34e8 ldr.w r3, [r4, #1256] ; 0x4e8 <- stack canary 9313 str r3, [sp, #76] ; 0x4c f8d4 8004 ldr.w r8, [r4, #4] <- preempt count Co-developed-by: Keith Packard <keithpac@amazon.com> Signed-off-by: Keith Packard <keithpac@amazon.com> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Reviewed-by: Linus Walleij <linus.walleij@linaro.org> Tested-by: Amit Daniel Kachhap <amit.kachhap@arm.com>
2021-09-18 08:44:37 +00:00
reload_current r7, r8
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@
@ Clear FP to mark the first stack frame
@
zero_fp
ARM: entry: consolidate trace_hardirqs_off into (svc|usr)_entry macros All handlers now call trace_hardirqs_off, so move this common code into the (svc|usr)_entry assembler macros. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 16:35:19 +00:00
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
.if \trace
ARM: fix lockdep unannotated irqs-off warning Wolfram Sang reported an unannotated irqs-off warning from lockdep: WARNING: CPU: 0 PID: 282 at kernel/locking/lockdep.c:3557 check_flags+0x84/0x1f4() DEBUG_LOCKS_WARN_ON(current->hardirqs_enabled) CPU: 0 PID: 282 Comm: rcS Tainted: G W 4.1.0-00002-g5b076054611833 #179 Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree) Backtrace: [<c0012c94>] (dump_backtrace) from [<c0012e3c>] (show_stack+0x18/0x1c) r6:c02dcc67 r5:00000009 r4:00000000 r3:00400000 [<c0012e24>] (show_stack) from [<c02510c8>] (dump_stack+0x20/0x28) [<c02510a8>] (dump_stack) from [<c0022c44>] (warn_slowpath_common+0x8c/0xb4) [<c0022bb8>] (warn_slowpath_common) from [<c0022cd8>] (warn_slowpath_fmt+0x38/0x40) r8:c780f470 r7:00000000 r6:00000000 r5:c03b0570 r4:c0b7ec04 [<c0022ca4>] (warn_slowpath_fmt) from [<c004cd38>] (check_flags+0x84/0x1f4) r3:c02e13d8 r2:c02dceaa [<c004ccb4>] (check_flags) from [<c0050e50>] (lock_acquire+0x4c/0xbc) r5:00000000 r4:60000193 [<c0050e04>] (lock_acquire) from [<c0256000>] (_raw_spin_lock+0x34/0x44) r9:000a8d5c r8:00000001 r7:c7806000 r6:c780f460 r5:c03b06a0 r4:c780f460 [<c0255fcc>] (_raw_spin_lock) from [<c005a8cc>] (handle_fasteoi_irq+0x20/0x11c) r4:c780f400 [<c005a8ac>] (handle_fasteoi_irq) from [<c0057a4c>] (generic_handle_irq+0x28/0x38) r6:00000000 r5:c03b038c r4:00000012 r3:c005a8ac [<c0057a24>] (generic_handle_irq) from [<c0057ae4>] (__handle_domain_irq+0x88/0xa8) r4:00000000 r3:00000026 [<c0057a5c>] (__handle_domain_irq) from [<c000a3cc>] (gic_handle_irq+0x40/0x58) r8:10c5347d r7:10c5347d r6:c35b1fb0 r5:c03a6304 r4:c8802000 r3:c35b1fb0 [<c000a38c>] (gic_handle_irq) from [<c0013bc8>] (__irq_usr+0x48/0x60) Exception stack(0xc35b1fb0 to 0xc35b1ff8) 1fa0: 00000061 00000000 000ab736 00000066 1fc0: 00000061 000aa1f0 000a8d54 000a8d54 000a8d88 000a8d5c 000a8cc8 000a8d68 1fe0: 72727272 bef8a528 000398c0 00031334 20000010 ffffffff r6:ffffffff r5:20000010 r4:00031334 r3:00000061 ---[ end trace cb88537fdc8fa202 ]--- possible reason: unannotated irqs-off. irq event stamp: 769 hardirqs last enabled at (769): [<c000f82c>] ret_fast_syscall+0x2c/0x54 hardirqs last disabled at (768): [<c000f80c>] ret_fast_syscall+0xc/0x54 softirqs last enabled at (0): [<c0020ec4>] copy_process.part.65+0x2e8/0x11dc softirqs last disabled at (0): [< (null)>] (null) His kernel configuration had: CONFIG_PROVE_LOCKING=y CONFIG_TRACE_IRQFLAGS=y but no IRQSOFF_TRACER, which means entry from userspace can result in the kernel seeing IRQs off without being notified of that change of state. Change the IRQSOFF ifdef in the usr_entry macro to TRACE_IRQFLAGS instead. Tested-by: Wolfram Sang <wsa+renesas@sang-engineering.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-07-03 11:42:36 +00:00
#ifdef CONFIG_TRACE_IRQFLAGS
ARM: entry: consolidate trace_hardirqs_off into (svc|usr)_entry macros All handlers now call trace_hardirqs_off, so move this common code into the (svc|usr)_entry assembler macros. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 16:35:19 +00:00
bl trace_hardirqs_off
#endif
ARM: 7688/1: add support for context tracking subsystem commit 91d1aa43 (context_tracking: New context tracking susbsystem) generalized parts of the RCU userspace extended quiescent state into the context tracking subsystem. Context tracking is then used to implement adaptive tickless (a.k.a extended nohz) To support the new context tracking subsystem on ARM, the user/kernel boundary transtions need to be instrumented. For exceptions and IRQs in usermode, the existing usr_entry macro is used to instrument the user->kernel transition. For the return to usermode path, the ret_to_user* path is instrumented. Using the usr_entry macro, this covers interrupts in userspace, data abort and prefetch abort exceptions in userspace as well as undefined exceptions in userspace (which is where FP emulation and VFP are handled.) For syscalls, the slow return path is covered by instrumenting the ret_to_user path. In addition, the syscall entry point is instrumented which covers the user->kernel transition for both fast and slow syscalls, and an additional instrumentation point is added for the fast syscall return path (ret_fast_syscall). Cc: Mats Liljegren <mats.liljegren@enea.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Signed-off-by: Kevin Hilman <khilman@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-03-28 21:54:40 +00:00
ct_user_exit save = 0
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
.endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.endm
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
.macro kuser_cmpxchg_check
ARM: remove user cmpxchg syscall Mark Brand reports that a NEEDS_SYSCALL_FOR_CMPXCHG enabled kernel would open a security hole in the ghost syscall used to implement cmpxchg, as it fails to validate the user pointer. However, in order for this option to be enabled, you'd need to be building a pre-ARMv6 kernel with SMP support. There is only one system known which fits that, which is an early ARM SMP FPGA implementation based on the ARM926T. In any case, the Kconfig does not allow SMP to be enabled for pre-ARMv6 systems. Moreover, even if NEEDS_SYSCALL_FOR_CMPXCHG were to be enabled, the kernel would not build as __ARM_NR_cmpxchg64 is not defined. The simple answer is to remove the buggy code. Reported-by: Mark Brand <markbrand@google.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-09-21 18:34:28 +00:00
#if !defined(CONFIG_CPU_32v6K) && defined(CONFIG_KUSER_HELPERS)
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
#ifndef CONFIG_MMU
#warning "NPTL on non MMU needs fixing"
#else
@ Make sure our user space atomic helper is restarted
@ if it was interrupted in a critical region. Here we
@ perform a quick test inline since it should be false
@ 99.9999% of the time. The rest is done out of line.
ARM: 9015/2: Define the virtual space of KASan's shadow region Define KASAN_SHADOW_OFFSET,KASAN_SHADOW_START and KASAN_SHADOW_END for the Arm kernel address sanitizer. We are "stealing" lowmem (the 4GB addressable by a 32bit architecture) out of the virtual address space to use as shadow memory for KASan as follows: +----+ 0xffffffff | | | | |-> Static kernel image (vmlinux) BSS and page table | |/ +----+ PAGE_OFFSET | | | | |-> Loadable kernel modules virtual address space area | |/ +----+ MODULES_VADDR = KASAN_SHADOW_END | | | | |-> The shadow area of kernel virtual address. | |/ +----+-> TASK_SIZE (start of kernel space) = KASAN_SHADOW_START the | | shadow address of MODULES_VADDR | | | | | | | | |-> The user space area in lowmem. The kernel address | | | sanitizer do not use this space, nor does it map it. | | | | | | | | | | | | | |/ ------ 0 0 .. TASK_SIZE is the memory that can be used by shared userspace/kernelspace. It us used for userspace processes and for passing parameters and memory buffers in system calls etc. We do not need to shadow this area. KASAN_SHADOW_START: This value begins with the MODULE_VADDR's shadow address. It is the start of kernel virtual space. Since we have modules to load, we need to cover also that area with shadow memory so we can find memory bugs in modules. KASAN_SHADOW_END This value is the 0x100000000's shadow address: the mapping that would be after the end of the kernel memory at 0xffffffff. It is the end of kernel address sanitizer shadow area. It is also the start of the module area. KASAN_SHADOW_OFFSET: This value is used to map an address to the corresponding shadow address by the following formula: shadow_addr = (address >> 3) + KASAN_SHADOW_OFFSET; As you would expect, >> 3 is equal to dividing by 8, meaning each byte in the shadow memory covers 8 bytes of kernel memory, so one bit shadow memory per byte of kernel memory is used. The KASAN_SHADOW_OFFSET is provided in a Kconfig option depending on the VMSPLIT layout of the system: the kernel and userspace can split up lowmem in different ways according to needs, so we calculate the shadow offset depending on this. When kasan is enabled, the definition of TASK_SIZE is not an 8-bit rotated constant, so we need to modify the TASK_SIZE access code in the *.s file. The kernel and modules may use different amounts of memory, according to the VMSPLIT configuration, which in turn determines the PAGE_OFFSET. We use the following KASAN_SHADOW_OFFSETs depending on how the virtual memory is split up: - 0x1f000000 if we have 1G userspace / 3G kernelspace split: - The kernel address space is 3G (0xc0000000) - PAGE_OFFSET is then set to 0x40000000 so the kernel static image (vmlinux) uses addresses 0x40000000 .. 0xffffffff - On top of that we have the MODULES_VADDR which under the worst case (using ARM instructions) is PAGE_OFFSET - 16M (0x01000000) = 0x3f000000 so the modules use addresses 0x3f000000 .. 0x3fffffff - So the addresses 0x3f000000 .. 0xffffffff need to be covered with shadow memory. That is 0xc1000000 bytes of memory. - 1/8 of that is needed for its shadow memory, so 0x18200000 bytes of shadow memory is needed. We "steal" that from the remaining lowmem. - The KASAN_SHADOW_START becomes 0x26e00000, to KASAN_SHADOW_END at 0x3effffff. - Now we can calculate the KASAN_SHADOW_OFFSET for any kernel address as 0x3f000000 needs to map to the first byte of shadow memory and 0xffffffff needs to map to the last byte of shadow memory. Since: SHADOW_ADDR = (address >> 3) + KASAN_SHADOW_OFFSET 0x26e00000 = (0x3f000000 >> 3) + KASAN_SHADOW_OFFSET KASAN_SHADOW_OFFSET = 0x26e00000 - (0x3f000000 >> 3) KASAN_SHADOW_OFFSET = 0x26e00000 - 0x07e00000 KASAN_SHADOW_OFFSET = 0x1f000000 - 0x5f000000 if we have 2G userspace / 2G kernelspace split: - The kernel space is 2G (0x80000000) - PAGE_OFFSET is set to 0x80000000 so the kernel static image uses 0x80000000 .. 0xffffffff. - On top of that we have the MODULES_VADDR which under the worst case (using ARM instructions) is PAGE_OFFSET - 16M (0x01000000) = 0x7f000000 so the modules use addresses 0x7f000000 .. 0x7fffffff - So the addresses 0x7f000000 .. 0xffffffff need to be covered with shadow memory. That is 0x81000000 bytes of memory. - 1/8 of that is needed for its shadow memory, so 0x10200000 bytes of shadow memory is needed. We "steal" that from the remaining lowmem. - The KASAN_SHADOW_START becomes 0x6ee00000, to KASAN_SHADOW_END at 0x7effffff. - Now we can calculate the KASAN_SHADOW_OFFSET for any kernel address as 0x7f000000 needs to map to the first byte of shadow memory and 0xffffffff needs to map to the last byte of shadow memory. Since: SHADOW_ADDR = (address >> 3) + KASAN_SHADOW_OFFSET 0x6ee00000 = (0x7f000000 >> 3) + KASAN_SHADOW_OFFSET KASAN_SHADOW_OFFSET = 0x6ee00000 - (0x7f000000 >> 3) KASAN_SHADOW_OFFSET = 0x6ee00000 - 0x0fe00000 KASAN_SHADOW_OFFSET = 0x5f000000 - 0x9f000000 if we have 3G userspace / 1G kernelspace split, and this is the default split for ARM: - The kernel address space is 1GB (0x40000000) - PAGE_OFFSET is set to 0xc0000000 so the kernel static image uses 0xc0000000 .. 0xffffffff. - On top of that we have the MODULES_VADDR which under the worst case (using ARM instructions) is PAGE_OFFSET - 16M (0x01000000) = 0xbf000000 so the modules use addresses 0xbf000000 .. 0xbfffffff - So the addresses 0xbf000000 .. 0xffffffff need to be covered with shadow memory. That is 0x41000000 bytes of memory. - 1/8 of that is needed for its shadow memory, so 0x08200000 bytes of shadow memory is needed. We "steal" that from the remaining lowmem. - The KASAN_SHADOW_START becomes 0xb6e00000, to KASAN_SHADOW_END at 0xbfffffff. - Now we can calculate the KASAN_SHADOW_OFFSET for any kernel address as 0xbf000000 needs to map to the first byte of shadow memory and 0xffffffff needs to map to the last byte of shadow memory. Since: SHADOW_ADDR = (address >> 3) + KASAN_SHADOW_OFFSET 0xb6e00000 = (0xbf000000 >> 3) + KASAN_SHADOW_OFFSET KASAN_SHADOW_OFFSET = 0xb6e00000 - (0xbf000000 >> 3) KASAN_SHADOW_OFFSET = 0xb6e00000 - 0x17e00000 KASAN_SHADOW_OFFSET = 0x9f000000 - 0x8f000000 if we have 3G userspace / 1G kernelspace with full 1 GB low memory (VMSPLIT_3G_OPT): - The kernel address space is 1GB (0x40000000) - PAGE_OFFSET is set to 0xb0000000 so the kernel static image uses 0xb0000000 .. 0xffffffff. - On top of that we have the MODULES_VADDR which under the worst case (using ARM instructions) is PAGE_OFFSET - 16M (0x01000000) = 0xaf000000 so the modules use addresses 0xaf000000 .. 0xaffffff - So the addresses 0xaf000000 .. 0xffffffff need to be covered with shadow memory. That is 0x51000000 bytes of memory. - 1/8 of that is needed for its shadow memory, so 0x0a200000 bytes of shadow memory is needed. We "steal" that from the remaining lowmem. - The KASAN_SHADOW_START becomes 0xa4e00000, to KASAN_SHADOW_END at 0xaeffffff. - Now we can calculate the KASAN_SHADOW_OFFSET for any kernel address as 0xaf000000 needs to map to the first byte of shadow memory and 0xffffffff needs to map to the last byte of shadow memory. Since: SHADOW_ADDR = (address >> 3) + KASAN_SHADOW_OFFSET 0xa4e00000 = (0xaf000000 >> 3) + KASAN_SHADOW_OFFSET KASAN_SHADOW_OFFSET = 0xa4e00000 - (0xaf000000 >> 3) KASAN_SHADOW_OFFSET = 0xa4e00000 - 0x15e00000 KASAN_SHADOW_OFFSET = 0x8f000000 - The default value of 0xffffffff for KASAN_SHADOW_OFFSET is an error value. We should always match one of the above shadow offsets. When we do this, TASK_SIZE will sometimes get a bit odd values that will not fit into immediate mov assembly instructions. To account for this, we need to rewrite some assembly using TASK_SIZE like this: - mov r1, #TASK_SIZE + ldr r1, =TASK_SIZE or - cmp r4, #TASK_SIZE + ldr r0, =TASK_SIZE + cmp r4, r0 this is done to avoid the immediate #TASK_SIZE that need to fit into a limited number of bits. Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: kasan-dev@googlegroups.com Cc: Mike Rapoport <rppt@linux.ibm.com> Reviewed-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Ard Biesheuvel <ardb@kernel.org> # QEMU/KVM/mach-virt/LPAE/8G Tested-by: Florian Fainelli <f.fainelli@gmail.com> # Brahma SoCs Tested-by: Ahmad Fatoum <a.fatoum@pengutronix.de> # i.MX6Q Reported-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Abbott Liu <liuwenliang@huawei.com> Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2020-10-25 22:53:46 +00:00
ldr r0, =TASK_SIZE
cmp r4, r0
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
blhs kuser_cmpxchg64_fixup
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
#endif
#endif
.endm
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.align 5
__dabt_usr:
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
usr_entry uaccess=0
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
kuser_cmpxchg_check
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
mov r2, sp
ARM: entry: data abort: tail-call the main data abort handler Tail-call the main C data abort handler code from the per-CPU helper code. Update the comments in the code wrt the new calling and return register state. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 15:01:26 +00:00
dabt_helper
b ret_from_exception
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnend )
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__dabt_usr)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.align 5
__irq_usr:
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
usr_entry
ARM: entry: instrument usr exception handlers with irqsoff tracing As we no longer re-enable interrupts in these exception handlers, add the irqsoff tracing calls to them so that the kernel tracks the state more accurately. Note that these calls are conditional on IRQSOFF_TRACER: kernel ----------> user ---------> kernel ^ irqs enabled ^ irqs disabled No kernel code can run on the local CPU until we've re-entered the kernel through one of the exception handlers - and userspace can not take any locks etc. So, the kernel doesn't care about the IRQ mask state while userspace is running unless we're doing IRQ off latency tracing. So, we can (and do) avoid the overhead of updating the IRQ mask state on every kernel->user and user->kernel transition. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 17:28:19 +00:00
kuser_cmpxchg_check
ARM: implement IRQ stacks Now that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well. Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Acked-by: Linus Walleij <linus.walleij@linaro.org> Tested-by: Keith Packard <keithpac@amazon.com> Acked-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-05 07:15:40 +00:00
irq_handler from_user=1
ARM: entry: no need to increase preempt count for IRQ handlers irq_enter() and irq_exit() already take care of the preempt_count handling for interrupts, which increment and decrement the hardirq bits of the preempt count. So we can remove the preempt count handing in our IRQ entry/exit assembly, like x86 did some 9 years ago. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 09:57:57 +00:00
get_thread_info tsk
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
mov why, #0
ARM: 6952/1: fix lockdep warning of "unannotated irqs-off" This patch fixes the lockdep warning of "unannotated irqs-off"[1]. After entering __irq_usr, arm core will disable interrupt automatically, but __irq_usr does not annotate the irq disable, so lockdep may complain the warning if it has chance to check this in irq handler. This patch adds trace_hardirqs_off in __irq_usr before entering irq_handler to handle the irq, also calls ret_to_user_from_irq to avoid calling disable_irq again. This is also a fix for irq off tracer. [1], lockdep warning log of "unannotated irqs-off" [ 13.804687] ------------[ cut here ]------------ [ 13.809570] WARNING: at kernel/lockdep.c:3335 check_flags+0x78/0x1d0() [ 13.816467] Modules linked in: [ 13.819732] Backtrace: [ 13.822357] [<c01cb42c>] (dump_backtrace+0x0/0x100) from [<c06abb14>] (dump_stack+0x20/0x24) [ 13.831268] r6:c07d8c2c r5:00000d07 r4:00000000 r3:00000000 [ 13.837280] [<c06abaf4>] (dump_stack+0x0/0x24) from [<c01ffc04>] (warn_slowpath_common+0x5c/0x74) [ 13.846649] [<c01ffba8>] (warn_slowpath_common+0x0/0x74) from [<c01ffc48>] (warn_slowpath_null+0x2c/0x34) [ 13.856781] r8:00000000 r7:00000000 r6:c18b8194 r5:60000093 r4:ef182000 [ 13.863708] r3:00000009 [ 13.866485] [<c01ffc1c>] (warn_slowpath_null+0x0/0x34) from [<c0237d84>] (check_flags+0x78/0x1d0) [ 13.875823] [<c0237d0c>] (check_flags+0x0/0x1d0) from [<c023afc8>] (lock_acquire+0x4c/0x150) [ 13.884704] [<c023af7c>] (lock_acquire+0x0/0x150) from [<c06af638>] (_raw_spin_lock+0x4c/0x84) [ 13.893798] [<c06af5ec>] (_raw_spin_lock+0x0/0x84) from [<c01f9a44>] (sched_ttwu_pending+0x58/0x8c) [ 13.903320] r6:ef92d040 r5:00000003 r4:c18b8180 [ 13.908233] [<c01f99ec>] (sched_ttwu_pending+0x0/0x8c) from [<c01f9a90>] (scheduler_ipi+0x18/0x1c) [ 13.917663] r6:ef183fb0 r5:00000003 r4:00000000 r3:00000001 [ 13.923645] [<c01f9a78>] (scheduler_ipi+0x0/0x1c) from [<c01bc458>] (do_IPI+0x9c/0xfc) [ 13.932006] [<c01bc3bc>] (do_IPI+0x0/0xfc) from [<c06b0888>] (__irq_usr+0x48/0xe0) [ 13.939971] Exception stack(0xef183fb0 to 0xef183ff8) [ 13.945281] 3fa0: ffffffc3 0001500c 00000001 0001500c [ 13.953948] 3fc0: 00000050 400b45f0 400d9000 00000000 00000001 400d9600 6474e552 bea05b3c [ 13.962585] 3fe0: 400d96c0 bea059c0 400b6574 400b65d8 20000010 ffffffff [ 13.969573] r6:00000403 r5:fa240100 r4:ffffffff r3:20000010 [ 13.975585] ---[ end trace efc4896ab0fb62cb ]--- [ 13.980468] possible reason: unannotated irqs-off. [ 13.985534] irq event stamp: 1610 [ 13.989044] hardirqs last enabled at (1610): [<c01c703c>] no_work_pending+0x8/0x2c [ 13.997131] hardirqs last disabled at (1609): [<c01c7024>] ret_slow_syscall+0xc/0x1c [ 14.005371] softirqs last enabled at (0): [<c01fe5e4>] copy_process+0x2cc/0xa24 [ 14.013183] softirqs last disabled at (0): [< (null)>] (null) Signed-off-by: Ming Lei <ming.lei@canonical.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-05 01:24:58 +00:00
b ret_to_user_from_irq
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnend )
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__irq_usr)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.ltorg
.align 5
__und_usr:
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
usr_entry uaccess=0
ARM: entry: instrument usr exception handlers with irqsoff tracing As we no longer re-enable interrupts in these exception handlers, add the irqsoff tracing calls to them so that the kernel tracks the state more accurately. Note that these calls are conditional on IRQSOFF_TRACER: kernel ----------> user ---------> kernel ^ irqs enabled ^ irqs disabled No kernel code can run on the local CPU until we've re-entered the kernel through one of the exception handlers - and userspace can not take any locks etc. So, the kernel doesn't care about the IRQ mask state while userspace is running unless we're doing IRQ off latency tracing. So, we can (and do) avoid the overhead of updating the IRQ mask state on every kernel->user and user->kernel transition. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 17:28:19 +00:00
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
mov r2, r4
mov r3, r5
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
@ r2 = regs->ARM_pc, which is either 2 or 4 bytes ahead of the
@ faulting instruction depending on Thumb mode.
@ r3 = regs->ARM_cpsr
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
@ The emulation code returns using r9 if it has emulated the
@ instruction, or the more conventional lr if we are to treat
@ this as a real undefined instruction
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@
ARM: replace BSYM() with badr assembly macro BSYM() was invented to allow us to work around a problem with the assembler, where local symbols resolved by the assembler for the 'adr' instruction did not take account of their ISA. Since we don't want BSYM() used elsewhere, replace BSYM() with a new macro 'badr', which is like the 'adr' pseudo-op, but with the BSYM() mechanics integrated into it. This ensures that the BSYM()-ification is only used in conjunction with 'adr'. Acked-by: Dave Martin <Dave.Martin@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-04-21 13:17:25 +00:00
badr r9, ret_from_exception
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
ARM: 8036/1: Enable IRQs before attempting to read user space in __und_usr The Undef abort handler in the kernel reads the undefined instruction from user space. If the page table was modified from another CPU, the user access could fail and do_page_fault() will be executed with interrupts disabled. This can potentially deadlock on ARM11MPCore or on Cortex-A15 with erratum 798181 workaround enabled (both implying IPI for TLB maintenance with page table lock held). This patch enables the IRQs in __und_usr before attempting to read the instruction from user space. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Arun KS <getarunks@gmail.com> Cc: Hartley Sweeten <hsweeten@visionengravers.com> Cc: Ryan Mallon <rmallon@gmail.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-04-22 15:14:29 +00:00
@ IRQs must be enabled before attempting to read the instruction from
@ user space since that could cause a page/translation fault if the
@ page table was modified by another CPU.
enable_irq
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
tst r3, #PSR_T_BIT @ Thumb mode?
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
bne __und_usr_thumb
sub r4, r2, #4 @ ARM instr at LR - 4
1: ldrt r0, [r4]
ARM: asm: Add ARM_BE8() assembly helper Add ARM_BE8() helper to wrap any code conditional on being compile when CONFIG_ARM_ENDIAN_BE8 is selected and convert existing places where this is to use it. Acked-by: Nicolas Pitre <nico@linaro.org> Reviewed-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
2013-02-12 18:59:57 +00:00
ARM_BE8(rev r0, r0) @ little endian instruction
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
uaccess_disable ip
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
@ r0 = 32-bit ARM instruction which caused the exception
@ r2 = PC value for the following instruction (:= regs->ARM_pc)
@ r4 = PC value for the faulting instruction
@ lr = 32-bit undefined instruction function
ARM: replace BSYM() with badr assembly macro BSYM() was invented to allow us to work around a problem with the assembler, where local symbols resolved by the assembler for the 'adr' instruction did not take account of their ISA. Since we don't want BSYM() used elsewhere, replace BSYM() with a new macro 'badr', which is like the 'adr' pseudo-op, but with the BSYM() mechanics integrated into it. This ensures that the BSYM()-ification is only used in conjunction with 'adr'. Acked-by: Dave Martin <Dave.Martin@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-04-21 13:17:25 +00:00
badr lr, __und_usr_fault_32
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
b call_fpe
__und_usr_thumb:
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
@ Thumb instruction
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
sub r4, r2, #2 @ First half of thumb instr at LR - 2
ARM: 7031/1: entry: Fix Thumb-2 undef handling for multi-CPU kernels When v6 and >=v7 boards are supported in the same kernel, the __und_usr code currently makes a build-time assumption that Thumb-2 instructions occurring in userspace don't need to be supported. Strictly speaking this is incorrect. This patch fixes the above case by doing a run-time check on the CPU architecture in these cases. This only affects kernels which support v6 and >=v7 CPUs together: plain v6 and plain v7 kernels are unaffected. Signed-off-by: Dave Martin <dave.martin@linaro.org> Reviewed-by: Jon Medhurst <tixy@yxit.co.uk> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-08-19 17:00:08 +00:00
#if CONFIG_ARM_THUMB && __LINUX_ARM_ARCH__ >= 6 && CONFIG_CPU_V7
/*
* Thumb-2 instruction handling. Note that because pre-v6 and >= v6 platforms
* can never be supported in a single kernel, this code is not applicable at
* all when __LINUX_ARM_ARCH__ < 6. This allows simplifying assumptions to be
* made about .arch directives.
*/
#if __LINUX_ARM_ARCH__ < 7
/* If the target CPU may not be Thumb-2-capable, a run-time check is needed: */
#define NEED_CPU_ARCHITECTURE
ldr r5, .LCcpu_architecture
ldr r5, [r5]
cmp r5, #CPU_ARCH_ARMv7
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
blo __und_usr_fault_16 @ 16bit undefined instruction
ARM: 7031/1: entry: Fix Thumb-2 undef handling for multi-CPU kernels When v6 and >=v7 boards are supported in the same kernel, the __und_usr code currently makes a build-time assumption that Thumb-2 instructions occurring in userspace don't need to be supported. Strictly speaking this is incorrect. This patch fixes the above case by doing a run-time check on the CPU architecture in these cases. This only affects kernels which support v6 and >=v7 CPUs together: plain v6 and plain v7 kernels are unaffected. Signed-off-by: Dave Martin <dave.martin@linaro.org> Reviewed-by: Jon Medhurst <tixy@yxit.co.uk> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-08-19 17:00:08 +00:00
/*
* The following code won't get run unless the running CPU really is v7, so
* coding round the lack of ldrht on older arches is pointless. Temporarily
* override the assembler target arch with the minimum required instead:
*/
.arch armv6t2
#endif
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
2: ldrht r5, [r4]
ARM: 7946/1: asm: __und_usr_thumb need byteswap instructions in BE case __und_usr_thumb function deals with thumb2 opcodes. In case of BE image, it needs to byteswap half word thumb2 encoded instructions before further processing them. Without this fix BE image user-land thread executing first VFP instruction encoded in thumb2 fails with SIGILL, because kernel does not recognize instruction and does not enable VFP. Reported-by: Corey Melton <comelton@cisco.com> Signed-off-by: Victor Kamensky <victor.kamensky@linaro.org> Tested-by: Stephen Boyd <sboyd@codeaurora.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-01-21 05:45:11 +00:00
ARM_BE8(rev16 r5, r5) @ little endian instruction
ARM: 7030/1: entry: Remove unnecessary masking when decoding Thumb-2 instructions When testing whether a Thumb-2 instruction is 32 bits long or not, the masking done in order to test bits 11-15 of the first instruction halfword won't affect the result of the comparison, so remove it. Signed-off-by: Dave Martin <dave.martin@linaro.org> Reviewed-by: Jon Medhurst <tixy@yxit.co.uk> Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-08-19 16:59:27 +00:00
cmp r5, #0xe800 @ 32bit instruction if xx != 0
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
blo __und_usr_fault_16_pan @ 16bit undefined instruction
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
3: ldrht r0, [r2]
ARM: 7946/1: asm: __und_usr_thumb need byteswap instructions in BE case __und_usr_thumb function deals with thumb2 opcodes. In case of BE image, it needs to byteswap half word thumb2 encoded instructions before further processing them. Without this fix BE image user-land thread executing first VFP instruction encoded in thumb2 fails with SIGILL, because kernel does not recognize instruction and does not enable VFP. Reported-by: Corey Melton <comelton@cisco.com> Signed-off-by: Victor Kamensky <victor.kamensky@linaro.org> Tested-by: Stephen Boyd <sboyd@codeaurora.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-01-21 05:45:11 +00:00
ARM_BE8(rev16 r0, r0) @ little endian instruction
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
uaccess_disable ip
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
add r2, r2, #2 @ r2 is PC + 2, make it PC + 4
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
str r2, [sp, #S_PC] @ it's a 2x16bit instr, update
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
orr r0, r0, r5, lsl #16
ARM: replace BSYM() with badr assembly macro BSYM() was invented to allow us to work around a problem with the assembler, where local symbols resolved by the assembler for the 'adr' instruction did not take account of their ISA. Since we don't want BSYM() used elsewhere, replace BSYM() with a new macro 'badr', which is like the 'adr' pseudo-op, but with the BSYM() mechanics integrated into it. This ensures that the BSYM()-ification is only used in conjunction with 'adr'. Acked-by: Dave Martin <Dave.Martin@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-04-21 13:17:25 +00:00
badr lr, __und_usr_fault_32
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
@ r0 = the two 16-bit Thumb instructions which caused the exception
@ r2 = PC value for the following Thumb instruction (:= regs->ARM_pc)
@ r4 = PC value for the first 16-bit Thumb instruction
@ lr = 32bit undefined instruction function
ARM: 7031/1: entry: Fix Thumb-2 undef handling for multi-CPU kernels When v6 and >=v7 boards are supported in the same kernel, the __und_usr code currently makes a build-time assumption that Thumb-2 instructions occurring in userspace don't need to be supported. Strictly speaking this is incorrect. This patch fixes the above case by doing a run-time check on the CPU architecture in these cases. This only affects kernels which support v6 and >=v7 CPUs together: plain v6 and plain v7 kernels are unaffected. Signed-off-by: Dave Martin <dave.martin@linaro.org> Reviewed-by: Jon Medhurst <tixy@yxit.co.uk> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-08-19 17:00:08 +00:00
#if __LINUX_ARM_ARCH__ < 7
/* If the target arch was overridden, change it back: */
#ifdef CONFIG_CPU_32v6K
.arch armv6k
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
#else
ARM: 7031/1: entry: Fix Thumb-2 undef handling for multi-CPU kernels When v6 and >=v7 boards are supported in the same kernel, the __und_usr code currently makes a build-time assumption that Thumb-2 instructions occurring in userspace don't need to be supported. Strictly speaking this is incorrect. This patch fixes the above case by doing a run-time check on the CPU architecture in these cases. This only affects kernels which support v6 and >=v7 CPUs together: plain v6 and plain v7 kernels are unaffected. Signed-off-by: Dave Martin <dave.martin@linaro.org> Reviewed-by: Jon Medhurst <tixy@yxit.co.uk> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-08-19 17:00:08 +00:00
.arch armv6
#endif
#endif /* __LINUX_ARM_ARCH__ < 7 */
#else /* !(CONFIG_ARM_THUMB && __LINUX_ARM_ARCH__ >= 6 && CONFIG_CPU_V7) */
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
b __und_usr_fault_16
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
#endif
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
UNWIND(.fnend)
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__und_usr)
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
* The out of line fixup for the ldrt instructions above.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
*/
ARM: 8322/1: keep .text and .fixup regions closer together This moves all fixup snippets to the .text.fixup section, which is a special section that gets emitted along with the .text section for each input object file, i.e., the snippets are kept much closer to the code they refer to, which helps prevent linker failure on large kernels. Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-24 09:41:09 +00:00
.pushsection .text.fixup, "ax"
ARM: 7425/1: extable: ensure fixup entries are 4-byte aligned Fixup entries in the kernel exception tables should be 4-byte aligned since we return directly to them when handling a faulting instruction in the kernel. This patch adds the missing align directives to the fixup entries. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-06-15 15:49:58 +00:00
.align 2
ARM: 8062/1: Modify ldrt fixup handler to re-execute the userspace instruction We will reach fixup handler when one thread(say cpu0) caused an undefined exception, while another thread(say cpu1) is unmmaping the page. Fixup handler returns to the next userspace instruction which has caused the undef execption, rather than going to the same instruction. ARM ARM says that after undefined exception, the PC will be pointing to the next instruction. ie +4 offset in case of ARM and +2 in case of Thumb And there is no correction offset passed to vector_stub in case of undef exception. File: arch/arm/kernel/entry-armv.S +1085 vector_stub und, UND_MODE During an undefined exception, in normal scenario(ie when ldrt instruction does not cause an abort) after resorting the context in VFP hardware, the PC is modified as show below before jumping to ret_from_exception which is in r9. File: arch/arm/vfp/vfphw.S +169 @ The context stored in the VFP hardware is up to date with this thread vfp_hw_state_valid: tst r1, #FPEXC_EX bne process_exception @ might as well handle the pending @ exception before retrying branch @ out before setting an FPEXC that @ stops us reading stuff VFPFMXR FPEXC, r1 @ Restore FPEXC last sub r2, r2, #4 @ Retry current instruction - if Thumb str r2, [sp, #S_PC] @ mode it's two 16-bit instructions, @ else it's one 32-bit instruction, so @ always subtract 4 from the following @ instruction address. But if ldrt results in an abort, we reach the fixup handler and return to ret_from_execption without correcting the pc. This patch modifes the fixup handler to re-execute the same instruction which caused undefined execption. Signed-off-by: Vinayak Menon <vinayakm.list@gmail.com> Signed-off-by: Arun KS <getarunks@gmail.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-05-19 10:43:00 +00:00
4: str r4, [sp, #S_PC] @ retry current instruction
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
ret r9
ARM: fix build error in arch/arm/kernel/process.c /tmp/ccJ3ssZW.s: Assembler messages: /tmp/ccJ3ssZW.s:1952: Error: can't resolve `.text' {.text section} - `.LFB1077' This is caused because: .section .data .section .text .section .text .previous does not return us to the .text section, but the .data section; this makes use of .previous dangerous if the ordering of previous sections is not known. Fix up the other users of .previous; .pushsection and .popsection are a safer pairing to use than .section and .previous. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-04-19 09:15:03 +00:00
.popsection
.pushsection __ex_table,"a"
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
.long 1b, 4b
ARM: 7170/2: fix compilation breakage in entry-armv.S Fix compilation failure, when Thumb support is not enabled: arch/arm/kernel/entry-armv.S: Assembler messages: arch/arm/kernel/entry-armv.S:501: Error: backward ref to unknown label "2:" arch/arm/kernel/entry-armv.S:502: Error: backward ref to unknown label "3:" make[2]: *** [arch/arm/kernel/entry-armv.o] Error 1 Signed-off-by: Guennadi Liakhovetski <g.liakhovetski@gmx.de> Reviewed-by: Dave Martin <dave.martin@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-11-22 22:42:12 +00:00
#if CONFIG_ARM_THUMB && __LINUX_ARM_ARCH__ >= 6 && CONFIG_CPU_V7
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
.long 2b, 4b
.long 3b, 4b
#endif
ARM: fix build error in arch/arm/kernel/process.c /tmp/ccJ3ssZW.s: Assembler messages: /tmp/ccJ3ssZW.s:1952: Error: can't resolve `.text' {.text section} - `.LFB1077' This is caused because: .section .data .section .text .section .text .previous does not return us to the .text section, but the .data section; this makes use of .previous dangerous if the ordering of previous sections is not known. Fix up the other users of .previous; .pushsection and .popsection are a safer pairing to use than .section and .previous. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-04-19 09:15:03 +00:00
.popsection
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*
* Check whether the instruction is a co-processor instruction.
* If yes, we need to call the relevant co-processor handler.
*
* Note that we don't do a full check here for the co-processor
* instructions; all instructions with bit 27 set are well
* defined. The only instructions that should fault are the
* co-processor instructions. However, we have to watch out
* for the ARM6/ARM7 SWI bug.
*
[ARM] 4584/2: ARMv7: Add Advanced SIMD (NEON) extension support This patch enables the use of the Advanced SIMD (NEON) extension on ARMv7. The NEON technology is a 64/128-bit hybrid SIMD architecture for accelerating the performance of multimedia and signal processing applications. The extension shares the registers with the VFP unit and enabling/disabling and saving/restoring follow the same rules. In addition, there are instructions that do not have the appropriate CP number encoded, the checks being made in the call_fpe function. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-01-10 18:16:17 +00:00
* NEON is a special case that has to be handled here. Not all
* NEON instructions are co-processor instructions, so we have
* to make a special case of checking for them. Plus, there's
* five groups of them, so we have a table of mask/opcode pairs
* to check against, and if any match then we branch off into the
* NEON handler code.
*
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
* Emulators may wish to make use of the following registers:
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
* r0 = instruction opcode (32-bit ARM or two 16-bit Thumb)
* r2 = PC value to resume execution after successful emulation
[ARM] Fix kernel-mode undefined instruction aborts If the kernel attempts to execute a CP1 or CP2 instruction and it aborts, and a FP emulator is not loaded, we try to return as if to a user context, instead of the proper kernel context. Since the fault came from kernel mode, we must use the kernel return paths. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-01-06 22:53:48 +00:00
* r9 = normal "successful" return address
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
* r10 = this threads thread_info structure
[ARM] Fix kernel-mode undefined instruction aborts If the kernel attempts to execute a CP1 or CP2 instruction and it aborts, and a FP emulator is not loaded, we try to return as if to a user context, instead of the proper kernel context. Since the fault came from kernel mode, we must use the kernel return paths. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-01-06 22:53:48 +00:00
* lr = unrecognised instruction return address
ARM: 8036/1: Enable IRQs before attempting to read user space in __und_usr The Undef abort handler in the kernel reads the undefined instruction from user space. If the page table was modified from another CPU, the user access could fail and do_page_fault() will be executed with interrupts disabled. This can potentially deadlock on ARM11MPCore or on Cortex-A15 with erratum 798181 workaround enabled (both implying IPI for TLB maintenance with page table lock held). This patch enables the IRQs in __und_usr before attempting to read the instruction from user space. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Arun KS <getarunks@gmail.com> Cc: Hartley Sweeten <hsweeten@visionengravers.com> Cc: Ryan Mallon <rmallon@gmail.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-04-22 15:14:29 +00:00
* IRQs enabled, FIQs enabled.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
*/
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
@
@ Fall-through from Thumb-2 __und_usr
@
#ifdef CONFIG_NEON
ARM: cleanup undefined instruction entry code We don't need to keep reloading the thread into into r10 - we can do this once and keep the value cached in the register. Also, schedule some instructions better so that the pipeline doesn't stall after a load in the neon code. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-02-23 17:53:52 +00:00
get_thread_info r10 @ get current thread
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
adr r6, .LCneon_thumb_opcodes
b 2f
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
call_fpe:
ARM: cleanup undefined instruction entry code We don't need to keep reloading the thread into into r10 - we can do this once and keep the value cached in the register. Also, schedule some instructions better so that the pipeline doesn't stall after a load in the neon code. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-02-23 17:53:52 +00:00
get_thread_info r10 @ get current thread
[ARM] 4584/2: ARMv7: Add Advanced SIMD (NEON) extension support This patch enables the use of the Advanced SIMD (NEON) extension on ARMv7. The NEON technology is a 64/128-bit hybrid SIMD architecture for accelerating the performance of multimedia and signal processing applications. The extension shares the registers with the VFP unit and enabling/disabling and saving/restoring follow the same rules. In addition, there are instructions that do not have the appropriate CP number encoded, the checks being made in the call_fpe function. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-01-10 18:16:17 +00:00
#ifdef CONFIG_NEON
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
adr r6, .LCneon_arm_opcodes
ARM: cleanup undefined instruction entry code We don't need to keep reloading the thread into into r10 - we can do this once and keep the value cached in the register. Also, schedule some instructions better so that the pipeline doesn't stall after a load in the neon code. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-02-23 17:53:52 +00:00
2: ldr r5, [r6], #4 @ mask value
[ARM] 4584/2: ARMv7: Add Advanced SIMD (NEON) extension support This patch enables the use of the Advanced SIMD (NEON) extension on ARMv7. The NEON technology is a 64/128-bit hybrid SIMD architecture for accelerating the performance of multimedia and signal processing applications. The extension shares the registers with the VFP unit and enabling/disabling and saving/restoring follow the same rules. In addition, there are instructions that do not have the appropriate CP number encoded, the checks being made in the call_fpe function. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-01-10 18:16:17 +00:00
ldr r7, [r6], #4 @ opcode bits matching in mask
ARM: cleanup undefined instruction entry code We don't need to keep reloading the thread into into r10 - we can do this once and keep the value cached in the register. Also, schedule some instructions better so that the pipeline doesn't stall after a load in the neon code. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-02-23 17:53:52 +00:00
cmp r5, #0 @ end mask?
beq 1f
and r8, r0, r5
[ARM] 4584/2: ARMv7: Add Advanced SIMD (NEON) extension support This patch enables the use of the Advanced SIMD (NEON) extension on ARMv7. The NEON technology is a 64/128-bit hybrid SIMD architecture for accelerating the performance of multimedia and signal processing applications. The extension shares the registers with the VFP unit and enabling/disabling and saving/restoring follow the same rules. In addition, there are instructions that do not have the appropriate CP number encoded, the checks being made in the call_fpe function. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-01-10 18:16:17 +00:00
cmp r8, r7 @ NEON instruction?
bne 2b
mov r7, #1
strb r7, [r10, #TI_USED_CP + 10] @ mark CP#10 as used
strb r7, [r10, #TI_USED_CP + 11] @ mark CP#11 as used
b do_vfp @ let VFP handler handle this
1:
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
tst r0, #0x08000000 @ only CDP/CPRT/LDC/STC have bit 27
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
tstne r0, #0x04000000 @ bit 26 set on both ARM and Thumb-2
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
reteq lr
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
and r8, r0, #0x00000f00 @ mask out CP number
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
THUMB( lsr r8, r8, #8 )
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
mov r7, #1
add r6, r10, #TI_USED_CP
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
ARM( strb r7, [r6, r8, lsr #8] ) @ set appropriate used_cp[]
THUMB( strb r7, [r6, r8] ) @ set appropriate used_cp[]
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
#ifdef CONFIG_IWMMXT
@ Test if we need to give access to iWMMXt coprocessors
ldr r5, [r10, #TI_FLAGS]
rsbs r7, r8, #(1 << 8) @ CP 0 or 1 only
ARM: 8844/1: use unified assembler in assembly files Use unified assembler syntax (UAL) in assembly files. Divided syntax is considered deprecated. This will also allow to build the kernel using LLVM's integrated assembler. Signed-off-by: Stefan Agner <stefan@agner.ch> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2019-02-17 23:57:38 +00:00
movscs r7, r5, lsr #(TIF_USING_IWMMXT + 1)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
bcs iwmmxt_task_enable
#endif
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
ARM( add pc, pc, r8, lsr #6 )
THUMB( lsl r8, r8, #2 )
THUMB( add pc, r8 )
nop
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
ret.w lr @ CP#0
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
W(b) do_fpe @ CP#1 (FPE)
W(b) do_fpe @ CP#2 (FPE)
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
ret.w lr @ CP#3
ret.w lr @ CP#4
ret.w lr @ CP#5
ret.w lr @ CP#6
ret.w lr @ CP#7
ret.w lr @ CP#8
ret.w lr @ CP#9
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
#ifdef CONFIG_VFP
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
W(b) do_vfp @ CP#10 (VFP)
W(b) do_vfp @ CP#11 (VFP)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
#else
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
ret.w lr @ CP#10 (VFP)
ret.w lr @ CP#11 (VFP)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
#endif
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
ret.w lr @ CP#12
ret.w lr @ CP#13
ret.w lr @ CP#14 (Debug)
ret.w lr @ CP#15 (Control)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: 7031/1: entry: Fix Thumb-2 undef handling for multi-CPU kernels When v6 and >=v7 boards are supported in the same kernel, the __und_usr code currently makes a build-time assumption that Thumb-2 instructions occurring in userspace don't need to be supported. Strictly speaking this is incorrect. This patch fixes the above case by doing a run-time check on the CPU architecture in these cases. This only affects kernels which support v6 and >=v7 CPUs together: plain v6 and plain v7 kernels are unaffected. Signed-off-by: Dave Martin <dave.martin@linaro.org> Reviewed-by: Jon Medhurst <tixy@yxit.co.uk> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-08-19 17:00:08 +00:00
#ifdef NEED_CPU_ARCHITECTURE
.align 2
.LCcpu_architecture:
.word __cpu_architecture
#endif
[ARM] 4584/2: ARMv7: Add Advanced SIMD (NEON) extension support This patch enables the use of the Advanced SIMD (NEON) extension on ARMv7. The NEON technology is a 64/128-bit hybrid SIMD architecture for accelerating the performance of multimedia and signal processing applications. The extension shares the registers with the VFP unit and enabling/disabling and saving/restoring follow the same rules. In addition, there are instructions that do not have the appropriate CP number encoded, the checks being made in the call_fpe function. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-01-10 18:16:17 +00:00
#ifdef CONFIG_NEON
.align 6
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
.LCneon_arm_opcodes:
[ARM] 4584/2: ARMv7: Add Advanced SIMD (NEON) extension support This patch enables the use of the Advanced SIMD (NEON) extension on ARMv7. The NEON technology is a 64/128-bit hybrid SIMD architecture for accelerating the performance of multimedia and signal processing applications. The extension shares the registers with the VFP unit and enabling/disabling and saving/restoring follow the same rules. In addition, there are instructions that do not have the appropriate CP number encoded, the checks being made in the call_fpe function. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-01-10 18:16:17 +00:00
.word 0xfe000000 @ mask
.word 0xf2000000 @ opcode
.word 0xff100000 @ mask
.word 0xf4000000 @ opcode
Linux Thumb-2 support for user-space applications This patch implements Thumb-2 application support in Linux. Original implementation by Paul Brook with fixes for VFP and Neon by Catalin Marinas. Signed-off-by: Paul Brook <paul@codesourcery.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2008-04-18 21:43:08 +00:00
.word 0x00000000 @ mask
.word 0x00000000 @ opcode
.LCneon_thumb_opcodes:
.word 0xef000000 @ mask
.word 0xef000000 @ opcode
.word 0xff100000 @ mask
.word 0xf9000000 @ opcode
[ARM] 4584/2: ARMv7: Add Advanced SIMD (NEON) extension support This patch enables the use of the Advanced SIMD (NEON) extension on ARMv7. The NEON technology is a 64/128-bit hybrid SIMD architecture for accelerating the performance of multimedia and signal processing applications. The extension shares the registers with the VFP unit and enabling/disabling and saving/restoring follow the same rules. In addition, there are instructions that do not have the appropriate CP number encoded, the checks being made in the call_fpe function. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-01-10 18:16:17 +00:00
.word 0x00000000 @ mask
.word 0x00000000 @ opcode
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
do_fpe:
ldr r4, .LCfp
add r10, r10, #TI_FPSTATE @ r10 = workspace
ldr pc, [r4] @ Call FP module USR entry point
/*
* The FP module is called with these registers set:
* r0 = instruction
* r2 = PC+4
* r9 = normal "successful" return address
* r10 = FP workspace
* lr = unrecognised FP instruction return address
*/
ARM: 6068/1: Fix build break with KPROBES enabled With CONFIG_KPROBES enabled two section are getting created which leads to below build break. LOG: AS arch/arm/kernel/entry-armv.o arch/arm/kernel/entry-armv.S: Assembler messages: arch/arm/kernel/entry-armv.S:431: Error: symbol ret_from_exception is in a different section arch/arm/kernel/entry-armv.S:490: Error: symbol ret_from_exception is in a different section arch/arm/kernel/entry-armv.S:491: Error: symbol __und_usr_unknown is in a different section This was introduced by commit 4260415f6a3b92c5c986398d96c314df37a4ccbf Reported-by: Anand Gadiyar <gadiyar@ti.com> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-04-30 09:45:46 +00:00
.pushsection .data
ARM: align .data section Robert Jarzmik reports that his PXA25x system fails to boot with 4.12, failing at __flush_whole_cache in arch/arm/mm/proc-xscale.S:215: 0xc0019e20 <+0>: ldr r1, [pc, #788] 0xc0019e24 <+4>: ldr r0, [r1] <== here with r1 containing 0xc06f82cd, which is the address of "clean_addr". Examination of the System.map shows: c06f22c8 D user_pmd_table c06f22cc d __warned.19178 c06f22cd d clean_addr indicating that a .data.unlikely section has appeared just before the .data section from proc-xscale.S. According to objdump -h, it appears that our assembly files default their .data alignment to 2**0, which is bad news if the preceding .data section size is not power-of-2 aligned at link time. Add the appropriate .align directives to all assembly files in arch/arm that are missing them where we require an appropriate alignment. Reported-by: Robert Jarzmik <robert.jarzmik@free.fr> Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-07-26 11:49:31 +00:00
.align 2
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ENTRY(fp_enter)
[ARM] Fix kernel-mode undefined instruction aborts If the kernel attempts to execute a CP1 or CP2 instruction and it aborts, and a FP emulator is not loaded, we try to return as if to a user context, instead of the proper kernel context. Since the fault came from kernel mode, we must use the kernel return paths. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-01-06 22:53:48 +00:00
.word no_fp
ARM: 6068/1: Fix build break with KPROBES enabled With CONFIG_KPROBES enabled two section are getting created which leads to below build break. LOG: AS arch/arm/kernel/entry-armv.o arch/arm/kernel/entry-armv.S: Assembler messages: arch/arm/kernel/entry-armv.S:431: Error: symbol ret_from_exception is in a different section arch/arm/kernel/entry-armv.S:490: Error: symbol ret_from_exception is in a different section arch/arm/kernel/entry-armv.S:491: Error: symbol __und_usr_unknown is in a different section This was introduced by commit 4260415f6a3b92c5c986398d96c314df37a4ccbf Reported-by: Anand Gadiyar <gadiyar@ti.com> Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-04-30 09:45:46 +00:00
.popsection
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
Thumb-2: Correctly handle undefined instructions in the kernel VFP instructions in the kernel may trigger undefined exceptions if VFP hardware is not present. This patch corrects the loading of such Thumb-2 instructions. It also marks the "no_fp" label as a function so that the linker generate a Thumb address. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-09-18 22:27:07 +00:00
ENTRY(no_fp)
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
ret lr
Thumb-2: Correctly handle undefined instructions in the kernel VFP instructions in the kernel may trigger undefined exceptions if VFP hardware is not present. This patch corrects the loading of such Thumb-2 instructions. It also marks the "no_fp" label as a function so that the linker generate a Thumb address. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-09-18 22:27:07 +00:00
ENDPROC(no_fp)
[ARM] Fix kernel-mode undefined instruction aborts If the kernel attempts to execute a CP1 or CP2 instruction and it aborts, and a FP emulator is not loaded, we try to return as if to a user context, instead of the proper kernel context. Since the fault came from kernel mode, we must use the kernel return paths. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-01-06 22:53:48 +00:00
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
__und_usr_fault_32:
mov r1, #4
b 1f
ARM: entry: provide uaccess assembly macro hooks Provide hooks into the kernel entry and exit paths to permit control of userspace visibility to the kernel. The intended use is: - on entry to kernel from user, uaccess_disable will be called to disable userspace visibility - on exit from kernel to user, uaccess_enable will be called to enable userspace visibility - on entry from a kernel exception, uaccess_save_and_disable will be called to save the current userspace visibility setting, and disable access - on exit from a kernel exception, uaccess_restore will be called to restore the userspace visibility as it was before the exception occurred. These hooks allows us to keep userspace visibility disabled for the vast majority of the kernel, except for localised regions where we want to explicitly access userspace. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-20 09:32:02 +00:00
__und_usr_fault_16_pan:
uaccess_disable ip
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
__und_usr_fault_16:
mov r1, #2
ARM: 8036/1: Enable IRQs before attempting to read user space in __und_usr The Undef abort handler in the kernel reads the undefined instruction from user space. If the page table was modified from another CPU, the user access could fail and do_page_fault() will be executed with interrupts disabled. This can potentially deadlock on ARM11MPCore or on Cortex-A15 with erratum 798181 workaround enabled (both implying IPI for TLB maintenance with page table lock held). This patch enables the IRQs in __und_usr before attempting to read the instruction from user space. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Arun KS <getarunks@gmail.com> Cc: Hartley Sweeten <hsweeten@visionengravers.com> Cc: Ryan Mallon <rmallon@gmail.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-04-22 15:14:29 +00:00
1: mov r0, sp
ARM: replace BSYM() with badr assembly macro BSYM() was invented to allow us to work around a problem with the assembler, where local symbols resolved by the assembler for the 'adr' instruction did not take account of their ISA. Since we don't want BSYM() used elsewhere, replace BSYM() with a new macro 'badr', which is like the 'adr' pseudo-op, but with the BSYM() mechanics integrated into it. This ensures that the BSYM()-ification is only used in conjunction with 'adr'. Acked-by: Dave Martin <Dave.Martin@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-04-21 13:17:25 +00:00
badr lr, ret_from_exception
ARM: Fix undefined instruction exception handling While trying to get a v3.5 kernel booted on the cubox, I noticed that VFP does not work correctly with VFP bounce handling. This is because of the confusion over 16-bit vs 32-bit instructions, and where PC is supposed to point to. The rule is that FP handlers are entered with regs->ARM_pc pointing at the _next_ instruction to be executed. However, if the exception is not handled, regs->ARM_pc points at the faulting instruction. This is easy for ARM mode, because we know that the next instruction and previous instructions are separated by four bytes. This is not true of Thumb2 though. Since all FP instructions are 32-bit in Thumb2, it makes things easy. We just need to select the appropriate adjustment. Do this by moving the adjustment out of do_undefinstr() into the assembly code, as only the assembly code knows whether it's dealing with a 32-bit or 16-bit instruction. Cc: <stable@vger.kernel.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-07-30 18:42:10 +00:00
b __und_fault
ENDPROC(__und_usr_fault_32)
ENDPROC(__und_usr_fault_16)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.align 5
__pabt_usr:
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
usr_entry
ARM: 5727/1: Pass IFSR register to do_PrefetchAbort() Instruction fault status register, IFSR, was introduced on ARMv6 to provide status information about the last insturction fault. It needed for proper prefetch abort handling. Now we have three prefetch abort model: * legacy - for CPUs before ARMv6. They doesn't provide neither IFSR nor IFAR. We simulate IFSR with section translation fault status for them to generalize code; * ARMv6 - provides IFSR, but not IFAR; * ARMv7 - provides both IFSR and IFAR. Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-09-25 12:39:47 +00:00
mov r2, sp @ regs
ARM: entry: prefetch abort: tail-call the main prefetch abort handler Tail-call the main C prefetch abort handler code from the per-CPU helper code. Also note that the helper function becomes ABI compliant in terms of the registers preserved. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-26 11:37:35 +00:00
pabt_helper
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnend )
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/* fall through */
/*
* This is the return code to user mode for abort handlers
*/
ENTRY(ret_from_exception)
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnstart )
UNWIND(.cantunwind )
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
get_thread_info tsk
mov why, #0
b ret_to_user
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnend )
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__pabt_usr)
ENDPROC(ret_from_exception)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
.align 5
__fiq_usr:
usr_entry trace=0
kuser_cmpxchg_check
mov r0, sp @ struct pt_regs *regs
bl handle_fiq_as_nmi
get_thread_info tsk
restore_user_regs fast = 0, offset = 0
UNWIND(.fnend )
ENDPROC(__fiq_usr)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*
* Register switch for ARMv3 and ARMv4 processors
* r0 = previous task_struct, r1 = previous thread_info, r2 = next thread_info
* previous and next are guaranteed not to be the same.
*/
ENTRY(__switch_to)
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnstart )
UNWIND(.cantunwind )
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
add ip, r1, #TI_CPU_SAVE
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
ARM( stmia ip!, {r4 - sl, fp, sp, lr} ) @ Store most regs on stack
THUMB( stmia ip!, {r4 - sl, fp} ) @ Store most regs on stack
THUMB( str sp, [ip], #4 )
THUMB( str lr, [ip], #4 )
ARM: 7735/2: Preserve the user r/w register TPIDRURW on context switch and fork Since commit 6a1c53124aa1 the user writeable TLS register was zeroed to prevent it from being used as a covert channel between two tasks. There are more and more applications coming to Windows RT, Wine could support them, but mostly they expect to have the thread environment block (TEB) in TPIDRURW. This patch preserves that register per thread instead of clearing it. Unlike the TPIDRURO, which is already switched, the TPIDRURW can be updated from userspace so needs careful treatment in the case that we modify TPIDRURW and call fork(). To avoid this we must always read TPIDRURW in copy_thread. Signed-off-by: André Hentschel <nerv@dawncrow.de> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Jonathan Austin <jonathan.austin@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-06-18 22:23:26 +00:00
ldr r4, [r2, #TI_TP_VALUE]
ldr r5, [r2, #TI_TP_VALUE + 4]
ARM: 6384/1: Remove the domain switching on ARMv6k/v7 CPUs This patch removes the domain switching functionality via the set_fs and __switch_to functions on cores that have a TLS register. Currently, the ioremap and vmalloc areas share the same level 1 page tables and therefore have the same domain (DOMAIN_KERNEL). When the kernel domain is modified from Client to Manager (via the __set_fs or in the __switch_to function), the XN (eXecute Never) bit is overridden and newer CPUs can speculatively prefetch the ioremap'ed memory. Linux performs the kernel domain switching to allow user-specific functions (copy_to/from_user, get/put_user etc.) to access kernel memory. In order for these functions to work with the kernel domain set to Client, the patch modifies the LDRT/STRT and related instructions to the LDR/STR ones. The user pages access rights are also modified for kernel read-only access rather than read/write so that the copy-on-write mechanism still works. CPU_USE_DOMAINS gets disabled only if the hardware has a TLS register (CPU_32v6K is defined) since writing the TLS value to the high vectors page isn't possible. The user addresses passed to the kernel are checked by the access_ok() function so that they do not point to the kernel space. Tested-by: Anton Vorontsov <cbouatmailru@gmail.com> Cc: Tony Lindgren <tony@atomide.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-09-13 15:03:21 +00:00
#ifdef CONFIG_CPU_USE_DOMAINS
ARM: domains: switch to keeping domain value in register Rather than modifying both the domain access control register and our per-thread copy, modify only the domain access control register, and use the per-thread copy to save and restore the register over context switches. We can also avoid the explicit initialisation of the init thread_info structure. This allows us to avoid needing to gain access to the thread information at the uaccess control sites. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-08-19 20:23:48 +00:00
mrc p15, 0, r6, c3, c0, 0 @ Get domain register
str r6, [r1, #TI_CPU_DOMAIN] @ Save old domain register
[ARM] Add thread_notify infrastructure Some machine classes need to allow VFP support to be built into the kernel, but still allow the kernel to run even though VFP isn't present. Unfortunately, the kernel hard-codes VFP instructions into the thread switch, which prevents this being run-time selectable. Solve this by introducing a notifier which things such as VFP can hook into to be informed of events which affect the VFP subsystem (eg, creation and destruction of threads, switches between threads.) Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-06-21 12:31:52 +00:00
ldr r6, [r2, #TI_CPU_DOMAIN]
[ARM] Support register switch in nommu mode This patch adds register switch support in nommu mode. Signed-off-by: Hyok S. Choi <hyok.choi@samsung.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-01-13 21:05:25 +00:00
#endif
ARM: 7735/2: Preserve the user r/w register TPIDRURW on context switch and fork Since commit 6a1c53124aa1 the user writeable TLS register was zeroed to prevent it from being used as a covert channel between two tasks. There are more and more applications coming to Windows RT, Wine could support them, but mostly they expect to have the thread environment block (TEB) in TPIDRURW. This patch preserves that register per thread instead of clearing it. Unlike the TPIDRURO, which is already switched, the TPIDRURW can be updated from userspace so needs careful treatment in the case that we modify TPIDRURW and call fork(). To avoid this we must always read TPIDRURW in copy_thread. Signed-off-by: André Hentschel <nerv@dawncrow.de> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Jonathan Austin <jonathan.austin@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-06-18 22:23:26 +00:00
switch_tls r1, r4, r5, r3, r7
ARM: entry: preserve thread_info pointer in switch_to Tweak the UP stack protector handling code so that the thread info pointer is preserved in R7 until set_current is called. This is needed for a subsequent patch that implements THREAD_INFO_IN_TASK and set_current for UP as well. This also means we will prefer the per-task protector on UP systems that implement the thread ID registers, so tweak the preprocessor conditionals to reflect this. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-11-24 11:09:48 +00:00
#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) && \
!defined(CONFIG_STACKPROTECTOR_PER_TASK)
ldr r9, [r2, #TI_TASK]
ARM: stack protector: change the canary value per task A new random value for the canary is stored in the task struct whenever a new task is forked. This is meant to allow for different canary values per task. On ARM, GCC expects the canary value to be found in a global variable called __stack_chk_guard. So this variable has to be updated with the value stored in the task struct whenever a task switch occurs. Because the variable GCC expects is global, this cannot work on SMP unfortunately. So, on SMP, the same initial canary value is kept throughout, making this feature a bit less effective although it is still useful. One way to overcome this GCC limitation would be to locate the __stack_chk_guard variable into a memory page of its own for each CPU, and then use TLB locking to have each CPU see its own page at the same virtual address for each of them. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org>
2010-06-08 01:50:33 +00:00
ldr r8, =__stack_chk_guard
ARM: Prepare for randomized task_struct With the new task struct randomization, we can run into a build failure for certain random seeds, which will place fields beyond the allow immediate size in the assembly: arch/arm/kernel/entry-armv.S: Assembler messages: arch/arm/kernel/entry-armv.S:803: Error: bad immediate value for offset (4096) Only two constants in asm-offset.h are affected, and I'm changing both of them here to work correctly in all configurations. One more macro has the problem, but is currently unused, so this removes it instead of adding complexity. Suggested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Arnd Bergmann <arnd@arndb.de> [kees: Adjust commit log slightly] Signed-off-by: Kees Cook <keescook@chromium.org>
2017-06-30 16:03:59 +00:00
.if (TSK_STACK_CANARY > IMM12_MASK)
ARM: entry: preserve thread_info pointer in switch_to Tweak the UP stack protector handling code so that the thread info pointer is preserved in R7 until set_current is called. This is needed for a subsequent patch that implements THREAD_INFO_IN_TASK and set_current for UP as well. This also means we will prefer the per-task protector on UP systems that implement the thread ID registers, so tweak the preprocessor conditionals to reflect this. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-11-24 11:09:48 +00:00
add r9, r9, #TSK_STACK_CANARY & ~IMM12_MASK
ARM: Prepare for randomized task_struct With the new task struct randomization, we can run into a build failure for certain random seeds, which will place fields beyond the allow immediate size in the assembly: arch/arm/kernel/entry-armv.S: Assembler messages: arch/arm/kernel/entry-armv.S:803: Error: bad immediate value for offset (4096) Only two constants in asm-offset.h are affected, and I'm changing both of them here to work correctly in all configurations. One more macro has the problem, but is currently unused, so this removes it instead of adding complexity. Suggested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Arnd Bergmann <arnd@arndb.de> [kees: Adjust commit log slightly] Signed-off-by: Kees Cook <keescook@chromium.org>
2017-06-30 16:03:59 +00:00
.endif
ARM: entry: preserve thread_info pointer in switch_to Tweak the UP stack protector handling code so that the thread info pointer is preserved in R7 until set_current is called. This is needed for a subsequent patch that implements THREAD_INFO_IN_TASK and set_current for UP as well. This also means we will prefer the per-task protector on UP systems that implement the thread ID registers, so tweak the preprocessor conditionals to reflect this. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-11-24 11:09:48 +00:00
ldr r9, [r9, #TSK_STACK_CANARY & IMM12_MASK]
ARM: stack protector: change the canary value per task A new random value for the canary is stored in the task struct whenever a new task is forked. This is meant to allow for different canary values per task. On ARM, GCC expects the canary value to be found in a global variable called __stack_chk_guard. So this variable has to be updated with the value stored in the task struct whenever a task switch occurs. Because the variable GCC expects is global, this cannot work on SMP unfortunately. So, on SMP, the same initial canary value is kept throughout, making this feature a bit less effective although it is still useful. One way to overcome this GCC limitation would be to locate the __stack_chk_guard variable into a memory page of its own for each CPU, and then use TLB locking to have each CPU see its own page at the same virtual address for each of them. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org>
2010-06-08 01:50:33 +00:00
#endif
ARM: entry: preserve thread_info pointer in switch_to Tweak the UP stack protector handling code so that the thread info pointer is preserved in R7 until set_current is called. This is needed for a subsequent patch that implements THREAD_INFO_IN_TASK and set_current for UP as well. This also means we will prefer the per-task protector on UP systems that implement the thread ID registers, so tweak the preprocessor conditionals to reflect this. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-11-24 11:09:48 +00:00
mov r7, r2 @ Preserve 'next'
ARM: 6384/1: Remove the domain switching on ARMv6k/v7 CPUs This patch removes the domain switching functionality via the set_fs and __switch_to functions on cores that have a TLS register. Currently, the ioremap and vmalloc areas share the same level 1 page tables and therefore have the same domain (DOMAIN_KERNEL). When the kernel domain is modified from Client to Manager (via the __set_fs or in the __switch_to function), the XN (eXecute Never) bit is overridden and newer CPUs can speculatively prefetch the ioremap'ed memory. Linux performs the kernel domain switching to allow user-specific functions (copy_to/from_user, get/put_user etc.) to access kernel memory. In order for these functions to work with the kernel domain set to Client, the patch modifies the LDRT/STRT and related instructions to the LDR/STR ones. The user pages access rights are also modified for kernel read-only access rather than read/write so that the copy-on-write mechanism still works. CPU_USE_DOMAINS gets disabled only if the hardware has a TLS register (CPU_32v6K is defined) since writing the TLS value to the high vectors page isn't possible. The user addresses passed to the kernel are checked by the access_ok() function so that they do not point to the kernel space. Tested-by: Anton Vorontsov <cbouatmailru@gmail.com> Cc: Tony Lindgren <tony@atomide.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-09-13 15:03:21 +00:00
#ifdef CONFIG_CPU_USE_DOMAINS
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
mcr p15, 0, r6, c3, c0, 0 @ Set domain register
#endif
[ARM] Add thread_notify infrastructure Some machine classes need to allow VFP support to be built into the kernel, but still allow the kernel to run even though VFP isn't present. Unfortunately, the kernel hard-codes VFP instructions into the thread switch, which prevents this being run-time selectable. Solve this by introducing a notifier which things such as VFP can hook into to be informed of events which affect the VFP subsystem (eg, creation and destruction of threads, switches between threads.) Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-06-21 12:31:52 +00:00
mov r5, r0
add r4, r2, #TI_CPU_SAVE
ldr r0, =thread_notify_head
mov r1, #THREAD_NOTIFY_SWITCH
bl atomic_notifier_call_chain
ARM: entry: preserve thread_info pointer in switch_to Tweak the UP stack protector handling code so that the thread info pointer is preserved in R7 until set_current is called. This is needed for a subsequent patch that implements THREAD_INFO_IN_TASK and set_current for UP as well. This also means we will prefer the per-task protector on UP systems that implement the thread ID registers, so tweak the preprocessor conditionals to reflect this. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-11-24 11:09:48 +00:00
#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) && \
!defined(CONFIG_STACKPROTECTOR_PER_TASK)
str r9, [r8]
ARM: stack protector: change the canary value per task A new random value for the canary is stored in the task struct whenever a new task is forked. This is meant to allow for different canary values per task. On ARM, GCC expects the canary value to be found in a global variable called __stack_chk_guard. So this variable has to be updated with the value stored in the task struct whenever a task switch occurs. Because the variable GCC expects is global, this cannot work on SMP unfortunately. So, on SMP, the same initial canary value is kept throughout, making this feature a bit less effective although it is still useful. One way to overcome this GCC limitation would be to locate the __stack_chk_guard variable into a memory page of its own for each CPU, and then use TLB locking to have each CPU see its own page at the same virtual address for each of them. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org>
2010-06-08 01:50:33 +00:00
#endif
[ARM] Add thread_notify infrastructure Some machine classes need to allow VFP support to be built into the kernel, but still allow the kernel to run even though VFP isn't present. Unfortunately, the kernel hard-codes VFP instructions into the thread switch, which prevents this being run-time selectable. Solve this by introducing a notifier which things such as VFP can hook into to be informed of events which affect the VFP subsystem (eg, creation and destruction of threads, switches between threads.) Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-06-21 12:31:52 +00:00
mov r0, r5
ARM: implement support for vmap'ed stacks Wire up the generic support for managing task stack allocations via vmalloc, and implement the entry code that detects whether we faulted because of a stack overrun (or future stack overrun caused by pushing the pt_regs array) While this adds a fair amount of tricky entry asm code, it should be noted that it only adds a TST + branch to the svc_entry path. The code implementing the non-trivial handling of the overflow stack is emitted out-of-line into the .text section. Since on ARM, we rely on do_translation_fault() to keep PMD level page table entries that cover the vmalloc region up to date, we need to ensure that we don't hit such a stale PMD entry when accessing the stack. So we do a dummy read from the new stack while still running from the old one on the context switch path, and bump the vmalloc_seq counter when PMD level entries in the vmalloc range are modified, so that the MM switch fetches the latest version of the entries. Note that we need to increase the per-mode stack by 1 word, to gain some space to stash a GPR until we know it is safe to touch the stack. However, due to the cacheline alignment of the struct, this does not actually increase the memory footprint of the struct stack array at all. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Keith Packard <keithpac@amazon.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-09-23 07:15:53 +00:00
#if !defined(CONFIG_THUMB2_KERNEL) && !defined(CONFIG_VMAP_STACK)
ARM: smp: Store current pointer in TPIDRURO register if available Now that the user space TLS register is assigned on every return to user space, we can use it to keep the 'current' pointer while running in the kernel. This removes the need to access it via thread_info, which is located at the base of the stack, but will be moved out of there in a subsequent patch. Use the __builtin_thread_pointer() helper when available - this will help GCC understand that reloading the value within the same function is not necessary, even when using the per-task stack protector (which also generates accesses via the TLS register). For example, the generated code below loads TPIDRURO only once, and uses it to access both the stack canary and the preempt_count fields. <do_one_initcall>: e92d 41f0 stmdb sp!, {r4, r5, r6, r7, r8, lr} ee1d 4f70 mrc 15, 0, r4, cr13, cr0, {3} 4606 mov r6, r0 b094 sub sp, #80 ; 0x50 f8d4 34e8 ldr.w r3, [r4, #1256] ; 0x4e8 <- stack canary 9313 str r3, [sp, #76] ; 0x4c f8d4 8004 ldr.w r8, [r4, #4] <- preempt count Co-developed-by: Keith Packard <keithpac@amazon.com> Signed-off-by: Keith Packard <keithpac@amazon.com> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Reviewed-by: Linus Walleij <linus.walleij@linaro.org> Tested-by: Amit Daniel Kachhap <amit.kachhap@arm.com>
2021-09-18 08:44:37 +00:00
set_current r7
ARM: switch_to: clean up Thumb2 code path The load-multiple instruction that essentially performs the switch_to operation in ARM mode, by loading all callee save registers as well the stack pointer and the program counter, is split into 3 separate loads for Thumb-2, with the IP register used as a temporary to capture the value of R4 before it gets overwritten. We can clean this up a bit, by sticking with a single LDMIA instruction, but one that pops SP and PC into IP and LR, respectively, and by using ordinary move register and branch instructions to get those values into SP and PC. This also allows us to move the set_current call closer to the assignment of SP, reducing the window where those are mutually out of sync. This is especially relevant for CONFIG_VMAP_STACK, which is being introduced in a subsequent patch, where we need to issue a load that might fault from the new stack while running from the old one, to ensure that stale PMD entries in the VMALLOC space are synced up. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Keith Packard <keithpac@amazon.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-16 16:00:01 +00:00
ldmia r4, {r4 - sl, fp, sp, pc} @ Load all regs saved previously
#else
mov r1, r7
ldmia r4, {r4 - sl, fp, ip, lr} @ Load all regs saved previously
ARM: implement support for vmap'ed stacks Wire up the generic support for managing task stack allocations via vmalloc, and implement the entry code that detects whether we faulted because of a stack overrun (or future stack overrun caused by pushing the pt_regs array) While this adds a fair amount of tricky entry asm code, it should be noted that it only adds a TST + branch to the svc_entry path. The code implementing the non-trivial handling of the overflow stack is emitted out-of-line into the .text section. Since on ARM, we rely on do_translation_fault() to keep PMD level page table entries that cover the vmalloc region up to date, we need to ensure that we don't hit such a stale PMD entry when accessing the stack. So we do a dummy read from the new stack while still running from the old one on the context switch path, and bump the vmalloc_seq counter when PMD level entries in the vmalloc range are modified, so that the MM switch fetches the latest version of the entries. Note that we need to increase the per-mode stack by 1 word, to gain some space to stash a GPR until we know it is safe to touch the stack. However, due to the cacheline alignment of the struct, this does not actually increase the memory footprint of the struct stack array at all. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Keith Packard <keithpac@amazon.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-09-23 07:15:53 +00:00
#ifdef CONFIG_VMAP_STACK
@
@ Do a dummy read from the new stack while running from the old one so
@ that we can rely on do_translation_fault() to fix up any stale PMD
@ entries covering the vmalloc region.
@
ldr r2, [ip]
#endif
ARM: switch_to: clean up Thumb2 code path The load-multiple instruction that essentially performs the switch_to operation in ARM mode, by loading all callee save registers as well the stack pointer and the program counter, is split into 3 separate loads for Thumb-2, with the IP register used as a temporary to capture the value of R4 before it gets overwritten. We can clean this up a bit, by sticking with a single LDMIA instruction, but one that pops SP and PC into IP and LR, respectively, and by using ordinary move register and branch instructions to get those values into SP and PC. This also allows us to move the set_current call closer to the assignment of SP, reducing the window where those are mutually out of sync. This is especially relevant for CONFIG_VMAP_STACK, which is being introduced in a subsequent patch, where we need to issue a load that might fault from the new stack while running from the old one, to ensure that stale PMD entries in the VMALLOC space are synced up. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Keith Packard <keithpac@amazon.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-10-16 16:00:01 +00:00
@ When CONFIG_THREAD_INFO_IN_TASK=n, the update of SP itself is what
@ effectuates the task switch, as that is what causes the observable
@ values of current and current_thread_info to change. When
@ CONFIG_THREAD_INFO_IN_TASK=y, setting current (and therefore
@ current_thread_info) is done explicitly, and the update of SP just
@ switches us to another stack, with few other side effects. In order
@ to prevent this distinction from causing any inconsistencies, let's
@ keep the 'set_current' call as close as we can to the update of SP.
set_current r1
mov sp, ip
ret lr
#endif
[ARM] 5385/2: unwind: Add unwinding information to exception entry points This is needed to allow or stop the unwinding at certain points in the kernel like exception entries. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2009-02-16 10:42:09 +00:00
UNWIND(.fnend )
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(__switch_to)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: implement support for vmap'ed stacks Wire up the generic support for managing task stack allocations via vmalloc, and implement the entry code that detects whether we faulted because of a stack overrun (or future stack overrun caused by pushing the pt_regs array) While this adds a fair amount of tricky entry asm code, it should be noted that it only adds a TST + branch to the svc_entry path. The code implementing the non-trivial handling of the overflow stack is emitted out-of-line into the .text section. Since on ARM, we rely on do_translation_fault() to keep PMD level page table entries that cover the vmalloc region up to date, we need to ensure that we don't hit such a stale PMD entry when accessing the stack. So we do a dummy read from the new stack while still running from the old one on the context switch path, and bump the vmalloc_seq counter when PMD level entries in the vmalloc range are modified, so that the MM switch fetches the latest version of the entries. Note that we need to increase the per-mode stack by 1 word, to gain some space to stash a GPR until we know it is safe to touch the stack. However, due to the cacheline alignment of the struct, this does not actually increase the memory footprint of the struct stack array at all. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Keith Packard <keithpac@amazon.com> Tested-by: Marc Zyngier <maz@kernel.org> Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
2021-09-23 07:15:53 +00:00
#ifdef CONFIG_VMAP_STACK
.text
.align 2
__bad_stack:
@
@ We've just detected an overflow. We need to load the address of this
@ CPU's overflow stack into the stack pointer register. We have only one
@ scratch register so let's use a sequence of ADDs including one
@ involving the PC, and decorate them with PC-relative group
@ relocations. As these are ARM only, switch to ARM mode first.
@
@ We enter here with IP clobbered and its value stashed on the mode
@ stack.
@
THUMB( bx pc )
THUMB( nop )
THUMB( .arm )
mrc p15, 0, ip, c13, c0, 4 @ Get per-CPU offset
.globl overflow_stack_ptr
.reloc 0f, R_ARM_ALU_PC_G0_NC, overflow_stack_ptr
.reloc 1f, R_ARM_ALU_PC_G1_NC, overflow_stack_ptr
.reloc 2f, R_ARM_LDR_PC_G2, overflow_stack_ptr
add ip, ip, pc
0: add ip, ip, #-4
1: add ip, ip, #0
2: ldr ip, [ip, #4]
str sp, [ip, #-4]! @ Preserve original SP value
mov sp, ip @ Switch to overflow stack
pop {ip} @ Original SP in IP
#if defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_GCC)
mov ip, ip @ mov expected by unwinder
push {fp, ip, lr, pc} @ GCC flavor frame record
#else
str ip, [sp, #-8]! @ store original SP
push {fpreg, lr} @ Clang flavor frame record
#endif
UNWIND( ldr ip, [r0, #4] ) @ load exception LR
UNWIND( str ip, [sp, #12] ) @ store in the frame record
ldr ip, [r0, #12] @ reload IP
@ Store the original GPRs to the new stack.
svc_entry uaccess=0, overflow_check=0
UNWIND( .save {sp, pc} )
UNWIND( .save {fpreg, lr} )
UNWIND( .setfp fpreg, sp )
ldr fpreg, [sp, #S_SP] @ Add our frame record
@ to the linked list
#if defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_GCC)
ldr r1, [fp, #4] @ reload SP at entry
add fp, fp, #12
#else
ldr r1, [fpreg, #8]
#endif
str r1, [sp, #S_SP] @ store in pt_regs
@ Stash the regs for handle_bad_stack
mov r0, sp
@ Time to die
bl handle_bad_stack
nop
UNWIND( .fnend )
ENDPROC(__bad_stack)
#endif
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
__INIT
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
/*
* User helpers.
*
* Each segment is 32-byte aligned and will be moved to the top of the high
* vector page. New segments (if ever needed) must be added in front of
* existing ones. This mechanism should be used only for things that are
* really small and justified, and not be abused freely.
*
docs: arm: convert docs to ReST and rename to *.rst Converts ARM the text files to ReST, preparing them to be an architecture book. The conversion is actually: - add blank lines and identation in order to identify paragraphs; - fix tables markups; - add some lists markups; - mark literal blocks; - adjust title markups. At its new index.rst, let's add a :orphan: while this is not linked to the main index.rst file, in order to avoid build warnings. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> Reviewed-by Corentin Labbe <clabbe.montjoie@gmail.com> # For sun4i-ss
2019-04-14 18:51:10 +00:00
* See Documentation/arm/kernel_user_helpers.rst for formal definitions.
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
*/
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
THUMB( .arm )
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
[ARM] 3746/2: Userspace helpers must be Thumb mode interworkable Patch from Nicolas Pitre The userspace helpers in clean/arch/arm/kernel/entry-armv.S are called directly in/from userspace. They need to cope with being called from Thumb code. Patch below uses the bx interworking instruction when CONFIG_ARM_THUMB=y. Based on an earlier patch from Paul Brook <paul@codesourcery.com> Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-08-18 16:20:15 +00:00
.macro usr_ret, reg
#ifdef CONFIG_ARM_THUMB
bx \reg
#else
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
ret \reg
[ARM] 3746/2: Userspace helpers must be Thumb mode interworkable Patch from Nicolas Pitre The userspace helpers in clean/arch/arm/kernel/entry-armv.S are called directly in/from userspace. They need to cope with being called from Thumb code. Patch below uses the bx interworking instruction when CONFIG_ARM_THUMB=y. Based on an earlier patch from Paul Brook <paul@codesourcery.com> Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-08-18 16:20:15 +00:00
#endif
.endm
ARM: poison memory between kuser helpers Poison the memory between each kuser helper. This ensures that any branch between the kuser helpers will be appropriately trapped. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 10:32:04 +00:00
.macro kuser_pad, sym, size
.if (. - \sym) & 3
.rept 4 - (. - \sym) & 3
.byte 0
.endr
.endif
.rept (\size - (. - \sym)) / 4
.word 0xe7fddef1
.endr
.endm
ARM: allow kuser helpers to be removed from the vector page Provide a kernel configuration option to allow the kernel user helpers to be removed from the vector page, thereby preventing their use with ROP (return orientated programming) attacks. This option is only visible for CPU architectures which natively support all the operations which kernel user helpers would normally provide, and must be enabled with caution. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-23 17:37:00 +00:00
#ifdef CONFIG_KUSER_HELPERS
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
.align 5
.globl __kuser_helper_start
__kuser_helper_start:
[ARM] 3210/1: add missing memory barrier helper for NPTL support Patch from Nicolas Pitre Strictly speaking, the NPTL kernel helpers are required for pre ARMv6 only. They are available on ARMv6+ as well for obvious compatibility reasons. However there are cases where extra memory barriers are needed when using an SMP ARMv6 machine but not on pre-ARMv6. This patch adds a memory barrier kernel helper that glibc can use as needed for pre-ARMv6 binaries to be forward compatible with an SMP kernel on ARMv6, as well as the necessary dmb instructions to the cmpxchg helper. Signed-off-by: Nicolas Pitre <nico@cam.org> Acked-by: Daniel Jacobowitz <dan@codesourcery.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-12-19 22:20:51 +00:00
/*
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
* Due to the length of some sequences, __kuser_cmpxchg64 spans 2 regular
* kuser "slots", therefore 0xffff0f80 is not used as a valid entry point.
[ARM] 3210/1: add missing memory barrier helper for NPTL support Patch from Nicolas Pitre Strictly speaking, the NPTL kernel helpers are required for pre ARMv6 only. They are available on ARMv6+ as well for obvious compatibility reasons. However there are cases where extra memory barriers are needed when using an SMP ARMv6 machine but not on pre-ARMv6. This patch adds a memory barrier kernel helper that glibc can use as needed for pre-ARMv6 binaries to be forward compatible with an SMP kernel on ARMv6, as well as the necessary dmb instructions to the cmpxchg helper. Signed-off-by: Nicolas Pitre <nico@cam.org> Acked-by: Daniel Jacobowitz <dan@codesourcery.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-12-19 22:20:51 +00:00
*/
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
__kuser_cmpxchg64: @ 0xffff0f60
ARM: remove user cmpxchg syscall Mark Brand reports that a NEEDS_SYSCALL_FOR_CMPXCHG enabled kernel would open a security hole in the ghost syscall used to implement cmpxchg, as it fails to validate the user pointer. However, in order for this option to be enabled, you'd need to be building a pre-ARMv6 kernel with SMP support. There is only one system known which fits that, which is an early ARM SMP FPGA implementation based on the ARM926T. In any case, the Kconfig does not allow SMP to be enabled for pre-ARMv6 systems. Moreover, even if NEEDS_SYSCALL_FOR_CMPXCHG were to be enabled, the kernel would not build as __ARM_NR_cmpxchg64 is not defined. The simple answer is to remove the buggy code. Reported-by: Mark Brand <markbrand@google.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-09-21 18:34:28 +00:00
#if defined(CONFIG_CPU_32v6K)
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
stmfd sp!, {r4, r5, r6, r7}
ldrd r4, r5, [r0] @ load old val
ldrd r6, r7, [r1] @ load new val
smp_dmb arm
1: ldrexd r0, r1, [r2] @ load current val
eors r3, r0, r4 @ compare with oldval (1)
ARM: 8844/1: use unified assembler in assembly files Use unified assembler syntax (UAL) in assembly files. Divided syntax is considered deprecated. This will also allow to build the kernel using LLVM's integrated assembler. Signed-off-by: Stefan Agner <stefan@agner.ch> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2019-02-17 23:57:38 +00:00
eorseq r3, r1, r5 @ compare with oldval (2)
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
strexdeq r3, r6, r7, [r2] @ store newval if eq
teqeq r3, #1 @ success?
beq 1b @ if no then retry
ARM: 6516/1: Allow SMP_ON_UP to work with Thumb-2 kernels. * __fixup_smp_on_up has been modified with support for the THUMB2_KERNEL case. For THUMB2_KERNEL only, fixups are split into halfwords in case of misalignment, since we can't rely on unaligned accesses working before turning the MMU on. No attempt is made to optimise the aligned case, since the number of fixups is typically small, and it seems best to keep the code as simple as possible. * Add a rotate in the fixup_smp code in order to support CPU_BIG_ENDIAN, as suggested by Nicolas Pitre. * Add an assembly-time sanity-check to ALT_UP() to ensure that the content really is the right size (4 bytes). (No check is done for ALT_SMP(). Possibly, this could be fixed by splitting the two uses ot ALT_SMP() (ALT_SMP...SMP_UP versus ALT_SMP...SMP_UP_B) into two macros. In the first case, ALT_SMP needs to expand to >= 4 bytes, not == 4.) * smp_mpidr.h (which implements ALT_SMP()/ALT_UP() manually due to macro limitations) has not been modified: the affected instruction (mov) has no 16-bit encoding, so the correct instruction size is satisfied in this case. * A "mode" parameter has been added to smp_dmb: smp_dmb arm @ assumes 4-byte instructions (for ARM code, e.g. kuser) smp_dmb @ uses W() to ensure 4-byte instructions for ALT_SMP() This avoids assembly failures due to use of W() inside smp_dmb, when assembling pure-ARM code in the vectors page. There might be a better way to achieve this. * Kconfig: make SMP_ON_UP depend on (!THUMB2_KERNEL || !BIG_ENDIAN) i.e., THUMB2_KERNEL is now supported, but only if !BIG_ENDIAN (The fixup code for Thumb-2 currently assumes little-endian order.) Tested using a single generic realview kernel on: ARM RealView PB-A8 (CONFIG_THUMB2_KERNEL={n,y}) ARM RealView PBX-A9 (SMP) Signed-off-by: Dave Martin <dave.martin@linaro.org> Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-12-01 14:39:23 +00:00
smp_dmb arm
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
rsbs r0, r3, #0 @ set returned val and C flag
ldmfd sp!, {r4, r5, r6, r7}
ARM: 7314/1: kuser: consistently use usr_ret for returning from helpers __kuser_cmpxchg64 has a return path using bx lr to get back to the caller. This is actually ok since the code in question is predicated on CONFIG_CPU_32v6K, but for the sake of consistency using the usr_ret macro is probably better. Acked-by: Dave Martin <dave.martin@linaro.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2012-02-03 10:08:05 +00:00
usr_ret lr
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
#elif !defined(CONFIG_SMP)
#ifdef CONFIG_MMU
/*
* The only thing that can break atomicity in this cmpxchg64
* implementation is either an IRQ or a data abort exception
* causing another process/thread to be scheduled in the middle of
* the critical sequence. The same strategy as for cmpxchg is used.
*/
stmfd sp!, {r4, r5, r6, lr}
ldmia r0, {r4, r5} @ load old val
ldmia r1, {r6, lr} @ load new val
1: ldmia r2, {r0, r1} @ load current val
eors r3, r0, r4 @ compare with oldval (1)
ARM: 8844/1: use unified assembler in assembly files Use unified assembler syntax (UAL) in assembly files. Divided syntax is considered deprecated. This will also allow to build the kernel using LLVM's integrated assembler. Signed-off-by: Stefan Agner <stefan@agner.ch> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2019-02-17 23:57:38 +00:00
eorseq r3, r1, r5 @ compare with oldval (2)
2: stmiaeq r2, {r6, lr} @ store newval if eq
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
rsbs r0, r3, #0 @ set return val and C flag
ldmfd sp!, {r4, r5, r6, pc}
.text
kuser_cmpxchg64_fixup:
@ Called from kuser_cmpxchg_fixup.
Merge branch 'devel-stable' into for-next Conflicts: arch/arm/kernel/entry-armv.S
2011-07-22 22:09:07 +00:00
@ r4 = address of interrupted insn (must be preserved).
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
@ sp = saved regs. r7 and r8 are clobbered.
@ 1b = first critical insn, 2b = last critical insn.
Merge branch 'devel-stable' into for-next Conflicts: arch/arm/kernel/entry-armv.S
2011-07-22 22:09:07 +00:00
@ If r4 >= 1b and r4 <= 2b then saved pc_usr is set to 1b.
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
mov r7, #0xffff0fff
sub r7, r7, #(0xffff0fff - (0xffff0f60 + (1b - __kuser_cmpxchg64)))
Merge branch 'devel-stable' into for-next Conflicts: arch/arm/kernel/entry-armv.S
2011-07-22 22:09:07 +00:00
subs r8, r4, r7
ARM: 8844/1: use unified assembler in assembly files Use unified assembler syntax (UAL) in assembly files. Divided syntax is considered deprecated. This will also allow to build the kernel using LLVM's integrated assembler. Signed-off-by: Stefan Agner <stefan@agner.ch> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2019-02-17 23:57:38 +00:00
rsbscs r8, r8, #(2b - 1b)
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
strcs r7, [sp, #S_PC]
#if __LINUX_ARM_ARCH__ < 6
bcc kuser_cmpxchg32_fixup
#endif
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
ret lr
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
.previous
#else
#warning "NPTL on non MMU needs fixing"
mov r0, #-1
adds r0, r0, #0
[ARM] 3746/2: Userspace helpers must be Thumb mode interworkable Patch from Nicolas Pitre The userspace helpers in clean/arch/arm/kernel/entry-armv.S are called directly in/from userspace. They need to cope with being called from Thumb code. Patch below uses the bx interworking instruction when CONFIG_ARM_THUMB=y. Based on an earlier patch from Paul Brook <paul@codesourcery.com> Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-08-18 16:20:15 +00:00
usr_ret lr
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
#endif
#else
#error "incoherent kernel configuration"
#endif
ARM: poison memory between kuser helpers Poison the memory between each kuser helper. This ensures that any branch between the kuser helpers will be appropriately trapped. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 10:32:04 +00:00
kuser_pad __kuser_cmpxchg64, 64
[ARM] 3210/1: add missing memory barrier helper for NPTL support Patch from Nicolas Pitre Strictly speaking, the NPTL kernel helpers are required for pre ARMv6 only. They are available on ARMv6+ as well for obvious compatibility reasons. However there are cases where extra memory barriers are needed when using an SMP ARMv6 machine but not on pre-ARMv6. This patch adds a memory barrier kernel helper that glibc can use as needed for pre-ARMv6 binaries to be forward compatible with an SMP kernel on ARMv6, as well as the necessary dmb instructions to the cmpxchg helper. Signed-off-by: Nicolas Pitre <nico@cam.org> Acked-by: Daniel Jacobowitz <dan@codesourcery.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-12-19 22:20:51 +00:00
__kuser_memory_barrier: @ 0xffff0fa0
ARM: 6516/1: Allow SMP_ON_UP to work with Thumb-2 kernels. * __fixup_smp_on_up has been modified with support for the THUMB2_KERNEL case. For THUMB2_KERNEL only, fixups are split into halfwords in case of misalignment, since we can't rely on unaligned accesses working before turning the MMU on. No attempt is made to optimise the aligned case, since the number of fixups is typically small, and it seems best to keep the code as simple as possible. * Add a rotate in the fixup_smp code in order to support CPU_BIG_ENDIAN, as suggested by Nicolas Pitre. * Add an assembly-time sanity-check to ALT_UP() to ensure that the content really is the right size (4 bytes). (No check is done for ALT_SMP(). Possibly, this could be fixed by splitting the two uses ot ALT_SMP() (ALT_SMP...SMP_UP versus ALT_SMP...SMP_UP_B) into two macros. In the first case, ALT_SMP needs to expand to >= 4 bytes, not == 4.) * smp_mpidr.h (which implements ALT_SMP()/ALT_UP() manually due to macro limitations) has not been modified: the affected instruction (mov) has no 16-bit encoding, so the correct instruction size is satisfied in this case. * A "mode" parameter has been added to smp_dmb: smp_dmb arm @ assumes 4-byte instructions (for ARM code, e.g. kuser) smp_dmb @ uses W() to ensure 4-byte instructions for ALT_SMP() This avoids assembly failures due to use of W() inside smp_dmb, when assembling pure-ARM code in the vectors page. There might be a better way to achieve this. * Kconfig: make SMP_ON_UP depend on (!THUMB2_KERNEL || !BIG_ENDIAN) i.e., THUMB2_KERNEL is now supported, but only if !BIG_ENDIAN (The fixup code for Thumb-2 currently assumes little-endian order.) Tested using a single generic realview kernel on: ARM RealView PB-A8 (CONFIG_THUMB2_KERNEL={n,y}) ARM RealView PBX-A9 (SMP) Signed-off-by: Dave Martin <dave.martin@linaro.org> Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-12-01 14:39:23 +00:00
smp_dmb arm
[ARM] 3746/2: Userspace helpers must be Thumb mode interworkable Patch from Nicolas Pitre The userspace helpers in clean/arch/arm/kernel/entry-armv.S are called directly in/from userspace. They need to cope with being called from Thumb code. Patch below uses the bx interworking instruction when CONFIG_ARM_THUMB=y. Based on an earlier patch from Paul Brook <paul@codesourcery.com> Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-08-18 16:20:15 +00:00
usr_ret lr
[ARM] 3210/1: add missing memory barrier helper for NPTL support Patch from Nicolas Pitre Strictly speaking, the NPTL kernel helpers are required for pre ARMv6 only. They are available on ARMv6+ as well for obvious compatibility reasons. However there are cases where extra memory barriers are needed when using an SMP ARMv6 machine but not on pre-ARMv6. This patch adds a memory barrier kernel helper that glibc can use as needed for pre-ARMv6 binaries to be forward compatible with an SMP kernel on ARMv6, as well as the necessary dmb instructions to the cmpxchg helper. Signed-off-by: Nicolas Pitre <nico@cam.org> Acked-by: Daniel Jacobowitz <dan@codesourcery.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-12-19 22:20:51 +00:00
ARM: poison memory between kuser helpers Poison the memory between each kuser helper. This ensures that any branch between the kuser helpers will be appropriately trapped. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 10:32:04 +00:00
kuser_pad __kuser_memory_barrier, 32
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
__kuser_cmpxchg: @ 0xffff0fc0
ARM: remove user cmpxchg syscall Mark Brand reports that a NEEDS_SYSCALL_FOR_CMPXCHG enabled kernel would open a security hole in the ghost syscall used to implement cmpxchg, as it fails to validate the user pointer. However, in order for this option to be enabled, you'd need to be building a pre-ARMv6 kernel with SMP support. There is only one system known which fits that, which is an early ARM SMP FPGA implementation based on the ARM926T. In any case, the Kconfig does not allow SMP to be enabled for pre-ARMv6 systems. Moreover, even if NEEDS_SYSCALL_FOR_CMPXCHG were to be enabled, the kernel would not build as __ARM_NR_cmpxchg64 is not defined. The simple answer is to remove the buggy code. Reported-by: Mark Brand <markbrand@google.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-09-21 18:34:28 +00:00
#if __LINUX_ARM_ARCH__ < 6
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
#ifdef CONFIG_MMU
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
/*
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
* The only thing that can break atomicity in this cmpxchg
* implementation is either an IRQ or a data abort exception
* causing another process/thread to be scheduled in the middle
* of the critical sequence. To prevent this, code is added to
* the IRQ and data abort exception handlers to set the pc back
* to the beginning of the critical section if it is found to be
* within that critical section (see kuser_cmpxchg_fixup).
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
*/
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
1: ldr r3, [r2] @ load current val
subs r3, r3, r0 @ compare with oldval
2: streq r1, [r2] @ store newval if eq
rsbs r0, r3, #0 @ set return val and C flag
usr_ret lr
.text
ARM: add a kuser_cmpxchg64 user space helper Some user space applications are designed around the ability to perform atomic operations on 64 bit values. Since this is natively possible only with ARMv6k and above, let's provide a new kuser helper to perform the operation with kernel supervision on pre ARMv6k hardware. Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org> Tested-by: Dave Martin <dave.martin@linaro.org>
2011-06-20 03:36:03 +00:00
kuser_cmpxchg32_fixup:
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
@ Called from kuser_cmpxchg_check macro.
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
@ r4 = address of interrupted insn (must be preserved).
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
@ sp = saved regs. r7 and r8 are clobbered.
@ 1b = first critical insn, 2b = last critical insn.
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
@ If r4 >= 1b and r4 <= 2b then saved pc_usr is set to 1b.
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
mov r7, #0xffff0fff
sub r7, r7, #(0xffff0fff - (0xffff0fc0 + (1b - __kuser_cmpxchg)))
ARM: entry: rejig register allocation in exception entry handlers This allows us to avoid moving registers twice to work around the clobbered registers when we add calls to trace_hardirqs_{on,off}. Ensure that all SVC handlers return with SPSR in r5 for consistency. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2011-06-25 14:44:20 +00:00
subs r8, r4, r7
ARM: 8844/1: use unified assembler in assembly files Use unified assembler syntax (UAL) in assembly files. Divided syntax is considered deprecated. This will also allow to build the kernel using LLVM's integrated assembler. Signed-off-by: Stefan Agner <stefan@agner.ch> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2019-02-17 23:57:38 +00:00
rsbscs r8, r8, #(2b - 1b)
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
strcs r7, [sp, #S_PC]
ARM: convert all "mov.* pc, reg" to "bx reg" for ARMv6+ ARMv6 and greater introduced a new instruction ("bx") which can be used to return from function calls. Recent CPUs perform better when the "bx lr" instruction is used rather than the "mov pc, lr" instruction, and this sequence is strongly recommended to be used by the ARM architecture manual (section A.4.1.1). We provide a new macro "ret" with all its variants for the condition code which will resolve to the appropriate instruction. Rather than doing this piecemeal, and miss some instances, change all the "mov pc" instances to use the new macro, with the exception of the "movs" instruction and the kprobes code. This allows us to detect the "mov pc, lr" case and fix it up - and also gives us the possibility of deploying this for other registers depending on the CPU selection. Reported-by: Will Deacon <will.deacon@arm.com> Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1 Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood Tested-by: Shawn Guo <shawn.guo@freescale.com> Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385 Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-06-30 15:29:12 +00:00
ret lr
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
.previous
[ARM] 3309/1: disable the pre-ARMv5 NPTL kernel helper in the non MMU case Patch from Nicolas Pitre The cmpxchg emulation on pre-ARMv5 relies on user code executed from a kernel address. If the operation cannot complete atomically, it is aborted from the usr_entry macro by clearing the Z flag. This clearing of the Z flag is done whenever the user pc is above TASK_SIZE. However this "pc >= TASK_SIZE" test cannot work in the non MMU case. Worse: the current code will corrupt the Z flag on every entry to the kernel. Let's disable it in the non MMU case for now. Using NPTL on non MMU targets needs to be worked out anyway. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-02-08 21:19:37 +00:00
#else
#warning "NPTL on non MMU needs fixing"
mov r0, #-1
adds r0, r0, #0
[ARM] 3746/2: Userspace helpers must be Thumb mode interworkable Patch from Nicolas Pitre The userspace helpers in clean/arch/arm/kernel/entry-armv.S are called directly in/from userspace. They need to cope with being called from Thumb code. Patch below uses the bx interworking instruction when CONFIG_ARM_THUMB=y. Based on an earlier patch from Paul Brook <paul@codesourcery.com> Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-08-18 16:20:15 +00:00
usr_ret lr
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
#endif
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
#else
ARM: 6516/1: Allow SMP_ON_UP to work with Thumb-2 kernels. * __fixup_smp_on_up has been modified with support for the THUMB2_KERNEL case. For THUMB2_KERNEL only, fixups are split into halfwords in case of misalignment, since we can't rely on unaligned accesses working before turning the MMU on. No attempt is made to optimise the aligned case, since the number of fixups is typically small, and it seems best to keep the code as simple as possible. * Add a rotate in the fixup_smp code in order to support CPU_BIG_ENDIAN, as suggested by Nicolas Pitre. * Add an assembly-time sanity-check to ALT_UP() to ensure that the content really is the right size (4 bytes). (No check is done for ALT_SMP(). Possibly, this could be fixed by splitting the two uses ot ALT_SMP() (ALT_SMP...SMP_UP versus ALT_SMP...SMP_UP_B) into two macros. In the first case, ALT_SMP needs to expand to >= 4 bytes, not == 4.) * smp_mpidr.h (which implements ALT_SMP()/ALT_UP() manually due to macro limitations) has not been modified: the affected instruction (mov) has no 16-bit encoding, so the correct instruction size is satisfied in this case. * A "mode" parameter has been added to smp_dmb: smp_dmb arm @ assumes 4-byte instructions (for ARM code, e.g. kuser) smp_dmb @ uses W() to ensure 4-byte instructions for ALT_SMP() This avoids assembly failures due to use of W() inside smp_dmb, when assembling pure-ARM code in the vectors page. There might be a better way to achieve this. * Kconfig: make SMP_ON_UP depend on (!THUMB2_KERNEL || !BIG_ENDIAN) i.e., THUMB2_KERNEL is now supported, but only if !BIG_ENDIAN (The fixup code for Thumb-2 currently assumes little-endian order.) Tested using a single generic realview kernel on: ARM RealView PB-A8 (CONFIG_THUMB2_KERNEL={n,y}) ARM RealView PBX-A9 (SMP) Signed-off-by: Dave Martin <dave.martin@linaro.org> Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-12-01 14:39:23 +00:00
smp_dmb arm
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
1: ldrex r3, [r2]
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
subs r3, r3, r0
strexeq r3, r1, [r2]
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
teqeq r3, #1
beq 1b
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
rsbs r0, r3, #0
[ARM] 4659/1: remove possibilities for spurious false negative with __kuser_cmpxchg The ARM __kuser_cmpxchg routine is meant to implement an atomic cmpxchg in user space. It however can produce spurious false negative if a processor exception occurs in the middle of the operation. Normally this is not a problem since cmpxchg is typically called in a loop until it succeeds to implement an atomic increment for example. Some use cases which don't involve a loop require that the operation be 100% reliable though. This patch changes the implementation so to reattempt the operation after an exception has occurred in the critical section rather than abort it. Here's a simple program to test the fix (don't use CONFIG_NO_HZ in your kernel as this depends on a sufficiently high interrupt rate): #include <stdio.h> typedef int (__kernel_cmpxchg_t)(int oldval, int newval, int *ptr); #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *)0xffff0fc0) int main() { int i, x = 0; for (i = 0; i < 100000000; i++) { int v = x; if (__kernel_cmpxchg(v, v+1, &x)) printf("failed at %d: %d vs %d\n", i, v, x); } printf("done with %d vs %d\n", i, x); return 0; } Signed-off-by: Nicolas Pitre <nico@marvell.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2007-11-20 16:20:29 +00:00
/* beware -- each __kuser slot must be 8 instructions max */
ARM: Allow SMP kernels to boot on UP systems UP systems do not implement all the instructions that SMP systems have, so in order to boot a SMP kernel on a UP system, we need to rewrite parts of the kernel. Do this using an 'alternatives' scheme, where the kernel code and data is modified prior to initialization to replace the SMP instructions, thereby rendering the problematical code ineffectual. We use the linker to generate a list of 32-bit word locations and their replacement values, and run through these replacements when we detect a UP system. Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-09-04 09:47:48 +00:00
ALT_SMP(b __kuser_memory_barrier)
ALT_UP(usr_ret lr)
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
#endif
ARM: poison memory between kuser helpers Poison the memory between each kuser helper. This ensures that any branch between the kuser helpers will be appropriately trapped. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 10:32:04 +00:00
kuser_pad __kuser_cmpxchg, 32
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
__kuser_get_tls: @ 0xffff0fe0
ARM: 6207/1: Replace CONFIG_HAS_TLS_REG with HWCAP_TLS and check for it on V6 The TLS register is only available on ARM1136 r1p0 and later. Set HWCAP_TLS flags if hardware TLS is available and test for it if CONFIG_CPU_32v6K is not set for V6. Note that we set the TLS instruction in __kuser_get_tls dynamically as suggested by Jamie Lokier <jamie@shareable.org>. Also the __switch_to code is optimized out in most cases as suggested by Nicolas Pitre <nico@fluxnic.net>. Reviewed-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Tony Lindgren <tony@atomide.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-07-05 13:53:10 +00:00
ldr r0, [pc, #(16 - 8)] @ read TLS, set in kuser_get_tls_init
[ARM] 3746/2: Userspace helpers must be Thumb mode interworkable Patch from Nicolas Pitre The userspace helpers in clean/arch/arm/kernel/entry-armv.S are called directly in/from userspace. They need to cope with being called from Thumb code. Patch below uses the bx interworking instruction when CONFIG_ARM_THUMB=y. Based on an earlier patch from Paul Brook <paul@codesourcery.com> Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2006-08-18 16:20:15 +00:00
usr_ret lr
ARM: 6207/1: Replace CONFIG_HAS_TLS_REG with HWCAP_TLS and check for it on V6 The TLS register is only available on ARM1136 r1p0 and later. Set HWCAP_TLS flags if hardware TLS is available and test for it if CONFIG_CPU_32v6K is not set for V6. Note that we set the TLS instruction in __kuser_get_tls dynamically as suggested by Jamie Lokier <jamie@shareable.org>. Also the __switch_to code is optimized out in most cases as suggested by Nicolas Pitre <nico@fluxnic.net>. Reviewed-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Tony Lindgren <tony@atomide.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-07-05 13:53:10 +00:00
mrc p15, 0, r0, c13, c0, 3 @ 0xffff0fe8 hardware TLS code
ARM: poison memory between kuser helpers Poison the memory between each kuser helper. This ensures that any branch between the kuser helpers will be appropriately trapped. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 10:32:04 +00:00
kuser_pad __kuser_get_tls, 16
.rep 3
ARM: 6207/1: Replace CONFIG_HAS_TLS_REG with HWCAP_TLS and check for it on V6 The TLS register is only available on ARM1136 r1p0 and later. Set HWCAP_TLS flags if hardware TLS is available and test for it if CONFIG_CPU_32v6K is not set for V6. Note that we set the TLS instruction in __kuser_get_tls dynamically as suggested by Jamie Lokier <jamie@shareable.org>. Also the __switch_to code is optimized out in most cases as suggested by Nicolas Pitre <nico@fluxnic.net>. Reviewed-by: Nicolas Pitre <nicolas.pitre@linaro.org> Signed-off-by: Tony Lindgren <tony@atomide.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2010-07-05 13:53:10 +00:00
.word 0 @ 0xffff0ff0 software TLS value, then
.endr @ pad up to __kuser_helper_version
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
__kuser_helper_version: @ 0xffff0ffc
.word ((__kuser_helper_end - __kuser_helper_start) >> 5)
.globl __kuser_helper_end
__kuser_helper_end:
ARM: allow kuser helpers to be removed from the vector page Provide a kernel configuration option to allow the kernel user helpers to be removed from the vector page, thereby preventing their use with ROP (return orientated programming) attacks. This option is only visible for CPU architectures which natively support all the operations which kernel user helpers would normally provide, and must be enabled with caution. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-23 17:37:00 +00:00
#endif
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
THUMB( .thumb )
[PATCH] ARM: 2651/3: kernel helpers for NPTL support Patch from Nicolas Pitre This patch entirely reworks the kernel assistance for NPTL on ARM. In particular this provides an efficient way to retrieve the TLS value and perform atomic operations without any instruction emulation nor special system call. This even allows for pre ARMv6 binaries to be forward compatible with SMP systems without any penalty. The problematic and performance critical operations are performed through segment of kernel provided user code reachable from user space at a fixed address in kernel memory. Those fixed entry points are within the vector page so we basically get it for free as no extra memory page is required and nothing else may be mapped at that location anyway. This is different from (but doesn't preclude) a full blown VDSO implementation, however a VDSO would prevent some assembly tricks with constants that allows for efficient branching to those code segments. And since those code segments only use a few cycles before returning to user code, the overhead of a VDSO far call would add a significant overhead to such minimalistic operations. The ARM_NR_set_tls syscall also changed number. This is done for two reasons: 1) this patch changes the way the TLS value was previously meant to be retrieved, therefore we ensure whatever library using the old way gets fixed (they only exist in private tree at the moment since the NPTL work is still progressing). 2) the previous number was allocated in a range causing an undefined instruction trap on kernels not supporting that syscall and it was determined that allocating it in a range returning -ENOSYS would be much nicer for libraries trying to determine if the feature is present or not. Signed-off-by: Nicolas Pitre Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-04-29 21:08:33 +00:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*
* Vector stubs.
*
ARM: move vector stubs Move the machine vector stubs into the page above the vector page, which we can prevent from being visible to userspace. Also move the reset stub, and place the swi vector at a location that the 'ldr' can get to it. This hides pointers into the kernel which could give valuable information to attackers, and reduces the number of exploitable instructions at a fixed address. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 10:40:32 +00:00
* This code is copied to 0xffff1000 so we can use branches in the
* vectors, rather than ldr's. Note that this code must not exceed
* a page size.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
*
* Common stub entry macro:
* Enter in IRQ mode, spsr = SVC/USR CPSR, lr = SVC/USR PC
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
*
* SP points to a minimal amount of processor-private memory, the address
* of which is copied into r0 for the mode specific abort handler.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
*/
[ARM] 3115/1: small optimizations to exception vector entry code Patch from Nicolas Pitre Since we know the value of cpsr on entry, we can replace the bic+orr with a single eor. Also remove a possible result delay (at least on XScale). Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-11-06 14:42:37 +00:00
.macro vector_stub, name, mode, correction=0
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.align 5
vector_\name:
.if \correction
sub lr, lr, #\correction
.endif
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
@
@ Save r0, lr_<exception> (parent PC) and spsr_<exception>
@ (parent CPSR)
@
stmia sp, {r0, lr} @ save r0, lr
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
mrs lr, spsr
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
str lr, [sp, #8] @ save spsr
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
@ Prepare for SVC32 mode. IRQs remain disabled.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
@
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
mrs r0, cpsr
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
eor r0, r0, #(\mode ^ SVC_MODE | PSR_ISETSTATE)
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
msr spsr_cxsf, r0
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
@
@ the branch table must immediately follow this code
@
and lr, lr, #0x0f
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
THUMB( adr r0, 1f )
THUMB( ldr lr, [r0, lr, lsl #2] )
[ARM] 3115/1: small optimizations to exception vector entry code Patch from Nicolas Pitre Since we know the value of cpsr on entry, we can replace the bic+orr with a single eor. Also remove a possible result delay (at least on XScale). Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-11-06 14:42:37 +00:00
mov r0, sp
Thumb-2: Implementation of the unified start-up and exceptions code This patch implements the ARM/Thumb-2 unified kernel start-up and exception handling code. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:54 +00:00
ARM( ldr lr, [pc, lr, lsl #2] )
[PATCH] ARM SMP: Fix vector entry The current vector entry system does not allow for SMP. In order to work around this, we need to eliminate our reliance on the fixed save areas, which breaks the way we enable alignment traps. This patch changes the way we handle the save areas such that we can have one per CPU. Signed-off-by: Russell King <rmk@arm.linux.org.uk>
2005-05-31 21:22:32 +00:00
movs pc, lr @ branch to handler in SVC mode
[ARM] 5227/1: Add the ENDPROC declarations to the .S files This declaration specifies the "function" type and size for various assembly functions, mainly needed for generating the correct branch instructions in Thumb-2. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2008-08-28 10:22:32 +00:00
ENDPROC(vector_\name)
Thumb-2: Add some .align statements to the .S files Since the Thumb-2 instructions can be 16-bit wide, data in the .text sections may not be aligned to a 32-bit word and this leads to unaligned exceptions. This patch does not affect the ARM code generation. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2009-07-24 11:32:52 +00:00
.align 2
@ handler addresses follow this label
1:
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.endm
ARM: use linker magic for vectors and vector stubs Use linker magic to create the vectors and vector stubs: we can tell the linker to place them at an appropriate VMA, but keep the LMA within the kernel. This gets rid of some unnecessary symbol manipulation, and have the linker calculate the relocations appropriately. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 11:03:31 +00:00
.section .stubs, "ax", %progbits
ARM: move vector stubs Move the machine vector stubs into the page above the vector page, which we can prevent from being visible to userspace. Also move the reset stub, and place the swi vector at a location that the 'ldr' can get to it. This hides pointers into the kernel which could give valuable information to attackers, and reduces the number of exploitable instructions at a fixed address. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 10:40:32 +00:00
@ This must be the first word
.word vector_swi
vector_rst:
ARM( swi SYS_ERROR0 )
THUMB( svc #0 )
THUMB( nop )
b vector_und
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*
* Interrupt dispatcher
*/
[ARM] 3115/1: small optimizations to exception vector entry code Patch from Nicolas Pitre Since we know the value of cpsr on entry, we can replace the bic+orr with a single eor. Also remove a possible result delay (at least on XScale). Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-11-06 14:42:37 +00:00
vector_stub irq, IRQ_MODE, 4
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.long __irq_usr @ 0 (USR_26 / USR_32)
.long __irq_invalid @ 1 (FIQ_26 / FIQ_32)
.long __irq_invalid @ 2 (IRQ_26 / IRQ_32)
.long __irq_svc @ 3 (SVC_26 / SVC_32)
.long __irq_invalid @ 4
.long __irq_invalid @ 5
.long __irq_invalid @ 6
.long __irq_invalid @ 7
.long __irq_invalid @ 8
.long __irq_invalid @ 9
.long __irq_invalid @ a
.long __irq_invalid @ b
.long __irq_invalid @ c
.long __irq_invalid @ d
.long __irq_invalid @ e
.long __irq_invalid @ f
/*
* Data abort dispatcher
* Enter in ABT mode, spsr = USR CPSR, lr = USR PC
*/
[ARM] 3115/1: small optimizations to exception vector entry code Patch from Nicolas Pitre Since we know the value of cpsr on entry, we can replace the bic+orr with a single eor. Also remove a possible result delay (at least on XScale). Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-11-06 14:42:37 +00:00
vector_stub dabt, ABT_MODE, 8
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.long __dabt_usr @ 0 (USR_26 / USR_32)
.long __dabt_invalid @ 1 (FIQ_26 / FIQ_32)
.long __dabt_invalid @ 2 (IRQ_26 / IRQ_32)
.long __dabt_svc @ 3 (SVC_26 / SVC_32)
.long __dabt_invalid @ 4
.long __dabt_invalid @ 5
.long __dabt_invalid @ 6
.long __dabt_invalid @ 7
.long __dabt_invalid @ 8
.long __dabt_invalid @ 9
.long __dabt_invalid @ a
.long __dabt_invalid @ b
.long __dabt_invalid @ c
.long __dabt_invalid @ d
.long __dabt_invalid @ e
.long __dabt_invalid @ f
/*
* Prefetch abort dispatcher
* Enter in ABT mode, spsr = USR CPSR, lr = USR PC
*/
[ARM] 3115/1: small optimizations to exception vector entry code Patch from Nicolas Pitre Since we know the value of cpsr on entry, we can replace the bic+orr with a single eor. Also remove a possible result delay (at least on XScale). Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-11-06 14:42:37 +00:00
vector_stub pabt, ABT_MODE, 4
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.long __pabt_usr @ 0 (USR_26 / USR_32)
.long __pabt_invalid @ 1 (FIQ_26 / FIQ_32)
.long __pabt_invalid @ 2 (IRQ_26 / IRQ_32)
.long __pabt_svc @ 3 (SVC_26 / SVC_32)
.long __pabt_invalid @ 4
.long __pabt_invalid @ 5
.long __pabt_invalid @ 6
.long __pabt_invalid @ 7
.long __pabt_invalid @ 8
.long __pabt_invalid @ 9
.long __pabt_invalid @ a
.long __pabt_invalid @ b
.long __pabt_invalid @ c
.long __pabt_invalid @ d
.long __pabt_invalid @ e
.long __pabt_invalid @ f
/*
* Undef instr entry dispatcher
* Enter in UND mode, spsr = SVC/USR CPSR, lr = SVC/USR PC
*/
[ARM] 3115/1: small optimizations to exception vector entry code Patch from Nicolas Pitre Since we know the value of cpsr on entry, we can replace the bic+orr with a single eor. Also remove a possible result delay (at least on XScale). Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2005-11-06 14:42:37 +00:00
vector_stub und, UND_MODE
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.long __und_usr @ 0 (USR_26 / USR_32)
.long __und_invalid @ 1 (FIQ_26 / FIQ_32)
.long __und_invalid @ 2 (IRQ_26 / IRQ_32)
.long __und_svc @ 3 (SVC_26 / SVC_32)
.long __und_invalid @ 4
.long __und_invalid @ 5
.long __und_invalid @ 6
.long __und_invalid @ 7
.long __und_invalid @ 8
.long __und_invalid @ 9
.long __und_invalid @ a
.long __und_invalid @ b
.long __und_invalid @ c
.long __und_invalid @ d
.long __und_invalid @ e
.long __und_invalid @ f
.align 5
ARM: move vector stubs Move the machine vector stubs into the page above the vector page, which we can prevent from being visible to userspace. Also move the reset stub, and place the swi vector at a location that the 'ldr' can get to it. This hides pointers into the kernel which could give valuable information to attackers, and reduces the number of exploitable instructions at a fixed address. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 10:40:32 +00:00
/*=============================================================================
* Address exception handler
*-----------------------------------------------------------------------------
* These aren't too critical.
* (they're not supposed to happen, and won't happen in 32-bit data mode).
*/
vector_addrexcptn:
b vector_addrexcptn
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
/*=============================================================================
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
* FIQ "NMI" handler
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
*-----------------------------------------------------------------------------
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
* Handle a FIQ using the SVC stack allowing FIQ act like NMI on x86
* systems.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
*/
ARM: 8150/3: fiq: Replace default FIQ handler This patch introduces a new default FIQ handler that is structured in a similar way to the existing ARM exception handler and result in the FIQ being handled by C code running on the SVC stack (despite this code run in the FIQ handler is subject to severe limitations with respect to locking making normal interaction with the kernel impossible). This default handler allows concepts that on x86 would be handled using NMIs to be realized on ARM. Credit: This patch is a near complete re-write of a patch originally provided by Anton Vorontsov. Today only a couple of small fragments survive, however without Anton's work to build from this patch would not exist. Thanks also to Russell King for spoonfeeding me a variety of fixes during the review cycle. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2014-09-17 16:12:06 +00:00
vector_stub fiq, FIQ_MODE, 4
.long __fiq_usr @ 0 (USR_26 / USR_32)
.long __fiq_svc @ 1 (FIQ_26 / FIQ_32)
.long __fiq_svc @ 2 (IRQ_26 / IRQ_32)
.long __fiq_svc @ 3 (SVC_26 / SVC_32)
.long __fiq_svc @ 4
.long __fiq_svc @ 5
.long __fiq_svc @ 6
.long __fiq_abt @ 7
.long __fiq_svc @ 8
.long __fiq_svc @ 9
.long __fiq_svc @ a
.long __fiq_svc @ b
.long __fiq_svc @ c
.long __fiq_svc @ d
.long __fiq_svc @ e
.long __fiq_svc @ f
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
ARM: 8515/2: move .vectors and .stubs sections back into the kernel VMA Commit b9b32bf70f2f ("ARM: use linker magic for vectors and vector stubs") updated the linker script to emit the .vectors and .stubs sections into a VMA range that is zero based and disjoint from the normal static kernel region. The reason for that was that this way, the sections can be placed exactly 4 KB apart, while the payload of the .vectors section is only 32 bytes. Since the symbols that are part of the .stubs section are emitted into the kallsyms table, they appear with zero based addresses as well, e.g., 00001004 t vector_rst 00001020 t vector_irq 000010a0 t vector_dabt 00001120 t vector_pabt 000011a0 t vector_und 00001220 t vector_addrexcptn 00001240 t vector_fiq 00001240 T vector_fiq_offset As this confuses perf when it accesses the kallsyms tables, commit 7122c3e9154b ("scripts/link-vmlinux.sh: only filter kernel symbols for arm") implemented a somewhat ugly special case for ARM, where the value of CONFIG_PAGE_OFFSET is passed to scripts/kallsyms, and symbols whose addresses are below it are filtered out. Note that this special case only applies to CONFIG_XIP_KERNEL=n, not because the issue the patch addresses exists only in that case, but because finding a limit below which to apply the filtering is not entirely straightforward. Since the .vectors and .stubs sections contain position independent code that is never executed in place, we can emit it at its most likely runtime VMA (for more recent CPUs), which is 0xffff0000 for the vector table and 0xffff1000 for the stubs. Not only does this fix the perf issue with kallsyms, allowing us to drop the special case in scripts/kallsyms entirely, it also gives debuggers a more realistic view of the address space, and setting breakpoints or single stepping through code in the vector table or the stubs is more likely to work as expected on CPUs that use a high vector address. E.g., 00001240 A vector_fiq_offset ... c0c35000 T __init_begin c0c35000 T __vectors_start c0c35020 T __stubs_start c0c35020 T __vectors_end c0c352e0 T _sinittext c0c352e0 T __stubs_end ... ffff1004 t vector_rst ffff1020 t vector_irq ffff10a0 t vector_dabt ffff1120 t vector_pabt ffff11a0 t vector_und ffff1220 t vector_addrexcptn ffff1240 T vector_fiq (Note that vector_fiq_offset is now an absolute symbol, which kallsyms already ignores by default) The LMA footprint is identical with or without this change, only the VMAs are different: Before: Idx Name Size VMA LMA File off Algn ... 14 .notes 00000024 c0c34020 c0c34020 00a34020 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE 15 .vectors 00000020 00000000 c0c35000 00a40000 2**1 CONTENTS, ALLOC, LOAD, READONLY, CODE 16 .stubs 000002c0 00001000 c0c35020 00a41000 2**5 CONTENTS, ALLOC, LOAD, READONLY, CODE 17 .init.text 0006b1b8 c0c352e0 c0c352e0 00a452e0 2**5 CONTENTS, ALLOC, LOAD, READONLY, CODE ... After: Idx Name Size VMA LMA File off Algn ... 14 .notes 00000024 c0c34020 c0c34020 00a34020 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE 15 .vectors 00000020 ffff0000 c0c35000 00a40000 2**1 CONTENTS, ALLOC, LOAD, READONLY, CODE 16 .stubs 000002c0 ffff1000 c0c35020 00a41000 2**5 CONTENTS, ALLOC, LOAD, READONLY, CODE 17 .init.text 0006b1b8 c0c352e0 c0c352e0 00a452e0 2**5 CONTENTS, ALLOC, LOAD, READONLY, CODE ... Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Chris Brandt <chris.brandt@renesas.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2016-02-10 10:41:08 +00:00
.globl vector_fiq
ARM: update FIQ support for relocation of vectors FIQ should no longer copy the FIQ code into the user visible vector page. Instead, it should use the hidden page. This change makes that happen. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-09 00:03:17 +00:00
ARM: use linker magic for vectors and vector stubs Use linker magic to create the vectors and vector stubs: we can tell the linker to place them at an appropriate VMA, but keep the LMA within the kernel. This gets rid of some unnecessary symbol manipulation, and have the linker calculate the relocations appropriately. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 11:03:31 +00:00
.section .vectors, "ax", %progbits
ARM: 8514/1: remove duplicate definitions of __vectors_start and __stubs_start Commit b9b32bf70f2f ("ARM: use linker magic for vectors and vector stubs") introduced new global definitions of __vectors_start and __stubs_start, and changed the existing ones to have internal linkage only. However, these symbols are still visible to kallsyms, and due to the way the .vectors and .stubs sections are emitted at the base of the VMA space, these duplicate definitions have conflicting values. $ nm -n vmlinux |grep -E __vectors|__stubs 00000000 t __vectors_start 00001000 t __stubs_start c0e77000 T __vectors_start c0e77020 T __stubs_start This is completely harmless by itself, since the wrong values are local symbols that cannot be referenced by other object files directly. However, since these symbols are also listed in the kallsyms symbol table in some cases (i.e., CONFIG_KALLSYMS_ALL=y and CONFIG_XIP_KERNEL=y), having these conflicting values can be confusing. So either remove them, or make them strictly local. Acked-by: Chris Brandt <chris.brandt@renesas.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2016-02-05 09:04:47 +00:00
.L__vectors_start:
ARM: use linker magic for vectors and vector stubs Use linker magic to create the vectors and vector stubs: we can tell the linker to place them at an appropriate VMA, but keep the LMA within the kernel. This gets rid of some unnecessary symbol manipulation, and have the linker calculate the relocations appropriately. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 11:03:31 +00:00
W(b) vector_rst
W(b) vector_und
ARM: 8514/1: remove duplicate definitions of __vectors_start and __stubs_start Commit b9b32bf70f2f ("ARM: use linker magic for vectors and vector stubs") introduced new global definitions of __vectors_start and __stubs_start, and changed the existing ones to have internal linkage only. However, these symbols are still visible to kallsyms, and due to the way the .vectors and .stubs sections are emitted at the base of the VMA space, these duplicate definitions have conflicting values. $ nm -n vmlinux |grep -E __vectors|__stubs 00000000 t __vectors_start 00001000 t __stubs_start c0e77000 T __vectors_start c0e77020 T __stubs_start This is completely harmless by itself, since the wrong values are local symbols that cannot be referenced by other object files directly. However, since these symbols are also listed in the kallsyms symbol table in some cases (i.e., CONFIG_KALLSYMS_ALL=y and CONFIG_XIP_KERNEL=y), having these conflicting values can be confusing. So either remove them, or make them strictly local. Acked-by: Chris Brandt <chris.brandt@renesas.com> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2016-02-05 09:04:47 +00:00
W(ldr) pc, .L__vectors_start + 0x1000
ARM: use linker magic for vectors and vector stubs Use linker magic to create the vectors and vector stubs: we can tell the linker to place them at an appropriate VMA, but keep the LMA within the kernel. This gets rid of some unnecessary symbol manipulation, and have the linker calculate the relocations appropriately. Cc: <stable@vger.kernel.org> Acked-by: Nicolas Pitre <nico@linaro.org> Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2013-07-04 11:03:31 +00:00
W(b) vector_pabt
W(b) vector_dabt
W(b) vector_addrexcptn
W(b) vector_irq
W(b) vector_fiq
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.data
ARM: align .data section Robert Jarzmik reports that his PXA25x system fails to boot with 4.12, failing at __flush_whole_cache in arch/arm/mm/proc-xscale.S:215: 0xc0019e20 <+0>: ldr r1, [pc, #788] 0xc0019e24 <+4>: ldr r0, [r1] <== here with r1 containing 0xc06f82cd, which is the address of "clean_addr". Examination of the System.map shows: c06f22c8 D user_pmd_table c06f22cc d __warned.19178 c06f22cd d clean_addr indicating that a .data.unlikely section has appeared just before the .data section from proc-xscale.S. According to objdump -h, it appears that our assembly files default their .data alignment to 2**0, which is bad news if the preceding .data section size is not power-of-2 aligned at link time. Add the appropriate .align directives to all assembly files in arch/arm that are missing them where we require an appropriate alignment. Reported-by: Robert Jarzmik <robert.jarzmik@free.fr> Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
2017-07-26 11:49:31 +00:00
.align 2
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-16 22:20:36 +00:00
.globl cr_alignment
cr_alignment:
.space 4
Reference in a new issue Copy permalink