2019-08-13 23:04:50 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0-only */
|
|
|
|
|
/*
|
|
|
|
|
* Linker script variables to be set after section resolution, as
|
|
|
|
|
* ld.lld does not like variables assigned before SECTIONS is processed.
|
|
|
|
|
*/
|
|
|
|
|
#ifndef __ARM64_KERNEL_IMAGE_VARS_H
|
|
|
|
|
#define __ARM64_KERNEL_IMAGE_VARS_H
|
|
|
|
|
|
|
|
|
|
#ifndef LINKER_SCRIPT
|
|
|
|
|
#error This file should only be included in vmlinux.lds.S
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_EFI
|
|
|
|
|
|
2020-02-17 11:44:37 +00:00
|
|
|
__efistub_kernel_size = _edata - _text;
|
2020-03-26 17:14:23 +00:00
|
|
|
__efistub_primary_entry_offset = primary_entry - _text;
|
2020-02-17 11:44:37 +00:00
|
|
|
|
2019-08-13 23:04:50 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The EFI stub has its own symbol namespace prefixed by __efistub_, to
|
|
|
|
|
* isolate it from the kernel proper. The following symbols are legally
|
|
|
|
|
* accessed by the stub, so provide some aliases to make them accessible.
|
|
|
|
|
* Only include data symbols here, or text symbols of functions that are
|
|
|
|
|
* guaranteed to be safe when executed at another offset than they were
|
|
|
|
|
* linked at. The routines below are all implemented in assembler in a
|
|
|
|
|
* position independent manner
|
|
|
|
|
*/
|
|
|
|
|
__efistub_memcmp = __pi_memcmp;
|
|
|
|
|
__efistub_memchr = __pi_memchr;
|
|
|
|
|
__efistub_memcpy = __pi_memcpy;
|
|
|
|
|
__efistub_memmove = __pi_memmove;
|
|
|
|
|
__efistub_memset = __pi_memset;
|
|
|
|
|
__efistub_strlen = __pi_strlen;
|
|
|
|
|
__efistub_strnlen = __pi_strnlen;
|
|
|
|
|
__efistub_strcmp = __pi_strcmp;
|
|
|
|
|
__efistub_strncmp = __pi_strncmp;
|
|
|
|
|
__efistub_strrchr = __pi_strrchr;
|
arm64: Rename arm64-internal cache maintenance functions
Although naming across the codebase isn't that consistent, it
tends to follow certain patterns. Moreover, the term "flush"
isn't defined in the Arm Architecture reference manual, and might
be interpreted to mean clean, invalidate, or both for a cache.
Rename arm64-internal functions to make the naming internally
consistent, as well as making it consistent with the Arm ARM, by
specifying whether it applies to the instruction, data, or both
caches, whether the operation is a clean, invalidate, or both.
Also specify which point the operation applies to, i.e., to the
point of unification (PoU), coherency (PoC), or persistence
(PoP).
This commit applies the following sed transformation to all files
under arch/arm64:
"s/\b__flush_cache_range\b/caches_clean_inval_pou_macro/g;"\
"s/\b__flush_icache_range\b/caches_clean_inval_pou/g;"\
"s/\binvalidate_icache_range\b/icache_inval_pou/g;"\
"s/\b__flush_dcache_area\b/dcache_clean_inval_poc/g;"\
"s/\b__inval_dcache_area\b/dcache_inval_poc/g;"\
"s/__clean_dcache_area_poc\b/dcache_clean_poc/g;"\
"s/\b__clean_dcache_area_pop\b/dcache_clean_pop/g;"\
"s/\b__clean_dcache_area_pou\b/dcache_clean_pou/g;"\
"s/\b__flush_cache_user_range\b/caches_clean_inval_user_pou/g;"\
"s/\b__flush_icache_all\b/icache_inval_all_pou/g;"
Note that __clean_dcache_area_poc is deliberately missing a word
boundary check at the beginning in order to match the efistub
symbols in image-vars.h.
Also note that, despite its name, __flush_icache_range operates
on both instruction and data caches. The name change here
reflects that.
No functional change intended.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20210524083001.2586635-19-tabba@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2021-05-24 08:30:01 +00:00
|
|
|
__efistub_dcache_clean_poc = __pi_dcache_clean_poc;
|
2019-08-13 23:04:50 +00:00
|
|
|
|
2020-12-22 20:02:06 +00:00
|
|
|
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
|
2019-08-13 23:04:50 +00:00
|
|
|
__efistub___memcpy = __pi_memcpy;
|
|
|
|
|
__efistub___memmove = __pi_memmove;
|
|
|
|
|
__efistub___memset = __pi_memset;
|
arm64: head: avoid relocating the kernel twice for KASLR
Currently, when KASLR is in effect, we set up the kernel virtual address
space twice: the first time, the KASLR seed is looked up in the device
tree, and the kernel virtual mapping is torn down and recreated again,
after which the relocations are applied a second time. The latter step
means that statically initialized global pointer variables will be reset
to their initial values, and to ensure that BSS variables are not set to
values based on the initial translation, they are cleared again as well.
All of this is needed because we need the command line (taken from the
DT) to tell us whether or not to randomize the virtual address space
before entering the kernel proper. However, this code has expanded
little by little and now creates global state unrelated to the virtual
randomization of the kernel before the mapping is torn down and set up
again, and the BSS cleared for a second time. This has created some
issues in the past, and it would be better to avoid this little dance if
possible.
So instead, let's use the temporary mapping of the device tree, and
execute the bare minimum of code to decide whether or not KASLR should
be enabled, and what the seed is. Only then, create the virtual kernel
mapping, clear BSS, etc and proceed as normal. This avoids the issues
around inconsistent global state due to BSS being cleared twice, and is
generally more maintainable, as it permits us to defer all the remaining
DT parsing and KASLR initialization to a later time.
This means the relocation fixup code runs only a single time as well,
allowing us to simplify the RELR handling code too, which is not
idempotent and was therefore required to keep track of the offset that
was applied the first time around.
Note that this means we have to clone a pair of FDT library objects, so
that we can control how they are built - we need the stack protector
and other instrumentation disabled so that the code can tolerate being
called this early. Note that only the kernel page tables and the
temporary stack are mapped read-write at this point, which ensures that
the early code does not modify any global state inadvertently.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20220624150651.1358849-21-ardb@kernel.org
Signed-off-by: Will Deacon <will@kernel.org>
2022-06-24 15:06:50 +00:00
|
|
|
|
|
|
|
|
__pi___memcpy = __pi_memcpy;
|
|
|
|
|
__pi___memmove = __pi_memmove;
|
|
|
|
|
__pi___memset = __pi_memset;
|
2019-08-13 23:04:50 +00:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
__efistub__text = _text;
|
|
|
|
|
__efistub__end = _end;
|
|
|
|
|
__efistub__edata = _edata;
|
|
|
|
|
__efistub_screen_info = screen_info;
|
efi/libstub: Clean up command line parsing routine
We currently parse the command non-destructively, to avoid having to
allocate memory for a copy before passing it to the standard parsing
routines that are used by the core kernel, and which modify the input
to delineate the parsed tokens with NUL characters.
Instead, we call strstr() and strncmp() to go over the input multiple
times, and match prefixes rather than tokens, which implies that we
would match, e.g., 'nokaslrfoo' in the stub and disable KASLR, while
the kernel would disregard the option and run with KASLR enabled.
In order to avoid having to reason about whether and how this behavior
may be abused, let's clean up the parsing routines, and rebuild them
on top of the existing helpers.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
2020-02-10 16:02:46 +00:00
|
|
|
__efistub__ctype = _ctype;
|
2019-08-13 23:04:50 +00:00
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
2020-06-25 13:14:08 +00:00
|
|
|
#ifdef CONFIG_KVM
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, to
|
|
|
|
|
* separate it from the kernel proper. The following symbols are legally
|
|
|
|
|
* accessed by it, therefore provide aliases to make them linkable.
|
|
|
|
|
* Do not include symbols which may not be safely accessed under hypervisor
|
|
|
|
|
* memory mappings.
|
|
|
|
|
*/
|
|
|
|
|
|
2020-06-25 13:14:11 +00:00
|
|
|
/* Alternative callbacks for init-time patching of nVHE hyp code. */
|
|
|
|
|
KVM_NVHE_ALIAS(kvm_patch_vector_branch);
|
|
|
|
|
KVM_NVHE_ALIAS(kvm_update_va_mask);
|
2020-10-24 15:33:38 +00:00
|
|
|
KVM_NVHE_ALIAS(kvm_get_kimage_voffset);
|
2021-03-22 12:09:51 +00:00
|
|
|
KVM_NVHE_ALIAS(kvm_compute_final_ctr_el0);
|
arm64: Mitigate spectre style branch history side channels
Speculation attacks against some high-performance processors can
make use of branch history to influence future speculation.
When taking an exception from user-space, a sequence of branches
or a firmware call overwrites or invalidates the branch history.
The sequence of branches is added to the vectors, and should appear
before the first indirect branch. For systems using KPTI the sequence
is added to the kpti trampoline where it has a free register as the exit
from the trampoline is via a 'ret'. For systems not using KPTI, the same
register tricks are used to free up a register in the vectors.
For the firmware call, arch-workaround-3 clobbers 4 registers, so
there is no choice but to save them to the EL1 stack. This only happens
for entry from EL0, so if we take an exception due to the stack access,
it will not become re-entrant.
For KVM, the existing branch-predictor-hardening vectors are used.
When a spectre version of these vectors is in use, the firmware call
is sufficient to mitigate against Spectre-BHB. For the non-spectre
versions, the sequence of branches is added to the indirect vector.
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: James Morse <james.morse@arm.com>
2021-11-10 14:48:00 +00:00
|
|
|
KVM_NVHE_ALIAS(spectre_bhb_patch_loop_iter);
|
|
|
|
|
KVM_NVHE_ALIAS(spectre_bhb_patch_loop_mitigation_enable);
|
|
|
|
|
KVM_NVHE_ALIAS(spectre_bhb_patch_wa3);
|
2021-12-10 14:32:56 +00:00
|
|
|
KVM_NVHE_ALIAS(spectre_bhb_patch_clearbhb);
|
2020-06-25 13:14:11 +00:00
|
|
|
|
|
|
|
|
/* Global kernel state accessed by nVHE hyp code. */
|
2020-06-25 13:14:14 +00:00
|
|
|
KVM_NVHE_ALIAS(kvm_vgic_global_state);
|
2020-06-25 13:14:11 +00:00
|
|
|
|
|
|
|
|
/* Kernel symbols used to call panic() from nVHE hyp code (via ERET). */
|
2021-03-18 14:33:11 +00:00
|
|
|
KVM_NVHE_ALIAS(nvhe_hyp_panic_handler);
|
2020-06-25 13:14:11 +00:00
|
|
|
|
2020-06-25 13:14:12 +00:00
|
|
|
/* Vectors installed by hyp-init on reset HVC. */
|
|
|
|
|
KVM_NVHE_ALIAS(__hyp_stub_vectors);
|
|
|
|
|
|
2020-06-25 13:14:13 +00:00
|
|
|
/* Kernel symbol used by icache_is_vpipt(). */
|
|
|
|
|
KVM_NVHE_ALIAS(__icache_flags);
|
|
|
|
|
|
2021-11-22 12:18:42 +00:00
|
|
|
/* VMID bits set by the KVM VMID allocator */
|
|
|
|
|
KVM_NVHE_ALIAS(kvm_arm_vmid_bits);
|
|
|
|
|
|
2020-06-25 13:14:13 +00:00
|
|
|
/* Kernel symbols needed for cpus_have_final/const_caps checks. */
|
|
|
|
|
KVM_NVHE_ALIAS(arm64_const_caps_ready);
|
|
|
|
|
KVM_NVHE_ALIAS(cpu_hwcap_keys);
|
|
|
|
|
|
2020-06-25 13:14:14 +00:00
|
|
|
/* Static keys which are set if a vGIC trap should be handled in hyp. */
|
|
|
|
|
KVM_NVHE_ALIAS(vgic_v2_cpuif_trap);
|
|
|
|
|
KVM_NVHE_ALIAS(vgic_v3_cpuif_trap);
|
|
|
|
|
|
|
|
|
|
/* Static key checked in pmr_sync(). */
|
|
|
|
|
#ifdef CONFIG_ARM64_PSEUDO_NMI
|
|
|
|
|
KVM_NVHE_ALIAS(gic_pmr_sync);
|
2020-09-12 15:37:07 +00:00
|
|
|
/* Static key checked in GIC_PRIO_IRQOFF. */
|
|
|
|
|
KVM_NVHE_ALIAS(gic_nonsecure_priorities);
|
2020-06-25 13:14:14 +00:00
|
|
|
#endif
|
|
|
|
|
|
2020-08-21 14:07:05 +00:00
|
|
|
/* EL2 exception handling */
|
|
|
|
|
KVM_NVHE_ALIAS(__start___kvm_ex_table);
|
|
|
|
|
KVM_NVHE_ALIAS(__stop___kvm_ex_table);
|
|
|
|
|
|
2020-12-02 18:41:09 +00:00
|
|
|
/* Array containing bases of nVHE per-CPU memory regions. */
|
|
|
|
|
KVM_NVHE_ALIAS(kvm_arm_hyp_percpu_base);
|
|
|
|
|
|
2021-03-05 18:52:51 +00:00
|
|
|
/* PMU available static key */
|
2021-11-11 02:07:36 +00:00
|
|
|
#ifdef CONFIG_HW_PERF_EVENTS
|
2021-03-05 18:52:51 +00:00
|
|
|
KVM_NVHE_ALIAS(kvm_arm_pmu_available);
|
2021-11-11 02:07:36 +00:00
|
|
|
#endif
|
2021-03-05 18:52:51 +00:00
|
|
|
|
2021-03-19 10:01:10 +00:00
|
|
|
/* Position-independent library routines */
|
|
|
|
|
KVM_NVHE_ALIAS_HYP(clear_page, __pi_clear_page);
|
|
|
|
|
KVM_NVHE_ALIAS_HYP(copy_page, __pi_copy_page);
|
|
|
|
|
KVM_NVHE_ALIAS_HYP(memcpy, __pi_memcpy);
|
|
|
|
|
KVM_NVHE_ALIAS_HYP(memset, __pi_memset);
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_KASAN
|
|
|
|
|
KVM_NVHE_ALIAS_HYP(__memcpy, __pi_memcpy);
|
|
|
|
|
KVM_NVHE_ALIAS_HYP(__memset, __pi_memset);
|
|
|
|
|
#endif
|
|
|
|
|
|
KVM: arm64: Prepare the creation of s1 mappings at EL2
When memory protection is enabled, the EL2 code needs the ability to
create and manage its own page-table. To do so, introduce a new set of
hypercalls to bootstrap a memory management system at EL2.
This leads to the following boot flow in nVHE Protected mode:
1. the host allocates memory for the hypervisor very early on, using
the memblock API;
2. the host creates a set of stage 1 page-table for EL2, installs the
EL2 vectors, and issues the __pkvm_init hypercall;
3. during __pkvm_init, the hypervisor re-creates its stage 1 page-table
and stores it in the memory pool provided by the host;
4. the hypervisor then extends its stage 1 mappings to include a
vmemmap in the EL2 VA space, hence allowing to use the buddy
allocator introduced in a previous patch;
5. the hypervisor jumps back in the idmap page, switches from the
host-provided page-table to the new one, and wraps up its
initialization by enabling the new allocator, before returning to
the host.
6. the host can free the now unused page-table created for EL2, and
will now need to issue hypercalls to make changes to the EL2 stage 1
mappings instead of modifying them directly.
Note that for the sake of simplifying the review, this patch focuses on
the hypervisor side of things. In other words, this only implements the
new hypercalls, but does not make use of them from the host yet. The
host-side changes will follow in a subsequent patch.
Credits to Will for __pkvm_init_switch_pgd.
Acked-by: Will Deacon <will@kernel.org>
Co-authored-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210319100146.1149909-18-qperret@google.com
2021-03-19 10:01:25 +00:00
|
|
|
/* Kernel memory sections */
|
|
|
|
|
KVM_NVHE_ALIAS(__start_rodata);
|
|
|
|
|
KVM_NVHE_ALIAS(__end_rodata);
|
|
|
|
|
KVM_NVHE_ALIAS(__bss_start);
|
|
|
|
|
KVM_NVHE_ALIAS(__bss_stop);
|
|
|
|
|
|
|
|
|
|
/* Hyp memory sections */
|
|
|
|
|
KVM_NVHE_ALIAS(__hyp_idmap_text_start);
|
|
|
|
|
KVM_NVHE_ALIAS(__hyp_idmap_text_end);
|
|
|
|
|
KVM_NVHE_ALIAS(__hyp_text_start);
|
|
|
|
|
KVM_NVHE_ALIAS(__hyp_text_end);
|
|
|
|
|
KVM_NVHE_ALIAS(__hyp_bss_start);
|
|
|
|
|
KVM_NVHE_ALIAS(__hyp_bss_end);
|
|
|
|
|
KVM_NVHE_ALIAS(__hyp_rodata_start);
|
|
|
|
|
KVM_NVHE_ALIAS(__hyp_rodata_end);
|
|
|
|
|
|
2021-03-19 10:01:43 +00:00
|
|
|
/* pKVM static key */
|
|
|
|
|
KVM_NVHE_ALIAS(kvm_protected_mode_initialized);
|
|
|
|
|
|
2020-06-25 13:14:08 +00:00
|
|
|
#endif /* CONFIG_KVM */
|
|
|
|
|
|
2019-08-13 23:04:50 +00:00
|
|
|
#endif /* __ARM64_KERNEL_IMAGE_VARS_H */
|