License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
2009-04-29 07:47:18 +00:00
|
|
|
/*
|
|
|
|
|
* ld script for the x86 kernel
|
|
|
|
|
*
|
|
|
|
|
* Historic 32-bit version written by Martin Mares <mj@atrey.karlin.mff.cuni.cz>
|
|
|
|
|
*
|
2009-04-29 08:58:38 +00:00
|
|
|
* Modernisation, unification and other changes and fixes:
|
|
|
|
|
* Copyright (C) 2007-2009 Sam Ravnborg <sam@ravnborg.org>
|
2009-04-29 07:47:18 +00:00
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* Don't define absolute symbols until and unless you know that symbol
|
|
|
|
|
* value is should remain constant even if kernel image is relocated
|
|
|
|
|
* at run time. Absolute symbols are not relocated. If symbol value should
|
|
|
|
|
* change if kernel is relocated, make the symbol section relative and
|
|
|
|
|
* put it inside the section definition.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#define LOAD_OFFSET __START_KERNEL_map
|
|
|
|
|
|
2020-03-26 19:30:20 +00:00
|
|
|
#define RUNTIME_DISCARD_EXIT
|
2019-10-29 21:13:30 +00:00
|
|
|
#define EMITS_PT_NOTE
|
2019-10-29 21:13:38 +00:00
|
|
|
#define RO_EXCEPTION_TABLE_ALIGN 16
|
2019-10-29 21:13:30 +00:00
|
|
|
|
2009-04-29 07:47:18 +00:00
|
|
|
#include <asm-generic/vmlinux.lds.h>
|
|
|
|
|
#include <asm/asm-offsets.h>
|
|
|
|
|
#include <asm/thread_info.h>
|
|
|
|
|
#include <asm/page_types.h>
|
2017-07-24 23:36:57 +00:00
|
|
|
#include <asm/orc_lookup.h>
|
2009-04-29 07:47:18 +00:00
|
|
|
#include <asm/cache.h>
|
|
|
|
|
#include <asm/boot.h>
|
|
|
|
|
|
|
|
|
|
#undef i386 /* in case the preprocessor is a 32bit one */
|
|
|
|
|
|
2019-01-09 16:32:10 +00:00
|
|
|
OUTPUT_FORMAT(CONFIG_OUTPUT_FORMAT)
|
2009-04-29 07:47:18 +00:00
|
|
|
|
|
|
|
|
#ifdef CONFIG_X86_32
|
|
|
|
|
OUTPUT_ARCH(i386)
|
|
|
|
|
ENTRY(phys_startup_32)
|
|
|
|
|
#else
|
|
|
|
|
OUTPUT_ARCH(i386:x86-64)
|
|
|
|
|
ENTRY(phys_startup_64)
|
|
|
|
|
#endif
|
|
|
|
|
|
x86_64: Fix jiffies ODR violation
'jiffies' and 'jiffies_64' are meant to alias (two different symbols that
share the same address). Most architectures make the symbols alias to the
same address via a linker script assignment in their
arch/<arch>/kernel/vmlinux.lds.S:
jiffies = jiffies_64;
which is effectively a definition of jiffies.
jiffies and jiffies_64 are both forward declared for all architectures in
include/linux/jiffies.h. jiffies_64 is defined in kernel/time/timer.c.
x86_64 was peculiar in that it wasn't doing the above linker script
assignment, but rather was:
1. defining jiffies in arch/x86/kernel/time.c instead via the linker script.
2. overriding the symbol jiffies_64 from kernel/time/timer.c in
arch/x86/kernel/vmlinux.lds.s via 'jiffies_64 = jiffies;'.
As Fangrui notes:
In LLD, symbol assignments in linker scripts override definitions in
object files. GNU ld appears to have the same behavior. It would
probably make sense for LLD to error "duplicate symbol" but GNU ld
is unlikely to adopt for compatibility reasons.
This results in an ODR violation (UB), which seems to have survived
thus far. Where it becomes harmful is when;
1. -fno-semantic-interposition is used:
As Fangrui notes:
Clang after LLVM commit 5b22bcc2b70d
("[X86][ELF] Prefer to lower MC_GlobalAddress operands to .Lfoo$local")
defaults to -fno-semantic-interposition similar semantics which help
-fpic/-fPIC code avoid GOT/PLT when the referenced symbol is defined
within the same translation unit. Unlike GCC
-fno-semantic-interposition, Clang emits such relocations referencing
local symbols for non-pic code as well.
This causes references to jiffies to refer to '.Ljiffies$local' when
jiffies is defined in the same translation unit. Likewise, references to
jiffies_64 become references to '.Ljiffies_64$local' in translation units
that define jiffies_64. Because these differ from the names used in the
linker script, they will not be rewritten to alias one another.
2. Full LTO
Full LTO effectively treats all source files as one translation
unit, causing these local references to be produced everywhere. When
the linker processes the linker script, there are no longer any
references to jiffies_64' anywhere to replace with 'jiffies'. And
thus '.Ljiffies$local' and '.Ljiffies_64$local' no longer alias
at all.
In the process of porting patches enabling Full LTO from arm64 to x86_64,
spooky bugs have been observed where the kernel appeared to boot, but init
doesn't get scheduled.
Avoid the ODR violation by matching other architectures and define jiffies
only by linker script. For -fno-semantic-interposition + Full LTO, there
is no longer a global definition of jiffies for the compiler to produce a
local symbol which the linker script won't ensure aliases to jiffies_64.
Fixes: 40747ffa5aa8 ("asmlinkage: Make jiffies visible")
Reported-by: Nathan Chancellor <natechancellor@gmail.com>
Reported-by: Alistair Delva <adelva@google.com>
Debugged-by: Nick Desaulniers <ndesaulniers@google.com>
Debugged-by: Sami Tolvanen <samitolvanen@google.com>
Suggested-by: Fangrui Song <maskray@google.com>
Signed-off-by: Bob Haarman <inglorion@google.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com> # build+boot on
Reviewed-by: Andi Kleen <ak@linux.intel.com>
Reviewed-by: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: stable@vger.kernel.org
Link: https://github.com/ClangBuiltLinux/linux/issues/852
Link: https://lkml.kernel.org/r/20200602193100.229287-1-inglorion@google.com
2020-06-02 19:30:59 +00:00
|
|
|
jiffies = jiffies_64;
|
x86/percpu: Introduce const-qualified const_pcpu_hot to micro-optimize code generation
Some variables in pcpu_hot, currently current_task and top_of_stack
are actually per-thread variables implemented as per-CPU variables
and thus stable for the duration of the respective task. There is
already an attempt to eliminate redundant reads from these variables
using this_cpu_read_stable() asm macro, which hides the dependency
on the read memory address. However, the compiler has limited ability
to eliminate asm common subexpressions, so this approach results in a
limited success.
The solution is to allow more aggressive elimination by aliasing
pcpu_hot into a const-qualified const_pcpu_hot, and to read stable
per-CPU variables from this constant copy.
The current per-CPU infrastructure does not support reads from
const-qualified variables. However, when the compiler supports segment
qualifiers, it is possible to declare the const-aliased variable in
the relevant named address space. The compiler considers access to the
variable, declared in this way, as a read from a constant location,
and will optimize reads from the variable accordingly.
By implementing constant-qualified const_pcpu_hot, the compiler can
eliminate redundant reads from the constant variables, reducing the
number of loads from current_task from 3766 to 3217 on a test build,
a -14.6% reduction.
The reduction of loads translates to the following code savings:
text data bss dec hex filename
25,477,353 4389456 808452 30675261 1d4113d vmlinux-old.o
25,476,074 4389440 808452 30673966 1d40c2e vmlinux-new.o
representing a code size reduction of -1279 bytes.
[ mingo: Updated the changelog, EXPORT(const_pcpu_hot). ]
Co-developed-by: Nadav Amit <namit@vmware.com>
Signed-off-by: Nadav Amit <namit@vmware.com>
Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://lore.kernel.org/r/20231020162004.135244-1-ubizjak@gmail.com
2023-10-20 16:19:20 +00:00
|
|
|
const_pcpu_hot = pcpu_hot;
|
x86_64: Fix jiffies ODR violation
'jiffies' and 'jiffies_64' are meant to alias (two different symbols that
share the same address). Most architectures make the symbols alias to the
same address via a linker script assignment in their
arch/<arch>/kernel/vmlinux.lds.S:
jiffies = jiffies_64;
which is effectively a definition of jiffies.
jiffies and jiffies_64 are both forward declared for all architectures in
include/linux/jiffies.h. jiffies_64 is defined in kernel/time/timer.c.
x86_64 was peculiar in that it wasn't doing the above linker script
assignment, but rather was:
1. defining jiffies in arch/x86/kernel/time.c instead via the linker script.
2. overriding the symbol jiffies_64 from kernel/time/timer.c in
arch/x86/kernel/vmlinux.lds.s via 'jiffies_64 = jiffies;'.
As Fangrui notes:
In LLD, symbol assignments in linker scripts override definitions in
object files. GNU ld appears to have the same behavior. It would
probably make sense for LLD to error "duplicate symbol" but GNU ld
is unlikely to adopt for compatibility reasons.
This results in an ODR violation (UB), which seems to have survived
thus far. Where it becomes harmful is when;
1. -fno-semantic-interposition is used:
As Fangrui notes:
Clang after LLVM commit 5b22bcc2b70d
("[X86][ELF] Prefer to lower MC_GlobalAddress operands to .Lfoo$local")
defaults to -fno-semantic-interposition similar semantics which help
-fpic/-fPIC code avoid GOT/PLT when the referenced symbol is defined
within the same translation unit. Unlike GCC
-fno-semantic-interposition, Clang emits such relocations referencing
local symbols for non-pic code as well.
This causes references to jiffies to refer to '.Ljiffies$local' when
jiffies is defined in the same translation unit. Likewise, references to
jiffies_64 become references to '.Ljiffies_64$local' in translation units
that define jiffies_64. Because these differ from the names used in the
linker script, they will not be rewritten to alias one another.
2. Full LTO
Full LTO effectively treats all source files as one translation
unit, causing these local references to be produced everywhere. When
the linker processes the linker script, there are no longer any
references to jiffies_64' anywhere to replace with 'jiffies'. And
thus '.Ljiffies$local' and '.Ljiffies_64$local' no longer alias
at all.
In the process of porting patches enabling Full LTO from arm64 to x86_64,
spooky bugs have been observed where the kernel appeared to boot, but init
doesn't get scheduled.
Avoid the ODR violation by matching other architectures and define jiffies
only by linker script. For -fno-semantic-interposition + Full LTO, there
is no longer a global definition of jiffies for the compiler to produce a
local symbol which the linker script won't ensure aliases to jiffies_64.
Fixes: 40747ffa5aa8 ("asmlinkage: Make jiffies visible")
Reported-by: Nathan Chancellor <natechancellor@gmail.com>
Reported-by: Alistair Delva <adelva@google.com>
Debugged-by: Nick Desaulniers <ndesaulniers@google.com>
Debugged-by: Sami Tolvanen <samitolvanen@google.com>
Suggested-by: Fangrui Song <maskray@google.com>
Signed-off-by: Bob Haarman <inglorion@google.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com> # build+boot on
Reviewed-by: Andi Kleen <ak@linux.intel.com>
Reviewed-by: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: stable@vger.kernel.org
Link: https://github.com/ClangBuiltLinux/linux/issues/852
Link: https://lkml.kernel.org/r/20200602193100.229287-1-inglorion@google.com
2020-06-02 19:30:59 +00:00
|
|
|
|
2016-02-17 22:41:14 +00:00
|
|
|
#if defined(CONFIG_X86_64)
|
2009-10-19 13:12:04 +00:00
|
|
|
/*
|
2016-02-17 22:41:14 +00:00
|
|
|
* On 64-bit, align RODATA to 2MB so we retain large page mappings for
|
|
|
|
|
* boundaries spanning kernel text, rodata and data sections.
|
2009-10-19 13:12:04 +00:00
|
|
|
*
|
|
|
|
|
* However, kernel identity mappings will have different RWX permissions
|
|
|
|
|
* to the pages mapping to text and to the pages padding (which are freed) the
|
|
|
|
|
* text section. Hence kernel identity mappings will be broken to smaller
|
|
|
|
|
* pages. For 64-bit, kernel text and kernel identity mappings are different,
|
2016-02-17 22:41:14 +00:00
|
|
|
* so we can enable protection checks as well as retain 2MB large page
|
|
|
|
|
* mappings for kernel text.
|
2009-10-19 13:12:04 +00:00
|
|
|
*/
|
2018-07-18 09:41:04 +00:00
|
|
|
#define X86_ALIGN_RODATA_BEGIN . = ALIGN(HPAGE_SIZE);
|
2009-10-14 21:46:56 +00:00
|
|
|
|
2018-07-18 09:41:04 +00:00
|
|
|
#define X86_ALIGN_RODATA_END \
|
2009-10-14 21:46:56 +00:00
|
|
|
. = ALIGN(HPAGE_SIZE); \
|
2018-07-18 09:41:04 +00:00
|
|
|
__end_rodata_hpage_align = .; \
|
|
|
|
|
__end_rodata_aligned = .;
|
2009-10-14 21:46:56 +00:00
|
|
|
|
2017-12-04 14:07:46 +00:00
|
|
|
#define ALIGN_ENTRY_TEXT_BEGIN . = ALIGN(PMD_SIZE);
|
|
|
|
|
#define ALIGN_ENTRY_TEXT_END . = ALIGN(PMD_SIZE);
|
|
|
|
|
|
2018-09-14 13:45:58 +00:00
|
|
|
/*
|
|
|
|
|
* This section contains data which will be mapped as decrypted. Memory
|
|
|
|
|
* encryption operates on a page basis. Make this section PMD-aligned
|
|
|
|
|
* to avoid splitting the pages while mapping the section early.
|
|
|
|
|
*
|
|
|
|
|
* Note: We use a separate section so that only this section gets
|
|
|
|
|
* decrypted to avoid exposing more than we wish.
|
|
|
|
|
*/
|
|
|
|
|
#define BSS_DECRYPTED \
|
|
|
|
|
. = ALIGN(PMD_SIZE); \
|
|
|
|
|
__start_bss_decrypted = .; \
|
|
|
|
|
*(.bss..decrypted); \
|
|
|
|
|
. = ALIGN(PAGE_SIZE); \
|
|
|
|
|
__start_bss_decrypted_unused = .; \
|
|
|
|
|
. = ALIGN(PMD_SIZE); \
|
|
|
|
|
__end_bss_decrypted = .; \
|
|
|
|
|
|
2009-10-14 21:46:56 +00:00
|
|
|
#else
|
|
|
|
|
|
2018-07-18 09:41:04 +00:00
|
|
|
#define X86_ALIGN_RODATA_BEGIN
|
|
|
|
|
#define X86_ALIGN_RODATA_END \
|
|
|
|
|
. = ALIGN(PAGE_SIZE); \
|
|
|
|
|
__end_rodata_aligned = .;
|
2009-10-14 21:46:56 +00:00
|
|
|
|
2017-12-04 14:07:46 +00:00
|
|
|
#define ALIGN_ENTRY_TEXT_BEGIN
|
|
|
|
|
#define ALIGN_ENTRY_TEXT_END
|
2018-09-14 13:45:58 +00:00
|
|
|
#define BSS_DECRYPTED
|
2017-12-04 14:07:46 +00:00
|
|
|
|
2009-10-14 21:46:56 +00:00
|
|
|
#endif
|
|
|
|
|
|
2009-04-29 07:47:19 +00:00
|
|
|
PHDRS {
|
|
|
|
|
text PT_LOAD FLAGS(5); /* R_E */
|
2010-11-16 21:31:26 +00:00
|
|
|
data PT_LOAD FLAGS(6); /* RW_ */
|
2009-04-29 07:47:19 +00:00
|
|
|
#ifdef CONFIG_X86_64
|
|
|
|
|
#ifdef CONFIG_SMP
|
2009-09-04 08:18:07 +00:00
|
|
|
percpu PT_LOAD FLAGS(6); /* RW_ */
|
2009-04-29 07:47:19 +00:00
|
|
|
#endif
|
2009-08-25 13:50:53 +00:00
|
|
|
init PT_LOAD FLAGS(7); /* RWE */
|
2009-04-29 07:47:19 +00:00
|
|
|
#endif
|
|
|
|
|
note PT_NOTE FLAGS(0); /* ___ */
|
|
|
|
|
}
|
2009-04-29 07:47:18 +00:00
|
|
|
|
2009-04-29 07:47:20 +00:00
|
|
|
SECTIONS
|
|
|
|
|
{
|
2024-03-13 07:58:39 +00:00
|
|
|
. = __START_KERNEL;
|
2009-04-29 07:47:20 +00:00
|
|
|
#ifdef CONFIG_X86_32
|
x86/kallsyms: fix GOLD link failure with new relative kallsyms table format
Commit 2213e9a66bb8 ("kallsyms: add support for relative offsets in
kallsyms address table") changed the default kallsyms symbol table
format to use relative references rather than absolute addresses.
This reduces the size of the kallsyms symbol table by 50% on 64-bit
architectures, and further reduces the size of the relocation tables
used by relocatable kernels. Since the memory footprint of the static
kernel image is always much smaller than 4 GB, these relative references
are assumed to be representable in 32 bits, even when the native word
size is 64 bits.
On 64-bit architectures, this obviously only works if the distance
between each relative reference and the chosen anchor point is
representable in 32 bits, and so the table generation code in
scripts/kallsyms.c scans the table for the lowest value that is covered
by the kernel text, and selects it as the anchor point.
However, when using the GOLD linker rather than the default BFD linker
to build the x86_64 kernel, the symbol phys_offset_64, which is the
result of arithmetic defined in the linker script, is emitted as a 'T'
rather than an 'A' type symbol, resulting in scripts/kallsyms.c to
mistake it for a suitable anchor point, even though it is far away from
the actual kernel image in the virtual address space. This results in
out-of-range warnings from scripts/kallsyms.c and a broken build.
So let's align with the BFD linker, and emit the phys_offset_[32|64]
symbols as absolute symbols explicitly. Note that the out of range
issue does not exist on 32-bit x86, but this patch changes both symbols
for symmetry.
Reported-by: Markus Trippelsdorf <markus@trippelsdorf.de>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Kees Cook <keescook@chromium.org>
Cc: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-03-18 09:04:37 +00:00
|
|
|
phys_startup_32 = ABSOLUTE(startup_32 - LOAD_OFFSET);
|
2009-04-29 07:47:20 +00:00
|
|
|
#else
|
x86/kallsyms: fix GOLD link failure with new relative kallsyms table format
Commit 2213e9a66bb8 ("kallsyms: add support for relative offsets in
kallsyms address table") changed the default kallsyms symbol table
format to use relative references rather than absolute addresses.
This reduces the size of the kallsyms symbol table by 50% on 64-bit
architectures, and further reduces the size of the relocation tables
used by relocatable kernels. Since the memory footprint of the static
kernel image is always much smaller than 4 GB, these relative references
are assumed to be representable in 32 bits, even when the native word
size is 64 bits.
On 64-bit architectures, this obviously only works if the distance
between each relative reference and the chosen anchor point is
representable in 32 bits, and so the table generation code in
scripts/kallsyms.c scans the table for the lowest value that is covered
by the kernel text, and selects it as the anchor point.
However, when using the GOLD linker rather than the default BFD linker
to build the x86_64 kernel, the symbol phys_offset_64, which is the
result of arithmetic defined in the linker script, is emitted as a 'T'
rather than an 'A' type symbol, resulting in scripts/kallsyms.c to
mistake it for a suitable anchor point, even though it is far away from
the actual kernel image in the virtual address space. This results in
out-of-range warnings from scripts/kallsyms.c and a broken build.
So let's align with the BFD linker, and emit the phys_offset_[32|64]
symbols as absolute symbols explicitly. Note that the out of range
issue does not exist on 32-bit x86, but this patch changes both symbols
for symmetry.
Reported-by: Markus Trippelsdorf <markus@trippelsdorf.de>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Kees Cook <keescook@chromium.org>
Cc: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-03-18 09:04:37 +00:00
|
|
|
phys_startup_64 = ABSOLUTE(startup_64 - LOAD_OFFSET);
|
2009-04-29 07:47:20 +00:00
|
|
|
#endif
|
|
|
|
|
|
2009-04-29 07:47:21 +00:00
|
|
|
/* Text and read-only data */
|
|
|
|
|
.text : AT(ADDR(.text) - LOAD_OFFSET) {
|
2009-09-16 20:44:28 +00:00
|
|
|
_text = .;
|
2016-09-21 21:04:07 +00:00
|
|
|
_stext = .;
|
2009-09-16 20:44:28 +00:00
|
|
|
/* bootstrapping code */
|
|
|
|
|
HEAD_TEXT
|
2009-04-29 07:47:21 +00:00
|
|
|
TEXT_TEXT
|
|
|
|
|
SCHED_TEXT
|
|
|
|
|
LOCK_TEXT
|
|
|
|
|
KPROBES_TEXT
|
2016-03-25 21:22:05 +00:00
|
|
|
SOFTIRQENTRY_TEXT
|
2023-11-21 16:07:32 +00:00
|
|
|
#ifdef CONFIG_MITIGATION_RETPOLINE
|
2023-07-11 09:19:51 +00:00
|
|
|
*(.text..__x86.indirect_thunk)
|
|
|
|
|
*(.text..__x86.return_thunk)
|
2018-01-18 16:14:21 +00:00
|
|
|
#endif
|
2022-09-15 11:10:46 +00:00
|
|
|
STATIC_CALL_TEXT
|
|
|
|
|
|
|
|
|
|
ALIGN_ENTRY_TEXT_BEGIN
|
2023-07-11 09:19:51 +00:00
|
|
|
*(.text..__x86.rethunk_untrain)
|
2022-09-15 11:10:46 +00:00
|
|
|
ENTRY_TEXT
|
2023-06-28 09:02:39 +00:00
|
|
|
|
2023-11-21 16:07:36 +00:00
|
|
|
#ifdef CONFIG_MITIGATION_SRSO
|
2023-06-28 09:02:39 +00:00
|
|
|
/*
|
2023-08-14 11:44:33 +00:00
|
|
|
* See the comment above srso_alias_untrain_ret()'s
|
2023-06-28 09:02:39 +00:00
|
|
|
* definition.
|
|
|
|
|
*/
|
2023-08-14 11:44:33 +00:00
|
|
|
. = srso_alias_untrain_ret | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 20);
|
2023-07-11 09:19:51 +00:00
|
|
|
*(.text..__x86.rethunk_safe)
|
2023-06-28 09:02:39 +00:00
|
|
|
#endif
|
2022-09-15 11:10:46 +00:00
|
|
|
ALIGN_ENTRY_TEXT_END
|
|
|
|
|
*(.gnu.warning)
|
|
|
|
|
|
2023-09-06 17:52:15 +00:00
|
|
|
} :text = 0xcccccccc
|
2019-04-23 18:38:27 +00:00
|
|
|
|
2019-10-29 21:13:37 +00:00
|
|
|
/* End of text section, which should occupy whole number of pages */
|
|
|
|
|
_etext = .;
|
2010-11-16 21:31:26 +00:00
|
|
|
. = ALIGN(PAGE_SIZE);
|
2019-10-29 21:13:37 +00:00
|
|
|
|
2018-07-18 09:41:04 +00:00
|
|
|
X86_ALIGN_RODATA_BEGIN
|
2009-08-25 13:50:53 +00:00
|
|
|
RO_DATA(PAGE_SIZE)
|
2018-07-18 09:41:04 +00:00
|
|
|
X86_ALIGN_RODATA_END
|
2009-04-29 07:47:22 +00:00
|
|
|
|
2009-04-29 07:47:23 +00:00
|
|
|
/* Data */
|
|
|
|
|
.data : AT(ADDR(.data) - LOAD_OFFSET) {
|
2009-05-11 12:22:00 +00:00
|
|
|
/* Start of data section */
|
|
|
|
|
_sdata = .;
|
2009-08-25 13:50:53 +00:00
|
|
|
|
|
|
|
|
/* init_task */
|
|
|
|
|
INIT_TASK_DATA(THREAD_SIZE)
|
2009-04-29 07:47:23 +00:00
|
|
|
|
2024-03-21 18:05:06 +00:00
|
|
|
/* equivalent to task_pt_regs(&init_task) */
|
2024-03-22 08:16:16 +00:00
|
|
|
__top_init_kernel_stack = __end_init_stack - TOP_OF_KERNEL_STACK_PADDING - PTREGS_SIZE;
|
2024-03-21 18:05:06 +00:00
|
|
|
|
2009-04-29 07:47:23 +00:00
|
|
|
#ifdef CONFIG_X86_32
|
2009-08-25 13:50:53 +00:00
|
|
|
/* 32 bit has nosave before _edata */
|
|
|
|
|
NOSAVE_DATA
|
2009-04-29 07:47:23 +00:00
|
|
|
#endif
|
|
|
|
|
|
2009-08-25 13:50:53 +00:00
|
|
|
PAGE_ALIGNED_DATA(PAGE_SIZE)
|
2009-04-29 07:47:23 +00:00
|
|
|
|
2009-11-13 11:54:40 +00:00
|
|
|
CACHELINE_ALIGNED_DATA(L1_CACHE_BYTES)
|
2009-04-29 07:47:23 +00:00
|
|
|
|
2009-08-25 13:50:53 +00:00
|
|
|
DATA_DATA
|
|
|
|
|
CONSTRUCTORS
|
|
|
|
|
|
|
|
|
|
/* rarely changed data like cpu maps */
|
2009-11-13 11:54:40 +00:00
|
|
|
READ_MOSTLY_DATA(INTERNODE_CACHE_BYTES)
|
2009-04-29 07:47:23 +00:00
|
|
|
|
|
|
|
|
/* End of data section */
|
|
|
|
|
_edata = .;
|
2009-08-25 13:50:53 +00:00
|
|
|
} :data
|
2009-04-29 07:47:23 +00:00
|
|
|
|
2017-03-30 15:49:27 +00:00
|
|
|
BUG_TABLE
|
2009-04-29 07:47:24 +00:00
|
|
|
|
2017-07-24 23:36:57 +00:00
|
|
|
ORC_UNWIND_TABLE
|
|
|
|
|
|
2011-08-03 13:31:50 +00:00
|
|
|
. = ALIGN(PAGE_SIZE);
|
|
|
|
|
__vvar_page = .;
|
|
|
|
|
|
|
|
|
|
.vvar : AT(ADDR(.vvar) - LOAD_OFFSET) {
|
2011-08-03 13:31:51 +00:00
|
|
|
/* work around gold bug 13023 */
|
|
|
|
|
__vvar_beginning_hack = .;
|
2011-08-03 13:31:50 +00:00
|
|
|
|
2011-08-03 13:31:51 +00:00
|
|
|
/* Place all vvars at the offsets in asm/vvar.h. */
|
2019-11-12 01:27:10 +00:00
|
|
|
#define EMIT_VVAR(name, offset) \
|
2011-08-03 13:31:51 +00:00
|
|
|
. = __vvar_beginning_hack + offset; \
|
2011-08-03 13:31:50 +00:00
|
|
|
*(.vvar_ ## name)
|
|
|
|
|
#include <asm/vvar.h>
|
|
|
|
|
#undef EMIT_VVAR
|
|
|
|
|
|
2014-03-17 22:22:11 +00:00
|
|
|
/*
|
|
|
|
|
* Pad the rest of the page with zeros. Otherwise the loader
|
|
|
|
|
* can leave garbage here.
|
|
|
|
|
*/
|
|
|
|
|
. = __vvar_beginning_hack + PAGE_SIZE;
|
2011-08-03 13:31:50 +00:00
|
|
|
} :data
|
|
|
|
|
|
2018-02-08 06:38:57 +00:00
|
|
|
. = ALIGN(__vvar_page + PAGE_SIZE, PAGE_SIZE);
|
2011-08-03 13:31:50 +00:00
|
|
|
|
2009-08-25 13:50:53 +00:00
|
|
|
/* Init code and data - will be freed after init */
|
|
|
|
|
. = ALIGN(PAGE_SIZE);
|
|
|
|
|
.init.begin : AT(ADDR(.init.begin) - LOAD_OFFSET) {
|
|
|
|
|
__init_begin = .; /* paired with __init_end */
|
2009-04-29 07:47:25 +00:00
|
|
|
}
|
|
|
|
|
|
2009-08-25 13:50:53 +00:00
|
|
|
#if defined(CONFIG_X86_64) && defined(CONFIG_SMP)
|
2009-04-29 07:47:25 +00:00
|
|
|
/*
|
2009-08-25 13:50:53 +00:00
|
|
|
* percpu offsets are zero-based on SMP. PERCPU_VADDR() changes the
|
|
|
|
|
* output PHDR, so the next output section - .init.text - should
|
|
|
|
|
* start another segment - init.
|
2009-04-29 07:47:25 +00:00
|
|
|
*/
|
2011-01-25 13:26:50 +00:00
|
|
|
PERCPU_VADDR(INTERNODE_CACHE_BYTES, 0, :percpu)
|
2014-11-04 08:50:48 +00:00
|
|
|
ASSERT(SIZEOF(.data..percpu) < CONFIG_PHYSICAL_START,
|
|
|
|
|
"per-CPU data too large - increase CONFIG_PHYSICAL_START")
|
2009-08-25 13:50:53 +00:00
|
|
|
#endif
|
2009-04-29 07:47:25 +00:00
|
|
|
|
2009-09-16 20:44:30 +00:00
|
|
|
INIT_TEXT_SECTION(PAGE_SIZE)
|
2009-08-25 13:50:53 +00:00
|
|
|
#ifdef CONFIG_X86_64
|
|
|
|
|
:init
|
|
|
|
|
#endif
|
2009-04-29 07:47:25 +00:00
|
|
|
|
2016-01-26 21:12:07 +00:00
|
|
|
/*
|
|
|
|
|
* Section for code used exclusively before alternatives are run. All
|
|
|
|
|
* references to such code must be patched out by alternatives, normally
|
|
|
|
|
* by using X86_FEATURE_ALWAYS CPU feature bit.
|
|
|
|
|
*
|
|
|
|
|
* See static_cpu_has() for an example.
|
|
|
|
|
*/
|
|
|
|
|
.altinstr_aux : AT(ADDR(.altinstr_aux) - LOAD_OFFSET) {
|
|
|
|
|
*(.altinstr_aux)
|
|
|
|
|
}
|
|
|
|
|
|
2009-09-16 20:44:30 +00:00
|
|
|
INIT_DATA_SECTION(16)
|
2009-04-29 07:47:25 +00:00
|
|
|
|
|
|
|
|
.x86_cpu_dev.init : AT(ADDR(.x86_cpu_dev.init) - LOAD_OFFSET) {
|
|
|
|
|
__x86_cpu_dev_start = .;
|
|
|
|
|
*(.x86_cpu_dev.init)
|
|
|
|
|
__x86_cpu_dev_end = .;
|
|
|
|
|
}
|
|
|
|
|
|
2013-10-17 22:35:35 +00:00
|
|
|
#ifdef CONFIG_X86_INTEL_MID
|
|
|
|
|
.x86_intel_mid_dev.init : AT(ADDR(.x86_intel_mid_dev.init) - \
|
|
|
|
|
LOAD_OFFSET) {
|
|
|
|
|
__x86_intel_mid_dev_start = .;
|
|
|
|
|
*(.x86_intel_mid_dev.init)
|
|
|
|
|
__x86_intel_mid_dev_end = .;
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2023-11-21 16:07:32 +00:00
|
|
|
#ifdef CONFIG_MITIGATION_RETPOLINE
|
2021-10-26 12:01:36 +00:00
|
|
|
/*
|
|
|
|
|
* List of instructions that call/jmp/jcc to retpoline thunks
|
|
|
|
|
* __x86_indirect_thunk_*(). These instructions can be patched along
|
|
|
|
|
* with alternatives, after which the section can be freed.
|
|
|
|
|
*/
|
|
|
|
|
. = ALIGN(8);
|
|
|
|
|
.retpoline_sites : AT(ADDR(.retpoline_sites) - LOAD_OFFSET) {
|
|
|
|
|
__retpoline_sites = .;
|
|
|
|
|
*(.retpoline_sites)
|
|
|
|
|
__retpoline_sites_end = .;
|
|
|
|
|
}
|
2022-06-14 21:15:37 +00:00
|
|
|
|
|
|
|
|
. = ALIGN(8);
|
|
|
|
|
.return_sites : AT(ADDR(.return_sites) - LOAD_OFFSET) {
|
|
|
|
|
__return_sites = .;
|
|
|
|
|
*(.return_sites)
|
|
|
|
|
__return_sites_end = .;
|
|
|
|
|
}
|
2022-09-15 11:11:09 +00:00
|
|
|
|
|
|
|
|
. = ALIGN(8);
|
|
|
|
|
.call_sites : AT(ADDR(.call_sites) - LOAD_OFFSET) {
|
|
|
|
|
__call_sites = .;
|
|
|
|
|
*(.call_sites)
|
|
|
|
|
__call_sites_end = .;
|
|
|
|
|
}
|
2021-10-26 12:01:36 +00:00
|
|
|
#endif
|
|
|
|
|
|
2022-03-08 15:30:55 +00:00
|
|
|
#ifdef CONFIG_X86_KERNEL_IBT
|
|
|
|
|
. = ALIGN(8);
|
|
|
|
|
.ibt_endbr_seal : AT(ADDR(.ibt_endbr_seal) - LOAD_OFFSET) {
|
|
|
|
|
__ibt_endbr_seal = .;
|
|
|
|
|
*(.ibt_endbr_seal)
|
|
|
|
|
__ibt_endbr_seal_end = .;
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2022-10-27 09:28:14 +00:00
|
|
|
#ifdef CONFIG_FINEIBT
|
|
|
|
|
. = ALIGN(8);
|
|
|
|
|
.cfi_sites : AT(ADDR(.cfi_sites) - LOAD_OFFSET) {
|
|
|
|
|
__cfi_sites = .;
|
|
|
|
|
*(.cfi_sites)
|
|
|
|
|
__cfi_sites_end = .;
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2010-08-27 18:19:33 +00:00
|
|
|
/*
|
|
|
|
|
* struct alt_inst entries. From the header (alternative.h):
|
|
|
|
|
* "Alternative instructions for different CPU types or capabilities"
|
|
|
|
|
* Think locking instructions on spinlocks.
|
|
|
|
|
*/
|
2009-04-29 07:47:26 +00:00
|
|
|
. = ALIGN(8);
|
|
|
|
|
.altinstructions : AT(ADDR(.altinstructions) - LOAD_OFFSET) {
|
|
|
|
|
__alt_instructions = .;
|
|
|
|
|
*(.altinstructions)
|
|
|
|
|
__alt_instructions_end = .;
|
|
|
|
|
}
|
|
|
|
|
|
2010-08-27 18:19:33 +00:00
|
|
|
/*
|
|
|
|
|
* And here are the replacement instructions. The linker sticks
|
|
|
|
|
* them as binary blobs. The .altinstructions has enough data to
|
|
|
|
|
* get the address and the length of them to patch the kernel safely.
|
|
|
|
|
*/
|
2009-04-29 07:47:26 +00:00
|
|
|
.altinstr_replacement : AT(ADDR(.altinstr_replacement) - LOAD_OFFSET) {
|
|
|
|
|
*(.altinstr_replacement)
|
|
|
|
|
}
|
|
|
|
|
|
2011-05-21 00:51:17 +00:00
|
|
|
. = ALIGN(8);
|
|
|
|
|
.apicdrivers : AT(ADDR(.apicdrivers) - LOAD_OFFSET) {
|
|
|
|
|
__apicdrivers = .;
|
|
|
|
|
*(.apicdrivers);
|
|
|
|
|
__apicdrivers_end = .;
|
|
|
|
|
}
|
|
|
|
|
|
2010-08-30 18:10:02 +00:00
|
|
|
. = ALIGN(8);
|
2009-04-29 07:47:27 +00:00
|
|
|
/*
|
2020-02-24 23:21:29 +00:00
|
|
|
* .exit.text is discarded at runtime, not link time, to deal with
|
|
|
|
|
* references from .altinstructions
|
2009-04-29 07:47:27 +00:00
|
|
|
*/
|
|
|
|
|
.exit.text : AT(ADDR(.exit.text) - LOAD_OFFSET) {
|
|
|
|
|
EXIT_TEXT
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
.exit.data : AT(ADDR(.exit.data) - LOAD_OFFSET) {
|
|
|
|
|
EXIT_DATA
|
|
|
|
|
}
|
|
|
|
|
|
2009-08-25 13:50:53 +00:00
|
|
|
#if !defined(CONFIG_X86_64) || !defined(CONFIG_SMP)
|
2011-03-24 17:50:09 +00:00
|
|
|
PERCPU_SECTION(INTERNODE_CACHE_BYTES)
|
2009-04-29 07:47:28 +00:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
. = ALIGN(PAGE_SIZE);
|
2009-04-29 10:56:58 +00:00
|
|
|
|
2009-04-29 07:47:28 +00:00
|
|
|
/* freed after init ends here */
|
2009-04-29 10:56:58 +00:00
|
|
|
.init.end : AT(ADDR(.init.end) - LOAD_OFFSET) {
|
|
|
|
|
__init_end = .;
|
|
|
|
|
}
|
2009-04-29 07:47:28 +00:00
|
|
|
|
2009-08-25 13:50:53 +00:00
|
|
|
/*
|
|
|
|
|
* smp_locks might be freed after init
|
|
|
|
|
* start/end must be page aligned
|
|
|
|
|
*/
|
|
|
|
|
. = ALIGN(PAGE_SIZE);
|
|
|
|
|
.smp_locks : AT(ADDR(.smp_locks) - LOAD_OFFSET) {
|
|
|
|
|
__smp_locks = .;
|
|
|
|
|
*(.smp_locks)
|
|
|
|
|
. = ALIGN(PAGE_SIZE);
|
2010-03-29 02:42:54 +00:00
|
|
|
__smp_locks_end = .;
|
2009-08-25 13:50:53 +00:00
|
|
|
}
|
|
|
|
|
|
2009-04-29 07:47:28 +00:00
|
|
|
#ifdef CONFIG_X86_64
|
|
|
|
|
.data_nosave : AT(ADDR(.data_nosave) - LOAD_OFFSET) {
|
2009-08-25 13:50:53 +00:00
|
|
|
NOSAVE_DATA
|
|
|
|
|
}
|
2009-04-29 07:47:28 +00:00
|
|
|
#endif
|
|
|
|
|
|
2009-04-29 07:47:29 +00:00
|
|
|
/* BSS */
|
|
|
|
|
. = ALIGN(PAGE_SIZE);
|
|
|
|
|
.bss : AT(ADDR(.bss) - LOAD_OFFSET) {
|
|
|
|
|
__bss_start = .;
|
2010-02-20 00:03:38 +00:00
|
|
|
*(.bss..page_aligned)
|
2020-07-21 09:34:48 +00:00
|
|
|
. = ALIGN(PAGE_SIZE);
|
2019-04-15 16:49:56 +00:00
|
|
|
*(BSS_MAIN)
|
2018-09-14 13:45:58 +00:00
|
|
|
BSS_DECRYPTED
|
2010-11-16 21:31:26 +00:00
|
|
|
. = ALIGN(PAGE_SIZE);
|
2009-04-29 07:47:29 +00:00
|
|
|
__bss_stop = .;
|
|
|
|
|
}
|
2009-04-29 07:47:28 +00:00
|
|
|
|
2019-06-19 18:40:57 +00:00
|
|
|
/*
|
|
|
|
|
* The memory occupied from _text to here, __end_of_kernel_reserve, is
|
|
|
|
|
* automatically reserved in setup_arch(). Anything after here must be
|
|
|
|
|
* explicitly reserved using memblock_reserve() or it will be discarded
|
|
|
|
|
* and treated as available memory.
|
|
|
|
|
*/
|
|
|
|
|
__end_of_kernel_reserve = .;
|
|
|
|
|
|
2009-04-29 07:47:29 +00:00
|
|
|
. = ALIGN(PAGE_SIZE);
|
2022-06-30 07:14:41 +00:00
|
|
|
.brk : AT(ADDR(.brk) - LOAD_OFFSET) {
|
2009-04-29 07:47:29 +00:00
|
|
|
__brk_base = .;
|
|
|
|
|
. += 64 * 1024; /* 64k alignment slop space */
|
x86/mm: Fix RESERVE_BRK() for older binutils
With binutils 2.26, RESERVE_BRK() causes a build failure:
/tmp/ccnGOKZ5.s: Assembler messages:
/tmp/ccnGOKZ5.s:98: Error: missing ')'
/tmp/ccnGOKZ5.s:98: Error: missing ')'
/tmp/ccnGOKZ5.s:98: Error: missing ')'
/tmp/ccnGOKZ5.s:98: Error: junk at end of line, first unrecognized
character is `U'
The problem is this line:
RESERVE_BRK(early_pgt_alloc, INIT_PGT_BUF_SIZE)
Specifically, the INIT_PGT_BUF_SIZE macro which (via PAGE_SIZE's use
_AC()) has a "1UL", which makes older versions of the assembler unhappy.
Unfortunately the _AC() macro doesn't work for inline asm.
Inline asm was only needed here to convince the toolchain to add the
STT_NOBITS flag. However, if a C variable is placed in a section whose
name is prefixed with ".bss", GCC and Clang automatically set
STT_NOBITS. In fact, ".bss..page_aligned" already relies on this trick.
So fix the build failure (and simplify the macro) by allocating the
variable in C.
Also, add NOLOAD to the ".brk" output section clause in the linker
script. This is a failsafe in case the ".bss" prefix magic trick ever
stops working somehow. If there's a section type mismatch, the GNU
linker will force the ".brk" output section to be STT_NOBITS. The LLVM
linker will fail with a "section type mismatch" error.
Note this also changes the name of the variable from .brk.##name to
__brk_##name. The variable names aren't actually used anywhere, so it's
harmless.
Fixes: a1e2c031ec39 ("x86/mm: Simplify RESERVE_BRK()")
Reported-by: Joe Damato <jdamato@fastly.com>
Reported-by: Byungchul Park <byungchul.park@lge.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: Joe Damato <jdamato@fastly.com>
Link: https://lore.kernel.org/r/22d07a44c80d8e8e1e82b9a806ddc8c6bbb2606e.1654759036.git.jpoimboe@kernel.org
2022-06-09 07:17:32 +00:00
|
|
|
*(.bss..brk) /* areas brk users have reserved */
|
2009-04-29 07:47:29 +00:00
|
|
|
__brk_limit = .;
|
|
|
|
|
}
|
|
|
|
|
|
x86/boot: Move compressed kernel to the end of the decompression buffer
This change makes later calculations about where the kernel is located
easier to reason about. To better understand this change, we must first
clarify what 'VO' and 'ZO' are. These values were introduced in commits
by hpa:
77d1a4999502 ("x86, boot: make symbols from the main vmlinux available")
37ba7ab5e33c ("x86, boot: make kernel_alignment adjustable; new bzImage fields")
Specifically:
All names prefixed with 'VO_':
- relate to the uncompressed kernel image
- the size of the VO image is: VO__end-VO__text ("VO_INIT_SIZE" define)
All names prefixed with 'ZO_':
- relate to the bootable compressed kernel image (boot/compressed/vmlinux),
which is composed of the following memory areas:
- head text
- compressed kernel (VO image and relocs table)
- decompressor code
- the size of the ZO image is: ZO__end - ZO_startup_32 ("ZO_INIT_SIZE" define, though see below)
The 'INIT_SIZE' value is used to find the larger of the two image sizes:
#define ZO_INIT_SIZE (ZO__end - ZO_startup_32 + ZO_z_extract_offset)
#define VO_INIT_SIZE (VO__end - VO__text)
#if ZO_INIT_SIZE > VO_INIT_SIZE
# define INIT_SIZE ZO_INIT_SIZE
#else
# define INIT_SIZE VO_INIT_SIZE
#endif
The current code uses extract_offset to decide where to position the
copied ZO (i.e. ZO starts at extract_offset). (This is why ZO_INIT_SIZE
currently includes the extract_offset.)
Why does z_extract_offset exist? It's needed because we are trying to minimize
the amount of RAM used for the whole act of creating an uncompressed, executable,
properly relocation-linked kernel image in system memory. We do this so that
kernels can be booted on even very small systems.
To achieve the goal of minimal memory consumption we have implemented an in-place
decompression strategy: instead of cleanly separating the VO and ZO images and
also allocating some memory for the decompression code's runtime needs, we instead
create this elaborate layout of memory buffers where the output (decompressed)
stream, as it progresses, overlaps with and destroys the input (compressed)
stream. This can only be done safely if the ZO image is placed to the end of the
VO range, plus a certain amount of safety distance to make sure that when the last
bytes of the VO range are decompressed, the compressed stream pointer is safely
beyond the end of the VO range.
z_extract_offset is calculated in arch/x86/boot/compressed/mkpiggy.c during
the build process, at a point when we know the exact compressed and
uncompressed size of the kernel images and can calculate this safe minimum
offset value. (Note that the mkpiggy.c calculation is not perfect, because
we don't know the decompressor used at that stage, so the z_extract_offset
calculation is necessarily imprecise and is mostly based on gzip internals -
we'll improve that in the next patch.)
When INIT_SIZE is bigger than VO_INIT_SIZE (uncommon but possible),
the copied ZO occupies the memory from extract_offset to the end of
decompression buffer. It overlaps with the soon-to-be-uncompressed kernel
like this:
|-----compressed kernel image------|
V V
0 extract_offset +INIT_SIZE
|-----------|---------------|-------------------------|--------|
| | | |
VO__text startup_32 of ZO VO__end ZO__end
^ ^
|-------uncompressed kernel image---------|
When INIT_SIZE is equal to VO_INIT_SIZE (likely) there's still space
left from end of ZO to the end of decompressing buffer, like below.
|-compressed kernel image-|
V V
0 extract_offset +INIT_SIZE
|-----------|---------------|-------------------------|--------|
| | | |
VO__text startup_32 of ZO ZO__end VO__end
^ ^
|------------uncompressed kernel image-------------|
To simplify calculations and avoid special cases, it is cleaner to
always place the compressed kernel image in memory so that ZO__end
is at the end of the decompression buffer, instead of placing t at
the start of extract_offset as is currently done.
This patch adds BP_init_size (which is the INIT_SIZE as passed in from
the boot_params) into asm-offsets.c to make it visible to the assembly
code.
Then when moving the ZO, it calculates the starting position of
the copied ZO (via BP_init_size and the ZO run size) so that the VO__end
will be at the end of the decompression buffer. To make the position
calculation safe, the end of ZO is page aligned (and a comment is added
to the existing VO alignment for good measure).
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
[ Rewrote changelog and comments. ]
Signed-off-by: Kees Cook <keescook@chromium.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Baoquan He <bhe@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Vivek Goyal <vgoyal@redhat.com>
Cc: lasse.collin@tukaani.org
Link: http://lkml.kernel.org/r/1461888548-32439-3-git-send-email-keescook@chromium.org
[ Rewrote the changelog some more. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-04-29 00:09:04 +00:00
|
|
|
. = ALIGN(PAGE_SIZE); /* keep VO_INIT_SIZE page aligned */
|
2009-12-14 21:55:20 +00:00
|
|
|
_end = .;
|
2009-04-29 07:47:29 +00:00
|
|
|
|
x86/mm: Create a workarea in the kernel for SME early encryption
In order for the kernel to be encrypted "in place" during boot, a workarea
outside of the kernel must be used. This SME workarea used during early
encryption of the kernel is situated on a 2MB boundary after the end of
the kernel text, data, etc. sections (_end).
This works well during initial boot of a compressed kernel because of
the relocation used for decompression of the kernel. But when performing
a kexec boot, there's a chance that the SME workarea may not be mapped
by the kexec pagetables or that some of the other data used by kexec
could exist in this range.
Create a section for SME in vmlinux.lds.S. Position it after "_end", which
is after "__end_of_kernel_reserve", so that the memory will be reclaimed
during boot and since this area is all zeroes, it compresses well. This
new section will be part of the kernel image, so kexec will account for it
in pagetable mappings and placement of data after the kernel.
Here's an example of a kernel size without and with the SME section:
without:
vmlinux: 36,501,616
bzImage: 6,497,344
100000000-47f37ffff : System RAM
1e4000000-1e47677d4 : Kernel code (0x7677d4)
1e47677d5-1e4e2e0bf : Kernel data (0x6c68ea)
1e5074000-1e5372fff : Kernel bss (0x2fefff)
with:
vmlinux: 44,419,408
bzImage: 6,503,136
880000000-c7ff7ffff : System RAM
8cf000000-8cf7677d4 : Kernel code (0x7677d4)
8cf7677d5-8cfe2e0bf : Kernel data (0x6c68ea)
8d0074000-8d0372fff : Kernel bss (0x2fefff)
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Baoquan He <bhe@redhat.com>
Reviewed-by: Dave Hansen <dave.hansen@intel.com>
Tested-by: Lianbo Jiang <lijiang@redhat.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Brijesh Singh <brijesh.singh@amd.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Joerg Roedel <jroedel@suse.de>
Cc: Kees Cook <keescook@chromium.org>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Rafael Ávila de Espíndola" <rafael@espindo.la>
Cc: Sami Tolvanen <samitolvanen@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "x86@kernel.org" <x86@kernel.org>
Link: https://lkml.kernel.org/r/3c483262eb4077b1654b2052bd14a8d011bffde3.1560969363.git.thomas.lendacky@amd.com
2019-06-19 18:40:59 +00:00
|
|
|
#ifdef CONFIG_AMD_MEM_ENCRYPT
|
|
|
|
|
/*
|
|
|
|
|
* Early scratch/workarea section: Lives outside of the kernel proper
|
|
|
|
|
* (_text - _end).
|
|
|
|
|
*
|
|
|
|
|
* Resides after _end because even though the .brk section is after
|
|
|
|
|
* __end_of_kernel_reserve, the .brk section is later reserved as a
|
|
|
|
|
* part of the kernel. Since it is located after __end_of_kernel_reserve
|
|
|
|
|
* it will be discarded and become part of the available memory. As
|
|
|
|
|
* such, it can only be used by very early boot code and must not be
|
|
|
|
|
* needed afterwards.
|
|
|
|
|
*
|
|
|
|
|
* Currently used by SME for performing in-place encryption of the
|
|
|
|
|
* kernel during boot. Resides on a 2MB boundary to simplify the
|
|
|
|
|
* pagetable setup used for SME in-place encryption.
|
|
|
|
|
*/
|
|
|
|
|
. = ALIGN(HPAGE_SIZE);
|
|
|
|
|
.init.scratch : AT(ADDR(.init.scratch) - LOAD_OFFSET) {
|
|
|
|
|
__init_scratch_begin = .;
|
|
|
|
|
*(.init.scratch)
|
|
|
|
|
. = ALIGN(HPAGE_SIZE);
|
|
|
|
|
__init_scratch_end = .;
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2018-02-08 06:38:57 +00:00
|
|
|
STABS_DEBUG
|
|
|
|
|
DWARF_DEBUG
|
2020-08-21 19:42:45 +00:00
|
|
|
ELF_DETAILS
|
linker script: unify usage of discard definition
Discarded sections in different archs share some commonality but have
considerable differences. This led to linker script for each arch
implementing its own /DISCARD/ definition, which makes maintaining
tedious and adding new entries error-prone.
This patch makes all linker scripts to move discard definitions to the
end of the linker script and use the common DISCARDS macro. As ld
uses the first matching section definition, archs can include default
discarded sections by including them earlier in the linker script.
ia64 is notable because it first throws away some ia64 specific
subsections and then include the rest of the sections into the final
image, so those sections must be discarded before the inclusion.
defconfig compile tested for x86, x86-64, powerpc, powerpc64, ia64,
alpha, sparc, sparc64 and s390. Michal Simek tested microblaze.
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Mike Frysinger <vapier@gentoo.org>
Tested-by: Michal Simek <monstr@monstr.eu>
Cc: linux-arch@vger.kernel.org
Cc: Michal Simek <monstr@monstr.eu>
Cc: microblaze-uclinux@itee.uq.edu.au
Cc: Sam Ravnborg <sam@ravnborg.org>
Cc: Tony Luck <tony.luck@intel.com>
2009-07-09 02:27:40 +00:00
|
|
|
|
|
|
|
|
DISCARDS
|
2009-04-29 07:47:20 +00:00
|
|
|
|
2020-08-21 19:43:04 +00:00
|
|
|
/*
|
|
|
|
|
* Make sure that the .got.plt is either completely empty or it
|
|
|
|
|
* contains only the lazy dispatch entries.
|
|
|
|
|
*/
|
|
|
|
|
.got.plt (INFO) : { *(.got.plt) }
|
|
|
|
|
ASSERT(SIZEOF(.got.plt) == 0 ||
|
|
|
|
|
#ifdef CONFIG_X86_64
|
|
|
|
|
SIZEOF(.got.plt) == 0x18,
|
|
|
|
|
#else
|
|
|
|
|
SIZEOF(.got.plt) == 0xc,
|
|
|
|
|
#endif
|
|
|
|
|
"Unexpected GOT/PLT entries detected!")
|
2020-08-21 19:43:05 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Sections that should stay zero sized, which is safer to
|
|
|
|
|
* explicitly check instead of blindly discarding.
|
|
|
|
|
*/
|
|
|
|
|
.got : {
|
|
|
|
|
*(.got) *(.igot.*)
|
|
|
|
|
}
|
|
|
|
|
ASSERT(SIZEOF(.got) == 0, "Unexpected GOT entries detected!")
|
|
|
|
|
|
|
|
|
|
.plt : {
|
|
|
|
|
*(.plt) *(.plt.*) *(.iplt)
|
|
|
|
|
}
|
|
|
|
|
ASSERT(SIZEOF(.plt) == 0, "Unexpected run-time procedure linkages detected!")
|
|
|
|
|
|
|
|
|
|
.rel.dyn : {
|
|
|
|
|
*(.rel.*) *(.rel_*)
|
|
|
|
|
}
|
|
|
|
|
ASSERT(SIZEOF(.rel.dyn) == 0, "Unexpected run-time relocations (.rel) detected!")
|
|
|
|
|
|
|
|
|
|
.rela.dyn : {
|
|
|
|
|
*(.rela.*) *(.rela_*)
|
|
|
|
|
}
|
|
|
|
|
ASSERT(SIZEOF(.rela.dyn) == 0, "Unexpected run-time relocations (.rela) detected!")
|
2020-08-21 19:43:04 +00:00
|
|
|
}
|
2009-04-29 07:47:18 +00:00
|
|
|
|
2009-10-16 05:18:46 +00:00
|
|
|
/*
|
|
|
|
|
* The ASSERT() sink to . is intentional, for binutils 2.14 compatibility:
|
|
|
|
|
*/
|
2009-08-03 21:44:54 +00:00
|
|
|
. = ASSERT((_end - LOAD_OFFSET <= KERNEL_IMAGE_SIZE),
|
|
|
|
|
"kernel image bigger than KERNEL_IMAGE_SIZE");
|
x86/build: Fix vmlinux size check on 64-bit
Commit
b4e0409a36f4 ("x86: check vmlinux limits, 64-bit")
added a check that the size of the 64-bit kernel is less than
KERNEL_IMAGE_SIZE.
The check uses (_end - _text), but this is not enough. The initial
PMD used in startup_64() (level2_kernel_pgt) can only map upto
KERNEL_IMAGE_SIZE from __START_KERNEL_map, not from _text, and the
modules area (MODULES_VADDR) starts at KERNEL_IMAGE_SIZE.
The correct check is what is currently done for 32-bit, since
LOAD_OFFSET is defined appropriately for the two architectures. Just
check (_end - LOAD_OFFSET) against KERNEL_IMAGE_SIZE unconditionally.
Note that on 32-bit, the limit is not strict: KERNEL_IMAGE_SIZE is not
really used by the main kernel. The higher the kernel is located, the
less the space available for the vmalloc area. However, it is used by
KASLR in the compressed stub to limit the maximum address of the kernel
to a safe value.
Clean up various comments to clarify that despite the name,
KERNEL_IMAGE_SIZE is not a limit on the size of the kernel image, but a
limit on the maximum virtual address that the image can occupy.
Signed-off-by: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20201029161903.2553528-1-nivedita@alum.mit.edu
2020-10-29 16:19:03 +00:00
|
|
|
|
|
|
|
|
#ifdef CONFIG_X86_64
|
2009-04-29 07:47:18 +00:00
|
|
|
/*
|
|
|
|
|
* Per-cpu symbols which need to be offset from __per_cpu_load
|
|
|
|
|
* for the boot processor.
|
|
|
|
|
*/
|
2018-12-19 19:01:43 +00:00
|
|
|
#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load
|
2009-04-29 07:47:18 +00:00
|
|
|
INIT_PER_CPU(gdt_page);
|
2019-04-14 16:00:06 +00:00
|
|
|
INIT_PER_CPU(fixed_percpu_data);
|
|
|
|
|
INIT_PER_CPU(irq_stack_backing_store);
|
2009-04-29 07:47:18 +00:00
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
2019-04-14 16:00:06 +00:00
|
|
|
. = ASSERT((fixed_percpu_data == 0),
|
|
|
|
|
"fixed_percpu_data is not at start of per-cpu area");
|
2009-04-29 07:47:18 +00:00
|
|
|
#endif
|
|
|
|
|
|
2023-11-21 16:07:34 +00:00
|
|
|
#ifdef CONFIG_MITIGATION_UNRET_ENTRY
|
2023-08-14 11:44:32 +00:00
|
|
|
. = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned");
|
2023-06-28 09:02:39 +00:00
|
|
|
#endif
|
|
|
|
|
|
2023-11-21 16:07:36 +00:00
|
|
|
#ifdef CONFIG_MITIGATION_SRSO
|
2023-09-05 05:05:00 +00:00
|
|
|
. = ASSERT((srso_safe_ret & 0x3f) == 0, "srso_safe_ret not cacheline-aligned");
|
2023-06-28 09:02:39 +00:00
|
|
|
/*
|
2023-08-09 16:40:26 +00:00
|
|
|
* GNU ld cannot do XOR until 2.41.
|
|
|
|
|
* https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=f6f78318fca803c4907fb8d7f6ded8295f1947b1
|
|
|
|
|
*
|
|
|
|
|
* LLVM lld cannot do XOR until lld-17.
|
|
|
|
|
* https://github.com/llvm/llvm-project/commit/fae96104d4378166cbe5c875ef8ed808a356f3fb
|
|
|
|
|
*
|
|
|
|
|
* Instead do: (A | B) - (A & B) in order to compute the XOR
|
2023-06-28 09:02:39 +00:00
|
|
|
* of the two function addresses:
|
|
|
|
|
*/
|
2023-08-14 11:44:33 +00:00
|
|
|
. = ASSERT(((ABSOLUTE(srso_alias_untrain_ret) | srso_alias_safe_ret) -
|
|
|
|
|
(ABSOLUTE(srso_alias_untrain_ret) & srso_alias_safe_ret)) == ((1 << 2) | (1 << 8) | (1 << 14) | (1 << 20)),
|
2023-06-28 09:02:39 +00:00
|
|
|
"SRSO function pair won't alias");
|
2023-05-15 14:07:26 +00:00
|
|
|
#endif
|
|
|
|
|
|
x86/build: Fix vmlinux size check on 64-bit
Commit
b4e0409a36f4 ("x86: check vmlinux limits, 64-bit")
added a check that the size of the 64-bit kernel is less than
KERNEL_IMAGE_SIZE.
The check uses (_end - _text), but this is not enough. The initial
PMD used in startup_64() (level2_kernel_pgt) can only map upto
KERNEL_IMAGE_SIZE from __START_KERNEL_map, not from _text, and the
modules area (MODULES_VADDR) starts at KERNEL_IMAGE_SIZE.
The correct check is what is currently done for 32-bit, since
LOAD_OFFSET is defined appropriately for the two architectures. Just
check (_end - LOAD_OFFSET) against KERNEL_IMAGE_SIZE unconditionally.
Note that on 32-bit, the limit is not strict: KERNEL_IMAGE_SIZE is not
really used by the main kernel. The higher the kernel is located, the
less the space available for the vmalloc area. However, it is used by
KASLR in the compressed stub to limit the maximum address of the kernel
to a safe value.
Clean up various comments to clarify that despite the name,
KERNEL_IMAGE_SIZE is not a limit on the size of the kernel image, but a
limit on the maximum virtual address that the image can occupy.
Signed-off-by: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20201029161903.2553528-1-nivedita@alum.mit.edu
2020-10-29 16:19:03 +00:00
|
|
|
#endif /* CONFIG_X86_64 */
|