Apart from bumping the toolchain Clang's and port's version, this commit
completely overhauls the way LLVM toolchain is built.
First, it gets rid of a complicated two-stage process of first compiling
clang and compiler-rt builtins and then building libunwind, libc++abi,
and libc++ -- it is possible to create a complete cross-compilation
toolchain in a single CMake invocation with a modern LLVM. Moreover, the
old method was inherently unsupported and subtly broken.
Next, it utilizes full potential of the Stubs "framework". Now we are
even able to compile Clang with -Wl,-z,defs which makes one of the
patches obsolete and the whole installation less error-prone. Note that
it comes at a cost of complicating the bootstrap process on a completely
novel architecture but this hopefully won't happen often.
Lastly, it fixes handling of the -no*lib* family of flags in the
Serenity LLVM driver and correctly uses -nostartfiles in conjunction
with stubs to make necessary CMake configure-time checks succeed.
A Python script is much easier to maintain than the organically grown
variable mess that was run.sh.
For now, the script inherits most environment variable modifiability
from the shell script, but this is not a requirement.
While porting this script, a couple of improvements have been made:
- Spaces (especially in paths) cannot break most arguments anymore.
Exceptions are environment variables specifying multiple arguments on
purpose, these should be replaced in the future anyways.
- Force control over virtualization is now possible with
SERENITY_VIRTUALIZATION_SUPPORT. If set to 0, this variable was
sometimes ignored before.
- Handling Windows native QEMU is much more robust. Multiple incorrect
checks for WSL, but not Windows native QEMU, were used before. This
would also allow disabling native Windows QEMU much more easily in the
future, which is necessary for GDB.
- Various machine types had wrong or outdated arguments, such as qn.
Co-Authored-By: Andrew Kaster <akaster@serenityos.org>
This makes CMake pass `-fpie` instead of `-fpic` to the compiler when
building the Kernel and userland *executables*. This allows the compiler
to make certain optimizations based on the fact that the code will be
used in an executable, such as not having to emit `.localalias` symbols.
This leads to a 450 KiB decrease in the size of the Kernel binary.
This is a minor bugfix release, which to my knowledge contains nothing
of importance to us. However, there is one QoL change to our patches.
We no longer force `-fpic` in the compiler driver, and instead use the
`--enable-default-pie` configure option to generate position-independent
code suitable for executables. For building shared libraries, the
`-fpic` flag must be specified explicitly.
While LLD and mold support RELR "packed" relocations on all
architectures, the BFD linker currently only implements them on x86-64
and POWER.
This fixes two issues:
- The Kernel had it enabled even for AArch64 + GCC, which led to the
following being printed: `warning: -z pack-relative-relocs ignored`.
- The userland always had it disabled, even in the supported AArch64 +
Clang/mold scenarios.
Since https://reviews.llvm.org/D131441, libc++ must be included before
LibC. As clang includes libc++ as one of the system includes, LibC
must be included after those, and the only correct way to do that is
to install LibC's headers into the sysroot.
Targets that don't link with LibC yet require its headers for one
reason or another must add install_libc_headers as a dependency to
ensure that the correct headers have been (re)installed into the
sysroot.
LibC/stddef.h has been dropped since the built-in stddef.h receives
a higher include priority.
In addition, string.h and wchar.h must
define __CORRECT_ISO_CPP_STRING_H_PROTO and
_LIBCPP_WCHAR_H_HAS_CONST_OVERLOADS respectively in order to tell
libc++ to not try to define methods implemented by LibC.
This is a mostly straight-forward rebase of our patches on top of
13.1.0. The spec files needed a change, as GCC no longer supports STABS
debug information, but we were building GCC with support for it.
Highlights of this release include static `operator()`, The Equality
Operator You Are Looking For and extended `constexpr` support.
The install() command used by 1e36d54493
installs the provided file into the *directory* named by the DESTINATION
parameter. So if we ask it to install pci.ids to /res/pci.ids, the final
destination will be /res/pci.ids/pci.ids.
`file(COPY_FILE ...)` fails if the target directory does not already
exist, whereas `install(FILES ... DESTINATION ...)` happily creates all
the required directories.
This ensures we only need to download these files once for all build
configurations. We similarly download the UCD, CLDR, and TZDB to this
cache directory as well.
"image" was an alias for "qemu-image".
I want to add an `image` userland utility, which clashes with that
shortname.
So remove the existing "image" target. It was just an alias for
"qemu-image".
If you use serenity.sh to build, nothing changes. This only affects you
if you run ninja manually -- you now have to say `ninja qemu-image` to
build the disk image.
Clean up the Wasm spec tests CMake rules to extract and compile the wat
files into wasm files in the LibWasm binary directory instead of its
source directory. Also make the rules more robust to missing host tools,
and use more CMake install rules for the test files rather than relying
on build-root-filesystem.sh. Add some FIXMEs for later, we really
shouldn't be doing installation of test files into /home/anon at the
build-root-filesystem stage in $CURRENT_YEAR. Tests go in /usr/Tests
Tell CMake to not create a new policy scope for the
(lagom|serenity|common)_options.cmake helpers, and lets us set common
policies for both projects in common_options.cmake that actually apply
to the rest of the project, instead of just common_options.cmake itself.
Problem: cmake cannot handle gzip files (see
https://gitlab.kitware.com/cmake/cmake/-/issues/23054 for more
details).
Instead of downloading the compressed (*.gz) USB and PCI ids,
we not download the raw uncompressed files. The sizes we "loose"
due to downloading such files are meaningless.
This are the file sizes:
```
diego@debian:~/$ ls -lh pci.ids{,.gz} usb.ids{,.gz}
-rw-r--r-- 1 diego diego 1.3M Aug 7 04:15 pci.ids
-rw-r--r-- 1 diego diego 300K Aug 7 04:15 pci.ids.gz
-rw-r--r-- 1 diego diego 700K May 20 22:34 usb.ids
-rw-r--r-- 1 diego diego 245K May 20 22:34 usb.ids.gz
```
Newer cmake's have internal functions to un-compress files. These
functions will work on pure windows - as well as linux. This
eliminates the need to search for external tools (TAR,GZIP,ZIP) - and
helps fixing #9866.
In order to finally fix#9866 we need to decide to bump the cmake
version requirements and remove the checks. If we demand a newer cmake
version, we will loose Ubuntu 20.04 as a build target - as it ships
with CMake 3.16.
For now - we keep compatibility with CMake 3.16 - and only if CMake
3.18 as been found - we use its new functionality.
Remove the Corrosion dependency, and use the now-builtin
add_jakt_executable function from the Jakt install rules to build our
example application.
By using find_package(Jakt), we now have to set ENABLE_JAKT manually on
both serenity and Lagom at the same time, so the preferred method to do
this for now is:
cmake -B Build/superbuild<arch><toolchain> \
-S Meta/CMake/Superbuild \
-DENABLE_JAKT=ON \
-DJAKT_SOURCE_DIR=/path/to/jakt
Where omitting JAKT_SOURCE_DIR will still pull from the main branch of
SerenityOS/jakt. This can be done after runing Meta/serenity.sh run.
The FLAC "spec tests", or rather the test suite by xiph that exercises
weird FLAC features and edge cases, can be found at
https://github.com/ietf-wg-cellar/flac-test-files and is a good
challenge for our FLAC decoder to become more spec compliant. Running
these tests is similar to LibWasm spec tests, you need to pass
INCLUDE_FLAC_SPEC_TESTS to CMake.
As of integrating these tests, 23 out of 63 fail. :yakplus:
The current emoji_txt.cmake does not handle download errors (which were
a common source of issues in the build problems channel) or Unicode
versioning. These are both handled by unicode_data.cmake. Move the
download to unicode_data.cmake so that we can more easily handle next
month's Unicode 15 release.
Instead of manually updating emoji.txt whenever new emoji are added,
we use Unicode's emoji-test.txt to generate emoji.txt on each build,
including only the emojis that Serenity supports at that time.
By using emoji-test.txt, we can also include all forms of each emoji
(fully-qualified, minimally-qualified, and unqualified) which can be
helpful when double-checking how certain forms are handled.
Not sure why that happens and how it worked until now, but we need to be
more precise about the location of PCI and USB IDs when decompressing
them while building the OS.
In commit 02e8f29560 we started exporting
the `CMAKE_INSTALL_*DIR` variables without prefix in order to make
better use of the actual `PREFIX` settings.
However, commands like `file(MAKE_DIRECTORY ...)` don't understand the
GNUInstallDirs way of building paths, so we ended up creating
directories in our main project directory by accident.
Fix that by manually adding the correct prefix onto the path.
In preparation for future refactoring of Lagom, let's use the variables
from GNUInstallDirs as much as possible for the helper macros and other
scripts used by the main build already.
This release brings support for various C++23 constructs like `if
consteval` and multidimensional subscript operators. Vectorization is
now enabled for O2 too, and `-ftrivial-auto-var-init` has been added
which can help us find and prevent security issues coming from
uninitialized variables.
Toolchain/Patches/gcc.patch is now significanly smaller as some unused,
autoconf-generated code has been removed.
Besides a version bump, the following changes have been made to our
toolchain infrastructure:
- LLVM/Clang is now built with -march=native if the host compiler
supports it. An exception to this is CI, as the toolchain cache is
shared among many different machines there.
- The LLVM tarball is not re-extracted if the hash of the applied
patches doesn't differ.
- The patches have been split up into atomic chunks.
- Port-specific patches have been integrated into the main patches,
which will aid in the work towards self-hosting.
- <sysroot>/usr/local/lib is now appended to the linker's search path by
default.
- --pack-dyn-relocs=relr is appended to the linker command line by
default, meaning ports take advantage of RELR relocations without any
patches or additional compiler flags.
The formatting of LLVM port's package.sh has been bothering me, so I
also indented the arguments to the CMake invocation.
We have seen some cases where the build fails for folks, and they are
missing unzip/tar/gzip etc. We can catch some of these in CMake itself,
so lets make sure to handle that uniformly across the build system.
The REQUIRED flag to `find_program` was only added on in CMake 3.18 and
above, so we can't rely on that to actually halt the program execution.
With regular builds, the generated IPC headers exist inside the Build
directory. The path Userland/Services under the build directory is
added to the include path.
For in-system builds the IPC headers are installed at /usr/include/.
To support this, we add /usr/include/Userland/Services to the build path
when building from Hack Studio.
Co-Authored-By: Andrew Kaster <akaster@serenityos.org>
This variable was originally called USE_MOLD_LINKER, but it was changed
to ENABLE_MOLD_LINKER during review to be consistent with other
configuration options. I branched off the commits that added RELR
support before this change, and I failed to update the variable name
there.
While playing around with getting serenity to run on my main desktop
machine I wanted a way of easily updating my physical serenity
partition.
To use it you just need to:
- Create and format your local partition to ext4
- Set `SERENITY_TARGET_INSTALL_PARTITION` to the partition /dev path.
- Run the `install-native-partition` build target.
Example:
$ export SERENITY_TARGET_INSTALL_PARTITION=/dev/nvme1n1p3
$ cd serenity/Build/x86_64
$ ninja install-native-partition
