It seems more correct to let each platform to define its own sequence of
initialization of the PCI bus, so let's remove the #if flags and just
put the entire Initializer.cpp file in the appropriate code directory.
The simple PCI::HostBridge class implements access to the PCI
configuration space by using x86 IO instructions. Therefore, it should
be put in the Arch/x86/PCI directory so it can be easily omitted for
non-x86 builds.
kprintf should not really care about the hardware-specific details of
each UART or serial port out there, so instead of using x86 specific
instructions, let's ensure that we will compile only the relevant code
for debug output for a targeted-specific platform.
The RTC and CMOS are currently only supported for x86 platforms and use
specific x86 instructions to produce only certain x86 plaform operations
and results, therefore, we move them to the Arch/x86 specific directory.
Many code patterns and hardware procedures rely on reliable delay in the
microseconds granularity, and since they are using such delays which are
valid cases, but should not rely on x86 specific code, we allow to
determine in compile time the proper platform-specific code to use to
invoke such delays.
We move QEMU and VirtualBox shutdown sequences to a separate file, as
well as moving the i8042 reboot code sequence too to another file.
This allows us to abstract specific methods from the power state node
code of the SysFS filesystem, to allow other architectures to put their
methods there too in the future.
Before of this patch, we supported two methods to address a boot device:
1. Specifying root=/dev/hdXY, where X is a-z letter which corresponds to
a boot device, and Y as number from 1 to 16, to indicate the partition
number, which can be omitted to instruct the kernel to use a raw device
rather than a partition on a raw device.
2. Specifying root=PARTUUID: with a GUID string of a GUID partition. In
case of existing storage device with GPT partitions, this is most likely
the safest option to ensure booting from persistent storage.
While option 2 is more advanced and reliable, the first option has 2
caveats:
1. The string prefix "/dev/hd" doesn't mean anything beside a convention
on Linux installations, that was taken into use in Serenity. In Serenity
we don't mount DevTmpFS before we mount the boot device on /, so the
kernel doesn't really access /dev anyway, so this convention is only a
big misleading relic that can easily make the user to assume we access
/dev early on boot.
2. This convention although resemble the simple linux convention, is
quite limited in specifying a correct boot device across hardware setup
changes, so option 2 was recommended to ensure the system is always
bootable.
With these caveats in mind, this commit tries to fix the problem with
adding more addressing options as well as to remove the first option
being mentioned above of addressing.
To sum it up, there are 4 addressing options:
1. Hardware relative address - Each instance of StorageController is
assigned with a index number relative to the type of hardware it handles
which makes it possible to address storage devices with a prefix of the
commandset ("ata" for ATA, "nvme" for NVMe, "ramdisk" for Plain memory),
and then the number for the parent controller relative hardware index,
another number LUN target_id, and a third number for LUN disk_id.
2. LUN address - Similar to the previous option, but instead we rely on
the parent controller absolute index for the first number.
3. Block device major and minor numbers - by specifying the major and
minor numbers, the kernel can simply try to get the corresponding block
device and use it as the boot device.
4. GUID string, in the same fashion like before, so the user use the
"PARTUUID:" string prefix and add the GUID of the GPT partition.
For the new address modes 1 and 2, the user can choose to also specify a
partition out of the selected boot device. To do that, the user needs to
append the semicolon character and then add the string "partX" where X
is to be changed for the partition number. We start counting from 0, and
therefore the first partition number is 0 and not 1 in the kernel boot
argument.
Now that the Spinlock code is not dependent on architectural specific
code anymore, we can move it back to the Locking folder. This also means
that the Spinlock implemenation is now used for the aarch64 kernel.
This commit adds the concept of an InterruptsState to the kernel. This
will be used to make the Spinlock code architecture independent. A new
Processor.cpp file is added such that we don't have to duplicate the
code.
This patch adds a new object to hold a Process's user credentials:
- UID, EUID, SUID
- GID, EGID, SGID, extra GIDs
Credentials are immutable and child processes initially inherit the
Credentials object from their parent.
Whenever a process changes one or more of its user/group IDs, a new
Credentials object is constructed.
Any code that wants to inspect and act on a set of credentials can now
do so without worrying about data races.
Unlike Clang, GCC does not support 8-byte atomics on i686 with the
-mno-80387 flag set, so until that is fixed, implement a minimal set of
atomics that are currently required.
We don't really support ATAPI (SCSI packets over ATA channels) and it's
uncertain if we ever will support such type of media. For this reason,
there's basically no reason to keep this code.
If we ever introduce ATAPI support into the Kernel, we can simply put
this back into the codebase.
Everything in Kernel/Storage/Partition but DiskPartition has been moved
into LibPartiton. This makes the Partition directory unnecessary so
DiskPartition is moved up into Kernel/Storage.
This commit creates a new library LibPartition which will contain
partition related code sharable between Kernel and Userland and
includes DiskPartitionMetadata as the first shared class.
This abstraction layer is mainly for ATA ports (AHCI ports, IDE ports).
The goal is to create a convenient and flexible framework so it's
possible to expand to support other types of controller (e.g. Intel PIIX
and ICH IDE controllers) and to abstract operations that are possible on
each component.
Currently only the ATA IDE code is affected by this, making it much
cleaner and readable - the ATA bus mastering code is moved to the
ATAPort code so more implementations in the near future can take
advantage of such functionality easily.
In addition to that, the hierarchy of the ATA IDE code resembles more of
the SATA AHCI code now, which means the IDEChannel class is solely
responsible for getting interrupts, passing them for further processing
in the ATAPort code to take care of the rest of the handling logic.
We do that to increase clarity of the major and secondary components in
the subsystem. To ensure it's even more understandable, we rename the
files to better represent the class within them and to remove redundancy
in the name.
Also, some includes are removed from the general components of the ATA
components' classes.
Under normal conditions (when mounting SysFS in /sys), there will be a
new directory in the /sys/devices directory called "graphics".
For now, under that directory there will be only a sub-directory called
"connectors" which will contain all DisplayConnectors' details, each in
its own sub-directory too, distinguished in naming with its minor
number.
Therefore, /sys/devices/graphics/connectors/MINOR_NUMBER/ will contain:
- General device attributes such as mutable_mode_setting_capable,
double_buffering_capable, flush_support, partial_flush_support and
refresh_rate_support. These values are exposed in the ioctl interface
of the DisplayConnector class too, but these can be useful later on
for command line utilities that want/need to expose these basic
settings.
- The EDID blob, simply named "edid". This will help userspace to fetch
the edid without the need of using the ioctl interface later on.
This change in fact does the following:
1. Use support for symlinks between /sys/dev/block/ storage device
identifier nodes and devices in /sys/devices/storage/{LUN}.
2. Add basic nodes in a /sys/devices/storage/{LUN} directory, to let
userspace to know about the device and its details.
LUN address is essentially how people used to address SCSI devices back
in the day we had these devices more in use. However, SCSI was taken as
an abstraction layer for many Unix and Unix-like systems, so it still
common to see LUN addresses in use. In Serenity, we don't really provide
such abstraction layer, and therefore until now, we didn't use LUNs too.
However (again), this changes, as we want to let users to address their
devices under SysFS easily. LUNs make sense in that regard, because they
can be easily adapted to different interfaces besides SCSI.
For example, for legacy ATA hard drive being connected to the first IDE
controller which was enumerated on the PCI bus, and then to the primary
channel as slave device, the LUN address would be 0:0:1.
To make this happen, we add unique ID number to each StorageController,
which increments by 1 for each new instance of StorageController. Then,
we adapt the ATA and NVMe devices to use these numbers and generate LUN
in the construction time.
This folder in the SysFS code represents everything related to /sys/dev,
which is a directory meant to be a convenient interface to track all IDs
of all block and character devices (ID = major:minor numbers).
The original intention was to support other types of consoles based on
standard VGA modes, but it never came to an implementation, nor we need
such feature at all.
Therefore, this class is not needed and can be removed.
We never supported VGA framebuffers and that folder was a big misleading
part of the graphics subsystem.
We do support bare-bones VGA text console (80x25), but that only happens
to be supported because we can't be 100% sure we can always initialize
framebuffer so in the worst scenario we default to plain old VGA console
so the user can still use its own machine.
Therefore, the only remaining parts of VGA is in the GraphicsManagement
code to help driving the VGA text console if needed.
In the same fashion like in the Linux kernel, we support pre-initialized
framebuffers that were set up by either the BIOS or the bootloader.
These framebuffers can be backed by any kind of video hardware, and are
not tied to VGA hardware at all. Therefore, this code should be in a
separate sub-folder in the Graphics subsystem to indicate this.
The flag will automatically initialize all variables to a pattern based
on it's type. The goal being here is to eradicate an entire bug class
of issues that can originate from uninitialized stack memory.
Some examples include:
- Kernel information disclosure, where uninitialized struct members
or struct padding is copied back to usermode, leaking kernel
information such as stack or heap addresses, or secret data like
stack cookies.
- Control flow based on uninitialized memory can cause a variety of
issues at runtime, including stack corruptions like buffer
overflows, heap corruptions due to deleting stray pointers.
Even basic logic bugs can result from control flow operating on
uninitialized data.
As of GCC 12 this flag is now supported.
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=a25e0b5e6ac8a77a71c229e0a7b744603365b0e9
Clang has already supported it for a few releases.
https://reviews.llvm.org/D54604
In a previous commit I moved everything into the new subdirectories in
FileSystem/SysFS directory without trying to actually make changes in
the code itself too much. Now it's time to split the code to make it
more readable and understandable, hence this change occurs now.
This new type of VMObject will be used to coordinate switching safely
from graphical mode to text mode and vice-versa, by supplying a way to
remap all Regions that were created with this object, so mappings can be
changed according to the given state of system mode. This makes it quite
easy to give applications like WindowServer the feeling of having full
access to the framebuffer device from a DisplayConnector, but still keep
the Kernel in control to be able to safely switch to text console.
These 2 classes currently contain much code that is x86(_64) specific.
Move them to the architecture specific directory. This also allows for a
simpler implementation for aarch64.
This requires us to add an Interrupts.h file in the Kernel/Arch
directory, which includes the architecture specific files.
The commit also stubs out the functions to be able to compile the
aarch64 Kernel.
With the update to GCC 12.1.0, the compiler now vectorizes code with
-O2. This causes vector ops to be emitted, which are not supported in
the Kernel. Add the -mgeneral-regs-only flag to force the compiler to
not emit floating-point and SIMD ops.
By default we enable the Kernel Undefined Behavior Sanitizer, which
checks for undefined behavior at runtime. However, sometimes a developer
might want to turn that off, so now there is a easy way to do that.
Most of the string.h and wchar.h functions are implemented quite naively
at the moment, and GCC's pattern recognition pass might realize what we
are trying to do, and transform them into libcalls. This is usually a
useful optimization, but not when we're implementing the functions
themselves :^)
Relevant discussion from the GCC Bugzilla:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102725
This prevents the infamous recursive `strlen`.
A more proper fix would be writing these functions in assembly. That
would likely give a small performance boost as well ;)
Instead of storing the current Processor into a core local register, we
currently just store it into a global, since we don't support SMP for
aarch64 anyway. This simplifies the initial implementation.
The DisplayConnector class is meant to replace the FramebufferDevice
class. The advantage of this class over the FramebufferDevice class is:
1. It removes the mmap interface entirely. This interface is unsafe, as
multiple processes could try to use it, and when switching to and from
text console mode, there's no "good" way to revoke a memory mapping from
this interface, let alone when there are multiple processes that call
this interface. Therefore, in the DisplayConnector class there's no
implementation for this method at all.
2. The class uses a new real-world structure called ModeSetting, which
takes into account the fact that real hardware requires more than width,
height and pitch settings to mode-set the display resolution.
3. The class assumes all instances should supply some sort of EDID,
so it facilitates such mechanism to do so. Even if a given driver does
not know what is the actual EDID, it will ask to create default-generic
EDID blob.
3. This class shifts the responsibilies of switching between console
mode and graphical mode from a GraphicsAdapter to the DisplayConnector
class, so when doing the switch, the GraphicsManagement code actually
asks each DisplayConnector object to do the switch and doesn't rely on
the GraphicsAdapter objects at all.
The code in Spinlock.h has no architectural specific logic, thus can be
moved to the Arch directory. This contains no functional change.
Also add the Spinlock.cpp file for aarch64 which contains stubs for the
lock and unlock functions.
This allows us to use the AK formatting functions in the aarch64 Kernel.
Also add FIXME to make sure that this file will be removed when the
proper abstractions are in place in the normal Kernel/kprintf.cpp.
The compiler figured out that the MemoryManager is not initialised, and
thus MemoryManager::the() cannot return a valid reference. Once the
necesarry code is in place, this compiler flag can be removed.
Coverage tools like LLVM's source-based coverage or GNU's --coverage
need to be able to write out coverage files from any binary, regardless
of its security posture. Not ignoring these pledges and veils means we
can't get our coverage data out without playing some serious tricks.
However this is pretty terrible for normal exeuction, so only skip these
checks when we explicitly configured userspace for coverage.
This creates all interfaces when the device is enumerated, with a link
to the configuration that it is a part of. As such, a new class,
`USBInterface` has been introduced to express this state.
This patch ports MemoryManager to RegionTree as well. The biggest
difference between this and the userspace code is that kernel regions
are owned by extant OwnPtr<Region> objects spread around the kernel,
while userspace regions are owned by the AddressSpace itself.
For kernelspace, there are a couple of situations where we need to make
large VM reservations that never get backed by regular VMObjects
(for example the kernel image reservation, or the big kmalloc range.)
Since we can't make a VM reservation without a Region object anymore,
this patch adds a way to create unbacked Region objects that can be
used for this exact purpose. They have no internal VMObject.)
RegionTree holds an IntrusiveRedBlackTree of Region objects and vends a
set of APIs for allocating memory ranges.
It's used by AddressSpace at the moment, and will be used by MM soon.
We're now able to detect all the regular CPUID feature flags from
ECX/EDX for EAX=1 :^)
None of the new ones are being used for anything yet, but they will show
up in /proc/cpuinfo and subsequently lscpu and SystemMonitor.
Note that I replaced the periods from the SSE 4.1 and 4.2 instructions
with underscores, which matches the internal enum names, Linux's
/proc/cpuinfo and the general pattern of replacing special characters
with underscores to limit feature names to [a-z0-9_].
The enum member stringification has been moved to a new function for
better re-usability and to avoid cluttering up Processor.cpp.
The obsolete ttyname and ptsname syscalls are removed.
LibC doesn't rely on these anymore, and it helps simplifying the Kernel
in many places, so it's an overall an improvement.
In addition to that, /proc/PID/tty node is removed too as it is not
needed anymore by userspace to get the attached TTY of a process, as
/dev/tty (which is already a character device) represents that as well.
This will replace the /dev/tty symlink created by SystemServer, so
instead of a symlink, a character device will be created. When doing
read(2), write(2) and ioctl(2) on this device, it will "redirect" these
operations to the attached TTY of the current process.
As there is no need for a Prekernel on aarch64, the Prekernel code was
moved into Kernel itself. The functionality remains the same.
SERENITY_KERNEL_AND_INITRD in run.sh specifies a kernel and an inital
ramdisk to be used by the emulator. This is needed because aarch64
does not need a Prekernel and the other ones do.
This commit flips VirtIOGPU back to using a Mutex for its operation
lock (instead of a spinlock). This is necessary for avoiding a few
system hangs when queuing actions on the driver from multiple
processes, which becomes much more of an issue when using VirGL from
multiple userspace process.
This does result in a few code paths where we inevitably have to grab
a mutex from inside a spinlock, the only way to fix both issues is to
move to issuing asynchronous virtio gpu commands.
If there's no PCI bus, then it's safe to assume that we run on a x86
machine that has an ISA IDE controller in the system. In such case, we
just instantiate a ISAIDEController object that assumes fixed locations
of IDE IO ports.
If there's no PCI bus, then it's safe to assume that the x86 machine we
run on supports VGA text mode console output with an ISA VGA adapter.
If this is the case, we just instantiate a ISAVGAAdapter object that
assumes this situation and allows us to boot into VGA text mode console.
This driver is not tested and probably not used on any modern hardware
machine, because it is plugged into the ISA bus and not the PCI bus.
Also, the run script doesn't utilize this device anymore, making it more
hard to test this driver and to ensure it doesn't rot.
We have 3 new components:
1. The AudioManagement singleton. This class like in other subsystems,
is responsible to find hardware audio controllers and keep a reference
to them.
2. AudioController class - this class is the parent class for hardware
controllers like the Sound Blaster 16 or Intel 82801AA (AC97). For now,
this class has simple interface for getting and controlling sample rate
of audio channels, as well a write interface for specific audio channel
but not reading from it. One AudioController object might have multiple
AudioChannel "child" objects to hold with reference counting.
3. AudioChannel class - this is based on the CharacterDevice class, and
represents hardware PCM audio channel. It facilitates an ioctl interface
which should be consistent across all supported hardware currently.
It has a weak reference to a parent AudioController, and when trying to
write to a channel, it redirects the data to the parent AudioController.
Each audio channel device should be added into a new directory under the
/dev filesystem called "audio".
Instead of seeing a black screen until GraphicsManagement was fully
initialized, this allows us to see the console output much earlier.
So, if the bootloader provided us with a framebuffer, set up a console
as early as possible.
Add polling support to NVMe so that it does not use interrupt to
complete a IO but instead actively polls for completion. This probably
is not very efficient in terms of CPU usage but it does not use
interrupts to complete a IO which is beneficial at the moment as there
is no MSI(X) support and it can reduce the latency of an IO in a very
fast NVMe device.
The NVMeQueue class has been made the base class for NVMeInterruptQueue
and NVMePollQueue. The factory function `NVMeQueue::try_create` will
return the appropriate queue to the controller based on the polling
boot parameter.
The polling mode can be enabled by adding an extra boot parameter:
`nvme_poll`.
This device will assist userspace to manage hotplug events.
A userspace application reads a DeviceEvent entry until the return value
is zero which indicates no events that are queued and waiting for
processing.
Trying to read with a buffer smaller than sizeof(DeviceEvent) results in
EOVERFLOW.
For now, there's no ioctl mechanism for this device but in the future an
acknowledgement mechanism can be implemented via ioctl(2) interface.
This was easily done, as the Kernel and Userland don't actually share
any of the APIs exposed by it, so instead the Kernel APIs were moved to
the Kernel, and the Userland APIs stayed in LibKeyboard.
This has multiple advantages:
* The non OOM-fallible String is not longer used for storing the
character map name in the Kernel
* The kernel no longer has to link to the userland LibKeyboard code
* A lot of #ifdef KERNEL cruft can be removed from LibKeyboard
Two classes are added - HostBridge and MemoryBackedHostBridge, which
both derive from HostController class. This allows the kernel to map
different busses from different PCI domains in the same time. Each
HostController implementation doesn't take the Address object to address
PCI devices but instead we take distinct numbers of the PCI bus, device
and function as it allows us to specify arbitrary PCI domains in the
Address structure and still to get the correct PCI devices. This also
matches the hardware behavior of PCI domains - the host bridge merely
takes memory operations or IO operations and translates them to
addressing of three components - PCI bus, device and function.
These changes also greatly simplify how enumeration of Host Bridges work
now - scanning of the hardware depends on what the Host bridges can do
for us, so in case we have multiple host bridges that expose a memory
mapped region or IO ports to access PCI configuration space, we simply
let the code of the host bridge to figure out how to fetch data for us.
Another semantical change is that a PCI domain structure is no longer
attached to a PhysicalAddress, so even in the case that the machine
doesn't implement PCI domains, we still treat that machine to contain 1
PCI domain to treat that one host bridge in the same way, like with a
machine with one or more PCI domains.
Add a basic NVMe driver support to serenity
based on NVMe spec 1.4.
The driver can support multiple NVMe drives (subsystems).
But in a NVMe drive, the driver can support one controller
with multiple namespaces.
Each core will get a separate NVMe Queue.
As the system lacks MSI support, PIN based interrupts are
used for IO.
Tested the NVMe support by replacing IDE driver
with the NVMe driver :^)
By using the binary from our build of binutils, we can be sure that `nm`
supports demangling symbols, so we can avoid spawning a separate
`c++filt` process.
We used to build with -Os in order to fit within a certain size, but
there isn't really a good reason for that kind of restriction.
Switching to -O2 yields a significant improvement in throughput,
for example `test-js` is roughly 20% faster on my machine. :^)
Now that the userland has a compatiblity wrapper for select(), the
kernel doesn't need to implement this syscall natively. The poll()
interface been around since 1987, any code still using select()
should be slapped silly.
Note: the SerenityOS source tree mostly uses select() and not poll()
despite SerenityOS having support for poll() since early 2019...
Like what happened with the PCI and USB code, this feels like the right
thing to do because we can improve on the ATA capabilities and keep it
distinguished from the rest of the subsystem.
We create a base class called GenericFramebufferDevice, which defines
all the virtual functions that must be implemented by a
FramebufferDevice. Then, we make the VirtIO FramebufferDevice and other
FramebufferDevice implementations inherit from it.
The most important consequence of rearranging the classes is that we now
have one IOCTL method, so all drivers should be committed to not
override the IOCTL method or make their own IOCTLs of FramebufferDevice.
All graphical IOCTLs are known to all FramebufferDevices, and it's up to
the specific implementation whether to support them or discard them (so
we require extensive usage of KResult and KResultOr, together with
virtual characteristic functions).
As a result, the interface is much cleaner and understandable to read.
A VirtIO graphics adapter is really the VirtIO GPU, so the virtualized
hardware has no distinction between both components so there's no
need to put such distinction in software.
We might need to split things in the future, but when we do so, we must
take proper care to ensure that the interface between the components
is correct and use the usual codebase patterns.
By enabling LTO for the kernel_heap object too, we open the door for
optimization opportunities that come from (partially) inlining `::new`
or kmalloc. Every software spends a non-trivial amount of its run time
on allocating memory, so hopefully this change will make LTO builds even
faster.
This commit updates the Clang toolchain's version to 13.0.0, which comes
with better C++20 support and improved handling of new features by
clang-format. Due to the newly enabled `-Bsymbolic-functions` flag, our
Clang binaries will only be 2-4% slower than if we dynamically linked
them, but we save hundreds of megabytes of disk space.
The `BuildClang.sh` script has been reworked to build the entire
toolchain in just three steps: one for the compiler, one for GNU
binutils, and one for the runtime libraries. This reduces the complexity
of the build script, and will allow us to modify the CI configuration to
only rebuild the libraries when our libc headers change.
Most of the compile flags have been moved out to a separate CMake cache
file, similarly to how the Android and Fuchsia toolchains are
implemented within the LLVM repo. This provides a nicer interface than
the heaps of command-line arguments.
We no longer build separate toolchains for each architecture, as the
same Clang binary can compile code for multiple targets.
The horrible mess that `SERENITY_CLANG_ARCH` was, has been removed in
this commit. Clang happily accepts an `i686-pc-serenity` target triple,
which matches what our GCC toolchain accepts.
LLD fails to define the _GLOBAL_OFFSET_TABLE_ symbol if all inputs to it
are LLVM bitcode files (i.e. those used for LTO). To allow the kernel to
be built with ThinLTO, the workaround suggested in the original LLVM bug
report (<https://bugs.llvm.org/show_bug.cgi?id=39634>) is added in this
commit.
There's basically no real difference in software between a SATA harddisk
and IDE harddisk. The difference in the implementation is for the host
bus adapter protocol and registers layout.
Therefore, there's no point in putting a distinction in software to
these devices.
This change also greatly simplifies and removes stale APIs and removes
unnecessary parameters in constructor calls, which tighten things
further everywhere.
This singleton simplifies many aspects that we struggled with before:
1. There's no need to make derived classes of Device expose the
constructor as public anymore. The singleton is a friend of them, so he
can call the constructor. This solves the issue with try_create_device
helper neatly, hopefully for good.
2. Getting a reference of the NullDevice is now being done from this
singleton, which means that NullDevice no longer needs to use its own
singleton, and we can apply the try_create_device helper on it too :)
3. We can now defer registration completely after the Device constructor
which means the Device constructor is merely assigning the major and
minor numbers of the Device, and the try_create_device helper ensures it
calls the after_inserting method immediately after construction. This
creates a great opportunity to make registration more OOM-safe.
Replace the old logic where we would start with a host build, and swap
all the CMake compiler and target variables underneath it to trick
CMake into building for Serenity after we configured and built the Lagom
code generators.
The SuperBuild creates two ExternalProjects, one for Lagom and one for
Serenity. The Serenity project depends on the install stage for the
Lagom build. The SuperBuild also generates a CMakeToolchain file for the
Serenity build to use that replaces the old toolchain file that was only
used for Ports.
To ensure that code generators are rebuilt when core libraries such as
AK and LibCore are modified, developers will need to direct their manual
`ninja` invocations to the SuperBuild's binary directory instead of the
Serenity binary directory.
This commit includes warning coalescing and option style cleanup for the
affected CMakeLists in the Kernel, top level, and runtime support
libraries. A large part of the cleanup is replacing USE_CLANG_TOOLCHAIN
with the proper CMAKE_CXX_COMPILER_ID variable, which will no longer be
confused by a host clang compiler.
This change removes the halt and reboot syscalls, and create a new
mechanism to change the power state of the machine.
Instead of how power state was changed until now, put a SysFS node as
writable only for the superuser, that with a defined value, can result
in either reboot or poweroff.
In the future, a power group can be assigned to this node (which will be
the GroupID responsible for power management).
This opens an opportunity to permit to shutdown/reboot without superuser
permissions, so in the future, a userspace daemon can take control of
this node to perform power management operations without superuser
permissions, if we enforce different UserID/GroupID on that node.
Both should reside in the SysFS firmware directory which is normally
located in /sys/firmware.
Also, apply some OOM-safety patterns when creating the BIOS and ACPI
directories.
This will somwhat help unify them also under the same SysFS directory in
the commit.
Also, it feels much more like this change reflects the reality that both
ACPI and the BIOS are part of the firmware on x86 computers.
These interfaces are broken for about 9 months, maybe longer than that.
At this point, this is just a dead code nobody tests or tries to use, so
let's remove it instead of keeping a stale code just for the sake of
keeping it and hoping someone will fix it.
To better justify this, I read that OpenBSD removed loadable kernel
modules in 5.7 release (2014), mainly for the same reason we do -
nobody used it so they had no good reason to maintain it.
Still, OpenBSD had LKMs being effectively working, which is not the
current state in our project for a long time.
An arguably better approach to minimize the Kernel image size is to
allow dropping drivers and features while compiling a new image.
Let's remove the DynamicParser class, as it really did nothing yet in
the Kernel. Instead, when we add support for AML parsing, we can figure
out how to do it properly without the need of a derived class that just
complicates everything for no good reason.
The current implementation of DevFS resembles the linux devtmpfs, and
not the traditional DevFS, so let's rename it to better represent the
direction of the development in regard to this filesystem.
The abbreviation for DevTmpFS is still "dev", because it doesn't add
value as a commandline option to make it longer.
In quick summary - DevFS in unix OSes is simply a static filesystem, so
device nodes are generated and removed by the kernel code. DevTmpFS
is a "modern reinvention" of the DevFS, so it is much more like a TmpFS
in the sense that not only it's stored entirely in RAM, but the userland
is responsible to add and remove devices nodes as it sees fit, and no
kernel code is directly being involved to keep the filesystem in sync.
A couple of things were changed:
1. Semantic changes - PCI segments are now called PCI domains, to better
match what they are really. It's also the name that Linux gave, and it
seems that Wikipedia also uses this name.
We also remove PCI::ChangeableAddress, because it was used in the past
but now it's no longer being used.
2. There are no WindowedMMIOAccess or MMIOAccess classes anymore, as
they made a bunch of unnecessary complexity. Instead, Windowed access is
removed entirely (this was tested, but never was benchmarked), so we are
left with IO access and memory access options. The memory access option
is essentially mapping the PCI bus (from the chosen PCI domain), to
virtual memory as-is. This means that unless needed, at any time, there
is only one PCI bus being mapped, and this is changed if access to
another PCI bus in the same PCI domain is needed. For now, we don't
support mapping of different PCI buses from different PCI domains at the
same time, because basically it's still a non-issue for most machines
out there.
2. OOM-safety is increased, especially when constructing the Access
object. It means that we pre-allocating any needed resources, and we try
to find PCI domains (if requested to initialize memory access) after we
attempt to construct the Access object, so it's possible to fail at this
point "gracefully".
3. All PCI API functions are now separated into a different header file,
which means only "clients" of the PCI subsystem API will need to include
that header file.
4. Functional changes - we only allow now to enumerate the bus after
a hardware scan. This means that the old method "enumerate_hardware"
is removed, so, when initializing an Access object, the initializing
function must call rescan on it to force it to find devices. This makes
it possible to fail rescan, and also to defer it after construction from
both OOM-safety terms and hotplug capabilities.
This change adds a static lock hierarchy / ranking to the Kernel with
the goal of reducing / finding deadlocks when running with SMP enabled.
We have seen quite a few lock ordering deadlocks (locks taken in a
different order, on two different code paths). As we properly annotate
locks in the system, then these facilities will find these locking
protocol violations automatically
The `LockRank` enum documents the various locks in the system and their
rank. The implementation guarantees that a thread holding one or more
locks of a lower rank cannot acquire an additional lock with rank that
is greater or equal to any of the currently held locks.
Add a dummy Arch/aarch64/boot.S that for now does nothing but
let all processor cores sleep.
For now, none of the actual Prekernel code is built for aarch64.
The pattern of having Prekernel inherit all of the build flags of the
Kernel, and then disabling some flags by adding `-fno-<flag>` options
to then disable those options doesn't work in all scenarios. For example
the ASAN flag `-fasan-shadow-offset=<offset>` has no option to disable
it once it's been passed, so in a future change where this flag is added
we need to be able to disable it cleanly.
The cleaner way is to just allow the Prekernel CMake logic to filter out
the COMPILE_OPTIONS specified for that specific target. This allows us
to remove individual options without trashing all inherited options.
Now that the old PCI::Device was removed, we can complete the PCI
changes by making the PCI::DeviceController to be named PCI::Device.
Really the entire purpose and the distinction between the two was about
interrupts, but since this is no longer a problem, just rename it to
simplify things further.
I created this class a long time ago just to be able to quickly make a
PCI device to also represent an interrupt handler (because PCI devices
have this capability for most devices).
Then after a while I introduced the PCI::DeviceController, which is
really almost the same thing (a PCI device class that has Address member
in it), but is not tied to interrupts so it can have no interrupts, or
spawn interrupt handlers however it wants to seems fit.
However I decided it's time to say goodbye for this class for
a couple of reasons:
1. It made a whole bunch of weird patterns where you had a PCI::Device
and a PCI::DeviceController being used in the topic of implementation,
where originally, they meant to be used mutually exclusively (you
can't and really don't want to use both).
2. We can really make all the classes that inherit from PCI::Device
to inherit from IRQHandler at this point. Later on, when we have MSI
interrupts support, we can go further and untie things even more.
3. It makes it possible to simplify the VirtIO implementation to a great
extent. While this commit almost doesn't change it, future changes
can untangle some complexity in the VirtIO code.
For UHCIController, E1000NetworkAdapter, NE2000NetworkAdapter,
RTL8139NetworkAdapter, RTL8168NetworkAdapter, E1000ENetworkAdapter we
are simply making them to inherit the IRQHandler. This makes some sense,
because the first 3 devices will never support anything besides IRQs.
For the last 2, they might have MSI support, so when we start to utilize
those, we might need to untie these classes from IRQHandler and spawn
IRQHandler(s) or MSIHandler(s) as needed.
The VirtIODevice class is also a case where we currently need to use
both PCI::DeviceController and IRQHandler classes as parents, but it
could also be untied from the latter.
The number of UHCI related files is starting to expand to the point
where it's best if we move this into their own subdirectory. It'll
also make it easier to manage when we decide to add some more
controller types (whenever that may be)
The `-z,text` linker flag causes the linker to reject shared libraries
and PIE executables that have textrels. Our code mostly did not use
these except in one place in LibC, which is changed in this commit.
This makes GNU ld match LLD's behavior, which has this option enabled by
default.
TEXTRELs pose a security risk, as performing these relocations require
executable pages to be written to by the dynamic linker. This can
significantly weaken W^X hardening mitigations.
Note that after this change, TEXTRELs can still be used in ports, as the
dynamic loader code is not changed. There are also uses of it in the
kernel, removing which are outside the scope of this PR. To allow those,
`-z,notext` is added.
We are not using this for anything and it's just been sitting there
gathering dust for well over a year, so let's stop carrying all this
complexity around for no good reason.
This allows us to 1) let go of the Process when an inode is ref'ing for
ProcFSExposedComponent related reasons, and 2) change our ref/unref
implementation.
Currently, to append a UTF-16 view to a StringBuilder, callers must
first convert the view to UTF-8 and then append the copy. Add a UTF-16
overload so callers do not need to hold an entire copy in memory.
This removes Pipes dependency on the UHCIController by introducing a
controller base class. This will be used to implement other controllers
such as OHCI.
Additionally, there can be multiple instances of a UHCI controller.
For example, multiple UHCI instances can be required for systems with
EHCI controllers. EHCI relies on using multiple of either UHCI or OHCI
controllers to drive USB 1.x devices.
This means UHCIController can no longer be a singleton. Multiple
instances of it can now be created and passed to the device and then to
the pipe.
To handle finding and creating these instances, USBManagement has been
introduced. It has the same pattern as the other management classes
such as NetworkManagement.
Taking a reference or a pointer to a value that's not aligned properly
is undefined behavior. While `[[gnu::packed]]` ensures that reads from
and writes to fields of packed structs is a safe operation, the
information about the reduced alignment is lost when creating pointers
to these values.
Weirdly enough, GCC's undefined behavior sanitizer doesn't flag these,
even though the doc of `-Waddress-of-packed-member` says that it usually
leads to UB. In contrast, x86_64 Clang does flag these, which renders
the 64-bit kernel unable to boot.
For now, the `address-of-packed-member` warning will only be enabled in
the kernel, as it is absolutely crucial there because of KUBSAN, but
might get excessively noisy for the userland in the future.
Also note that we can't append to `CMAKE_CXX_FLAGS` like we do for other
flags in the kernel, because flags added via `add_compile_options` come
after these, so the `-Wno-address-of-packed-member` in the root would
cancel it out.
This commit implements the ISO 9660 filesystem as specified in ECMA 119.
Currently, it only supports the base specification and Joliet or Rock
Ridge support is not present. The filesystem will normalize all
filenames to be lowercase (same as Linux).
The filesystem can be mounted directly from a file. Loop devices are
currently not supported by SerenityOS.
Special thanks to Lubrsi for testing on real hardware and providing
profiling help.
Co-Authored-By: Luke <luke.wilde@live.co.uk>
...and also RangeAllocator => VirtualRangeAllocator.
This clarifies that the ranges we're dealing with are *virtual* memory
ranges and not anything else.
This enables further work on implementing KASLR by adding relocation
support to the pre-kernel and updating the kernel to be less dependent
on specific virtual memory layouts.
GCC and Clang allow us to inject a call to a function named
__sanitizer_cov_trace_pc on every edge. This function has to be defined
by us. By noting down the caller in that function we can trace the code
we have encountered during execution. Such information is used by
coverage guided fuzzers like AFL and LibFuzzer to determine if a new
input resulted in a new code path. This makes fuzzing much more
effective.
Additionally this adds a basic KCOV implementation. KCOV is an API that
allows user space to request the kernel to start collecting coverage
information for a given user space thread. Furthermore KCOV then exposes
the collected program counters to user space via a BlockDevice which can
be mmaped from user space.
This work is required to add effective support for fuzzing SerenityOS to
the Syzkaller syscall fuzzer. :^) :^)
We don't need an entirely separate VMObject subclass to influence the
location of the physical pages.
Instead, we simply allocate enough physically contiguous memory first,
and then pass it to the AnonymousVMObject constructor that takes a span
of physical pages.
This patch changes the semantics of purgeable memory.
- AnonymousVMObject now has a "purgeable" flag. It can only be set when
constructing the object. (Previously, all anonymous memory was
effectively purgeable.)
- AnonymousVMObject now has a "volatile" flag. It covers the entire
range of physical pages. (Previously, we tracked ranges of volatile
pages, effectively making it a page-level concept.)
- Non-volatile objects maintain a physical page reservation via the
committed pages mechanism, to ensure full coverage for page faults.
- When an object is made volatile, it relinquishes any unused committed
pages immediately. If later made non-volatile again, we then attempt
to make a new committed pages reservation. If this fails, we return
ENOMEM to userspace.
mmap() now creates purgeable objects if passed the MAP_PURGEABLE option
together with MAP_ANONYMOUS. anon_create() memory is always purgeable.
GCC-11 added a new option `-fzero-call-used-regs` which causes the
compiler to zero function arguments before return of a function. The
goal being to reduce the possible attack surface by disarming ROP
gadgets that might be potentially useful to attackers, and reducing
the risk of information leaks via stale register data. You can find
the GCC commit below[0].
This is a mitigation I noticed on the Linux KSPP issue tracker[1] and
thought it would be useful mitigation for the SerenityOS Kernel.
The reduction in ROP gadgets is observable using the ropgadget utility:
$ ROPgadget --nosys --nojop --binary Kernel | tail -n1
Unique gadgets found: 42754
$ ROPgadget --nosys --nojop --binary Kernel.RegZeroing | tail -n1
Unique gadgets found: 41238
The size difference for the i686 Kernel binary is negligible:
$ size Kernel Kernel.RegZerogin
text data bss dec hex filename
13253648 7729637 6302360 27285645 1a0588d Kernel
13277504 7729637 6302360 27309501 1a0b5bd Kernel.RegZeroing
We don't have any great workloads to measure regressions in Kernel
performance, but Kees Cook mentioned he measured only around %1
performance regression with this enabled on his Linux kernel build.[2]
References:
[0] d10f3e900b
[1] https://github.com/KSPP/linux/issues/84
[2] https://lore.kernel.org/lkml/20210714220129.844345-1-keescook@chromium.org/
