We should be documenting required pacakges elsewhere and installing them
in the setup step of CI.
This also fixes a problem where the run script would fail if you already
had a cloned wpt directory.
The proper syntax for defining user-defined literals does not require a
space between the `operator""` token and the operator name:
> error: identifier 'sv' preceded by whitespace in a literal operator
> declaration is deprecated
This is a minimal set of changes to allow `serenity.sh build riscv64` to
successfully generate the build environment and start building. This
includes some, but not all, assembly stubs that will be needed later on;
they are currently empty.
LibC is always guaranteed to be loaded at program start, so its
thread-local variables live in the static TLS block. This permits us to
use the more optimal initial-exec TLS access model.
IFF was a generic container fileformat that was popular on the Amiga
since it was the only file format supported by Deluxe Paint.
ILBM is an image format popular in the late eighties/nineties
that uses the IFF container.
This is a very first version of the decoder that only supports
(byterun) compressed files with bpp <= 8.
Only the minimal chunks are decoded: CMAP, BODY, BMHD.
I am planning to add support for the following variants:
- EHB (32 colours + lighter 32 colours)
- HAM6 / HAM8 (special mode that allowed to display the whole Amiga
4096 colours / 262 144 colours palette)
- TrueColor (24bit)
Things that could be fun to do:
- Still images could be animated using color cycle information
Stop worrying about tiny OOMs. Work towards #20449.
While going through these, I also changed the function signature in many
places where returning ThrowCompletionOr<T> is no longer necessary.
The test runner script sets the `halt_on_error=1` `UBSAN_OPTIONS` flag
already, this just makes it a compile-time decision. This should
alleviate some of the slowness of running on-target tests without
hardware acceleration.
Instead of hardcoding bin/, we should use the INSTALL_BINDIR directly
to reflect the intent of the way we're doing the settings.
This should have the benefit of fixing the build when a distro sets the
GNUInstallDirs variables to exciting values, like on NixOS.
This is accomplished by detecting the Userland subdirectory from the GML
file's own path. Thereby, any Userland program, e.g. applet,
application, or even service, can use GML and the include paths of the
generated files will be correct.
As with other special cases, this depends on the property name.
The UIDimension code is kept separate since it can handle both integers
and strings, and the string names for special dimensions don't match
with the enum names.
This encoder can handle all integer formats and sample rates, though
only two channels well. It uses fixed LPC and performs a
close-to-optimal parameter search on the LPC order and residual Rice
parameter, leading to decent compression already.
I couldn't run the parser in a debugger like I normally would, so I
added printouts to understand where the parser is failing.
More could be added, but these are enough to get a good idea of what
the parser is doing. It's very spammy, though, so enable it by flicking
the IMAP_PARSER_DEBUG switch :^)
Two non-functional changes:
- Remove pointless `-latomic` flag. It was specified via
`add_compile_options`, which only affects compilation and not linking,
so the library was never actually linked into the kernel. In fact, we
do not even build `libatomic` for our toolchain.
- Do not disable `-Wnonnull`. The warning-causing code was fixed at some
point.
This commit also removes `-mstrict-align` from the userland. Our target
AArch64 hardware natively supports unaligned accesses without a
significant performance penalty. Allowing the compiler to insert
unaligned accesses into aligned-as-written code allows for some
performance optimizations in fact. We keep this option turned on in the
kernel to preserve correctness for MMIO, as that might be sensitive to
alignment.
We now apply MathML's default user agent style sheet along with other
default styles. This sheet is not mixed in with the other styles in
CSS/Default.css because it is a namespaced stylesheet and so has to
be its own sheet.
This does the exact same thing as the runtime initializer,
except it is faster and can catch some errors much earlier.
The code generator includes these important features:
- Automatic include generation where necessary
- Special-casing for TabWidget and ScrollableContainerWidget
- No use of DeprecatedString where possible
We weren't installing a lot of generated sources for the top level Lagom
build or for LibWeb, making it impossible to use LibWeb from a
find_package. ...And also Kernel/API/KeyCode.h, which is included by
no less than 8 different files in Userland/Libraries. We also weren't
installing any Ladybird header files.
This will help a lot with developing chromes for different UI frameworks
where we can see which helper classes and processes are really using Qt
vs just using it to get at helper data.
As a bonus, remove Qt dependency from WebDriver.
This will ensure that we don't leak any memory while playing back
audio.
There is an expectation value in the test that is only set to true when
PulseAudio is present for the moment. When any new implementation is
added for other libraries/platforms, we should hopefully get a CI
failure due to unexpected success in creating the `PlaybackStream`.
To ensure that we clean up our PulseAudio connection whenever audio
output is not needed, add `PulseAudioContext::weak_instance()` to allow
us to check whether an instance exists without creating one.
We got some errors while loading https://twinings.co.uk/ about this
interface missing, and it looked fairly simple so I sketched it out.
Note that I did leave some FIXMEs where it's not clear exactly which
metrics we should be returning.
This object is available as `window.internals` (or just `internals`) and
is only accessible while running in "test mode".
This first version only has one API: gc(), which triggers a garbage
collection immediately.
In the future, we can add more APIs here to help us test parts of the
engine that are hard or impossible to reach via public web APIs.
The pretty-print gdb helpers were not updated since the
DeprecatedString change. This commit introduces a new printer for String
and renames the old one to DeprecatedString.
This adds an abstract `Audio::PlaybackStream` class to allow cross-
platform audio playback to be done in an opaque manner by applications
in both Serenity and Lagom.
Currently, the only supported audio API is PulseAudio, but a Serenity
implementation should be added shortly as well.
Since the existing Promise class is designed with deferred tasks on the
main thread only, we need a new class that will ensure we can handle
promises that are resolved/rejected off the main thread.
This new class ensures that the callbacks are only called on the same
thread that the promise is fulfilled from. If the callbacks are not set
before the thread tries to fulfill the promise, it will spin until they
are so that they will run on that thread.
Rework the write_if_changed logic to properly truncate the output file.
The original logic would not truncate the file, leading to extra junk
at the end of files if the IDL files shrunk between builds.
3dd3120a8a changed open mode from Write to
ReadWrite, which stopped truncating files.
This could be noticed by building a slightly older branch, as compiling
it gave compile errors.
Now that all the classes for Ladybird are in the Ladybird namespace, we
don't need them named as Ladybird::FooBarLadybird. For the Qt-specific
classes, we can tack on a Qt at the end for clarity, but FontPlugin and
ImageCodecPlugin no longer have anything to do with Qt.
LibTLS still can't access many parts of the web, so let's hide this
behind a flag (with all the plumbing that entails).
Hopefully this can encourage folks to improve LibTLS's algorithm support
:^).
Re-organize our helper files here a bit, to make a clearer distinction
between Qt-specific helpers and generic non-serenity helpers.
A future commit will move Lagom specific code from LibSQL to ladybird
as well, so that we can see about future generic apis for spawning
helper procesess.
The Qt relationship was removed in de31a8a4, so let's acknowledge that
and make it clearer to potential non-Qt ports that this file is usable
by them :^).
This reduces the number of tasks to schedule, and the complexity of the
build system integrations for the BindingsGenerator. As a bonus, we move
the "only write if changed" feature into the generator to reduce the
build system load on generated files for this generator.
The LocaleData generator currently stores vectors of unique instances of
CLDR data (e.g. languages, currencies, etc.). For each CLDR file that we
parse, we linearly search through those vectors to decide if the current
field being parsed is unique. Given the size of the CLDR, this adds up
to quite a bit of time.
Augment these vectors with a hash map to store the index of each unique
instance in those vectors. This allows for quickly checking if a field
is unique, and to later look up those indices.
We do not apply this technique to every bit of CLDR data here. For
example, CLDR::character_orders contains only 2 entries. In that case,
it is quicker to search the vector than it is to hash a string key.
This reduces the runtime of GenerateLocaleData from to 2.03s to 1.09s.
Similar to languages and currencies, extract the loop to collect the
unique set of date fields to a preprocessing function. This alone does
not yield any performance improvement, but combined with an upcoming
patch will make the parse_locale_date_fields() a bit faster.
We currently parse each CLDR calendar, then decide based on its primary
key whether we want to skip it. Instead, we can decide to skip it based
on its file name.
This reduces the runtime of GenerateLocaleData from 2.03s to 1.97s.
The LocaleData generator has to read a few of the CLDR files more than
once, to e.g. prepare some data up front (for reasons why, see commits
c86f7a6 and 0b69e9f). This takes non-neglible time, especially for large
JSON files such as currencies.json. So in these cases, cache the parsed
JSON in a map.
This reduces the runtime of GenerateLocaleData from 2.32s to 2.03s.
These were used to generate specialized tables. Now that those tables
have been migrated to general 2-stage lookup tables, these fields are
all unused.
Similar to commit 0652cc4, we now generate 2-stage lookup tables for
case conversion information. Only about 1500 code points are actually
cased. This means that case information is rather highly compressible,
as the blocks we break the code points into will generally all have no
casing information at all.
In total, this change:
* Does not change the size of libunicode.so (which is nice because,
generally, the 2-stage lookup tables are expected to trade a bit
of size for performance).
* Reduces the runtime of the new benchmark test case added here from
1.383s to 1.127s (about an 18.5% improvement).
There is no functional change here. This information will compose the
upcoming multistage casing tables in an upcoming patch. Extract it to
its own struct to prepare for that.
There is no functional change here. This just adjusts the changes made
in commit 0652cc4 to be a bit more generic for code point casing tables.
We currently only generate property tables, which boil down to a vector
of booleans. Casing tables will be a struct of varying types, so this
generalizes some of the generator to prepare for that ahead of time, to
make the upcoming casing patch smaller / easier to grok.
When generating code point property tables, we currently binary search
the code point range lists for each property to decide if a code point
has that property. However, we are both iterating over the code points
and through the sorted properties in order. This means we do not need
to search code point ranges that are below the current code point at
all. We can even remove the code point ranges that fall below the
current code point, as we will not see a code point in those ranges
again.
On my machine, this reduces the run time of GenerateUnicodeData from
3.4 seconds to 1.2 seconds.
The idl file lists are used for two things:
1. As inputs for `generate_window_or_worker_interfaces`
2. In a loop in `generate_idl_bindings` and the loop variable
is passed to `rebase_path`
Both these cases can handle a normal fully-qualified GN path,
so there's no need for the "abspath".
No behavior change.
Currently, the `isobmff` utility will only print the media file type
info from the FileTypeBox (major brand and compatible brands), as well
as the names and sizes of top-level boxes.
This disables running benchmark test cases on Lagom in CI. When we run
unit tests on-board Serenity, run-tests sets this environment variable.
We do not use that utility to run unit tests on Lagom, so we've been
running benchmark tests unnecessarily.
When markdown-check is built, it outputs hundreds of lines of "ignoring
this and that link because reasons". This is extremely not helpful when
trying to figure out exactly which check failed on your commit. Also
remove the timing numbers from lint-ci.sh These are just noise and also
don't help to figure out which pre-commit check failed. Ideally the
output on fail should be "[OK]: Check A" for all the passing checks and
"[FAIL] Check N" with the required context for the failed check.
We currently produce a single table for all categories of code point
properties (GeneralCategory, Script, etc.). Each row contains a field
indicating the range of code points to which that property applies. At
runtime, we then do a binary search through that table to decide if a
code point has a property.
This changes our approach to generate a 2-stage lookup table for each of
those categories. There is an in-depth explanation of these tables above
the new `create_code_point_tables` method. The end effect is that code
point property lookup is reduced from a binary search to constant-time
array lookups.
In total, this change:
* Increases the size of libunicode.so from 2.7 MB to 2.9 MB.
* Reduces the runtime of the new benchmark test case added here from
3.576s to 1.020s (a 3.5x speedup).
* In a profile of resizing a TextEditor window with a 3MB file open,
the runtime of checking if a code point has a word break property
reduces from ~81% to ~56%.
The next commit will need a type from LibUnicode/CharacterTypes.h. To
avoid conflicts between that header's CodePointRange and the one that is
defined in the code generator, just use the public definition.
We started generating this data in commit 0505e03, but it was unused.
It's still not used, so let's remove it, rather than bloating the size
of libunicode.so with unused data. If we need it in the future, it's
trivial to add back.
Note we *have* always used the block name data from that commit, and
that is still present here.
The AST interpreter is still available behind a new `--ast` flag.
We switch to testing with bytecode in the big run-tests battery on
SerenityOS. Lagom CI continues running both AST and BC.
Rather than splitting the Iterator type and its AOs into two files,
let's combine them into one file to match every other JS runtime object
that we have.
This is an editorial change in the ECMA-262 spec. See:
https://github.com/tc39/ecma262/commit/956e5af
This splits the GetIterator AO into two AOs, to remove some recursion
and to (soon) remove optional parameters.
fddbd11baa made it so the command executed
read `sh -c -- '"script" args*'`, the -- in this command is redundant as
the script name never starts with a dash and can never be interpreted as
an option or a flag.
The actually meaningful placement for -- here is after `$SUDO`, to make
sure `$SUDO` does not incorrectly treat `-c` as an option to itself, and
`$SHELL` cannot be interpreted as an option/flag in the extremely
unlikely event that it starts with a dash.
The dollar sign is a special character in POSIX shells and in the Ninja
build file format. If the file name contains a `$`, something goes wrong
in the escaping/unescaping of this symbol, and CMake/GCC/Clang generate
invalid dependency files where not all instances of `$` are escaped
properly. Because of this, Ninja fails to rebuild `$262Object.cpp` if
the headers included by it have changed.
Stale `$262Object.cpp.o` files have been the cause of mysterious crashes
multiple times which only go away after doing a clean build. Let's
prevent these from happening again by removing the `$` from the
filename.
Recreating the previous screenshot in a current build of the system will
show many, usually subtle, changes in comparison.
This patch adds a new screenshot of the SerenityOS desktop with
Terminal, File Manager, System Monitor and Ladybird visible.
This allows opening the ladybird.app app bundle on macOS, using Xcode
tools like Instruments on the applications in the app bundle, and even
installing the app bundle into /Applications :^)
This commit makes it possible to let properties accept easing functions
as values, which will be used in a later commit to implement
animation-timing-function.
A lot of code gen happening here. These generators are kind of
awkward to work with, and the fact that the CLDR data download
extracts over 8,000 files makes it hard to fit into the explicit
patterns GN expects of us.
We depend on GNU-specific du switch `--apparent-size`. Busybox has this
implemented, but as `-b` instead.
Another part of the build system uses `cp --preserve=timestamps`. This
can be replaced by `rsync -t`, and rsync is already used through the
file.
Thanks to those changes, Serenity can be built on a Busybox system,
without GNU coreutils.
This includes a few new options to the .clang-format configuration file
to A) adhere to option changes within clang-format 16 (namely the option
AlignTrailingComments), and B) enforce existing style guide rules with
new clang-format rules.
The main missing features are rootMargin, proper nested browsing
context support and content clip/clip-path support.
This makes images appear on some sites, such as YouTube and
howstuffworks.com.
Using the cross-page links, we can generate a directed graph showing the
topology of which pages refer to other pages. This is not just for fun:
the links show how often a page is linked (since links are not
deduplicated on purpose), which pairs of pages only have links in one
direction (where a link in the other direction may be useful), which
groups of closely-interlinked pages exist, and which pages have few or
no links to other pages.
The EXTRA_MARKDOWN_CHECK_ARGS argument to the check-markdown script can
be used to inject the -g flag for generating the graph on all manpages.
Apart from the class used audio fuzzers have identical behavior: Create
a memory stream from the fuzzer input and pass this to the loader, then
try to load audio until an error occurs. Since the loader plugins need
to have the same static create() function anyways for LibAudio itself,
we can unify the fuzzer implementations and reduce code duplication.
This utility will learn tricks such as extracting images from PDFs and
dumping tables from PDFs so that we can create code from specs.
It also allows testing LibPDF things in lagom, and allows testing
reading large amounts of PDFs using a shell script.
This was missed in 02b74e5a70
We need to disable consteval in AK::String as well as AK::StringView,
and we need to disable it when building both the tools build and the
fuzzer build.
This removes a lot of duplicated stream creation code from the plugins,
and also simplifies the way that the appropriate plugin is found. This
mirrors the ImageDecoderPlugin design and necessitates new sniffing
methods on the loaders.
Once LibC is installed to the sysroot and its conflicts with libc++
are resolved, including LibC headers in such a way will cause errors
with a modern LLVM-based toolchain.
This is needed to avoid including LibC headers in Lagom builds.
Unfortunately, we cannot rely on the build machine to provide a
fully POSIX-compatible ELF header for Lagom builds, so we have to
use our own.
The JS::VM now owns the one Bytecode::Interpreter. We no longer have
multiple bytecode interpreters, and there is no concept of a "current"
bytecode interpreter.
If you ask for VM::bytecode_interpreter_if_exists(), it will return null
if we're not running the program in "bytecode enabled" mode.
If you ask for VM::bytecode_interpreter(), it will return a bytecode
interpreter in all modes. This is used for situations where even the AST
interpreter switches to bytecode mode (generators, etc.)
This was meant to be a temporary unit testuntil we could run test-js
in bytecode mode. This has been possible for a long time now, so let's
remove the unnecessary extra program.
GCC's build fails in `libisl`'s configure step if `CC` is set to
Homebrew Clang with the message "Link Time Optimisation is not
supported". This is likely due to the fact that it tries to use ranlib
from Xcode, which is not compatible with the newer LLVM version's
bitcode format.
The toolchain build runs after `pick_host_compiler` is called, which
selects Homebrew Clang if the installed Xcode version is too old. We
need to unset `CC` and `CXX` for the toolchain build to sidestep the
issue.
This reduced the runtime of `lint-ports` from 72s down to 0.5s on
my computer (a reduction of ~99.2%)! While it may not be as
accurate as running `package.sh showproperty` it makes using
`precommit` while editing ports much nicer!
Don't try to implement this AO in bytecode. Instead, the bytecode
Interpreter class now has a run() API with the same inputs as the AST
interpreter. It sets up the necessary environments etc, including
invoking the GlobalDeclarationInstantiation AO.
This creates (and installs upon WebContent startup) a platform plugin to
play audio data.
On Serenity, we use AudioServer to play audio over IPC. Unfortunately,
AudioServer is currently coupled with Serenity's audio devices, and thus
cannot be used in Ladybird on Lagom. Instead, we use a Qt audio device
to play the audio, which requires the Qt multimedia package.
While we use Qt to play the audio, note that we can still use LibAudio
to decode the audio data and retrieve samples - we simply send Qt the
raw PCM signals.
This does a few things:
- The decoder uses a 32- or 64-bit integer as a reservoir of the data
being decoded, rather than one single byte as it was previously.
- `read_bool()` only refills the reservoir (value) when the size drops
below one byte. Previously, it would read out a bit-sized range from
the data to completely refill the 8-bit value, doing much more work
than necessary for each individual read.
- VP9-specific code for reading the marker bit was moved to its own
function in Context.h.
- A debug flag `VPX_DEBUG` was added to optionally enable checking of
the final bits in a VPX ranged arithmetic decode and ensure that it
contains all zeroes. These zeroes are a bitstream requirement for
VP9, and are also present for all our lossy WebP test inputs
currently. This can be useful to test whether all the data present in
the range has been consumed.
A lot of the size of this diff comes from the removal of error handling
from all the range decoder reads in LibVideo/VP9 and LibGfx/WebP (VP8),
since it is now checked only at the end of the range.
In a benchmark decoding `Tests/LibGfx/test-inputs/4.webp`, decode times
are improved by about 22.8%, reducing average runtime from 35.5ms±1.1ms
down to 27.4±1.1ms.
This should cause no behavioral changes.
On macOS, CMake incorrectly tries to add and/or remove rpaths from files
that it has already processed when it performs installation. Setting the
rpaths during the build process ensures that they are only set once, and
as a bonus, makes installation slightly more performant.
Fixes#10055.
This broke in c2e24a2fa1 where the boot
drive was removed from `SERENITY_MACHINE`. We now add the boot drive to
the common Qemu arguments, so it gets included in the aarch64 run
configuration as well.
The PNP IDs data file was recently updated with an accidental duplicate
entry (HONOR Device Co., Ltd.). Rather than breaking everyone's build,
let's just ignore duplicates.
Generated iterator prototypes already have the IteratorPrototype as
their prototype, but we were incorrectly hijacking them and rerouting
to ObjectPrototype.
Regressed in cfe663435e.
If a local include does not point to a file in the repository, it should
be a system include instead. This is now checked on every commit.
While this does introduce significant overhead in terms of percentage,
I think that an additional 10ms on huge commits (or less on smaller
commits) are acceptable:
hyperfine -w1 './Meta/check-style.py AK/*.h AK/*.cpp' # Before
Benchmark 1: ./Meta/check-style.py AK/*.h AK/*.cpp
Time (mean ± σ): 20.3 ms ± 0.4 ms [User: 17.1 ms, System: 3.
5 ms]
Range (min … max): 19.5 ms … 21.6 ms 128 runs
hyperfine -w1 './Meta/check-style.py AK/*.h AK/*.cpp' # After
Benchmark 1: ./Meta/check-style.py AK/*.h AK/*.cpp
Time (mean ± σ): 32.3 ms ± 0.4 ms [User: 27.9 ms, System: 4.
4 ms]
Range (min … max): 31.4 ms … 34.9 ms 91 runs
Previously this was compiled to require an object despite the IDL file
specifying 'optional'.
This commit makes IDLGenerator respect this modifier, and fixes the only
affected instance.
This partially implements CSS-Animations-1 (though there are references
to CSS-Animations-2).
Current limitations:
- Multi-selector keyframes are not supported.
- Most animation properties are ignored.
- Timing functions are not applied.
- Non-absolute values are not interpolated unless the target is also of
the same non-absolute type (e.g. 10% -> 25%, but not 10% -> 20px).
- The JavaScript interface is left as an exercise for the next poor soul
looking at this code.
With those said, this commit implements:
- Interpolation for most common types
- Proper keyframe resolution (including the synthetic from-keyframe
containing the initial state)
- Properly driven animations, and proper style invalidation
Co-Authored-By: Andreas Kling <kling@serenityos.org>
This will make it a lot easier to understand what went wrong, especially
when the failure occurs on CI but not at home.
And of course, use LibDiff to generate the diff! :^)
It does not make sense to test known-working code that is deprecated and
in the process of being removed. Also, this test becomes too cumbersome
to write without using read_all or line iteration in some form, and
migrating the test is just silly.
That's what this class really is; in fact that's what the first line of
the comment says it is.
This commit does not rename the main files, since those will contain
other time-related classes in a little bit.
There have been multiple reports of Xcode 14.0 (based on upstream LLVM
14) segfaulting when compiling `LibCore/Process.cpp`. Let's require
Xcode 14.3, which is a known good version based on LLVM 15.
Note that Xcode 14.3 requires macOS Ventura, so users of Monterey or
older are expected to get Homebrew Clang instead.
Homebrew Clang 13 also suffers from the same crash. Although I have not
tested on Linux, the backtrace points to the middle-end, so x86_64 is
also likely to be affected. LLVM 14 was released 14 months ago, so it's
not an unreasonable requirement.
It's not safe to allocate from the GC heap while in the constructor of a
GC heap cell. (Because if this ends up triggering a collection, we may
end up trying to call through an uninitialized vtable).
This was already done safely in the initialize() virtual in much of
LibJS and LibWeb. This patch moves the logic for prototypes, mixins,
and CSSStyleDeclaration as well.
Fixes a long-standing GC crash that was pretty easy to reproduce by
refreshing https://vercel.com/
