Upload results from CIFuzz using SARIF.
This will allow CIFuzz to report issues in the security tab.
This is a better UI than having to look through logs.
TODO(google/oss-fuzz#10452): Add proper descriptions of UBSAN bugs.
We only really care about lowering the device timeout so we get to
a shell faster when the root device doesn't appear so let's only
lower that timeout instead of lowering all default timeouts.
Let's start moving towards a more involved partitioning setup to
test our stuff more when using mkosi.
The root partition is generated on boot with systemd-repart.
CentOS supports neither erofs nor btrfs so we use squashfs and xfs
instead.
We also enable SecureBoot= locally for additional coverage. This
and the use of verity means users need to run `mkosi genkey` once
to generate the keys necessary to do secure boot and verity.
Let's make sure we log more of what mkosi's doing so we can debug
issues better. Note this also makes mkosi set SYSTEMD_LOG_LEVEL=debug
when running programs so we'll get all the systemd debug logging as
well.
Instead of building the initrds for the mkosi images with dracut,
let's switch to using mkosi presets to build the initrd with mkosi
as well.
This commit splits up our single image build into three separate
mkosi presets:
1. The "base" preset. This image contains systemd and all its runtime
dependencies. The sole purpose of this image is to serve as a base image
for the initrd and the final image. It's also responsible for building
systemd from source with the build script. The results are installed into
the base image. Note that we install the systemd and udev packages into this
image as well to prevent package managers from overriding the systemd we built
from source with the distro packaged systemd if it's pulled in as a dependency
by another package from the initrd or final profiles.
2. The "initrd" preset. This image provides the initrd. It's trivial and does
nothing more than packaging the base image up as a zstd compressed initramfs and
adds /init and /etc/initrd-release symlinks to the image.
3. The "final" preset. This image builds on top of the base image and adds
a kernel and extra packages that are useful for testing and debugging.
We also split out the optional kernel build into a separate set of config files
that are only included if a kernel to build is actually provided.
Note that this commit doesn't really change anything about how mkosi is used.
The commands remain the same, except that mkosi will now build all the presets
in order. "mkosi summary" will show the summary of all the presets. "mkosi qemu,
boot, shell" will always boot the final preset. With "-f", all presets will be
built and the final one is booted. "-i" makes a cache of each preset.
The only thing to keep in mind is that specifying config via the mkosi CLI will
apply to each of the presets. e.g. any extra packages added with "-p" will be
installed in both the initrd and the final image. To apply local configuration
to a single preset, create a file 00-local.conf in
mkosi.presets/<profile>/mkosi.conf.d and put all the preset specific configuration
in there.
Let's use the new support for matching against any distribution in
a list of distributions to start sharing most things between the
ubuntu/debian configs and centos/fedora configs.
follow-up to https://github.com/systemd/systemd/pull/27071
in order to create Github Releases, the job needs permissions to write
contents
also:
- pinned the `softprops/action-gh-release` action to a specific commit
- made it only active on the `systemd` organization repos (so not on
forks)
This also migrates the configuration to the new format that was
just merged in mkosi. Specifically, we make use of the new [Match]
sections to only include specific config snippets per distro.
We have some missing coverage in the CI, all builds enable these features,
but there are often changes and they cover a lot of code. Do one build
without them to ensure we don't break builds.
make a github release for every tag that starts with `v*`,
and a pre-release if the tag contains "-rc".
on the 'systemd/systemd' repo, the "Release" will be draft, so that the
release manager can fill up the notes manually.
on 'systemd/systemd-stable' the release will be created immediately.
info about the action used:
https://github.com/softprops/action-gh-release
- Drop Netdev= as it was removed in mkosi
- Always install python-psutil in the final image (required for networkd tests)
- Always Install python-pytest in the final image (required for ukify tests)
- Use the narrow glob for all centos python packages
- Drop the networkd mkosi config files (the default image can be used instead)
- Use ".conf" as the mkosi config file suffix everywhere
- Copy src/ to /root/src in the final image and set gdb substitute path in
.gdbinit to make gdb work properly
- ACLs are not set on generated directories anymore by default, so
we enable them explictly now so that when running unprivileged mkosi,
the user running mkosi can remove all generated files and directories.
- We don't explicitly set QemuHeadless= anymore as the option was removed
and made the default.
- We set the loglevel= kernel cmdline argument explicitly now as mkosi
doesn't set it by default anymore.
This drops all mentions of gnu-efi and its manual build machinery. A
future commit will bring bootloader builds back. A new bootloader meson
option is now used to control whether to build sd-boot and its userspace
tooling.
"Failed to dissect image: connection timed out" messages have been
appearing sporadically in mkosi CI. Let's enable debug logging to
help figure out why.
Let's make sure we're testing unprivileged builds properly. Usage
of SourceFileTransfer= and SourceFileTransferFinal= are removed as
they were dropped by mkosi. SourceFileTransfer=mount is now the
default in mkosi so behavior for the build script is unchanged. We
stop copying sources in the final image until mkosi adds support
for virtiofs.
`github.event.issue.pull_request` is an object, not a boolean.
This is the root cause of why the step that is supposed to remove labels
is always skipped. Having this condition in place is not necessary since
the workflow is run on the `pull_request_target` event.
(The one case that is left unchanged is '< <(subcommand)'.)
This way, the style with no gap was already dominant. This way, the reader
immediately knows that ' < ' is a comparison operator and ' << ' is a shift.
In a few cases, replace custom EOF replacement by just EOF. There is no point
in using someting like "_EOL" unless "EOF" appears in the text.
Due to the limitation of `GITHUB_TOKEN` when running workflows from forks,
it's required to split the `development_freeze` workflow in two.
* First workflow will run on the `pull_request` trigger and save the PR
number in the artifact. This workflow is running with read-only permissions
on `GITHUB_TOKEN`.
* Second workflow will get triggered on `workflow_run`. It will be run
directly in the `systemd/systemd` context and can get permission to be
able to create comments on PR.
GITHUB_TOKEN limitations:
* https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token
GitHub Security Labs Article - How to correctly and safely overcome GITHUB_TOKEN limitations:
* https://securitylab.github.com/research/github-actions-preventing-pwn-requests/
This gives us some nice test coverage for secure boot enrolling and the
stub secure boot workound. The authenticated EFI variables are already
created by mkosi, all we need to do is request secure boot to be used.
It's still useful to test the EFI handover logic in systemd-boot.
We use a mkosi.prepare script to install a newer python and update
the system to use it.
We also add the necessary deps for ukify to the mkosi configs.
CentOS Stream 8 is dropped from CI because its python version is too
old (3.6) to be able to run ukify.
Let's make sure we're testing with the latest changes in mkosi. This
includes both the switch to systemd-repart and ukify, making sure we
get extra testing coverage for those components.
This also drops options from the centos config that have been removed
in the newer mkosi.
For some reason idmapping runs into some issues so we disable it for
now.
Meson started warning when 'setup' is not used:
WARNING: Running the setup command as `meson [options]` instead of `meson setup [options]` is ambiguous and deprecated.
Also add more quoting in output to make the message clearer.
This erase_obj() machinery looks like voodoo and creates an awful lot of
noise as soon as we get back to building with -O0. We can do this in a
more simple way by introducing a struct that holds the information we
need on cleanup. When building with optimization enabled, all this gets
inlined and the eraser vanishes.
- Mention "/please-review" in the contributing guide
- Remove "needs-rebase" on push
- Don't add "please-review" if a green label is set
- Don't add please-review label to draft PRs
- Add please-review when a PR moves out of draft
When a pull request is opened/updated, add "please-review" and
remove a few other labels.
When a comment is made with /please-review on a PR. Add the
"please-review" label to the PR.
Rather than passing seeds up to userspace via EFI variables, pass seeds
directly to the kernel's EFI stub loader, via LINUX_EFI_RANDOM_SEED_TABLE_GUID.
EFI variables can potentially leak and suffer from forward secrecy
issues, and processing these with userspace means that they are
initialized much too late in boot to be useful. In contrast,
LINUX_EFI_RANDOM_SEED_TABLE_GUID uses EFI configuration tables, and so
is hidden from userspace entirely, and is parsed extremely early on by
the kernel, so that every single call to get_random_bytes() by the
kernel is seeded.
In order to do this properly, we use a bit more robust hashing scheme,
and make sure that each input is properly memzeroed out after use. The
scheme is:
key = HASH(LABEL || sizeof(input1) || input1 || ... || sizeof(inputN) || inputN)
new_disk_seed = HASH(key || 0)
seed_for_linux = HASH(key || 1)
The various inputs are:
- LINUX_EFI_RANDOM_SEED_TABLE_GUID from prior bootloaders
- 256 bits of seed from EFI's RNG
- The (immutable) system token, from its EFI variable
- The prior on-disk seed
- The UEFI monotonic counter
- A timestamp
This also adjusts the secure boot semantics, so that the operation is
only aborted if it's not possible to get random bytes from EFI's RNG or
a prior boot stage. With the proper hashing scheme, this should make
boot seeds safe even on secure boot.
There is currently a bug in Linux's EFI stub in which if the EFI stub
manages to generate random bytes on its own using EFI's RNG, it will
ignore what the bootloader passes. That's annoying, but it means that
either way, via systemd-boot or via EFI stub's mechanism, the RNG *does*
get initialized in a good safe way. And this bug is now fixed in the
efi.git tree, and will hopefully be backported to older kernels.
As the kernel recommends, the resultant seeds are 256 bits and are
allocated using pool memory of type EfiACPIReclaimMemory, so that it
gets freed at the right moment in boot.
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-including-and-excluding-paths
> If you define a path with the ! character, you must also define at least one
> path without the ! character. If you only want to exclude paths, use
> paths-ignore instead.
>
> The order that you define patterns matters:
> A matching negative pattern (prefixed with !) after a positive match will
> exclude the path.
> A matching positive pattern after a negative match will include the path
> again.
Even if some of the exluded paths *could* impact the build, generally it's a
waste of time to do mkosi builds on them. Let's skip to releave the builders a
bit.
GitHub Action `devel-freezer` helps with development freeze notifications
during the RC phase. It will create comments using predefined messages on
newly created and updated PRs when the RC tag has been released.
Also, it will update comments once a new major version has been released.
Documentation available at: https://github.com/redhat-plumbers-in-action/devel-freezer
In the Xen case, it's the hypervisor which manages kexec. We thus
have to ask it whether a kernel is loaded, instead of relying on
/sys/kernel/kexec_loaded.
Also, unify the string quotation a bit and drop one unnecessary
expression syntax (as everything in `if` statements is automatically
evaluated as an expression).
* chore: enable scorecard action
* chore: add badge to the README file
* chore: enable on config file update
* chore: update scorecard to 2.0.4
* chore: run scorecard on PR at main branch
* chore: add condition to publish_result key
* chore: skip upload to code scanning if PR
* chore: only runs scorecard in the main repo
Resolves: #25042
Since LGTM is no longer enabled for the systemd repo (as it's going to
be discontinued by the EOY), let's run CodeQL on every PR instead to
replace it.
It's still failing in Ubuntu:
● isc-dhcp-server.service loaded failed failed ISC DHCP IPv4 server
● isc-dhcp-server6.service loaded failed failed ISC DHCP IPv6 server
Just disable them via the kernel command line masking.
Explicitly enable all features/components in the mkosi build to
ensure they all get built and we get an error if they can't be built.
We also rework the packages sections of all mkosi configs to reduce
duplication and cover all the dependencies necessary to build/use all
systemd features.
Note that for the final image, since systemd is installed by default
in base images, we rely on that to install the base library dependencies
and we only list extra optional dependencies and tools that aren't already
installed by default into the base image.
We also drop the centos stream 8 mkosi build as dependencies on that
distro are too out-of-date to be able to build all systemd features.
Since centos stream 9 has been out for a while, let's focus on that
and leave it to downstream to keep systemd building on centos stream 8.
Finally, there's a few additions to the mkosi scripts to make sure
services don't start by default on boot.
The warning isn't that serious and mostly there to inform the user that
lld/mold cannot build efi binaries. It is also better to build test with
fatal meson warnings.
With this update, Arch Linux keyring updates will be automatically
pulled in instead of having to update to a new mkosi commit every
time the keyring gets outdated.
It should turn on the "restricted" mode by default regardless of
whether the global setting is on or not. New jobs in this action
should have to overwrite it explicitly to gain write access in any
way.
It should also make the action consistent with the other actions
writing various stuff like 'labeler' and 'codeql'.
Issue forms templates allow us to add automation in place.
This patch replaces old markdown issue templates with new issue forms.
It also adds workflow to automatically mark issues by component label
based on reported data.
This change could help with initial triaging of issues.
This is what's most likely used to build systemd with clang in
practice so let's test it as well.
Preparation for reverting 0bd292567a
(which replaced bogus buffer overflow found with _FORTIFY_SOURCE=3
with actual segfaults).
Allows to 'meson install --tags systemd-boot --no-rebuild' to install only the EFI
binaries, skipping the rest, for a very quick build:
$ ninja src/boot/efi/linuxx64.efi.stub
[21/21] Generating src/boot/efi/linuxx64.efi.stub with a custom command
$ ninja src/boot/efi/systemd-bootx64.efi
[10/10] Generating src/boot/efi/systemd-bootx64.efi with a custom command
$ DESTDIR=/tmp/foo meson install --tags systemd-boot --no-rebuild
Installing src/boot/efi/systemd-bootx64.efi to /tmp/foo/usr/lib/systemd/boot/efi
Requires Meson 0.60 to be used, prints a warning for unknown keyword
in earlier versions, but there's no failure
https://mesonbuild.com/Installing.html#installation-tags
The idea was to catch CFLite regressions but since the action itself
pulls the latest docker images it can't be pinned properly and issues
like https://github.com/google/clusterfuzzlite/issues/91 are going to
pop up anyway. Let's unpin it by analogy with CIFuzz and hope it doesn't
break very often.
We recently added caching for the dependencies we build from source
in mkosi's github action which speeds up builds by +-10 minutes. Let's
update to the latest commit so we benefit from this in systemd's mkosi
CI as well.
This makes sure that building ia32 EFI binaries on x86_64 works.
We force gnu-efi support to ensure it's not skipped by accident
and provide the lib32 dir manually, because clang does not support
'--print-multi-os-directory', which is used to auto-detect it.
Make sure our RPM macros work as intended. Based on the original PR
(#16464) by Mikhail Novosyolov.
Co-authored-by: Mikhail Novosyolov <m.novosyolov@rosalinux.ru>
Now that mkosi has centos-stream 9 support, let's add a config in
the repo so that the mkosi CI tests that configuration as well.
Centos doesn't support btrfs so we use xfs instead. For some reason,
building --hostonly-initrd centos images breaks the qemu boot so I
disabled that option for centos.
We update the mkosi commit hash to 0dd39c20a4
which adds the PowerTools repo to CentOS Stream 8 which is required
to make all the necessary packages required to build systemd on
CentOS Stream 8 available.
For some reason Ubuntu Focal repositories now have `llvm-13` virtual
package which can't be installed, but successfully fools our check,
resulting in no clang/llvm being installed...
```
$ apt show llvm-13
Package: llvm-13
State: not a real package (virtual)
N: Can't select candidate version from package llvm-13 as it has no candidate
N: Can't select versions from package 'llvm-13' as it is purely virtual
N: No packages found
$ apt install --dry-run llvm-13
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package llvm-13 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'llvm-13' has no installation candidate
```
The packages are installed to provide the dhcpd binary, used by
test/test-network/systemd-networkd-tests.py, but we don't need the units
to run, and in fact in some cases the image fails to boot because of
them:
Spawning container image on /home/runner/work/systemd/systemd/image.raw.
Press ^] three times within 1s to kill container.
● isc-dhcp-server.service loaded failed failed ISC DHCP IPv4 server
● isc-dhcp-server6.service loaded failed failed ISC DHCP IPv6 server
Container image failed with error code 1.
Error: Process completed with exit code 1.
Mask the units with an --extra-tree.
to, hopefully, get rid of the following error:
```
2022-02-13 13:32:12 [ERROR] Failed to get [GITHUB_TOKEN]!
2022-02-13 13:32:12 [ERROR] []
2022-02-13 13:32:12 [ERROR] Please set a [GITHUB_TOKEN] from the main workflow environment to take advantage of multiple status reports!
```
The 'slim' version drops certain storage-heavy linters[0] which we don't
use anyway, so let's make the job a bit faster by downloading and using
a smaller image.
[0] https://github.com/github/super-linter#slim-image
CIFuzz has been kind of broken for a couple months because
coverage reports downloaded from OSS-Fuzz contain absolute
paths while paths to files changed in PRs are relative and they
don't match. It makes it kind of hard for CIFuzz to figure out
what it should run so it runs either all fuzz targets or just new
fuzz targets. Until that issue is fixed let's just always predictably run
all fuzz targets.
Judging by
ERROR! Failed to call GitHub Status API!
it doesn't seem to work. Even if it did it would just clutter the status
checks I think so let's just remove MULTI_STATUS along with
GITHUB_TOKEN.
some actions like Coverity and CFLite aren't run on every PR so to make
sure they are more or less fine when they are changed it makes sense to
at least check them with superlinter/actionlint: https://github.com/rhysd/actionlint
The following warnings were fixed along the way:
```
.github/workflows/mkosi.yml:55:7: shellcheck reported issue in this script: SC2086:info:6:14: Double quote to prevent globbing and word splitting [shellcheck]
|
55 | run: |
| ^~~~
.github/workflows/mkosi.yml:55:7: shellcheck reported issue in this script: SC2046⚠️6:40: Quote this to prevent word splitting [shellcheck]
|
55 | run: |
| ^~~~
.github/workflows/mkosi.yml:55:7: shellcheck reported issue in this script: SC2006:style:6:40: Use $(...) notation instead of legacy backticked `...` [shellcheck]
|
55 | run: |
| ^~~~
```
```
.github/workflows/coverity.yml:31:9: shellcheck reported issue in this script: SC2086:info:1:93: Double quote to prevent globbing and word splitting [shellcheck]
|
31 | run: echo "COVERITY_SCAN_NOTIFICATION_EMAIL=$(git log -1 ${{ github.sha }} --pretty=\"%aE\")" >> $GITHUB_ENV
| ^~~~
```
The idea behind this action is to make it possible to compare the
latest fuzz targets with PRs to figure out whether bugs are really
reproducible in PRs only. Since forks (including systemd-stable) are
usually based on the upstream repository where almost all the bugs
are fixed before releases are cut it should be safe to assume that
if CFLite finds bugs in PRs they are most likely introduced in those
PRs.
It should probably be brought back once https://github.com/google/clusterfuzzlite/issues/84
is fixed.
It's like CIFuzz but unlike CIFuzz it's compatible with forks and
it should make it possible to run the fuzzers to make sure that
patches backported to them are backported correctly without introducing
new bugs and regressions.
