We can't use the systemd-journal-upload user here, since it's created
dynamically by DynamicUser=yes. However, we can use the group specified
in SupplementaryGroups=, so do exactly that.
Turns out that redirecting a lot of output to the console can have some
funny effects, like random kernel soft lockups. I spotted this in
various CIs, but it remained almost entirely hidden thanks to
`softlockup_panic=1`, until 1a36d2672f which introduced a couple of
tests that log quite a lot in a short amount of time. This, in
combination with newer kernel version, which, for some reason, seem to
be more susceptible to such soft lockups, made the Arch Linux jobs soft
lockup quite a lot, see [0].
While debugging this I also noticed that runs which don't redirect
stdout/stderr to the console are noticeably faster, e.g.:
# TEST-71 nspawn + QEMU (KVM), StandardOutput=journal+console
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:24.64
# TEST-71 nspawn + QEMU (KVM), StandardOutput=journal
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:17.95
# TEST-71 nspawn + QEMU, StandardOutput=journal+console
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:04.70
# TEST-71 nspawn + QEMU, StandardOutput=journal
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:44.48
# TEST-04 QEMU, StandardOutput=journal+console
Elapsed (wall clock) time (h:mm:ss or m:ss): 4:22.70
# TEST-04 QEMU, StandardOutput=console
Elapsed (wall clock) time (h:mm:ss or m:ss): 5:04.67
Given all this, let's effectively revert ba7abf79a5, and dump the
testsuite-related journal messages only after the test finishes, so they
don't go through the slow console.
Resolves: systemd/systemd-centos-ci#660
[0] https://github.com/systemd/systemd-centos-ci/issues/660
Let's save all journals from the test machine instead of calling export
on each journal file separately, which makes the code less complicated
(and probably faster).
I'm pretty sure this is not the only case, but it's the one I recently
noticed. Even though we call ddebug() from a function, that function is
called before ddebug() is defined, resulting in the same issue as if we
called just ddebug() in its place, i.e.:
..//test-functions: line 276: ddebug: command not found
Currently the test works only with policy shipped by Fedora, which makes
it pretty much useless in most of our CIs. Let's drop the custom module
and make the test more generic, so it works with the refpolicy as well,
which should allow us to run it on Arch and probably even in Ubuntu CI.
Let's just rely on the word splitting done by bash instead of messing
with that ourselves, as it's just adding extra complexity to appease one
ShellCheck check. Also, this apparently never worked for the nspawn
stuff anyway, since I forgot to set $IFS to an appropriate value, so it
always put all arguments from $KERNEL_APPEND into a single array item
with an extra newline, which then made systemd sad:
~# readarray arr <<< "foo bar baz"; for i in "${arr[@]}"; do echo "'$i'"; done
'foo bar baz
'
~# make -C test/TEST-45-TIMEDATE/ clean setup run BUILD_DIR=$PWD/build TEST_NO_QEMU=1 KERNEL_APPEND="systemd.log_level=console"
...
~# journalctl -o short-monotonic --no-hostname --file /var/tmp/systemd-tests/systemd-test.XaDX67/system.journal --grep "Failed to parse" -p info --no-pager
[551138.986882] systemd-tmpfiles[21]: Failed to parse log level 'console
[551138.987179] systemd-remount-fs[20]: Failed to parse log level 'console
[551138.993125] systemd-sysusers[23]: Failed to parse log level 'console
[551138.998685] journalctl[29]: Failed to parse log level 'console
Resolves: #29945
Let's put this back in, as it could help with occasional machine lock ups
on overloaded systems (and it didn't help with the original issue
anyway).
This reverts commit 3a89904e45.
On slower/overloaded systems it may take a bit for the swtpm socket
to show up:
I: Started swtpm as PID 189419 with state dir /tmp/tmp.pWqUutuGUj
I: Configured emulated TPM2 device tpm-spapr
+ tee /var/tmp/systemd-test-TEST-70-TPM2_1/console.log
+ timeout --foreground 1200 /bin/qemu-system-ppc64le -smp 4 ...
qemu-system-ppc64le: -chardev socket,id=chrtpm,path=/tmp/tmp.pWqUutuGUj/sock: Failed to connect to '/tmp/tmp.pWqUutuGUj/sock': No such file or directory
E: qemu failed with exit code 1
Spotted regularly in the ppc64le cron job and in some Ubuntu CI/CentOS CI
pr runs [0].
[0] https://github.com/systemd/systemd/pull/29183#issuecomment-1721727927
We can't do anything about them anyway, and most importantly this seems
to alleviate systemd/systemd-centos-ci#660, which should make the CIs
a bit less angry (at least until the issue is addressed properly).
The pipe stuff introduced in 701e0c2660 causes nspawn to switch the
console from 'interactive' into 'read-only' which is a bit useless when
debugging. Let's set --console=interactive explicitly in such case.
Follow-up to 701e0c2660.
This metadata (EXTENSION_RELOAD_MANAGER) can be set to "1" to reload the manager
when merging/refreshing/unmerging a system extension image. This can be useful in case the sysext
image provides systemd units that need to be loaded.
With `--no-reload`, one can deactivate the EXTENSION_RELOAD_MANAGER metadata interpretation.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
Same scenario as with libsystemd - ldd might use unprefixed RPATH, and
we install our own stuff into the image unconditionally anyway.
Also, bail out early if we hit a missing DSO with a possibly helpful
message.
This extends the test framework a bit, and allows adding additional
initrds to the qemu invocation, which we use here to place credentials
in the new /run/systemd/@initrd/ credentials dir which are then passed
to the host.
So we're able to detect memory leaks in our NSS modules.
An example after introducing a memory leak in nss-myhostname.c:
testsuite-71.sh[2881]: =================================================================
testsuite-71.sh[2881]: ==2880==ERROR: LeakSanitizer: detected memory leaks
testsuite-71.sh[2881]: Direct leak of 2 byte(s) in 1 object(s) allocated from:
testsuite-71.sh[2881]: #0 0x7fa28907243b in strdup (/usr/lib64/libasan.so.8.0.0+0x7243b)
testsuite-71.sh[2881]: #1 0x7fa286a7bc10 in gethostname_full ../src/basic/hostname-util.c:67
testsuite-71.sh[2881]: #2 0x7fa286a74af9 in gethostname_malloc ../src/basic/hostname-util.h:24
testsuite-71.sh[2881]: #3 0x7fa286a756f4 in _nss_myhostname_gethostbyname4_r ../src/nss-myhostname/nss-myhostname.c:79
testsuite-71.sh[2881]: #4 0x7fa288f17588 in getaddrinfo (/lib64/libc.so.6+0xf4588)
testsuite-71.sh[2881]: #5 0x7fa2890a4d93 in __interceptor_getaddrinfo.part.0 (/usr/lib64/libasan.so.8.0.0+0xa4d93)
testsuite-71.sh[2881]: #6 0x55a54b2b7159 in ahosts_keys_int.part.0 (/usr/bin/getent.orig+0x4159)
testsuite-71.sh[2881]: SUMMARY: AddressSanitizer: 2 byte(s) leaked in 1 allocation(s).
As hitting an ASan/UBSan error in PID1 results in a crash (and a kernel
panic when running under qemu), we usually lose the stack trace which
makes debugging quite painful. Let's mitigate this by forwarding the
stack trace to multiple places - namely to a file and the syslog.
