2022-05-04 10:29:31 +00:00
|
|
|
The extended testsuite only works with UID=0. It consists of the subdirectories
|
|
|
|
named "test/TEST-??-*", each of which contains a description of an OS image and
|
|
|
|
a test which consists of systemd units and scripts to execute in this image.
|
|
|
|
The same image is used for execution under `systemd-nspawn` and `qemu`.
|
2012-05-25 16:44:42 +00:00
|
|
|
|
|
|
|
To run the extended testsuite do the following:
|
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
$ ninja -C build # Avoid building anything as root later
|
|
|
|
$ sudo test/run-integration-tests.sh
|
|
|
|
ninja: Entering directory `/home/zbyszek/src/systemd/build'
|
|
|
|
ninja: no work to do.
|
|
|
|
--x-- Running TEST-01-BASIC --x--
|
2020-11-17 20:39:37 +00:00
|
|
|
+ make -C TEST-01-BASIC clean setup run
|
2018-10-09 12:37:00 +00:00
|
|
|
make: Entering directory '/home/zbyszek/src/systemd/test/TEST-01-BASIC'
|
2020-03-25 12:15:37 +00:00
|
|
|
TEST-01-BASIC CLEANUP: Basic systemd setup
|
|
|
|
TEST-01-BASIC SETUP: Basic systemd setup
|
2012-05-25 16:44:42 +00:00
|
|
|
...
|
2020-03-25 12:15:37 +00:00
|
|
|
TEST-01-BASIC RUN: Basic systemd setup [OK]
|
2018-10-09 12:37:00 +00:00
|
|
|
make: Leaving directory '/home/zbyszek/src/systemd/test/TEST-01-BASIC'
|
|
|
|
--x-- Result of TEST-01-BASIC: 0 --x--
|
|
|
|
--x-- Running TEST-02-CRYPTSETUP --x--
|
2020-11-17 20:39:37 +00:00
|
|
|
+ make -C TEST-02-CRYPTSETUP clean setup run
|
2012-05-25 16:44:42 +00:00
|
|
|
|
|
|
|
If one of the tests fails, then $subdir/test.log contains the log file of
|
|
|
|
the test.
|
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
To run just one of the cases:
|
2012-05-25 16:44:42 +00:00
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
$ sudo make -C test/TEST-01-BASIC clean setup run
|
2012-05-25 16:44:42 +00:00
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
Specifying the build directory
|
|
|
|
==============================
|
2013-12-07 23:01:53 +00:00
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
If the build directory is not detected automatically, it can be specified
|
|
|
|
with BUILD_DIR=:
|
2012-05-25 16:44:42 +00:00
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
$ sudo BUILD_DIR=some-other-build/ test/run-integration-tests
|
2012-05-25 16:44:42 +00:00
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
or
|
2012-05-25 16:44:42 +00:00
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
$ sudo make -C test/TEST-01-BASIC BUILD_DIR=../../some-other-build/ ...
|
2013-12-07 23:01:53 +00:00
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
Note that in the second case, the path is relative to the test case directory.
|
|
|
|
An absolute path may also be used in both cases.
|
|
|
|
|
2020-11-17 20:55:47 +00:00
|
|
|
Testing installed binaries instead of built
|
|
|
|
===========================================
|
|
|
|
|
|
|
|
To run the extended testsuite using the systemd installed on the system instead
|
|
|
|
of the systemd from a build, use the NO_BUILD=1:
|
|
|
|
|
|
|
|
$ sudo NO_BUILD=1 test/run-integration-tests
|
|
|
|
|
2018-10-09 12:37:00 +00:00
|
|
|
Configuration variables
|
|
|
|
=======================
|
|
|
|
|
2019-03-16 17:32:38 +00:00
|
|
|
TEST_NO_QEMU=1
|
2022-05-04 10:29:31 +00:00
|
|
|
Don't run tests under qemu
|
2018-10-09 12:37:00 +00:00
|
|
|
|
2020-12-26 20:11:55 +00:00
|
|
|
TEST_QEMU_ONLY=1
|
2022-05-04 10:29:31 +00:00
|
|
|
Run only tests that require qemu
|
2020-12-26 20:11:55 +00:00
|
|
|
|
2019-03-16 17:32:38 +00:00
|
|
|
TEST_NO_NSPAWN=1
|
|
|
|
Don't run tests under systemd-nspawn
|
2013-12-07 23:01:53 +00:00
|
|
|
|
2020-12-31 17:29:58 +00:00
|
|
|
TEST_PREFER_NSPAWN=1
|
|
|
|
Run all tests that do not require qemu under systemd-nspawn
|
|
|
|
|
2019-03-16 17:32:38 +00:00
|
|
|
TEST_NO_KVM=1
|
2022-05-04 10:29:31 +00:00
|
|
|
Disable qemu KVM auto-detection (may be necessary when you're trying to run the
|
|
|
|
*vanilla* qemu and have both qemu and qemu-kvm installed)
|
2019-03-16 17:32:38 +00:00
|
|
|
|
2019-09-26 21:29:38 +00:00
|
|
|
TEST_NESTED_KVM=1
|
|
|
|
Allow tests to run with nested KVM. By default, the testsuite disables
|
|
|
|
nested KVM if the host machine already runs under KVM. Setting this
|
|
|
|
variable disables such checks
|
|
|
|
|
2019-03-16 17:32:38 +00:00
|
|
|
QEMU_MEM=512M
|
2022-05-04 10:29:31 +00:00
|
|
|
Configure amount of memory for qemu VMs (defaults to 512M)
|
2019-03-16 17:32:38 +00:00
|
|
|
|
|
|
|
QEMU_SMP=1
|
2022-05-04 10:29:31 +00:00
|
|
|
Configure number of CPUs for qemu VMs (defaults to 1)
|
2019-03-16 17:32:38 +00:00
|
|
|
|
|
|
|
KERNEL_APPEND='...'
|
|
|
|
Append additional parameters to the kernel command line
|
|
|
|
|
|
|
|
NSPAWN_ARGUMENTS='...'
|
|
|
|
Specify additional arguments for systemd-nspawn
|
|
|
|
|
|
|
|
QEMU_TIMEOUT=infinity
|
2022-11-15 10:00:22 +00:00
|
|
|
Set a timeout for tests under qemu (defaults to 1800 sec)
|
2019-03-16 17:32:38 +00:00
|
|
|
|
|
|
|
NSPAWN_TIMEOUT=infinity
|
2022-11-15 10:00:22 +00:00
|
|
|
Set a timeout for tests under systemd-nspawn (defaults to 1800 sec)
|
2019-03-16 17:32:38 +00:00
|
|
|
|
|
|
|
INTERACTIVE_DEBUG=1
|
|
|
|
Configure the machine to be more *user-friendly* for interactive debuggung
|
|
|
|
(e.g. by setting a usable default terminal, suppressing the shutdown after
|
|
|
|
the test, etc.)
|
2018-10-09 12:37:00 +00:00
|
|
|
|
2023-06-05 08:47:21 +00:00
|
|
|
TEST_MATCH_SUBTEST=subtest
|
|
|
|
If the test makes use of `run_subtests` use this variable to provide
|
|
|
|
a POSIX extended regex to run only subtests matching the expression
|
|
|
|
|
|
|
|
TEST_MATCH_TESTCASE=testcase
|
|
|
|
Same as $TEST_MATCH_SUBTEST but for subtests that make use of `run_testcases`
|
|
|
|
|
2022-09-23 12:59:02 +00:00
|
|
|
The kernel and initrd can be specified with $KERNEL_BIN and $INITRD. (Fedora's
|
2023-05-29 21:18:55 +00:00
|
|
|
or Debian's default kernel path and initrd are used by default.)
|
2013-12-07 23:01:53 +00:00
|
|
|
|
2022-05-04 10:29:31 +00:00
|
|
|
A script will try to find your qemu binary. If you want to specify a different
|
2018-10-09 12:37:00 +00:00
|
|
|
one with $QEMU_BIN.
|
|
|
|
|
|
|
|
Debugging the qemu image
|
|
|
|
========================
|
|
|
|
|
2023-05-29 21:18:55 +00:00
|
|
|
If you want to log in the testsuite virtual machine, use INTERACTIVE_DEBUG=1
|
|
|
|
and log in as root:
|
2018-10-09 12:37:00 +00:00
|
|
|
|
2023-05-29 21:18:55 +00:00
|
|
|
$ sudo make -C test/TEST-01-BASIC INTERACTIVE_DEBUG=1 run
|
2013-12-07 23:01:53 +00:00
|
|
|
|
2023-05-29 21:18:55 +00:00
|
|
|
The root password is empty.
|
2019-10-02 08:16:01 +00:00
|
|
|
|
|
|
|
Ubuntu CI
|
|
|
|
=========
|
|
|
|
|
|
|
|
New PR submitted to the project are run through regression tests, and one set
|
|
|
|
of those is the 'autopkgtest' runs for several different architectures, called
|
|
|
|
'Ubuntu CI'. Part of that testing is to run all these tests. Sometimes these
|
2020-07-10 20:13:31 +00:00
|
|
|
tests are temporarily deny-listed from running in the 'autopkgtest' tests while
|
2019-10-02 08:16:01 +00:00
|
|
|
debugging a flaky test; that is done by creating a file in the test directory
|
2020-07-10 20:13:31 +00:00
|
|
|
named 'deny-list-ubuntu-ci', for example to prevent the TEST-01-BASIC test from
|
2019-10-02 08:16:01 +00:00
|
|
|
running in the 'autopkgtest' runs, create the file
|
2020-07-10 20:13:31 +00:00
|
|
|
'TEST-01-BASIC/deny-list-ubuntu-ci'.
|
2019-10-02 08:16:01 +00:00
|
|
|
|
2020-07-10 20:13:31 +00:00
|
|
|
The tests may be disabled only for specific archs, by creating a deny-list file
|
2019-10-02 08:16:01 +00:00
|
|
|
with the arch name at the end, e.g.
|
2020-07-10 20:13:31 +00:00
|
|
|
'TEST-01-BASIC/deny-list-ubuntu-ci-arm64' to disable the TEST-01-BASIC test
|
2019-10-02 08:16:01 +00:00
|
|
|
only on test runs for the 'arm64' architecture.
|
|
|
|
|
|
|
|
Note the arch naming is not from 'uname -m', it is Debian arch names:
|
|
|
|
https://wiki.debian.org/ArchitectureSpecificsMemo
|
|
|
|
|
2020-07-10 20:13:31 +00:00
|
|
|
For PRs that fix a currently deny-listed test, the PR should include removal
|
|
|
|
of the deny-list file.
|
2020-10-16 16:51:58 +00:00
|
|
|
|
|
|
|
In case a test fails, the full set of artifacts, including the journal of the
|
|
|
|
failed run, can be downloaded from the artifacts.tar.gz archive which will be
|
|
|
|
reachable in the same URL parent directory as the logs.gz that gets linked on
|
|
|
|
the Github CI status.
|
|
|
|
|
|
|
|
To add new dependencies or new binaries to the packages used during the tests,
|
|
|
|
a merge request can be sent to: https://salsa.debian.org/systemd-team/systemd
|
|
|
|
targeting the 'upstream-ci' branch.
|
2021-01-05 14:05:15 +00:00
|
|
|
|
|
|
|
The cloud-side infrastructure, that is hooked into the Github interface, is
|
|
|
|
located at:
|
|
|
|
|
|
|
|
https://git.launchpad.net/autopkgtest-cloud/
|
2021-09-15 12:49:04 +00:00
|
|
|
|
|
|
|
In case of infrastructure issues with this CI, things might go wrong in two
|
|
|
|
places:
|
|
|
|
|
|
|
|
- starting a job: this is done via a Github webhook, so check if the HTTP POST
|
|
|
|
are failing on https://github.com/systemd/systemd/settings/hooks
|
|
|
|
- running a job: all currently running jobs are listed at
|
|
|
|
https://autopkgtest.ubuntu.com/running#pkg-systemd-upstream in case the PR
|
|
|
|
does not show the status for some reason
|
|
|
|
- reporting the job result: this is done on Canonical's cloud infrastructure,
|
|
|
|
if jobs are started and running but no status is visible on the PR, then it is
|
|
|
|
likely that reporting back is not working
|
|
|
|
|
|
|
|
For infrastructure help, reaching out to Canonical via the #ubuntu-devel channel
|
|
|
|
on libera.chat is an effective way to receive support in general.
|
2022-02-10 10:59:27 +00:00
|
|
|
|
|
|
|
Manually running a part of the Ubuntu CI test suite
|
|
|
|
===================================================
|
|
|
|
|
|
|
|
In some situations one may want/need to run one of the tests run by Ubuntu CI
|
|
|
|
locally for debugging purposes. For this, you need a machine (or a VM) with
|
2023-01-19 12:54:19 +00:00
|
|
|
the same Ubuntu release as is used by Ubuntu CI (Jammy ATTOW).
|
2022-02-10 10:59:27 +00:00
|
|
|
|
|
|
|
First of all, clone the Debian systemd repository and sync it with the code of
|
|
|
|
the PR (set by the $UPSTREAM_PULL_REQUEST env variable) you'd like to debug:
|
|
|
|
|
|
|
|
# git clone https://salsa.debian.org/systemd-team/systemd.git
|
|
|
|
# cd systemd
|
|
|
|
# git checkout upstream-ci
|
|
|
|
# TEST_UPSTREAM=1 UPSTREAM_PULL_REQUEST=12345 ./debian/extra/checkout-upstream
|
|
|
|
|
|
|
|
Now install necessary build & test dependencies:
|
|
|
|
|
|
|
|
## PPA with some newer Ubuntu packages required by upstream systemd
|
|
|
|
# add-apt-repository -y ppa:upstream-systemd-ci/systemd-ci
|
|
|
|
# apt build-dep -y systemd
|
2023-01-19 12:54:19 +00:00
|
|
|
# apt install -y autopkgtest debhelper genisoimage git qemu-system-x86
|
2022-02-10 10:59:27 +00:00
|
|
|
|
|
|
|
Build systemd deb packages with debug info:
|
|
|
|
|
2023-01-19 12:54:19 +00:00
|
|
|
# TEST_UPSTREAM=1 DEB_BUILD_OPTIONS="nocheck nostrip" dpkg-buildpackage -us -uc
|
2022-02-10 10:59:27 +00:00
|
|
|
# cd ..
|
|
|
|
|
|
|
|
Prepare a testbed image for autopkgtest (tweak the release as necessary):
|
|
|
|
|
2023-01-19 12:54:19 +00:00
|
|
|
# autopkgtest-buildvm-ubuntu-cloud --ram-size 1024 -v -a amd64 -r jammy
|
2022-02-10 10:59:27 +00:00
|
|
|
|
|
|
|
And finally run the autopkgtest itself:
|
|
|
|
|
|
|
|
# autopkgtest -o logs *.deb systemd/ \
|
|
|
|
--timeout-factor=3 \
|
|
|
|
--test-name=boot-and-services \
|
|
|
|
--shell-fail \
|
2023-01-19 12:54:19 +00:00
|
|
|
-- autopkgtest-virt-qemu --ram-size 2048 autopkgtest-jammy-amd64.img
|
2022-02-10 10:59:27 +00:00
|
|
|
|
|
|
|
where --test-name= is the name of the test you want to run/debug. The
|
|
|
|
--shell-fail option will pause the execution in case the test fails and shows
|
|
|
|
you the information how to connect to the testbed for further debugging.
|
2022-03-11 15:51:53 +00:00
|
|
|
|
2022-09-22 22:54:12 +00:00
|
|
|
Manually running CodeQL analysis
|
2022-03-11 15:51:53 +00:00
|
|
|
=====================================
|
|
|
|
|
2022-09-22 22:54:12 +00:00
|
|
|
This is mostly useful for debugging various CodeQL quirks.
|
2022-03-11 15:51:53 +00:00
|
|
|
|
|
|
|
Download the CodeQL Bundle from https://github.com/github/codeql-action/releases
|
|
|
|
and unpack it somewhere. From now the 'tutorial' assumes you have the `codeql`
|
|
|
|
binary from the unpacked archive in $PATH for brevity.
|
|
|
|
|
|
|
|
Switch to the systemd repository if not already:
|
|
|
|
|
|
|
|
$ cd <systemd-repo>
|
|
|
|
|
|
|
|
Create an initial CodeQL database:
|
|
|
|
|
|
|
|
$ CCACHE_DISABLE=1 codeql database create codeqldb --language=cpp -vvv
|
|
|
|
|
|
|
|
Disabling ccache is important, otherwise you might see CodeQL complaining:
|
|
|
|
|
|
|
|
No source code was seen and extracted to /home/mrc0mmand/repos/@ci-incubator/systemd/codeqldb.
|
|
|
|
This can occur if the specified build commands failed to compile or process any code.
|
|
|
|
- Confirm that there is some source code for the specified language in the project.
|
|
|
|
- For codebases written in Go, JavaScript, TypeScript, and Python, do not specify
|
|
|
|
an explicit --command.
|
|
|
|
- For other languages, the --command must specify a "clean" build which compiles
|
|
|
|
all the source code files without reusing existing build artefacts.
|
|
|
|
|
2022-09-22 22:54:12 +00:00
|
|
|
If you want to run all queries systemd uses in CodeQL, run:
|
2022-03-11 15:51:53 +00:00
|
|
|
|
2022-09-22 22:54:12 +00:00
|
|
|
$ codeql database analyze codeqldb/ --format csv --output results.csv .github/codeql-custom.qls .github/codeql-queries/*.ql -vvv
|
2022-03-11 15:51:53 +00:00
|
|
|
|
|
|
|
Note: this will take a while.
|
|
|
|
|
|
|
|
If you're interested in a specific check, the easiest way (without hunting down
|
|
|
|
the specific CodeQL query file) is to create a custom query suite. For example:
|
|
|
|
|
|
|
|
$ cat >test.qls <<EOF
|
|
|
|
- queries: .
|
|
|
|
from: codeql/cpp-queries
|
|
|
|
- include:
|
|
|
|
id:
|
|
|
|
- cpp/missing-return
|
|
|
|
EOF
|
|
|
|
|
|
|
|
And then execute it in the same way as above:
|
|
|
|
|
|
|
|
$ codeql database analyze codeqldb/ --format csv --output results.csv test.qls -vvv
|
|
|
|
|
|
|
|
More about query suites here: https://codeql.github.com/docs/codeql-cli/creating-codeql-query-suites/
|
|
|
|
|
|
|
|
The results are then located in the `results.csv` file as a comma separated
|
|
|
|
values list (obviously), which is the most human-friendly output format the
|
|
|
|
CodeQL utility provides (so far).
|
2023-04-01 10:15:42 +00:00
|
|
|
|
|
|
|
Code coverage
|
|
|
|
=============
|
|
|
|
|
|
|
|
We have a daily cron job in CentOS CI which runs all unit and integration tests,
|
|
|
|
collects coverage using gcov/lcov, and uploads the report to Coveralls[0]. In
|
|
|
|
order to collect the most accurate coverage information, some measures have
|
|
|
|
to be taken regarding sandboxing, namely:
|
|
|
|
|
|
|
|
- ProtectSystem= and ProtectHome= need to be turned off
|
|
|
|
- the $BUILD_DIR with necessary .gcno files needs to be present in the image
|
|
|
|
and needs to be writable by all processes
|
|
|
|
|
|
|
|
The first point is relatively easy to handle and is handled automagically by
|
|
|
|
our test "framework" by creating necessary dropins.
|
|
|
|
|
|
|
|
Making the $BUILD_DIR accessible to _everything_ is slightly more complicated.
|
|
|
|
First, and foremost, the $BUILD_DIR has a POSIX ACL that makes it writable
|
|
|
|
to everyone. However, this is not enough in some cases, like for services
|
|
|
|
that use DynamicUser=yes, since that implies ProtectSystem=strict that can't
|
|
|
|
be turned off. A solution to this is to use ReadWritePaths=$BUILD_DIR, which
|
|
|
|
works for the majority of cases, but can't be turned on globally, since
|
|
|
|
ReadWritePaths= creates its own mount namespace which might break some
|
|
|
|
services. Hence, the ReadWritePaths=$BUILD_DIR is enabled for all services
|
|
|
|
with the `test-` prefix (i.e. test-foo.service or test-foo-bar.service), both
|
|
|
|
in the system and the user managers.
|
|
|
|
|
|
|
|
So, if you're considering writing an integration test that makes use
|
|
|
|
of DynamicUser=yes, or other sandboxing stuff that implies it, please prefix
|
|
|
|
the test unit (be it a static one or a transient one created via systemd-run),
|
|
|
|
with `test-`, unless the test unit needs to be able to install mount points
|
|
|
|
in the main mount namespace - in that case use IGNORE_MISSING_COVERAGE=yes
|
|
|
|
in the test definition (i.e. TEST-*-NAME/test.sh), which will skip the post-test
|
|
|
|
check for missing coverage for the respective test.
|
|
|
|
|
|
|
|
[0] https://coveralls.io/github/systemd/systemd
|