From a79505909ff755ad69a8b89281a5cf7da2d6baeb Mon Sep 17 00:00:00 2001 From: Dan Streetman Date: Mon, 16 Sep 2019 12:34:55 -0400 Subject: [PATCH 1/2] test: add temporarily blacklisted tests This temporarily blacklists some tests when run under Ubuntu CI. This is the upstream side of the Debian 'upstream' test MR: https://salsa.debian.org/systemd-team/systemd/merge_requests/52 The tests blacklisted here should only be temporarily blacklisted until they can be fixed; the intention is that these blacklist files will be added and removed over time while debugging/fixing flaky and/or regressed tests, without causing test failure noise for other PRs. --- test/TEST-24-UNIT-TESTS/blacklist-ubuntu-ci-ppc64el | 0 test/TEST-25-IMPORT/blacklist-ubuntu-ci | 0 test/TEST-30-ONCLOCKCHANGE/blacklist-ubuntu-ci-i386 | 0 test/TEST-36-NUMAPOLICY/blacklist-ubuntu-ci-s390x | 0 test/TEST-40-EXEC-COMMAND-EX/blacklist-ubuntu-ci-arm64 | 0 5 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 test/TEST-24-UNIT-TESTS/blacklist-ubuntu-ci-ppc64el create mode 100644 test/TEST-25-IMPORT/blacklist-ubuntu-ci create mode 100644 test/TEST-30-ONCLOCKCHANGE/blacklist-ubuntu-ci-i386 create mode 100644 test/TEST-36-NUMAPOLICY/blacklist-ubuntu-ci-s390x create mode 100644 test/TEST-40-EXEC-COMMAND-EX/blacklist-ubuntu-ci-arm64 diff --git a/test/TEST-24-UNIT-TESTS/blacklist-ubuntu-ci-ppc64el b/test/TEST-24-UNIT-TESTS/blacklist-ubuntu-ci-ppc64el new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/TEST-25-IMPORT/blacklist-ubuntu-ci b/test/TEST-25-IMPORT/blacklist-ubuntu-ci new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/TEST-30-ONCLOCKCHANGE/blacklist-ubuntu-ci-i386 b/test/TEST-30-ONCLOCKCHANGE/blacklist-ubuntu-ci-i386 new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/TEST-36-NUMAPOLICY/blacklist-ubuntu-ci-s390x b/test/TEST-36-NUMAPOLICY/blacklist-ubuntu-ci-s390x new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/TEST-40-EXEC-COMMAND-EX/blacklist-ubuntu-ci-arm64 b/test/TEST-40-EXEC-COMMAND-EX/blacklist-ubuntu-ci-arm64 new file mode 100644 index 0000000000..e69de29bb2 From b4532d650b69993faef92d198fac69b1520528a8 Mon Sep 17 00:00:00 2001 From: Dan Streetman Date: Wed, 2 Oct 2019 04:16:01 -0400 Subject: [PATCH 2/2] test/README.testsuite: add section for Ubuntu CI blacklist files Add some docs explaining how blacklist individual test/TEST-* tests in Ubuntu CI using per-test blacklist files. --- test/README.testsuite | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/test/README.testsuite b/test/README.testsuite index 2753ed6e3a..9e0c36512c 100644 --- a/test/README.testsuite +++ b/test/README.testsuite @@ -97,3 +97,26 @@ kernel command line parameter with $KERNEL_APPEND and then log in as root. $ sudo make -C test/TEST-01-BASIC KERNEL_APPEND="systemd.unit=multi-user.target" run Root password is empty. + +Ubuntu CI +========= + +New PR submitted to the project are run through regression tests, and one set +of those is the 'autopkgtest' runs for several different architectures, called +'Ubuntu CI'. Part of that testing is to run all these tests. Sometimes these +tests are temporarily blacklisted from running in the 'autopkgtest' tests while +debugging a flaky test; that is done by creating a file in the test directory +named 'blacklist-ubuntu-ci', for example to prevent the TEST-01-BASIC test from +running in the 'autopkgtest' runs, create the file +'TEST-01-BASIC/blacklist-ubuntu-ci'. + +The tests may be disabled only for specific archs, by creating a blacklist file +with the arch name at the end, e.g. +'TEST-01-BASIC/blacklist-ubuntu-ci-arm64' to disable the TEST-01-BASIC test +only on test runs for the 'arm64' architecture. + +Note the arch naming is not from 'uname -m', it is Debian arch names: +https://wiki.debian.org/ArchitectureSpecificsMemo + +For PRs that fix a currently blacklisted test, the PR should include removal +of the blacklist file.